09 Dec Insights From How Humans Play Borderlands Science
Throughout the past year, we’ve been collecting and analyzing millions of solutions submitted by the player base. We have already made a post that examines how the player solutions help us create sequences that differ from current algorithms, and how much diversity the solutions have. Today, we’ll be looking into how players solve Borderlands Science puzzles and why it is useful for us to know. Specifically, we’re analyzing the strategies and patterns that emerge from solutions and what insights can be gleaned from them.
To start, let’s look at a simple correlation matrix of a few selected features of a particular sample puzzle. A correlation matrix describes the relationship between two features – in this case we’ve picked a few features of note. The closer the number is to 1, the most correlated the two variables are.
Undo Moves describes the number of times the player removes a gap that they previously placed. We can observe a high correlation with time – this makes sense, a solution with more corrections likely took more time to do.
The Pareto score was discussed in length in a previous blog post. But briefly, it is a feature that describes how close to optimality a solution is. Unfortunately, there aren’t any significantly interesting correlation between any individual features and the pareto score – meaning there’s no single feature we can pinpoint that helps us create more near optimal solutions.
But there’s more to puzzle solving than the features that can be extracted from a completed solution – what about the patterns that emerge when we look at solutions over time? Let’s take a look:
Aha! We can definitely see that there is a dominant pattern in solving the puzzles – adding gaps from left to right at the bottom row! You may be wondering if this strategy is just a cheap way to solve the puzzles and doesn’t provide valuable information for us, but worry not. We have designed the puzzles to have an offset – so it is expected that this would be a potential solution for the puzzle!
On top of that, we have actually shuffled the columns of each puzzle so that every player will play a different version of the puzzle! The true solution distribution looks more like this:
And thus, we can look through the background noise and find the true patterns behind the solutions. There are certain “hotspots” in the distribution – places where players are more likely to place their gaps. These hotspots can be significantly different from what conventional algorithms propose as solutions. As an example;
If we were to compare the distribution to the solution of the greedy algorithm (in red) used to generate the solutions described in the previous blog post ,we can see that there is a significant difference in where the players have put their gaps in their solutions.
We analyze the differences between our algorithms and the player solutions of thousands of different puzzles to figure out exactly how a player solution differs from ours. This allows us to look into what makes human solutions special – leading to multiple potential avenues in the BLS project.
How does this knowledge help with the rest of the project?
Now what’s the point of looking at the player solutions in such depth? The whole idea of citizen science revolves around the idea that human provided data is an untapped potential for furthering science. We think that the diversity and different patterns that emerge from human player solutions can be beneficial to the genomic sequencing task.
We won’t necessarily be taking only the most near-optimal player solutions, but also the ones that represent the entire player solution distribution the best to develop algorithms that learn from specific human strategies to provide new alignments.
Whether this is through attempting to realign the sequences with specifically filtered player solutions, or training a reinforcement learning model using player solutions over time – it’s not just the solutions that you are submitting that are important, but also the strategies you are using that are helping the Borderlands Science team move forward.