How good are humans at identifying the role of randomness? Not very, apparently.
As individuals, we like to look for concrete causes when events occur. We naturally pick out patterns in observations. We search for explanations, relationships, correlations – anything to help us make sense of how the universe works. It’s an instinctual endeavor:
You can see why assuming causality could have had evolutionary advantages. It is part of the general vigilance that we have inherited from ancestors. We are automatically on the lookout for the possibility that the environment has changed. Lions may appear on the plain at random times, but it would be safer to notice and respond to an apparent increase in the rate of appearance of prides of lions, even if it is actually due to the fluctuations of a random process. (Kahneman p. 115)
Unfortunately, this tendency can lead us to grossly underestimate the prevalence of randomness. Consider this example from Daniel Kahneman:
Take the sex of six babies born in a sequence at a hospital. The sequence of boys and girls is obviously random; the events are independent of each other [the outcome of one birth in the sequence does not influence the outcome of another], and the number of boys and girls who were born in the hospital in the last few hours has no effect whatsoever on the sex of the next baby. Now consider three possible sequences:
Are the sequences equally likely? The intuitive answer – “of course not!” – is false. Because the events are independent and because the outcomes B and G are (approximately) equally likely, then any possible sequence of six births is as likely as any other. Even now that you know this conclusion is true, it remains counterintuitive, because only the third sequence appears random. As expected, BGBBGB is judged much more likely than the other two sequences. We are pattern seekers, believers in a coherent world, in which regularities (such as a sequence of six girls) appear not by accident but as a result of mechanical causality or of someone’s intention. We do not expect to see regularity produces by a random process, and when we detect what appears to be a rule, we quickly reject the idea that the process is truly random. Random processes produce many sequences that convince people the process is not random after all.
Our intuition is often fooled by the presence of randomness in an otherwise-seemingly-random series of events. This can sometimes be described via the representativeness heuristic, wherein individuals perceive patterns to be representative of certain groupings, stereotypes, or non-random occurrences – when in fact the observation is a misrepresentation. Here’s another famous example, as presented by Sunstein and Thaler:
A less trivial example [of the representativeness heuristic], from the Cornell psychologist Tom Gilovich (1991), comes from the experience of London residents during the German bombing campaigns of World War II. London newspapers published maps, such as the one [below], displaying the location of the strikes from German V-1 and V-2 missiles that landed in central London. As you can see, the patters does not seem at all random. Bombs appear to be clustered around the River Thames and also in the northwest sector of the map. People in London expressed concern at the time because the pattern seemed to suggest that the Germans could aim their bombs with great precision. Some Londoners even speculated that the blank spaces were probably the neighborhoods where German spies live. They were wrong. In fact the Germans could do no better than aim their bombs at Central London and hope for the best. […]
Still, the location of the bomb strikes does not look random. What is going on here? We often see patterns because we construct our informal tests only after looking at the evidence. The World War II example is an excellent illustration of this problem. Suppose we divide the map into quadrants [as in the second image below]. If we then do a formal statistical test – or, for the less statistically inclined, just count the number of hits in each quadrant – we do find evidence of a nonrandom pattern. However, nothing in nature suggests that this is the right way to test for randomness. Suppose instead we form the quadrants diagonally [as in the third image]. We are now unable to reject the hypothesis that the bombs land at random. (Nudge p. 27-8)
We can be very easily misled by our misinterpretation of randomness as distinct, deliberate, and orderly behavior. As in the Londoners’ example, such misinterpretation can then lead to irrational reactions (such as accusing innocent neighbors of being German spies).
How easy is it to be tricked? Rather easy, it turns out. It is possible to produce an entirely-randomly-generated academic paper in mathematics – and have it accepted and published in an academic journal. This occurred recently, prompting much larger concerns about the state of academic journal referees and the degree of rigor with which they examine submissions, but simultaneously highlighting the conclusion that randomness can very often look very real. [If you want to randomly generate papers yourself, the source is here.]
Nassim Taleb describes this particular phenomenon as a reverse Turing Test, where a computer-generated work (non-intelligent) fools a reader into thinking it has been produced by a (intelligent) human. Taleb (p. 76) even remarks on a rather beautiful sentence generated by a random poem composition exercise performed by surrealist poets (who each contributed certain words and then combined them in a Mad Lib fashion): “The exquisite cadavers shall drink the new wine.”
It might be nice to think of such an oddly wonderful sentence as intentionally constructed, but often, events which we wish to be deliberate are, in fact, random. While it can leave our appetite for causality and explanation unsatisfied, it is the way of the universe. David Pizarro recommends we think more about this phenomenon and how it influences our way of thinking everyday:
[…] when our pattern-detection systems misfire they tend to err in the direction of perceiving patterns where none actually exist.
The German neurologist Klaus Conrad coined the term “Apophenia” to describe this tendency in patients suffering from certain forms of mental illness. But it is increasingly clear from a variety of findings in the behavioral sciences that this tendency is not limited to ill or uneducated minds; healthy, intelligent people make similar errors on a regular basis: a superstitious athlete sees a connection between victory and a pair of socks, a parent refuses to vaccinate her child because of a perceived causal connection between inoculation and disease, a scientist sees hypothesis-confirming results in random noise, and thousands of people believe the random “shuffle” function on their music software is broken because they mistake spurious coincidence for meaningful connection.
Why do we suffer from this “everyday apophenia” as Pizarro calls it? Sometimes it may just be our inability to inhibit our instinct tendency to find causality in the events we observe. Other times, it may be a result of the law of small numbers: our willingness to accept a very small sample size as sufficient to draw a conclusion about the non-random nature of observations. Think of the baby pattern GGGGGG having come up in a string of 1000 observations of B’s and G’s – if the entire thousand-baby sequence was not otherwise dominated by girls, would you still suspect the 6-baby sequence GGGGGG to be non-random? Still, it may be a matter of banking on superstition to believe in our ability to influence how the universe functions (such as the athlete misperceiving a connection between socks and performance). We may also perceive non-randomness as a consequence of our own optimism about our skills: we are often more than willing to attribute our successes to skill and our failures to “bad luck.”
Regardless of the reason, it is essential to be aware of the ubiquity of randomness – and to recognize our own bias toward searching for causality. Because very often, what we seek isn’t there. It’s just random.