A Sample of the Literature on Heuristics* and Biases

A LECTURE TO TYBBA HONOURS STUDENTS OF BKMIBA BY DR MUNISH Y ALAGH HL COLLEGE CAMPUS, 30th JANUARY 2014.

What is this lecture about?

This lecture is a Sample of the Literature on Heuristics* and Biases(Shortcuts in Thinking* and Biases Related To Them)the law of small numbers (and its related heuristic: representativeness) and errors in statistical thinking related to it demystified. What is all this? I will present this slide again in the end and ask you to explain it.

The law of small numbers demystified

A study of the incidence of kidney cancer in Counties of United States. Kidney Cancer is lowest in counties which are mostly rural and sparsely populated and located in traditionally Republican States. You probably ignored the Republican Part. Did you focus on the Rural Part? Did you think that rural lifestyle leads to low kidney cancer? Probably.

Now comes the surprise!

The counties in the United States which have the highest incidence of kidney cancer also tend to be in mostly rural, sparsely populated counties and Republican States! Maybe the poverty of the rural lifestyle caused this! Something is wrong here! Rural Lifestyle cannot explain both high and low incidence of kidney cancer!

What is the solution to this riddle?lets begin to think.

Well, what could be the solution to this riddle?, Lets atleast begin the process of exploration. Infact, the key factor is not that the counties were rural or predominantly Republican, it is that rural counties have small populations. More about this explanation later*, first lets investigate certain features of this conondrum.

The Problem is In fact: Errors in Statistical Reasoning

From whatever I told you above you must have thought O My God! Here, we have another average lecture on some academic topic, but the lecture I am about to give you is not academic at all, infact it is a lecture whose basis is on the difficult relationship between our mind and statistics.

What was the shortcut that your mind got trapped in at the beginning of this story?
We automatically and effortlessly identify causal connections between events, sometimes even when the connection is spurious. What was the shortcut that your mind got into? When told about the high incidence counties, you immediately assume that these counties are different from other counties for a reason, that there must be a cause that explains this difference.

Our Mind and Statistics.

Our mind, specially the intuitive part of our mind is inept when faced with merely statistical facts, which change the probability of outcomes but do not cause them to happen. How can we justify the above? Here we go As I had promised earlier there is an explanation to this story of Kidney Cancer. What is it?

The Explanation:

Imagine the population of the United States as marbles in a giant urn. Some marbles are marked KC, for kidney cancer. You draw samples of marbles and populate each county in turn. Rural Samples are smaller than other samples. Extreme Outcomes (very high and/or very low cancer rates) are most likely to be found in sparsely populated counties. This is all there is to the story.

Going Deeper into the Explanation.

Imagine a large urn filled with marbles. Half the marbles are red, half are white. Next, draw 4 marbles from the urn, record the number of red balls in the sample, throw the balls back into the urn, and then do it again, many times. If you summarize the results, you will find that the outcome "2 red, 2 white" occurs (almost exactly) 6 times as often as the outcome "4 red" or "4 white." This relationship is a mathematical fact.

A related statistical fact

A related statistical fact is relevant to the cancer example. From the same urn, Jack draws 4 marbles on each trial, Jill draws 7. They both record each time they observe a homogeneous sampleall white or all red. If they go on long enough, Jack will observe such extreme outcomes more often than Jillby a factor of 8.

Just an accident of sampling

The small population of a county neither causes nor prevents cancer; it merely allows the incidence of cancer to be much higher (or much lower than in the largerpopulation. The deeper truth is that there is nothing to explain. The incidence of cancer is not truly lower or higher than normal in a countv with a small population it just appears to be so in a particular year because of an accident of sampling.

Law of large and small numbers

large samples- deserve more trust than smaller samples.

you may find that the following statements apply to you: "sparsely populated" did not immediately stand out as relevant when you read the kidney cancer story. You were at least mildly surprised by the size of the difference between samples of 4 and samples of 7. The following two statements mean exactly the same thing: Large samples are more precise than small samples.

Small samples yield extreme results more often than large samples do.
The first statement has a clear ring of truth, but until the second version makes intuitive sense, you have not truly understood the first. The bottom line: yes, you did know that the results of large samples are more precise, but you may now realize that you did not know it very well.

Sampling Variation

you wish to confirm the hypothesis that the vocabulary of the average six-year-old girl is larger than the vocabulary of an average boy of the same age. The hypothesis is true in the population; Girls and boys vary a great deal, however, and by the luck of the draw you could select a sample in which the difference is inconclusive, or even one in which boys actually score higher. Using a sufficiently large sample is the only way to reduce the risk. Researchers who pick too small a sample leave themselves at the mercy of sampling

Content versus reliability

In a poll of 300 seniors 60% support the President summarize in exactly three words, you would choose The elderly support the President.'' These words provide the gist of the story. Your summary would be the same if the sample size had been different. Of course, a completely absurd number of sample size, small or big, would draw your attention.

Halo Effect

believing that small samples closely resemble the population from which they are drawn implies: we are prone to exaggerate the consistency and coherence of what we see, The exaggerated faith of researchers in what can be learned from a few observations is closely related to the halo effect, the sense we often get that we know and understand a person about whom we actually know very little.

There has to be a reason for everything

Take the sex of six babies born in sequence at a hospital. The sequence of boys and girls is obviously random; the events are independent of each other, and the number of boys and girls who were born in the hospital in the last few hours has no effect whatsoever on the sex of the next. However we will not consider the fact that if in the sequence all events are independent and outcome boy and girl are approximately equally likely, then any possible sequence of six births is as likely as any other. We are pattern seekers, believers in a coherent world, in which regularities (such as a sequence of six girls) appear as a result of someone's intention. Lions may appear on the plain at random times, but it would be safer to notice and respond to an apparent increase in the rate of appearance of prides of lions, even if it is actually due to the fluctuations of a random process.

Search for certainty, search for causality- a waste of time.

War broke out in 1973. Squadrons flying from the same base, one of which had lost four planes while the other had lost none. An inquiry was initiated. There was no prior reason to believe that one of the squadrons was more effective than the other, and no operational differences were found, but of course the lives of the pilots differed in many random ways, including, how often they went home between missions.. Rationally the command should accept that the different outcomes were due to blind luck, a random search for a non obvious cause was hopeless.

Finding Patterns in Sports

The assumed hot hand in sports is very usual, if in basketball a player sinks three or four baskets in a row, defense starts guarding him more, his players start passing more to him, even his coach thinks he has a temporary hot hand, we are too quick to perceive order and causality in randomness.

In a small sample there will be more extreme results.

A research study showed that more small schools had done well, so the Gates foundation started funding small schools and a causal story can easily be linked to this saying that attention to students is more in small schools, actually larger schools empirically, if anything, do better possibly because of greater curriculum options . And so Unfortunately the causal analysis is wrong, the actual fact is which could have been pointed out had the Gates foundation taken statistics seriously is that more small schools had also done badly. Clearly on average small schools are not better just more variable.

Journal Articles
In further reading 1) if you read the article I have given you Belief in the law of small numbers.Tversky, Amos; Kahneman, Daniel, Psychological Bulletin, Vol 76(2), Aug 1971, 105-110. This article gives many examples which show that the fact that extreme outcomes result from small samples more often than large samples is a statistical fact and can lead to a bias or statistical error in misinterpreting, what is a result of sample size, as a factor related to the content of the story. Unfortunately we focus more on content than reliability.

Journal Articles
In further reading 1) if you read the article I have given you On the psychology of prediction.Kahneman, Daniel; Tversky, Amos Psychological Review, Vol 80(4), Jul 1973, 237-251. You will find examples of how people see patterns where none exist. People judge even extreme and rare outcomes as more probable if the content of the story indicates that a particular outcome is more representative, even if that outcome is extreme or rare. This is the heuristic of representativeness.

Journal Articles
In further reading 1) if you read the article I have given you

How to Make Cognitive Illusions Disappear: Beyond Heuristics and Biases, Gerd Gigerenzer, European Review of Social Psychology,Volume 2, Issue 1, 1991 Special Issue: European Review of Social Psychology.

You will find that this statistical error of not considering the sample size and focussing on the content of the story, itself has limitations, as the author explains: by considering a more detailed view of statistics ie by considering, for example: relative frequency rather than single frequency case, and also, whether or not the observation was randomly selected or self selected itself; this kind of a nuanced technical view of statistics could possibly remove this statistical error itself.

What was this lecture about?Explain below statement

This lecture is a Sample of the Literature on Heuristics* and Biases(Shortcuts in Thinking* and Biases Related To Them)the law of small numbers (and its related heuristic: representativeness) and errors in statistical thinking related to it demystified.