Simpson’s Paradox

[chengjun]

Judea Pearl, Madelyn Glymour, and Nicholas P. Jewell. （2016）Causal Inference in Statistics: A Primer, First Edition. John Wiley & Sons, Ltd. Published 2016 by John Wiley & Sons, Ltd.
Companion Website: www.wiley.com/go/Pearl/Causality

Named after Edward Simpson (born 1922), the statistician who first popularized it, the paradox refers to

the existence of data in which a statistical association that holds for an entire population is reversed in every subpopulation.

For instance, we might discover that students who smoke get higher grades, on average, than nonsmokers get. But when we take into account the students’ age, we might find that, in every age group, smokers get lower grades than nonsmokers get. Then, if we take into account both age and income, we might discover that smokers once again get higher grades than nonsmokers of the same age and income. The reversals may continue indefinitely, switching back and forth as we consider more and more attributes.

In this context, we want to decide whether smoking causes grade increases and in which direction and by how much, yet it seems hopeless to obtain the answers from the data.

In the classical example used by Simpson (1951), a group of sick patients are given the option to try a new drug. Among those who took the drug, a lower percentage recovered than among those who did not. However, when we partition by gender, we see that more men taking the drug recover than do men are not taking the drug, and more women taking the drug recover than do women are not taking the drug! In other words, the drug appears to help men and women, but hurt the general population. It seems nonsensical, or even impossible—which is why, of course, it is considered a paradox. Some people find it hard to believe that numbers could even be combined in such a way. To make it believable, then, consider the following example:

Remarkably, though the numbers are the same in the gender and blood pressure examples, the correct result lies in the segregated data for the former and the aggregate data for the latter.

1. A working definition of “causation.”
2. A method by which to formally articulate causal assumptions—that is, to create causal
   models.
3. A method by which to link the structure of a causal model to features of data.
4. A method by which to draw conclusions from the combination of causal assumptions
   embedded in a model and data.

Study question 1.2.1

What is wrong with the following claims?

• (a) “Data show that income and marriage have a high positive correlation. Therefore, your earnings will increase if you get married.”
• (b) “Datashowthatasthenumberoffiresincrease,sodoesthenumberoffirefighters.There- fore, to cut down on fires, you should reduce the number of fire fighters.”
• (c) “Datashowthatpeoplewhohurrytendtobelatetotheirmeetings.Don’thurry,oryou’ll be late.”

Study question 1.2.2

A baseball batter Tim has a better batting average than his teammate Frank. However, some- one notices that Frank has a better batting average than Tim against both right-handed and left-handed pitchers. How can this happen? (Present your answer in a table.)

Study question 1.2.3

Determine, for each of the following causal stories, whether you should use the aggregate or the segregated data to determine the true effect.

• (a) There are two treatments used on kidney stones: Treatment A and Treatment B. Doctors are more likely to use Treatment A on large (and therefore, more severe) stones and more likely to use Treatment B on small stones. Should a patient who doesn’t know the size of his or her stone examine the general population data, or the stone size-specific data when determining which treatment will be more effective?
• (b) There are two doctors in a small town. Each has performed 100 surgeries in his career, which are of two types: one very difficult surgery and one very easy surgery. The first doctor performs the easy surgery much more often than the difficult surgery and the second doctor performs the difficult surgery more often than the easy surgery. You need surgery, but you do not know whether your case is easy or difficult. Should you consult the success rate of each doctor over all cases, or should you consult their success rates for the easy and difficult cases separately, to maximize the chance of a successful surgery?

Study question 1.2.4

In an attempt to estimate the effectiveness of a new drug, a randomized experiment is con- ducted. In all, 50% of the patients are assigned to receive the new drug and 50% to receive a placebo. A day before the actual experiment, a nurse hands out lollipops to some patients who show signs of depression, mostly among those who have been assigned to treatment the next day (i.e., the nurse’s round happened to take her through the treatment-bound ward). Strangely, the experimental data revealed a Simpson’s reversal: Although the drug proved beneficial to the population as a whole, drug takers were less likely to recover than nontakers, among both lollipop receivers and lollipop nonreceivers. Assuming that lollipop sucking in itself has no effect whatsoever on recovery, answer the following questions:

• (a) Is the drug beneficial to the population as a whole or harmful?
• (b) Does your answer contradict our gender example, where sex-specific data was deemed more appropriate?
• (c) Draw a graph (informally) that more or less captures the story. (Look ahead to Section 1.4 if you wish.)
• (d) How would you explain the emergence of Simpson’s reversal in this story?
• (e) Would your answer change if the lollipops were handed out (by the same criterion) a day after the study?

[Hint: Use the fact that receiving a lollipop indicates a greater likelihood of being assigned to drug treatment, as well as depression, which is a symptom of risk factors that lower the likelihood of recovery.]