Selection Bias

Imagine stopping people at the door of a gym at 6 a.m. and asking how many days a week they work out. What would you hear? Most would say four or five days a week, or more. Now suppose you announced, "80 percent of Americans are fitness fanatics." Nobody would believe you.

That's because people at a gym at dawn already love exercise, or are very disciplined about it. The lives of people who work out now and then, or hardly ever, never made it into your survey. You gathered people in a special place, so your conclusion was bound to bend in the wrong direction.

This mistake is called selection bias: picking a lopsided sample instead of looking at what the whole group is really like. If the way you choose your subjects is crooked from the start, even perfect math gives you nonsense.

During World War II, the military wanted to add armor to bombers and other warplanes to help more of them survive. Researchers collected the planes that had come back safely from combat and studied the bullet holes closely. According to a story often told, the returning planes had holes clustered in places like the wings and the body, and people assumed those most-hit areas should get thicker armor.

But the statistician Abraham Wald argued the opposite. He saw that planes hit in truly critical spots, like the engine or the cockpit, never made it home. They had already been shot down.

The moment researchers studied only the planes that returned, the most important information, about the planes that crashed, went missing completely. This distortion, caused by looking only at what survived to the end, is called survivorship bias. It is one of the best-known kinds of selection bias, and one of its best lessons.

A common misconception is that you can make the error disappear just by surveying more people. But if the method is wrong, 100 people or 100,000 people will give you the same distortion.

For example, say you run a poll on one particular website. Even if hundreds of thousands vote, you still won't learn what the whole country thinks. The voices that get captured are exaggerated: certain age groups who are heavy internet users, or people who were so unhappy about the topic that they clicked the vote button themselves. This tilt, caused by only the people who eagerly chose to respond, is called self-selection bias.

To be more precise, selection bias doesn't need anyone to cheat on purpose. It can arise all by itself from the setting or method of a survey. That's why scientists and statisticians use random sampling, which gives everyone an equal chance of being picked, so the study doesn't start off tilted.