The Holes You Cannot See: World War II Aircraft and Survivorship Bias

Imagine standing on the runway of an air base. Fighter planes return from their missions one after another, their fuselages riddled with bullet holes. A group of engineers and statisticians gathers around each plane, marks the exact location of every hole on a diagram of the aircraft's body, and gradually a map emerges showing dense clusters of bullet holes across the wings, the main fuselage and the tail. The conclusion of this map, at least at first glance, seems obvious: these are the spots to armor, because this is where the planes have taken the most damage.

This was precisely the problem the United States military faced during World War II: many warplanes were being shot down, and there was neither the budget nor the weight allowance to armor the entire body of an aircraft; every extra kilogram of armor made a plane heavier, slower and less maneuverable. So a choice had to be made: which parts should be reinforced? The data, at least on the surface, offered a clear answer: the wings and the fuselage.

In the well-known telling of this story, a mathematician and statistician named Abraham Wald saw something others had missed; not in the data, but in what was absent.

A Discovery Hidden in an Absence

Wald asked a simple but foundational question: where had the planes under examination come from? The answer was plain: from the planes that had managed to return to base. But this was exactly where the problem began. The planes whose vital parts, such as the engine, the cockpit or the fuel tank, had been hit never made it back to base to be counted in this survey. They had gone down before reaching base, in the sky, at sea or in enemy territory.

So that map riddled with holes was in fact a map of the places where a plane could be hit and still fly and return. The absence of bullet damage in the engine area did not mean the engine was safe. On the contrary, planes damaged in that area usually never returned to base to be recorded in the statistics. The conclusion drawn from the survey was therefore the opposite of the initial assumption: the parts to armor were those that showed the least damage on the returning planes, not those bearing the most traces of bullet impacts. For damage to those very parts usually meant the plane was lost, leaving no chance of returning and being entered into the data.

This subtle error now has a name: survivorship bias.

When We See Only the Survivors

Survivorship bias occurs when we examine only what has come through a selection process intact and overlook the cases that were eliminated or destroyed in the course of that same process. In such conditions we may arrive at mistaken conclusions. The problem is that the eliminated cases often stay hidden from our sight with ease; not necessarily because someone has concealed them, but because they no longer exist to be seen.

There are simple reasons why we usually see only what remains: failures tend to leave the stage quietly. Less is written about them, less is recorded, and fewer people talk about them. A company that has gone bankrupt no longer publishes an annual report. A patient who dropped out of a medical study before the end of treatment does not show up in the final figures. An artist who was never discovered has no opportunity to give an interview and describe their fruitless efforts. For this reason, what comes before our eyes is often a rosier picture than reality; not because anyone intends to deceive us, but because failures leave our field of vision sooner than successes do.

From Aircraft to Everyday Life

The story of the World War II aircraft is only one instance of a far broader phenomenon, one that appears in domains very far from the battlefield.

One clear example can be seen in the familiar accounts of entrepreneurial success. The story of the person who left university unfinished and later built a billion-dollar company is told again and again; but of the thousands who made the same decision, set their studies aside and invested in a high-risk idea, only to fail, we hear very little. When we hear only the stories of the successful, we may conclude that "dropping out" or "risk-taking" is the path to success. Yet all we actually know from these stories is that some of those who made such a decision succeeded; and that is very different from claiming that those decisions caused their success.

The same pattern can be seen in historical narratives. We study the civilizations, governments and ideas that endured, whose written works have been preserved or whose documents and archives are available to us. But countless societies, organizations and ideas that perished without leaving a trace almost never become the subject of serious research. Not necessarily because they were less important, but because nothing of them remains for us to study.

In examining human lives, the same error recurs in another form. We usually narrate the lives of great thinkers from their peak: the books they wrote, the theories they developed and the influence they had on others. But the thousands of unfinished ideas, rejected papers, discarded drafts and years of doubt and failure they went through before reaching that peak are dropped from the story of their lives. It is as though only the path of success has survived, with no sign of the countless roads that led nowhere; roads in which those same thinkers themselves had failed again and again.

In medical research, too, if we examine only the patients who remained in a study until its end, we may obtain a false picture of a treatment's effectiveness. Patients who left the study because of side effects, the worsening of their illness or other reasons are part of the reality, and overlooking them can make the findings misleading.

Even in our everyday judgments about others we face this same bias. When we see someone who is successful, we usually attend only to the outcome they have achieved; not the failures they have been through, not the opportunities they have lost, and not the support they may have received along the way. On social media this bias has become part of our daily experience: we compare our own life, full of ups and downs, with a curated and polished image of other people's lives; an image that was meant from the outset to show only the best moments of their lives.

A Question That Goes Beyond Statistics

What lifts the story of the World War II aircraft above a purely statistical problem is that it concerns something far more human than mathematics: the limits of our view of the world. Our minds usually judge on the basis of what is in front of us, not on the basis of the whole of reality; a reality parts of which may remain hidden from us forever. This limitation is not a sign of stupidity or carelessness on our part; it is part of the mind's natural way of making sense of the world. But if we remain unaware of it, it may lead our decisions and judgments astray; from financial investment and career choice to our verdicts on others and even our judgments about ourselves.

The central lesson of the story of those aircraft is to keep asking ourselves: what is not visible in this picture? Which people, which ideas and which efforts never had the chance to reach this point and come into view?

Intellectual maturity means asking about the things that are absent. Perhaps, then, the most important question is this:

How much of our judgment rests on what we have seen, and how much on what never had the chance to be seen?