How Survivors Deceive You

During World War II, aerial bombardment was one of the Allies' effective tools for destroying Germany's industrial and military infrastructure and ultimately defeating that country; however, this approach came with heavy casualties for the British and American air forces. Estimates indicate that the life expectancy of a bomber crew was between 12 and 15 missions. At one point in the war, if you were a bomber pilot in Europe, your chance of returning safely from a mission was about 50 percent. Military leaders had concluded that they needed to add more protective armor to their aircraft to shield them from anti-aircraft fire and fighters; but adding armor to all parts of the plane was not feasible, as it would greatly reduce speed and make it more vulnerable; therefore, they had to decide which parts of the aircraft to armor.

For this reason, the air force began collecting data on the points where aircraft were damaged. After each mission, they carefully examined the planes that had returned and mapped the number of damages from shrapnel and bullets and their locations on the aircraft. Gradually, it became clear that there was a specific pattern in the distribution of damage on the planes.

Figure-1 shows that most of the damage was on the wing and fuselage areas of the aircraft.

Based on this, military experts concluded that since the most bullets hit the wing and fuselage areas of the aircraft, these parts needed more protective armor. At first glance, this conclusion seems correct.

But Abraham Wald completely disagreed with this conclusion. He was among the mathematicians who worked for the U.S. Army during World War II. Wald showed that a significant error had occurred in the analyses because the conclusion was based only on data from aircraft that returned from missions; but what do we know about the planes that crashed during the mission? In fact, the statistical sample was biased toward the planes that returned. Wald demonstrated that, exactly the opposite, those parts of the aircraft that received the fewest hits in Figure-1 needed protection. In fact, the damage points on the returning aircraft indicate that if a plane is hit in these areas, it has a higher probability of returning safely. Wald's suggestions practically helped improve the return rate of aircraft.

Since Wald's way of thinking about the problem mentioned above is very interesting, in the appendix of this article I will briefly refer to his computational method regarding the vulnerability of different parts of the aircraft. Non-technical readers can ignore it.

What is Survivorship Bias?

Survivorship Bias is an error in reasoning that occurs when you focus only on the individuals or things that have passed through a selection process and overlook those that did not, primarily because they are no longer observable.

For example, someone might conclude, based on a limited number of graduates from a high school who were admitted to good universities, that the high school provides superior educational services. This may be true, but such an argument cannot be made without considering the admission status of the other graduates of that high school.

As another example, buildings with solid construction, beautiful architecture, good functionality, and proper maintenance endure and remain for several generations. By comparing only the surviving old buildings with today's buildings, people might conclude that better buildings were constructed in the past; but they do not factor into their conclusion the thousands of other structures from the past that were not well-built, have perished over time, and are no longer observable. This bias can apply to outstanding works of art from the past that have triumphed over competition through time, when compared with contemporary works of art. One of the reasons for a sense of nostalgia for the past is this type of comparison.

Secrets of Success

These days, books on the secrets of success, which address the factors behind the success of entrepreneurs or top organizations, have many fans. One such example is the book Good to Great by Jim Collins, which is also one of the best-selling books published in the field of management.

He selected eleven companies from among 1,435 companies that had managed to outperform the stock market average over the past forty years, and then sought the common characteristics that, in his belief, made these companies successful. The list of these eleven companies is below:

Abbott Laboratories	Kimberly-Clark	Pitney Bowes
Circuit City	Kroger	Walgreens
Fannie Mae	Nucor	Wells Fargo
Gillette	Philip Morris

But the problem is that the backward-looking approach exposes Collins's study to survivorship bias. Collins should have started with a list of companies at the beginning of the study's time period and chosen acceptable criteria for selecting the top eleven companies. These criteria should have been applied objectively and impartially, without considering how these companies would perform over the next forty years. It is meaningless to predict which companies will perform well over a period after you have already seen which ones did perform well during that time! This is not prediction; it is recounting history.

In fact, frame the question this way: What is the probability that you would find 11 companies exhibiting common characteristics purely by chance? Collins answers in his book that this probability is 1 in 17 million; but the correct answer is 100 percent!

Suppose someone goes to register their car and receives the license plate 333S33. The probability of receiving such a plate is about 1 in 2 million. If I had predicted before they received this plate that they could get this plate, it would have been extraordinary. But once they have received the plate, the probability is 100 percent!

When you look at any group of companies with a backward-looking approach, you can always find a set of common characteristics. For example, in the list above, all the companies have the letter i or r in their name. Can one say that the presence of these two letters in the companies' names caused their success? Of course not!

After the book's publication, a study of these companies' stocks between 2001 and 2012 revealed that six of the eleven companies above had financial performance below the market average. This shows why the backward-looking approach is systematically wrong.

What is the message of this discussion for managers?

These examples make it clear that in order to draw conclusions, you need to pay attention to all instances, even those you cannot immediately observe. They also make it clear that learning from failures is not always a straightforward process. Learning requires careful observation and examination and going beyond superficial assumptions. When you look only at successful instances, you may overlook the fatal behaviors and mistakes that befell the unsuccessful ones. Perhaps this is why, when that sage was asked, “From whom did you learn manners?” he replied: “From the ill-mannered.”

.

---

.

Appendix: A Review of Wald’s Work on Aircraft Survivability

Wald was interested in knowing the distribution of damage on returning aircraft in order to determine which parts of the plane should be reinforced with armor to increase survivability. Note that there was no access to data on aircraft that had crashed during missions.

In the absence of such data, he tried to estimate the probability that a plane hit by a certain number of bullets could continue its flight after receiving one more bullet. He also attempted to calculate the probability of the aircraft’s survival after being hit in various parts.

Suppose 400 aircraft are sent on a mission and 380 return. The number of aircraft hit times is . The following information is available:

Each aircraft is divided into four sections: 1) Engines, 2) Fuselage, 3) Fuel system, and 4) Other parts. indicates what proportion of the aircraft’s total area each of these sections occupies. Figure-2 shows the distribution of bullet strikes on the various sections of the returning aircraft ().

Based on the available data, the fraction of aircraft that returned after receiving bullets can be obtained (). Wald assumed that if an aircraft is hit by a greater number, such as bullets, it will certainly crash:

Thus, the percentage of lost aircraft is calculated from the following relation:

Suppose represents the conditional probability that an aircraft will crash after being hit by the -th bullet, given that it has received bullets but has not crashed.

Also, represents the fraction of aircraft that have crashed upon receiving the -th bullet. It is assumed that if an aircraft is not hit, it will certainly return (). Therefore, the following relation holds:

Thus, the fraction of aircraft that crash due to the -th hit is obtained from the following relation:

The goal is to calculate the desired probabilities from the observable data (). It can be proven that the following equation holds:

In the above relation, is the conditional probability that the aircraft does not crash after being hit by the -th bullet, given that it has received bullets and has not crashed (). As a simplifying assumption, one can consider this probability to be constant (). Therefore, for the sample data, the above relation leads to solving the following equation to find :

By solving the above equation, is obtained, and can be calculated.

The assumption that is constant may be restrictive in practice. In his subsequent work, Wald addressed how to solve the problem without this assumption. That discussion is beyond the scope of this article.

In another section, Wald discusses how to calculate the vulnerability probability of different parts of the aircraft. If represents the probability that region is hit, given that it receives only that single bullet and does not crash, the following relation holds:

This relation is, in fact, Bayes' Law. Assuming is constant, the above equation takes the following form:

 represents the ratio of hits on each section to the total hits on the returning aircraft, calculated from the available data. is the probability of a bullet hitting a section of the aircraft, which can be considered equivalent to the area that section occupies relative to the total surface area of the aircraft (Figure-2). The values of  for different sections of the aircraft are shown in Figure-3.

As can be seen, the most vulnerable point is the aircraft's engines. Compare these results with Figure-2, which shows the distribution of bullet hits, where the engines of the returning aircraft were among the points that received the fewest hits.

.

---

.

References:

Mangel, M., & Samaniego, F. J. (1984). “Abraham Wald’s Work on Aircraft Survivability”. Journal of the American Statistical Association, 79(386), 259-267

Smith, G. (2014). “Standard Deviations: Flawed Assumptions, Tortured Data, and Other Ways to Lie with Statistics”, Overlook Duckworth, Peter Mayer Publishers, Inc. New York

Syed, M. (2015). “Black Box Thinking: Why Some People Never Learn from Their Mistakes – But Some Do”, Portfolio / Penguin, New York

.

---

.

How Survivors Deceive You

Author: Farzad Minoui

.

---

.