Survivorship Bias

Don't just study those who were successful; also study those who failed.

Let's tip over a sacred cow here: take everything world champions say with a grain of salt.  They don't always know what they're talking about.

This statement might sound a bit controversial.  After all, our intuition tells us that to be successful, we should study those who were successful.  But our intuition can be wrong due to survivorship bias.

Survivorship bias is a logical error where we overstudy those who were successful but understudy those who failed.

A classic example of survivorship bias is Abraham Wald's study of bomber planes in World War II.  In an attempt to reduce bomber losses due to gunfire, Wald plotted where bullet holes appeared on returning bomber aircraft.  Wald observed that bullet holes were clustered in certain areas, as shown in the figure below.

If we want to reduce bomber losses, our intuition might tell us to add reinforcing armor in the areas where the bullet holes are clustered.  However, Wald proposed the opposite: to add extra armor where there were no bullet holes.  This is because of survivorship bias: the reason for the “empty spots” on Wald's bullet hole map was that the planes shot in those areas never made it home.

The lesson here is extremely important:

Don't just study those who were successful; also study those who failed.

Studying only successful people is dangerous because: for every success story, there may be hundreds who tried the same approach and failed.  That one wildly successful person might be an extremely lucky outlier.

Successful people are often overconfident in their advice.  They'll assume anyone can be successful by following their method...and that if you're not successful, it must be your fault.  But successful people are often unaware of the massive advantages they had, such as wealth, first mover advantages, unearned talent, or just plain luck.  They only see the bullet holes, not the “empty spots” where the planes went down.

On the podcast:

• Ep. 181: BJJ for Old F***s, feat. Rob Biernacki & Stephan Kesting
• Ep. 195: Idle Time, feat. Dominyka Obelenyte

Further study:

• Survivorship bias — Wikipedia