Statistical Illusions nobody has to lie
ILLUSION 04 · AN UNCOUNTED POPULATION

The Missing Planes

You armor the plane where the bullets are. The bullets are where it lived.

A wave of bombers flies a raid and most of them come back. You get to look the survivors over, count where they are hit, and decide where to bolt on armor plate for the next wave. Armor is heavy, so you cannot cover the whole plane; you have to choose.

The damage is right there in front of you, measured and complete. You will still put the armor in the wrong place, for a reason that is invisible precisely because it worked.

Setup

The planes that came back

Here is your fleet after the raid: the planes that made it home, with every hit on them marked. The fuselage is peppered. The wings are peppered. The engines are almost clean. You have armor for two parts of the plane, no more, and the question is where it goes.

Before you start, one honest note

What is on the table, and what is not

The map is complete for the planes in front of you. Nothing about them is hidden and nothing is revealed later that was withheld now. Count the holes, plate the two zones you judge most exposed, and send up the next wave.

The only thing held back is the wave itself: what happens when you fly the fleet with your armor on it, and one thing about the planes that are not in front of you.

Operate

Plate the plane

Each panel below is a zone, with the number of hits your surviving planes took there. Plate the two you think will save the most planes. The obvious move is to cover the zones with the most holes, and that is exactly the move to make with your eyes open.

Your returning bombers – every hit on the planes that came home

plate two zones, then fly
a hit that came home
a hit that did not
your armor
Fig. 1 · your surviving bombers, with every hit marked. This map is generated in your browser, not a record of any real raid. The engines look clean; hold that thought until the wave is up.
Commit

Send up the fleet

· unlocks once you plate a zone

This is the armor the next wave flies with. There is no calling them back to re-plate once they are up, here or over Europe. Commit to it.

One wave. The count comes back as it comes back, and it does not move to spare you.
Your loadout

–

Reveal

Now fly it, and count the missing

· unlocks when you commit

You armored where the holes were. The holes were where the plane could take a hit and still come home.

Planes home, with your armor
–
out of the 400 you sent up
Better than no armor by
–
the planes your plate actually saved
Engine hits: seen vs flew
–
the map understated the engines this much

Repair

So where does the armor go

· unlocks after the reveal

Armor the parts of the survivors that are not hit. The gaps in the returning planes are not lucky; they are the wounds that were fatal, so the planes that got them are not in your sample. Plate the clean zones, especially the engines, and fly again.

Same fleet, same hits, the plate moved to where the map said it was safe.
What Abraham Wald actually did, 1943

Wald worked from exactly this problem for the Statistical Research Group. In one worked example, of 400 planes sent up, 380 came back carrying this many hits each. From the survivors alone he estimated the chance a plane survived a given hit at about 0.85.

400 flew, 380 returned, 20 did not; his per-hit survival estimate was 0.851. These are his own numbers from a hypothetical worked example, not a real damage table; no such table was ever published, which is the point.

Debrief

What just happened

· unlocks after the repair

You read a complete, accurate map and armored the wrong plane, the same way a careful person would. Nothing was rigged. The map was true about the planes it showed. The trouble is which planes it showed: only the ones that lived, which is the same as saying only the ones whose wounds were survivable.

What was absent from the table

The planes that did not come home. Every hole in your survivors is a hit a plane walked away from; the hits that did not get walked away from left with the plane. The danger was not in the map and there was nothing in the map to point at, because the evidence for it had removed itself. It is the population nobody counted.

Who caught this

Abraham Wald, working for the Statistical Research Group in 1943, built a way to estimate damage he could not observe from the survivors he could (A Method of Estimating Plane Vulnerability Based on Damage of Survivors, reprinted 1980). W. Allen Wallis recorded the group's work (JASA 75(370), 1980); Mangel and Samaniego gave it a full modern reading (JASA 79(386), 1984); and Bill Casselman separated the record from the legend (AMS Feature Column, 2016).

One honest caveat

And the famous version of this story is itself a survivor. The bullet-riddled diagram, the officers, the retort, most of it is not in the record; the vivid telling outlived its own caveats, which is the same lesson one more time. The essay's closing panel, the planes that came back, sets the story beside the record.