The Royal Air Force’s only airworthy Second World War-era Lancaster bomber flying over eastern England. Powered by four of the Rolls Royce Merlin engines that powered such fighter aircraft as the Supermarine Spitfire, the Hawker Hurricane, and the North American Aviation Mustang, the Avro Lancaster was the subject of an off-target statistical analysis intended to improve the survivability of the aircraft in combat.

Walking home from a meeting recently I heard a strangely familiar growl growing over my left shoulder. I finally looked back to see a Boeing B-17 Flying Fortress — the mainstay of the strategic bombing campaign of the American Eight Air Force During the Second World War — flying toward me at a surprisingly low altitude.

The plane sounded pretty much as they do in the movies except, since this plane presumably wasn’t carrying several tons of high explosives, the engines weren’t working as hard as the sound effects in the movies so they sounded less stressed. I watched until the plane disappeared in the haze to the north.

Seeing this septuagenarian aircraft (the last one was built in 1945) reminded me of the use of statistics during the war to try to assess the effectiveness of various war-making efforts. It was, of course, during the war that Edwards Deming put into practice in American industry the sort of statistical controls he later introduced in Japan.

I once read a story about a statistical survey done by the RAF on the survivability of the Avro Lancaster bomber, the main British four-engine heavy bomber during the war. The survey illustrated the importance of knowing exactly what we are trying to measure with statistics, and why.

The survey assessed the location and type of battle damage received by Lancasters on missions over a Europe occupied by National Socialist (Nazi) Germany. In the report I read, the RAF was trying to determine how the Lancaster might be strengthened so it could better survive combat.

Eventually somebody pointed out that this survey actually made little sense. The survey was of damage to planes that had returned safely, bringing home at least part of the crew.

The real point of such a survey should have been to find out not where the planes were damaged, but where the planes weren’t damaged. The survey only showed what sort of battle damage the aircraft could survive. What the RAF really needed to know was what kind of battle damage the aircraft couldn’t survive.

To generalize this lesson, it’s important to know what we are trying to measure with statistics and, more importantly, why we want to measure it.

For the RAF, the why was obvious. The air war over Europe was a brutal one. A stunningly high number of bomber crews — sixty percent — didn’t survive the war. Few people today know that a quarter of American deaths in the Second World War — 100,000 of them, 40 times the number of Americans killed in the Normandy landings — came in the air war over Europe.

The “why” of the survey, then, was not to save aircraft (more could and would be built) but to save the lives of allied airmen. To be sure, there was a high correlation between aircraft survival and aircrew survival, but the fundamental need was to bring home the aircrews to fight another day. The survivability survey as originally structured couldn’t give the information that was required.

I mention this survey because it often seems obvious what we ought to measure, and why, but I have often seen managers in business make the same type of mistake the RAF brass originally made. (At least the RAF were measuring. I’m shocked at the number of managers I have known who measured essentially nothing, relying instead on hunches and guesswork.) If the RAF could make such a mistake when the stakes were so high, rest assured you or I could make that same mistake in our businesses if we allow complacency to slip in when the stakes are much, much lower.

Copyright 2017 by Paul G. Spring. All rights reserved.