Survivorship bias is super interesting. This is the notion whereby people over-focus on the ‘survived’ outcomes of a process, and use those outcomes as basis to improve it, thereby failing to understand that you learn more from the invisible, ‘dead’ parts that didn’t make it until the end.
This term was coined by mathematician Abraham Wald during WWII. The military was analyzing fighter planes that came back from war, by inspecting their bullet holes and opting for reinforcement at those places.
However, what the military failed to understand, is that the very planes they analyzed actually made it back safely to the ground (although heavily damaged), whereas many other planes had been destroyed in air — thus being the ’strong ones’. With other words: rather than reinforcing the hit areas of returning jets, it would make more sense to reinforce the areas that were NOT hit (because those areas caused the other planes to crash).
So this is survivorship bias:
“You’re skewed towards a one data set, and base your conclusions on that data set, without taking into account the ‘invisible’ data that affects the overall results just as much.”
“They don’t make ’em anymore like they used to!
So being aware of this fallacy, the well known saying “they don’t make ’em anymore like they used to”, makes sense: old products that are still used today, have proven to be strong and passed the test of time. Their properties might truly be better than the majority of products in their class, as the 99% that didn’t survive until today have been damaged goods and ended up in landfill (due to whatever defect). So the products you’re looking at today, are basically ’the best of the breed’ — the survivors.
Development, design & the survivor bias
So what does this have to do with design and product or technical development?
Well, often we find ourselves trying to learn from customer behavior and customer/user feedback during or after a certain experience. More often that not, we tend to focus on the people that ‘finished’ a process, such as booking a flight, ordering a book on Amazon or buying furniture on IKEA.com. However, these are considered the ’survivors’: those customers decided to pursue the experience until the very end and complete it. It is far more interesting to look at all the cases whereby people did not finish the (buying) process, and dropped out at some point in time.
Likewise, developers and testers should be careful not to focus only on the data that comes from users who finished a certain task, but more so pay attention to users that didn’t. The reasons for drop-off (faulty tech, bad UX, 404’s, etc.) are the easiest starting point for product improvement.
So don’t just focus on the visible ‘bullet holes’, but put effort in discovering why the other planes went down. This will truly solve your problems and improve your overall customer journey — just as Abraham Wald did during WWII.
🌍 🌍 🌍 🌍 This article was published by Umvel. We help challengers design, build and scale their business
