Episode 3 : Quick! Hide in that percentage!

Did you know that an astounding 92% of survey respondents said that chocolate chip cookie dough ice cream was the best flavor?  Or that only a mere 11% of people prefer sausage over peperoni as their primary pizza topping.  If you are a fan of cookie dough and peperoni then these numbers look great.  Shout them from the rooftops!

You already know what’s coming next, don’t you?  You’re at least 80% sure I’m about to say “but”.  Well, congratulations, you’re right.

But wait, what do those percentages even mean?  We often see numbers that look fine and seem like they were based on something good.  They’re numbers, right?  Numbers are good, scientific and mathematical things.  Aren’t they?

Percentages are tricky because they often hide what went into producing them.  On the cookie dough survey, just how many respondents were there?  What if there were only 13 respondents and 11 of those liked cookie dough.  While it’s a small group, still most liked cookie dough. Then I tell you the survey was done at the International Chocolate Chip Cookie Festival (wouldn’t that be awesome, by the way!) and you wonder if there was a bit of bias.  You should also be saying, um, 11 divided by 13 is 85%, not 92%, and you’d be right.  I also didn’t mention that one of the respondents said anchovies was the best ice cream flavor – because that person got the surveys confused and I excluded the response as an invalid outlier.  But should I have excluded it?  I never told you what my survey criteria was, or possible responses were, or what I considered valid.  That 92% isn’t looking so credible anymore is it?

The peperoni numbers shared above - let’s say there were 100 respondents (a little bit bigger population) and the question was do you prefer peperoni or sausage on your pizza.  That leaves a nice binary answer, so nothing weird can show up.  And we know that just 11 people checked sausage, so the math works out.  But who did this survey?  When you find out it was sponsored by the Mid Atlantic Peperoni Foundation, then maybe even those nice clean numbers are suspect.  Perhaps they had a bit of bias in who was surveyed.

When you see a nice clean percentage given to you, the first thing you should ask is what are the underlying numbers.  If those aren’t given, then right away it’s a suspect number.  If you see the underlying numbers, then ask if those seem valid?  If it was a survey, were there enough respondents, what was the methodology, were values removed or corrected.    The last big piece is who created the number or commissioned the study.  Who created it might reveal a bias that wasn’t apparent.

Is every percentage you see untrustworthy?  Well, no.  However, you shouldn’t just blindly accept a number without seeing what’s behind it.  There might be more lurking in there than you thought.