I recently completed a first of its kind, revolutionary, ground breaking, extraordinary study to once and for all determine exactly what the best kind of cookie really is. And the winner is … chocolate chip! Yes, that’s right, the humble chocolate chip cookie is the hands down favorite. Let us all rejoice and rest easy now that we’ve finally put that great mystery to bed.
If your first question when reading this is “Where can I get some cookies?” well then, you’re right. But you should also ask “Hey, how did you conduct this study?”. Likely you’ll get an answer that falls into one of three types. Depending on which type you get could change how much trust you put in the results of a survey or study.
It’s entirely possible I will tell you that the methodology used in my study is confidential and a trade secret. If I was to tell you, you could do your own studies and thereby take money out of my pocket and then I couldn’t buy cookies anymore. Finding out the methodology for a study is somehow secret doesn’t automatically make the study suspect, but it should set off some loud warning klaxons. When you know that you can’t review the methods used it means that you have to take the provider’s results pretty much on faith. Maybe the operator of the study is trustworthy and accurate and this is fine. However, this tact is also used to hide problems or lead recipients into a desired conclusion. It’s also possible the study was flawed or perhaps wasn’t even done in the first place. Since you can’t see it, you can’t tell.
I could be totally cool and give you all the details of the study. In this case, you’ll know that I used 24 types of cookies, sourced from award winning bakeries that used only top ingredients and each cookie was made within 24 hours of eating. The participants were a group of 61 people from diverse cultures and genders and age ranges. Oh, and it was in space. Like outer space. You are perfectly free to repeat my study, but it was conducted in such a way that it is difficult if not impossible to ever repeat the exact conditions. Since you know all the details perhaps you can trust the results. Since you know the study isn’t reproduceable it is fine to consider the results a bit suspect. Perhaps there were reasons the study was designed to make reproducing it difficult.
Finally, I could tell you that I purchased 12 packages of common brand name cookies in differing varieties (and provide the details of each) and that I assembled a group of 25 randomly diverse people (again, giving the details of the group like gender, age, ethnicity) and asked them to sample each cookie and rank their top 5 in order. Then I picked the most often occurring one. This is great, you have everything you need if you want to conduct this study, and it’s entirely possible to do so. This is what we like to hear. However, what happens far too often is you find out there was only one study done. No one else did this to see if the results were the same. When multiple people do the study and get the same result, then it’s trustworthiness goes up. Since there was just the one study, how much do you trust it? If you are planning on starting a cookie business and investing your life’s savings on this one result, you might want to rethink that decision. Wondering what cookie to buy this week at the store, maybe you’re OK.
We see a lot of news that starts with something like “Latest study shows …” but we’re not often told much about the studies themselves. Some could fall into the first category where the method is purposely hidden. You might see this in cases where someone purports to have defied gravity or created a perpetual motion machine – only you can’t see how it was done. Be wary. While my second type seems a bit fantastical, it’s pretty common in the science world simply because some things are hard to do. A number of studies are literally done in space, so doing them again is extremely cost prohibitive. Or they require very specific conditions and knowledge. This doesn’t immediately invalidate the results, but it does mean we should interpret it as not fully verified or even likely to change in the future. The last type is also very common, and very pervasive. Quite a number of studies were done once, got a result and never repeated. There was no nefarious reason for not repeating them, it just never happened. We should certainly be aware of these so we can more accurately make decisions using their results. A number of psychological studies have wound up here, and many of their results are used commonly.
The moral of the story here is that you might need to study the studies. For now, I need some cookies.