Spotting A Lie, Bad Data in Scientific Studies

"Drinking soda will rot your teeth at the same rate as meth."

The above line was the conclusion of a scientific study released a few years ago. While the study makes for a sensational headline, the data was deeply flawed. Spotting a sensationalized study can be difficult with so many headlines flying about today. Having the ability to spot a fake or misrepresented article doesn't have to be difficult though, and I'm going to show you some things to look for.

First off is the "too good to be true" measure. If it sounds way too good to be true, it likely is, trust that instinct and look it up. The study may very well find some amazing conclusion we never knew about, but the only way to tell is to look at the referenced article. Personally, I can usually get an idea if the study or headline is good in under 2 minutes, not a huge drain on our time.

Study Size

If the study doesn't trigger the too good to be true warning bells, there are some things we can look for. When looking at the efficacy of a study, the first thing I'm going to look for is the sample size, or how many subjects were involved in the study.

Let's take my first headline in this article as an example. Let me quote some actual headlines to reinforce my point.

"Diet Soda Erodes Teeth As Much As Meth, Crack: Case Study" - CBS News

"Soda Addiction As Bad For Your Teeth As Meth Or Crack" - Daily Mail Online

The headlines are numerous for this study, but is it true? Headlines like these may not set off your alarm bells at first, but my alarm went off and I did follow up on this study..... surprise, the study is flawed from the start.  The study size was not adequate... well that may be way understating the problem.

Dr Mohamed Bassiouny studied 3 different people, one that drank soda, one did meth, and one did crack cocaine. Already this study should be thrown out since the sample size was 3 people. The reason we never do studies this small is that a single person does not represent the general populace. Can you imagine if your performance at work was based on 3 people in the entire company? This could be good or bad, if they chose 3 really bad employee's it may not work out well for everyone else, however, if they looked at 1,000 employees, it's likely to be closer to the reality of the entire company.

PIXNIO-205885-1824x1216.jpg

I don't drink alcohol almost at all, you likely do as most people do in the world. That fundamentally makes me different from you. In order for a study to attempt to represent the populace of a large country or the world, the sample size of participants must be as large as possible to attempt to eliminate as many factors as possible that may impact that study and are outside of the scope of study.

As an example, the conclusion of this study says that meth and crack are equal to diet soda when it came to teeth destruction. Since we are only looking at a single person, then it becomes relevant to consider other factors. Does this person brush their teeth at all, do they visit a dentist? There are so many factors that may impact their teeth that we can not conclude that soda was the one that caused this.

Most valid studies have participants in the 1000+ range. I normally will even question studies in the mid to low hundreds of participants. Sample size should be one of the first things your look for when it comes to studies like this.

Get Yourself Some Control

Control, control, control. If you've ever heard about a true scientific study, then likely you've heard about the control group. If you don't already know, the control is the group that is given nothing, or a placebo that essentially acts like nothing. Why do we do this though?

Control groups give us a baseline reading. What would happen if we did absolutely nothing? This question is answered by a control, this is where we see how the item being tested would work compared to a group (control) of things or people that have not taken the drug, or really been given nothing related to the study. We are just looking to have a set of data that shows how the item being studied is compared to what happens with no intervention.

So why use a placebo in the control?

Humans are complicated creatures, we have the ability to make ourselves feel better even if we aren't and also to feel poorly even if we're fine. In studies if you give someone a placebo, there is a chance that just because someone is taking something, they may think they are getting better... even if it's just a sugar pill. In order to account for this effect, the control group may be given a placebo. This has two purposes, first, this gives a baseline of how a drug might make people feel even if it isn't working and allows the study of the real drug to be compared against the placebo effect. Second is that it does not let the patient know if they are or are not getting the drug and allows the study to eliminate the influence of a patient on the study.

So, next time you are looking at a medical study, look for a control group too.

There are many different factors that we can examine in order to differentiate a good study from a bad one. It may be a good idea to treat all studies with a healthy dose of skepticism if you read it in the news. News organizations want viewership and often blow up the true findings of studies for a few more eyes on the article. At the end of the day, reviewing a headline a little deeper may help combat misleading articles and keep you from eating pounds of chocolate that is supposed to make us live longer.