Spotting A Lie, Bad Data in Scientific Studies

"Drinking soda will rot your teeth at the same rate as meth."

The above line was the conclusion of a scientific study released a few years ago. While the study makes for a sensational headline, the data was deeply flawed. Spotting a sensationalized study can be difficult with so many headlines flying about today. Having the ability to spot a fake or misrepresented article doesn't have to be difficult though, and I'm going to show you some things to look for.

First off is the "too good to be true" measure. If it sounds way too good to be true, it likely is, trust that instinct and look it up. The study may very well find some amazing conclusion we never knew about, but the only way to tell is to look at the referenced article. Personally, I can usually get an idea if the study or headline is good in under 2 minutes, not a huge drain on our time.

Study Size

If the study doesn't trigger the too good to be true warning bells, there are some things we can look for. When looking at the efficacy of a study, the first thing I'm going to look for is the sample size, or how many subjects were involved in the study.

Let's take my first headline in this article as an example. Let me quote some actual headlines to reinforce my point.

"Diet Soda Erodes Teeth As Much As Meth, Crack: Case Study" - CBS News

"Soda Addiction As Bad For Your Teeth As Meth Or Crack" - Daily Mail Online

The headlines are numerous for this study, but is it true? Headlines like these may not set off your alarm bells at first, but my alarm went off and I did follow up on this study..... surprise, the study is flawed from the start.  The study size was not adequate... well that may be way understating the problem.

Dr Mohamed Bassiouny studied 3 different people, one that drank soda, one did meth, and one did crack cocaine. Already this study should be thrown out since the sample size was 3 people. The reason we never do studies this small is that a single person does not represent the general populace. Can you imagine if your performance at work was based on 3 people in the entire company? This could be good or bad, if they chose 3 really bad employee's it may not work out well for everyone else, however, if they looked at 1,000 employees, it's likely to be closer to the reality of the entire company.


I don't drink alcohol almost at all, you likely do as most people do in the world. That fundamentally makes me different from you. In order for a study to attempt to represent the populace of a large country or the world, the sample size of participants must be as large as possible to attempt to eliminate as many factors as possible that may impact that study and are outside of the scope of study.

As an example, the conclusion of this study says that meth and crack are equal to diet soda when it came to teeth destruction. Since we are only looking at a single person, then it becomes relevant to consider other factors. Does this person brush their teeth at all, do they visit a dentist? There are so many factors that may impact their teeth that we can not conclude that soda was the one that caused this.

Most valid studies have participants in the 1000+ range. I normally will even question studies in the mid to low hundreds of participants. Sample size should be one of the first things your look for when it comes to studies like this.

Get Yourself Some Control

Control, control, control. If you've ever heard about a true scientific study, then likely you've heard about the control group. If you don't already know, the control is the group that is given nothing, or a placebo that essentially acts like nothing. Why do we do this though?

Control groups give us a baseline reading. What would happen if we did absolutely nothing? This question is answered by a control, this is where we see how the item being tested would work compared to a group (control) of things or people that have not taken the drug, or really been given nothing related to the study. We are just looking to have a set of data that shows how the item being studied is compared to what happens with no intervention.

So why use a placebo in the control?

Humans are complicated creatures, we have the ability to make ourselves feel better even if we aren't and also to feel poorly even if we're fine. In studies if you give someone a placebo, there is a chance that just because someone is taking something, they may think they are getting better... even if it's just a sugar pill. In order to account for this effect, the control group may be given a placebo. This has two purposes, first, this gives a baseline of how a drug might make people feel even if it isn't working and allows the study of the real drug to be compared against the placebo effect. Second is that it does not let the patient know if they are or are not getting the drug and allows the study to eliminate the influence of a patient on the study.

So, next time you are looking at a medical study, look for a control group too.

There are many different factors that we can examine in order to differentiate a good study from a bad one. It may be a good idea to treat all studies with a healthy dose of skepticism if you read it in the news. News organizations want viewership and often blow up the true findings of studies for a few more eyes on the article. At the end of the day, reviewing a headline a little deeper may help combat misleading articles and keep you from eating pounds of chocolate that is supposed to make us live longer.

Anti-Vaxxers, Stop! Just Stop!

If you live in the United States, then it's likely you've run across an anti-vaxxer or at least heard stories about them. Proclaiming themselves as saviors of the human race, anti-vaxxers were at one time harmless idiots that were so rare you need not worry about them. Problem is, this group has grown, and with it, the danger to society has grown as well.

Anti-Vaxx History

Since 1796 when Edward Jenner first tested the use of Cowpox as a protective measure against smallpox, we have been developing vaccines and in turn protecting the general population from a range of horrid diseases. In the short 200 year history of vaccinations, we've managed to completely eradicate smallpox, and almost completely eliminate polio, measles, mumps, rubella, whooping cough, and other diseases. All of these diseases are preventable and for a very low cost. 

Unfortunately, we are now seeing the return of some diseases as young children and adults who were never vaccinated start to come in contact with these diseases at an increasing rate. We can trace this trend back to one person in particular, Andrew Wakefield. If you know the background of Andrew Wakefield, then you have looked upon someone who might as well be the next incarnation of the devil. 

Astronaut Scott Kelly vaccinates himself in orbit

Astronaut Scott Kelly vaccinates himself in orbit

What makes Andrew such a deplorable human being? It's the way he started the anti-vaxxers down their ill-fated path. Andrew authored a scientific paper in the journal "The Lancet" in 1998.

Wakefield and 12 other authors proclaimed that there was a tie between certain vaccines and autism. In the publication, the authors pointed the finger at the MMR vaccine, in particular, claiming that 8 of the 12 children in the study developed autistic behaviors within 2 weeks of the MMR vaccine being administered.

Wakefield also took to the press to demand the MMR vaccine be withdrawn and this act alone was a major catalyst to the current anti-vaxx crazies that roam the streets. Wakefield had not even published the paper when he took the story to the press. Wakefield claimed to be doing this for the interest of the public. He was hiding a secret about the study though... he made up the data! 

Andrew Wakefield had performed the study after he was hired by a lawyer. The lawyer was preparing a lawsuit against MMR vaccine producers and wanted a link between the vaccine and side effects in an effort to make their suit more powerful. This major conflict of interest alone should have been enough to call into question the findings of the study. Wakefield took things a step further though and changed the data in the study to fit what the desired outcome was.

In 2010, the journal retracted the study and Wakefield lost his license. Charges against Wakefield included the following:

  • He was paid for the results of the study to match lawyers desires
  • Ordering medical tests not indicated by patient history and without approval
  • Withheld how patients were recruited into the study
  • Paid for children's blood samples at his son's birthday party
  • Showed disregard for distress or pain of children in the study
  • Did not gain ethics approval for the study

7 years after Wakefield lost his license, we are still battling an anti-vaccine movement based completely on myth and a made up study.

The New Anti-Vaxxer

Now anti-vaxxers have expanded their excuses for deciding not to vaccinate their children and they are all just as flawed as the original study that sparked this craze.

One of the top excuses I've heard floated is that current vaccines have mercury in them and that causes autism. 

This flawed logic likely stems from the fact that mercury causes neurological disorders, but completely ignores the fact that there is no mercury in modern vaccines. What anti-vaxxers have glommed on to is that in the past, vaccines have used thimerosal to prevent to growth of bacteria and fungus within the vaccine itself. Mercury is one of a long line of atoms that makes up thimerosal. 

The chemical makeup for thimerosal is C9H9HgNaO2S, as you can see Mercury (Hg) is only one part of the makeup of this compound. Having mercury as part of a compound does not automatically give the compound the same health effects of the atoms that make up the compound. A great example is Chlorine, in gaseous form Chlorine is very toxic to breath in, but bind it with sodium and you create table salt. Salt is harmless in most applications and when eaten in small doses. 

Anti-vaxxers position on thimerosal gets even worse when you realize that the CDC had thimerosal removed from all vaccines in 1999 as an additional precaution in case it was harmful. So every vaccine administered since 1999 has contained no mercury at all. 

More information can be found on the CDC website here.  

Vaccine to Autism Rates

Locking down the rates of autism claimed by vaccine detractors is a difficult number to find since scientific articles and studies do not support this idea. However, even if it was true that vaccines caused autism, then it would still be worth administering vaccines. 

When accounting for all of the deadly or disabling diseases we vaccinate against, current autism rates would not justify us eliminating vaccines. Since the link isn't actually true, we should not give the idea any thought. 

Community Protection

Anti-vaxxers often focus on a single made up figure, autism rates. When this is your sole focus, you are going to miss the bigger picture. Vaccines are not just protecting the individual that it is administered to, there is such thing as herd immunity. 

Throughout the world there are people who are unable to receive vaccines. Such cases are often the result of individual allergies to either the vaccine, or the person could be immunocomprimised.

People that are unable to receive a vaccine must rely on this herd immunity concept. Protection for such people is completely reliant on as many people in society being vaccinated as possible. If society is immune, then the disease has little chance to be carried to a person unable to be vaccinated. Here in lies a large flaw in the anti-vaxxers position. By not taking a vaccine, you are not just putting yourself or kids at risk, but you can now act as a carrier to those unable to receive a vaccine. 

 Growth in Autism Numbers

If there is no link between autism and vaccines, where is the increased number of autism cases coming from?  

There really is a simple answer for this. Over the last few decades, autism has gained more publicity and better detection methods. Prior to being called autism, many cases likely were misdiagnosed or just written off under a colloquial term. As detection methods became more accurate, and as autism knowledge became more widespread, the more cases were diagnosed. This really is a case of a disease being better defined and easier to diagnose. 


Knowledge is power. If you want to read the studies that back up what I've said in this article, here is a list of studies you can read, and please, for everyone's safety, get yourself and your kids vaccinated. 

Study Disproving a Link Between Vaccines and Psychological Disorders in the First Year of Life

Study Finds No Link Between MMR Vaccine and Autism

Webpage with Index of Studies on Vaccine and Negative Effects