THE LEVELS OF SCIENTIFIC PROOF: I’VE DONE MY RESEARCH
This post is one that I hold dear to my heart. Probably because it affects all of us personally. Misinformation. The constant flood of false information dilutes the truth so much that we have trouble telling the two apart.
It takes exponentially more energy to deconstruct and debunk misinformation that it does to produce it and that is precisely why it’s unlikely to ever go away. One can spend days researching a single question or subject, but in seconds a non-expert can post a comment that totally puts the author's credibility into question.
Plant-based, vegan or vegetarian diets have lots of myths and misinformation surrounding them. Rabbit food, protein deficiencies, nutrient deficiencies and soy phobia are just some of the common false beliefs that come with plant-based diets. Believe me, even with extremely strong science backing up the benefits of a plant predominant pattern of eating, some will never buy in to my "plant-based propaganda".
Recently, I posted an article about keto diets. I knew I would get haters, and I anticipated and pre-planned my reaction. So when someone attacked and accused me of cherry-picking the data to force people to obey my “agenda”, I kind of chuckled and choked on my coffee. It was funny for a second, and then it wasn’t anymore. The last thing I want to do is divide people. The opposite actually. My whole mission is to help my community get healthier and to bring the teaching of nutritional science to medical schools. Doctors need to learn more about preventing disease through nutrition. Period.
For those unaware of the term “cherry-picking”, I’m basically being accused of choosing to reference only studies that have conclusions that align with my values. Firstly, my values are what they are because of the science, not the other way around. And secondly, the absolute opposite of cherry picking data is relying on peer reviewed, randomized controlled trials and then looking at meta-analysis of these trials. A meta-analysis is a statistical analysis of multiple different studies addressing the same question, aka the exact opposite of cherry picking. Anyways, I spent 14 hours researching, writing and publishing a review on keto, for one single person to take 5 seconds to reply and I quote: “Typical plant based “expert” telling us we’ll all die if we don’t bend to his will and go plant based too. No thanks. I’ll stick with what works for me.” And just like that, with zero effort, zero research and zero scientific background, a random guy deconstructs my work and puts my credibility into question. Believe me, I creeped him and his profile and I confirm he is not a closet nutrition expert. I don’t blame people for not knowing who to believe, not one bit. I went to medical school and also had trouble deciphering what was considered a healthy eating pattern. Doctors are poorly trained in nutrition and that’s a huge problem, but the main issue is our food system and how big corporations control the messages diffused to us and our children. Remember that just a few generations ago, processed foods didn’t even exist, yet today, more than 50% of our children’s calories come from ultra-processed foods, and we are unlikely to ever live in a world without them. Somehow, companies control what information we see and hear and they've done a great job in normalizing the consumption of unhealthy foods. Marketing helps keep us blinded to where our processed and animal foods really come from.
Nothing amps me up more than people saying they've "done their research", unless they've really done their research. Google is a great tool for many types of questions, and in these situations it will provide the right answers. For many other questions, it will redirect you towards the answer you’re looking for instead of the truth. I’m willing to bet that the overwhelming majority of people reading this aren’t very familiar with terms like statistical power, p scores or confidence intervals. It’s nothing to be embarrassed about if you don’t understand these terms, because once, I didn’t either. If you think you might have research interpretation figured out, you may simply be a victim of the Dunning-Kruger effect, a well studied cognitive bias that states that people with low ability at a task overestimate their own ability, and that people with high ability at a task underestimate their ability. In simpler terms, some people know so little about a subject that they have no way of truly knowing how little they really know. It's only when you study a subject for years that you can truly appreciate its vastness.
In 2018, a group of MIT researchers scoured through millions of tweets from 2006 to 2017. They made a shocking discovery. Their analysis concluded that: “Falsehood diffused significantly farther, faster, deeper, and more broadly than the truth in all categories of information.” One of their statements made it pretty clear: “IT TOOK THE TRUTH ABOUT SIX TIMES AS LONG AS FALSEHOOD TO REACH 1,500 PEOPLE”. How can we be sure that this conclusion is true? Well, the millions of study subjects, aka the tweets, being studied makes it almost irrefutable and the long study timeline, aka looking from 2006 to 2017, makes this type of data invaluable. That’s why recommandations change, guidelines evolve and time gives science time to get it right, when at first it got it wrong. You shouldn’t get mad when newer studies disprove old findings. And you shouldn’t lose trust in the scientific process. Be glad that science is always pushing to dig deeper in the data, although I do recognize that all studies aren’t created equal. If tomorrow I post that broccoli helps keep you healthy, I wouldn’t be surprised if none of you shared that post. Now imagine that I post that broccoli causes cancer. Everyone wants to be the first to share new information, even if it risks being false. That’s why it’s important to turn on your BS detector before sharing information that could be false.
Here are the main differences between google research, social media research and scientific research. This is my chance to nerd out. Feel free to join and to learn how to harness the power of your BS detector.
Firstly, one must absolutely understand that anyone can find research that supports their point of view. Let me use smoking for example. Did you know that multiple studies have found health benefits from smoking? That's right! If you google "benefits of smoking cigarettes", you'll soon notice thousands of studies bragging about the positive health effects of smoking. If you read these articles on specific social media platforms, the algorithm will naturally suggest related articles that further promote smoking and other bad habits as healthy lifestyle choices. While thousands of studies show benefits, millions show the opposite. That's why it's crucial that people understand the importance of confounding factors like industry bias, industry funding and research tools like meta-analysis. Peer reviewed, p scores and confidence intervals are also some of the concepts that should broadly be understood before one can correctly interpret data. And that’s why people that have “done their research” tend to find results that agree with their point of view. Real research includes leaving our egos at the door and looking at data coming from peer-reviewed studies of high quality, done on a large number of people and reproduced by other experts in the field. The quality of evidence will also depend on the study structure and this will be addressed later in this article.
You may have been duped
Did you know that many corporations will avoid publishing data that could be detrimental to their products? Did you know that studies on cigarette smoking were 88 times more likely to yield a positive or beneficial conclusion when the study was funded by tobacco companies or their affiliates? Did you know that it was possible to fund studies through distant holding companies in order to hide from readers where the money actually came from? There are many layers to analyze when looking at studies and the data they produce. Is the source reliable? Is it trustworthy? Do they have something to gain or to lose that depends on the study results? Is it peer reviewed, meaning is it critiqued and analyzed by other experts of the same field before being distributed to the masses? These are just a few of the many things to take into account before accepting a study conclusion or a google statement as fact. I used cigarette smoking to paint this picture since the evidence linking smoking to poor health outcomes is basically impossible to refute, but what about lesser studied subjects? Some questions will always remained unanswered. Think about a company selling a magical weight loss pill. They will likely never invest millions to study their product, knowing there’s a possibility the results could show it doesn't work. And if they do study it, they can find clever ways to publish results in their favor. Just creating the perfect amount of confusion to keep readers from knowing the truth is their strategy. The meat and dairy industry have been doing this for years. Eggs were a staple food in our house, and I thought they were superfoods. Then I read the literature published by independent researchers and saw that there was a whole other side to the story. Funny how all the studies published by the egg industry always yield positive outcomes. This is a frequently used tactic by big corporations wanting to guarantee big results. By comparing butter to bacon and sausages, it's possible to show that butter is healthy. By comparing soda to chocolate covered treats, it's possible to conclude that soda doesn't lead to excess weight gain. Studies done by the sugar, meat and dairy industries have relied on multiple tactics similar to these to make their products look less unhealthy by simply comparing them to worse ones.
In addition to the many ways to distort a study's findings, there are also many different study types. Each type of study has different pros and cons. They also have different "power”. Statistical power means that the study will detect a difference between two groups being studied, if there's a difference to be detected. For example, if a particular outcome, like a medication side effect, happens in 1/10000 people, it is likely that this will not be detected if the study has only 100 patients in it. In this scenario, the study lacks enough "power" to make that detection. Statistical power (the capacity to detect an effect, like a side effect for example), depends primarily on two things: 1) the size of the effect (is the side effect rare or common) and 2) the size of the sample used to detect the effect (does the study have 100 patients, or 100000).
Another major factor affecting statistical power is the study design. To use evidence as a means of proof depends on the quality of the evidence itself. Here's a quick breakdown of the different types of evidence.
Types of scientific evidence
Proof or research comes in many forms, all with different levels of power. Here, I'll rank the different levels of proof and explain the strength of the data they produce. This is the nerding out part that I warned you about. Feel free to jump to the conclusion if you've already done your research!
Most scientific studies can be broken down into observational (observing something that happens) and experimental (where study designers control some of the variables). In general, experimental studies are considered to provide stronger evidence and clearer cause and effect.
The strength of evidence will be listed from weakest to strongest.
Anecdotal & Expert Opinions
Anecdotal evidence is a person’s own personal experience or view. This is one of the most frequently used levels of evidence by those making claims that have not been thoroughly studied for any reason. This one pulls at people's heart strings as they play on emotion, rather than logical and rational thinking. One's anecdotal experience with a particular weight loss diet for example, might sway you into trying it, even if other stronger levels of proof show otherwise. An expert’s opinion, or someone reporting their personal experience, is considered a weak form of evidence. Unfortunately, when opinions and anecdotes are rampant, people listen and tend to get influenced.
Animal & Cell Studies (experimental)
Animal research can be useful, and often offers a guide as to which studies are useful to conduct in humans. However, it's irresponsible to guarantee that results seen in mice or test tubes will be those seen in humans, even more if these interventions are likely to have adverse effects.
Case Reports & Case Series (observational)
A case report is a documented record of a specific subject. Though low on the hierarchy of evidence, they can aid detection of new diseases, or side effects of treatments. A case series is similar, but tracks multiple subjects. Both types of study indicate correlation, not causation.
Case-Control Studies (observational)
Case-control studies are retrospective, involving two groups of subjects, one with a particular condition or symptom, and one without. For example, we can study people with lung cancer, and then look retrospectively in the past to see if they have ever smoked. They also indicate correlation more than causation.
Cohort Studies (observational)
A cohort study is similar to a case-control study. It involves selection of a group of people sharing a certain characteristic (e.g. ingestion of a specific food ), and compares them over time to a group of people who do not have this characteristic. When done on large groups of people for extended periods of time, they provide great insight. The AHS-2 (Adventist Health Studies-2), and Epic Oxford study are 2 huge cohort studies that provided great and powerful proof of a plant-based diet's benefits.
Randomized Controlled Trials (experimental)
Subjects are randomly assigned to an intervention group, that receives the treatment or intervention, or a control group, which commonly receives a placebo. The trial is said to be blinded if the participants don't know which group they're in. In trials said to be "double blind", the experimenters do not know either. Blinding helps remove bias and when these trials are done on large groups, their power to detect small effects (like rare side effects, or small benefits) increases dramatically. Keep in mind that not all studies can be done in certain situations. For example, it would be kind of tough putting a cup of broccoli in a placebo, or blinding patients from a particular diet.
Systematic Review
Systematic reviews draw on multiple randomized controlled trials to draw their conclusions, and also take into consideration the quality of the studies included. Reviews help reduce bias from individual studies and give us a clearer picture, making them the best form of evidence.
You can probably appreciate that for logistical, financial or ethical reasons, it can be a challenge to gather proper evidence for a certain question. Just like broccoli doesn't fit in a placebo pill, or patients can't be blinded into eating a keto diet, some questions will get answered with imperfect studies. Innumerable variables then affect the study results and personal biases affect their interpretation. Some people are profoundly disturbed by the fact that logic and reason alone can't generate truths. Lots of people claim that common sense should prevail, but common sense often sends us in the wrong direction. Logic and common sense are, by definition, deeply intertwined with our emotions, biases and personal beliefs. This is why many people having views or beliefs different from those proven by science tend to be “anti-a lot of things”. There are infinite examples of situations where logical thinking and common sense have proved us very wrong. By no means am I saying that science is always irrefutable. Science evolves, gets better, proves things and disproves things that were widely accepted as fact. All one can do is trust the best balance of scientific evidence available at the present time. If studies show that 1000mcg is the best dose of B12 supplementation, that's what I'll take and that's what I'll recommend. If the science changes in 10 years and shows that 100mcg is better, then instead of losing trust in science, I'll simply be glad that science is progressing and evolving, because I understand that levels of proof change, and that a single study can't answer all of life's questions. It takes hundreds or thousands of studies on a single subject, done on thousands of patients, for extended periods of time to consider their results as irrefutable facts. That level of proof is rare, so all we can do is follow what the best proof says at the time.
When recommendations are formulated from scientific research, it’s important to consider the type of study, its source as well as its level of proof. You can safely assume that nutrition studies comparing food against placebos are pretty difficult to design. How would you compare an apple a day against placebo in a double blind study? Spices are perfect for these types of studies, since they can be compressed in a capsule and compared against placebo. Also, just participating in a study will change people’s behaviors enough to impact the measured outcomes.
That’s why it’s important to recognize that expert opinions, patient anecdotes, although not negligible, are considered weak levels of proof, poorly reproducible, and one must be careful before generalizations can be made to larger populations.
Health care professionals and the general public need to be educated on the scientific literature surrounding nutrition. There needs to be complete transparency regarding the source and type of study. This will help protect the vulnerable person looking to improve their health, and help them avoid the innumerable recommendations out there that have been disproved or are part of the large amount of pseudoscience, or “bro-science” out there. I see plenty of social media posts saying take this vitamin or supplement and buy this product, when very often, there’s large amounts of scientific literature on that very subject. Some recommendations are legit, and some have absolutely no scientific backing. When someone wants to sell you a product promising health and longevity, question it. Mother Nature created most of these things already, and hid them directly in the whole foods we evolved with. I encourage you all to check out YouTube for tips on how to interpret scientific data. This may heighten the sensitivity of your spider senses and your BS detector.
Check out my website plantbaseddrjules.com and look for the “How To” section in the menu. There, you’ll find tips and tricks that helped me on my journey towards a plant-predominant diet. Everything there is completely free, no catches! If you're looking for quick, easy and healthy plant-based recipes, check out plantbaseddrjules.com and download my free recipe eBook!
Look for me on the socials, @plantbased_dr_jules on Instagram and go like my Facebook Page, Plant-based Dr. Jules. If you’re looking for some fitness motivation and are curious to see what a plant-based athlete can accomplish, follow me, @maritimeninja, on my fitness account on Instagram or check out my fitness group on Facebook, called Maritime Ninja Warrior. I'm a two-time world championship qualified athlete and you can follow my fitness journey there! You can even access the resources section by becoming a member. It's free and there, you can download free resources like my plant-based recipe eBook!
You also check out my YouTube channel here for more tips and tricks on how to embark on a plant-based journey!
Thanks so much for reading! Please consider sharing this article!
Plant-Based Dr. Jules 💚🌱
Keep taking care of yourselves and others!
Plant-based Dr Jules 💚🌱
Please show me you’ve read this post by dropping a like, and engage with me with a comment. I can’t wait to see what you guys have to say!
