We have all had the experience of standing in the checkout line and glancing at the tabloids advertising fad diets and miracle foods telling readers that if they eat this or that, they will be able to lose 10 pounds in a week. While most of the time it is easy to identify headlines likely that lack much truth and reliability what about news and headlines regarding scientific studies? Even highly-regarded news and media outlets often promote false or misleading claims about food and health. While doing so may be an advertising tactic, it is also likely an unintended consequence of condensing the highly complicated, ever-changing science of nutrition into a one-size-fits-all assertion.
While it’s easier to trust publications promoting scientific research than the media, both can be sources of inaccurate information and can spread misleading claims. Below are several tactics to employ the next time you come across what might be a major headline-breaking study.
Step 1 – Consider the story covering the study.
Source: Ask yourself where you learned about the study. Was it posted by a friend on social media? Highlighted on the nightly news? Published in a major newspaper? Consider whether or not the study and its findings are being interpreted and promoted to create a catchy tagline that optimizes viewership. Is it too good to be true? Or maybe it’s the opposite, maybe its generating fear?
Who is the author? Is the article written by a trained scientist a food blogger, or a journalist? Do they have a history of writing balanced, scientifically-sound articles, or have they historically written more emotionally-charged content? Consider both the credentials as well as the potential motives of the author.
What’s the full story? It is impossible to capture all of the findings, strengths, and weaknesses of a study in a single headline, so it’s important to dig for details. In addition to bypassing the headline for the full article, you should also check out any source material to ensure it supports the headline and article. Not all writers are scientists who have the expertise to both adequately evaluate a study, and put the study’s findings into the broader context of related evidence on the topic. Further, whether the writer does so intentionally or not, they may look for parts of studies that best fit within the context of their own story. Make sure to seek out and evaluate the study, not just the writer’s take on it.
Step 2 – Evaluate the study
Consider if the study was peer-reviewed and published in a reputable journal. The peer-review process ensures unbiased experts rate the quality of how the study was organized and raise questions as needed. Further, peer-reviewed studies that are published in a reputable journals and reviews or guidelines published by international health agencies and policy-making bodies are considered to have more reliable findings.
Check the timing. Sometimes stories reference studies that have been published for a year or longer in order to try and raise attention about an issue. Make sure to check the study’s publication date to determine if the findings are current and relevant.
Identify the study type. Studies vary greatly in their quality and design. It is important to remember that association does not equal causation. Some study designs can identify associations between a behavior or an exposure and a health outcome, but can show that is it caused by that behavior or exposure. Further, as with all study types, it is essential to consider confounding variables (i.e. something that wasn’t controlled for by the researcher, but may impact the outcome of the study) and potential sources of bias, which could skew study findings.
Reputable research takes steps to reduce the risk that the researchers’ or participants’ preference or bias, or even chance could affect the study’s results. It is important to be aware of the fact that not all research is created equal, and thus their results are not equal. This guide from FoodInsight may help you navigate the different types of studies and what the implications of the results may be, when the studies are carried out appropriately.
In following these steps to properly evaluate headline-breaking studies, you could be part of the solution to the growing problem of the dissemination of bad science and deceiving headlines. Time to put your white hat (and reading glasses) on.