Research studies help advance scientific and public understanding of a vast number of public health topics. In the same voice, one research topic could have a plethora of studies that all come to different conclusions. If one study finds that meat is bad for heart health, there could be another that shows the complete opposite. This is often due to varying approaches to study design, methodology, result analysis, and funding.
Considering how these issues can bring the integrity of scientists into question, Harvard Medical School (HMS) computational scientists generated a tool called vibration of effects (VoE). It acts as a computational approach to testing the validity of research findings, and is encouraged for use by researchers to audit their own findings before publishing. It’s even available and free for use online.
It began when Chirag Patel, associate professor of biomedical informatics in the Blavatnik Institute at HMS, developed the tool several years ago. To dig deeper, he collaborated with Brad Tierney, post-doctoral student at Weill Cornell Medical College, and Aleksandar Kostic, assistant professor of microbiology at HMS and assistant investigator at Joslin Diabetes Center. Together, they developed a three-part publication series to determine how microbial species and genetic expression relate to disease development.
The first study explores the genetic makeup of human gut bacteria, linking groups of bacterial genes to certain diseases. The second study builds off this by officially introducing the VoE tool, using it to audit and compare statistical analyses.
In the latest study, which unifies all parts of the series, the scientists explained how the tool was valuable for determining microbe-disease pairings across prevalent conditions like inflammatory bowel disease (IBD) and cirrhosis.
Additionally, the team showed that the VoE tool can even be used to determine possible “extra” or confounding variables that weren’t accounted for in study design. This was done by running 6+ million statistical modeling strategies on previous study findings, and testing different variable combinations to not only identify problems with results, but to determine the cause. “It can help them understand why they may be getting conflicting findings on the same research question, and it can help them dig deeper and find links they may otherwise overlook,” Tierney says in a statement.
The results of the latest study show that one-third of 581 reported microbe-disease associations were inconsistent, often due to other variables and differences in study design. Most shockingly, the scientists found that for the link between gut bacteria and both types of diabetes, over 90% of findings were inconsistent.
From this trio series, the researchers urge that modeling VoE become a crucial part of data analysis. “This approach is one way to maximize researchers’ confidence in the results they are getting from their analyses before they even publish them,” Tierney adds.
Further work will be done to refine the tool for maximum influence.
The study is published on PLoS Biology.