BP_blog
Skeptical Science

Questionable Data Plagues Scientific Research



Those who think science is the measure of all truth might want to check the data first.

Listen Now | Download


Eric Metaxas

Here’s a quote for you: “A lot of what is published [in scientific journals] is incorrect.” Care to guess where those words appeared? Not on a website that questions the “consensus of experts on climate change.” Nor do they appear in a publication associated with intelligent design or other critiques of Neo-Darwinism.

They appeared in the April 11, 2015, issue of the Lancet, the prestigious British medical journal.

The writer, Richard Horton, was quoting a participant at a recent symposium on the “reproducibility and reliability of biomedical research.” Specifically, the symposium discussed one of the “most sensitive issues in science today: the idea that something has gone fundamentally wrong with one of our greatest human creations.”

And he’s referring to scientific research—the research that not only purports to tell us how the world works, but, increasingly, how people should order their lives and societies.daily_commentary_06_01_15

As Horton told Lancet readers, “The case against science is straightforward: much of the scientific literature, perhaps half, may simply be untrue. Afflicted by studies with small sample sizes, tiny effects, invalid exploratory analyses, and flagrant conflicts of interest, together with an obsession for pursuing fashionable trends of dubious importance, science has taken a turn towards darkness.”

He continues, “In their quest for telling a compelling story, scientists too often sculpt data to fit their preferred theory of the world.”

We recently saw an example of this in a story about a much-publicized study purporting to show that voters were likely to change their minds about same-sex marriage if they were visited by gay pollsters who shared their stories with them.

Researchers seeking to reproduce the findings found discrepancies in the data and asked the original researcher for the original data. The researcher was unable to produce the original data. This led the lead researcher to request that the study be withdrawn. Even supporters of same-sex marriage acknowledged that the study and the conclusions drawn from it were fraudulent.

While this was the most-publicized example of a study that turned out to be untrue, it’s far from the only one. In late April, researchers published the results of their efforts to replicate 100 of “psychology’s biggest experiments.” They were only able to get the same results in 39 of them.

Commenting on the failure, Daniele Fanelli of Stanford told the prestigious journal “Nature” that “reproducibility rates in cancer biology and drug discovery could be even lower.” She added, “From my expectations, these are not bad at all.”

Not bad at all?

It’s true that to err is human, but insisting that science is the best and most reliable guide to life despite its checkered track record is more than error. It’s chutzpah.

According to Horton, the participants at the symposium he attended agreed that “something needed to be done” about the “bad scientific practices” he describes, but none of them were sure about what that “something” might be.

Newsletter_Gen_180x180_B

Well, I have a suggestion: stop telling everyone else that science is the best, if not only, way to answer life’s big questions. While this advice may not fix the problems in the lab, it keeps the “turn toward darkness” from spreading misinformation to the rest of society.

One more suggestion: if a lot of the stuff being published is “incorrect” or “untrue,” please refrain from comparing people who question the scientific consensus to Holocaust deniers and flat-earthers.

A little bit of humility would not be bad at all.

Further Reading and Information

Skeptical Science: Questionable Data Plagues Scientific Research
A healthy dose of skepticism can often be a good thing—even when it comes to science. As Christians, we must oppose the worldview of scientism—the notion that science has the answers to all that matters. After all, scientists are fallen humans as well. And as Eric implied, just because the word "science" is included in a study or research report does not make it less subject to error or misinformation. Check out the links below for in-depth information on this topic.

Resources

Doubts About Study of Gay Canvassers Rattle the Field
Benedict Carey, Pam Belluck | New York Times | May 25, 2015

Offline: What is medicine’s 5 Sigma?
Richard Horton | The Lancet | April 11, 2015

Why Most Published Research Findings Are False
John P. A. Ioannidis | PLOS Medicine | August 30, 2005

Retracted Scientific Studies: A Growing List
Michael Roston | New York Times | May 28, 2015

How often are scientific studies retracted?
Amanda Schupak | CBS News | May 26, 2015


Comments:

Thanks for pointing that out. I've made the correction.
Typo
You need a "to" added in this sentence I think:

"They were only able get the same results in 39 of them."

Then I will be ready to share it. :)




Commentaries