Null Results, No Headlines

article image

Many scientific studies have been so embraced by the media, they’ve become facts we often regard with, “Well, duh.” Consider bilingualism: It’s the full-proof edge! Hire a Spanish-speaking nanny! Pop in Rosetta Stone and double your salary!

Despite it being an invaluable skill, it’d be wise to hold off on the “duh.” A new metastudy suggests that there’s a publication bias to what scientific journals choose to headline—ones that eventually make their way through the mainstream media and into the public’s bank of “common sense” facts. That’s because there’s more meat to a headline that claims there’s a link between two things (*blank* causes cancer, for example) than a study that finds the same research inconclusive or that denies the connection entirely.

One research team from Edinburgh University looked at 104 papers on bilingualism that were presented at academic conferences (typical venues for researchers to receive edits before attempting to get published in peer-reviewed journals). Despite there being an even split between research that either supported or questioned the advantages to bilingualism, only 36 percent of papers that challenged the popular notion got published, compared to 63 percent that supported it.

But publications are not the only ones to blame; another study looked at how often researchers reported their null results. One Stanford team analyzed projects run through the National Science Foundation’s Timeshare Experiments in the Social Sciences (TESS), which keeps records of all experiments it’s conducted. The team could then figure out what has yielded positive or negative results and if those findings have been published, written but never published, or never written at all. While they found that publication bias is, indeed, real, they found that researchers aren’t submitting their null results, either. Of the 47 negative outcomes in the TESS data, only 38 percent were even written. Of the 170 positive results, roughly 93 percent were reported. 

Neil Malhotra, co-author of a new study on publication bias and a professor at Stanford’s Graduate School of Business, said it’s important for researchers to at least write down their null results so that other scientists can be aware of them. But it’ll take institutional changes, he said, such as new journals dedicated to negative findings so scientists feel encouraged to report these results.

Although inconclusive studies lack drama, they provide science with context, Malhotra said. The Edinburgh research team summed it up in their press release: “All data, not just selected data that supports a particular theory, should be shared, and this is especially true when it comes to data regarding issues that have enormous societal relevance and implications, such as bilingualism.”

Image by Sebastian Wiertz, licensed under Creative Commons.

In-depth coverage of eye-opening issues that affect your life.