How much can we trust peer reviewed scientific literature?

Question

One of the distinguishing characteristics of skeptics.SE is that we expect answers to be referenced preferably to peer reviews scientific publications. But do we also need to differentiate among good science and sloppy science? To express this point another way, can we trust all peer reviewed science equally well?

As an example that points to the nature of the problem, I recently posted a (now closed) question based on some quotes from James Lovelock where he alleged that as much as 80% of the published work on CFC science (and possibly also climate science) was sloppy or fraudulent: https://skeptics.stackexchange.com/questions/14602/does-the-quality-of-published-scientific-research-decline-when-there-is-a-consen. There are issues with the way I posed the question (not least because I attached a reason to Lovelock's claimed effect) but it is clear that Lovelock thinks the majority of the publications in some areas are rubbish.

Lovelock isn't the only person to raise the issue. Attempts to replicate much of the medical literature often result in low success rates and the whole literature reporting clinical trials of pharmaceuticals is systematically biased by selective publication (see Ben Goldacre's book Bad Pharma).

I'm going to attempt to redo the question on the main site, but the point of raising the issue here is that it seems to me to pose a very significant problem for skeptics.SE. If a large proportion of the literature is sloppy, fraudulent or wrong, how does the community differentiate among well-referenced answers quoting dodgy science and well-referenced answers quoting good science?

I don't have any good suggestions other than to somehow improve the weight of answers using meta-analyses. I'd be interested to see whether others think the problem is serious and whether there are other ways this community could deal with it.

---

Update

I came across this article from The New York Times and though it worth including as it gives a remarkably clear insight into why some published types of scientific study produce less reliable results than others (and why poor reporting of the differences results in many misleading newspaper and media stories). In fact it would almost be worth including is on skeptics.SE as standard advice for how to report scientific results. The article argues (my emphasis):

R.C.T.s [randomised controlled trials] are often very difficult to set up properly and can take many years to carry out. As a result, most research we read about involves just correlational studies. John Ioannidis, in a series of highly regarded analyses, has shown that, in published medical research, 80 percent of non-randomized studies (by far the most common) are later found to be wrong. Even 25 percent of randomized studies and 15 percent of large randomized studies — the best of the best — turn out to be inadequate...

Media tend to present almost any scientific result they report as valuable for guiding our lives... Too many news reports present experimental results as providing good advice on which we can reliably act. In most cases those results would be better viewed as mistakes pointing to a next step that will be a bit less mistaken.

Science reporting would be much improved if we had a labeling system that made clear a given study’s place in the scientific process. Is it merely a preliminary result (a small-scale heuristic study meant to suggest a hypothesis that will itself require many stages of further testing before we have a reliable conclusion)? Is it a larger-scale observational study (showing a correlation but by no means establishing a causal connection)? Is it a large-sample randomized controlled test (establishing a causal connection, given specific conditions)? Or, finally, is it a well-established scientific law that we know how to apply in a wide range of conditions?

Paying attention to this (crude) classification would greatly improve many answers here.

Answer 1

Can we trust a single peer-reviewed article?

In general, no.

Peer review isn't a guarantee. It's a first-stage junk filter, nothing more than that.

And it's an imperfect filter - sometimes junk gets through.

However, there are processes within the literature, after something gets published. It gets discussed. People try to replicate the results. They sometimes succeed, they sometimes fail. Either way, things get learnt, and written up.

People also look at conflicts of interest, in the post-publication cross-examination. And a lot of work sponsored by pharma, fossil-fuel and tobacco has been identified as fraudulent: so there are mechanisms in place to pick those things up. It's imbalanced: the corporate pseudo-science tends to be backed by billions of pounds/dollars/euro, whereas those working to expose the fraud have small research budgets. But they can publish.

So although a single peer-reviewed article can't be trusted, a body of articles gets more weight. Articles that reinforce each other across different universities, different journals, different publishers, paid for by different funders, have greater weight, because they're coming from competing places.

What does this mean for us?

The biggest difficulty is dealing with controversy. The stock in trade of those in the anti-science business, for many years, has been manufacturing controversy from nothing. The same PR smearing gets used again and again

• the creationists / cdesign proponentists / intelligent design advocate "teaching the controversy" to shoehorn creationist anti-science alongside well-established scientific evolution teaching.

• Tobacco-sponsored anti-science lobbyists have as their aim the manufacture of doubt, to cloud the issue with the public and with policy-makers, to get their pro-tobacco anti-science onto the same platform alongside the well-established science of causal links from smoking to health damage.

• fossil-fuel sponsored anti-science lobbyists use those same techniques, to cloud the issue with public and policy-makers, to try to manufacture doubt from nothing, setting anti-science against the well-established science of causal links from greenhouse gas emissions to global climate change.

So we must be very careful when answering, not to present as controversy something which has been manufactured by the anti-science PR. You can find some kind of reference for any bit of pseudo-science, whether it's anti-gravity, perpetual motion machines, luke-warm global warming, safe tobacco, creationism or whatever. And where the religious stakes or sunk infrastructure costs are high, then there will be big money, and thus a lot of publicity, backing the anti-science.

And although it doesn't work when writing a journal paper, when writing a Skeptics answer, we can do a lot worse than refer to the pedigree of authors and of journals. We can also use corroboration between papers to make stronger answers, which is a staple of scientific publishing.

Pedigree

Pedigree matters. Not as much as some think. More than some think. But it does matter. Pedigree of authors, pedigree of journal.

A technical paper from bottom-tier journals such as Energy & Environment is pretty much worthless. Their pedigree, although nominally a peer-reviewed journal, is one of repeatedly publishing absolute tosh (e.g. their paper on how the Sun is made of iron), and of not retracting stuff that's been show to be worthless or worse.

Top tier journals such as Nature and Science do slip up occasionally; but they of very high pedigree in general@ their standards are very high, their review process is rigorous, and they do care about publishing corrections and the like. So a single paper in Nature, in general, counts for more than a single paper in Energy & Environment. That's not necessarily true for every single possible combination of a paper from each, but it is true in general.

Similarly, some authors get a pedigree for high-quality work in a particular field. That's almost always a very narrow field. Within that field, that makes their papers more reliable. However, as soon as they go outside their field, the may be no more credible than the next lay person. James Lovelock is a good example here: his work in popularising the ecosystem & symbiosis research of Lynn Margulis in the late 1960s was very good pop science. However, he has frequently gone outside that speciality, and written an awful lot of tosh, which, at various times subsequently, he has admitted he was very wrong on, such as his 1970s refusal to accept the science of how CFCs were damaging the ozone layer, creating a major hazard. His pedigree as a populariser of ecosystem science doesn't bring any credibility to other kinds of work. And some lobbyists will cultivate more than one pseudo-controversy: Stephen Milloy is infamous for his junk science on both tobacco and climate change; Roy Spencer is an "intelligent design" proponent infamous for his climage change denial and publishing of climate "science" that was unfit for peer-reviewed publication.

Corroboration

Where there are conflicts in the literature, answerers need to do a lot more work to put together a strong answer. The problem there is that to do really good job at that, you need to do much of the work of a review paper.

So why not just write that review paper, and get proper academic credit for it, rather than putting together an answer here?

Of course, that would just result in a single peer-reviewed article, and those can't be trusted ...

_{More notes on Lovelock, just in case the question on the main site gets deleted as off-topic:
Yes, Lovelock continued well after the damage was known, denying the significant damage to the Ozone layer from CFCs.
Here he is in 1978, at Fluorostrat 78, a conference sponsored by the chemical industry: (New Scientist, 12 October 1978, p94)}

_{Professor Jim Lovelock ... commented that already observations of stratospheric ozone show that as yet no detachable [sic] influence of CFCs on the ozone layer had been found.}

_{And here he is again in 1989 (New Scientist, 23 September 1989, p64)}

_{... how about the ozone layer? Most of the world believes that CFCs from aerosol cans, refrigerators and the like are causing a hole to form in the ozone layer over Antarctica (and perhaps the Arctic too) each spring. But not Lovelock. He believes that the giant volcanic eruption of El Chichon in Mexico in 1982 threw so much dust into the stratosphere that chemical reactions involving these particles were primarily to blame for the opening of the ozone hole.}

Answer 2

You don't become a skeptic by changing your sources. Nobody should believe something just because it's a highly upvoted answer on this website. A reader should always evaluate the argument that an answer makes before he starts believing in the argument.

After evaluating an argument you should not judge it as being true or false but rate your confidence in the belief. Put a percentage on the likelihood that the claim is true.

It would be interesting to have a website that mixes elements of prediction book with elements of skeptic.SE. For better or worse skeptics.SE can't be that website. It lacks the necessary feature to automate the process. On the plus side it's much easier for a newbie to understand how skeptics.SE works than it would be to understand a prediction book/skeptics.SE hybrid.

Very few answer on skeptics.SE should encourage you to believe that the likelihood of a claim being true is 99.9% if it was 50% before you read the answer. A single academic paper just doesn't give you that evidence.

If have to ask yourself why you need evidence. If someone advocates cutting down CO2 emission than I'm okay if he gives me evidence that suggest that climate change happens with 90% probability. It's enough that I trust the people who advocate cutting CO2.

If the person advocates large scale geoengineering 90% are not that I trust him and support his proposal. A huge problem in the public discussion of science is that few people can mentally distinguish an event having a 95% likelihood of being true from it having a 99.999% of being true.

As science progresses and we have an increasing ability to do things that might have a 5% chance of destroying our earth it's increasingly important that influential people learn that distinction.

Answer 3

Counterargument: http://xkcd.com/54/

This community is made of experts in evaluating the validity of claims - which includes trying to understand how good is the literature.

I don't see what you mention as a problem at all here, and actually that's why we have a voting system. Answers with bad references are often down voted, including answers with poor studies. Ergo, we do self correct.

---

I would like to expand my answer a bit with a series of quick and easy rules of thumb that could be enacted by the community.

Verify the source and its origin.

We accept answers with no papers on very many topics, for example on history. In those cases we look at the reliability history of the publisher of the reference. I don't think the situation you describe is any worse — particular attention is needed.

Describe controversies, do not take part in them.

Is there a significant controversy surrounding a topic or a claim? The answer should describe it the controversy and not take part in it. This includes discredited papers on the matter or significant disagreement on the specific claim by experts in the field. This needs to be very specific. Surely there are negative results being stashed away as Dr. Goldacre says. Let's mention it only where there is evidence of a controversy, not as a general disclaimer on all pharmacology questions.

Answer "we do not know" when there is not enough evidence.

Is the topic insufficiently studied? Are the papers not particularly good in that regard? The answer should mention that and not take a strong position without evidence.

Answer 4

I agree that this is a fundamental problem. Every idiot can get a paper published, and even in a peer-reviewed journal. That’s unfortunately a fact, and it goes way beyond shoddy research.

For an example just look at the latest Séralini paper which has been characterised quite accurately as blatant fraud.

If even obviously fraudulent “research” can pass peer review, is Skeptics doomed?

Probably not. Even though there are exceptions, the scientific community usually quite quickly finds a consensus in such situations – either after some follow-up research or after intensive discussion in letters to journals.

The more interesting the contentious results, the more active the discussion. As uninvolved people we can use that as a gauge of the truthfulness and reliability of the research. In the case of the Séralini paper, dismissal from the scientific community was immediate and final. Nobody who refers to the discussion can get any other conclusion than that the paper was worthless.

In other cases, there may not immediately emerge a consensus – consider the faster-than-light neutrinos – but here, too, the research community dealt with the issue in such a way that even outsiders can draw conclusions.

There are very few cases where the situation is less clear cut.

• One is the current debate about the existence of, and the human contribution to, global warming. However, any honest outside observer must be aware of the overwhelming consensus in the field (even more overwhelming now!), and the fact that although there is a vocal minority contesting that consensus, their alternative theories have been disproved, time and time again. And as more data is collected, the predictions become more extreme, rather than less so.

  In summary, although the situation here is chaotic, any honest outside appraisal cannot fail to notice the great weight behind the research

• Yes, most published research is probably false, and goes unchallenged for years. However, that’s actually a good hint that the results simply aren’t that interesting, with very few exceptions¹. For Skeptics, this part of published science is mostly irrelevant (it is a problem in science itself though).

As a hint: when encountering a paper with controversial results there’s a very simple litmus test to assess its credibility: is it published in a high-profile journal? It’s a bit unfortunate but research that is published in high-profile journals is simply less likely to be completely false (alas, there are exceptions here, too). The reason is simple: controversial results from a solid study are always first submitted to the “god tier” journals: researchers want it published there. If those journals reject it, the next lower one is tried. And so on. So if all of the big journals reject a paper with intriguing results, there’s something wrong, and lots of researchers (namely the reviewers) agree on that.

Finally, remember that science simply has uncertainty built in: there’s always the possibility of results being overturned and consensus changing. But that shouldn’t be seen as a problem – in fact, it’s the highest quality of science, and what sets it apart from all other Attempts At Explaining Nature.

---

¹ There was a horrible mistake in the late 80s where a few studies wrongly concluded that a given heart medication was effective, killing thousands of patients over the next decade until the error was found out. Can’t find the reference now.