Should we close question about claims taken from reputable peer-reviewed publications?

Question

I'm seeing an increase in questions that cite as their source of claim an article that would itself count as a reputable source for this site. It's from a reputable source, it's peer reviewed, or it cites plenty of other sources. More usually many of the above.

The question then takes a claim from the source and says "Is this true?".

The trouble with this is that 1) it's very unlikely that any contributors here are going to be a better position to judge the truth of the claim than the original source 2) since the original source is reputable and referenced we could theoretically say "Yes it is" simply by citing the original claim 3) Usually the only reason to doubt the claim is that the question personally doesn't believe it, or simply doesn't like the conclusion.

Can we have a close reason or something else to deal with question like this?

Answer 1

I'm pretty sure that these questions are on-topic and have been for most of Skeptics' existence. This was discussed back in 2011 and the highest voted answer by Mad Scientist says "I don't think we should shy away from critically examining scientific papers".

The questions should be reworded as needed to focus on the claim instead of the article so that answers can address the real problem (whether something is true or not) instead of just learning if the article's results are useless or not. Also, saying "debunk this" makes it sound like you've already chosen a side.

I have my own reasoning for why these questions should be allowed:

• Many of these claims can be found repeated elsewhere. It's basically a waste of time to shut them down since there's a chance someone will ask the same thing later on.
• Not everyone can read scientific articles. Even if you understand the words, you might not know the difference between blind and unblind experiments; statistical significance and chance; or something peer reviewed and something predatory.
• On a similar note, some articles are paywalled and nobody wants to pay.
• Then there's the fact that so many results cannot be reproduced. Science isn't about doing something once and for all. A good answer will address the article that's being asked about and all the other good research out there on the subject.
• I learned about just how bad the problem of reproducibility is from an episode of Adam Ruins Everything (sources, archived alternative).

See also How much can we trust peer reviewed scientific literature?

Answer 2

Just to add to what Laurel said, the reason why meta-analyses, reviews and so forth exist is that a single study might not be so conclusive. (The better meta-analyses nowadays also estimate publication bias etc.)

Furthermore, in areas like epidemiology (and not only) there are a lot of observational studies. It can be tricky to infer causation from these and studies will sometimes disagree in their conclusions, e.g. by including different confounders.

Never mind allegiance bias etc.

Answer 3

These questions should not be allowed here. Take for example, this question. This basically says, "Is this study sound?" The current answer then just says, here are some ways to evaluate a study's soundness without actually analyzing the study. So that answer could essentially be added to any question on the site that refers to a study's soundness. Questions that all get the same answer are not useful.

Worse, it creates a situation where we might have cross-question debates. Question one asks about a popular source claim. It gets answered with a reference to one or more studies. Another question now asks if that study is sound. Someone mentions a counter study, which then gets its own question...

For questions like "Is this study sound?" we should generally favor people asking on a more specific stack like Physics.SE, Economics.SE, Chemistry.SE, Politics.SE, CrossValidated.SE, etc. where domain-specific experts are more common. And we might direct them to a FAQ where soundness is discussed in general, preferably in the form of a claim if it is on the main site. Or it may be a Meta FAQ if we can't find a notable claim that more directly justifies that type of answer.

The main point of this site is to say, "Yes, that claim reasonably represents what this study said." Or "No, that claim misrepresents what the study said." Or "You can look at that both ways, because..." We should not try to be a singular arbiter of all truth. We create footnotes, not do peer review.