Should we tackle already-referenced claims?

Question

I am torn on how we should treat cases where the claim is of the form:

I read in this article that:

Ninety percent of slithy toves gyre more than they gimble. (Andersen, 1980)

Is it true that so many slithy toves gyre so much?

Researching such a question is mundane: just look up the article by Andersen.

At first blush, it seems quite pointless to crowdsource that task; it seems to show too little effort by the OP.

Should we put such questions on hold until they provide a reason for not following up themselves?

Sometimes, the reason given is that the information is in a book or behind a paywall - i.e. not immediately available for free. I don't find this particularly convincing; if the OP isn't motivated enough to pay the article writer for their research, why should anyone else be motivated enough to repeat the research?

Sometimes, the reason given is that the OP doesn't know whether to trust an article (or how to decide whether to trust an article). We need to know that information in order to answer at the appropriate level. Otherwise, we fall into the vicious circle of:

Yes. It is true that ninety percent of slithy toves gyre more than they gimble. (Andersen, 1980)

Answer 1

Yes because:

• I thought that a reason for the site is to examine evidence, not just to find evidence.

• Doesn't the need for (existence of) meta-studies imply that single-studies can be misleading?

And do you want the OP to "show effort"? I thought that this answer implied that you want it to be easy to post a question:

The hurdles for asking a valid question on Skeptics.SE are high, and we almost certainly lose potential new users because of it. I don't propose that we should let through bad questions just to make it simpler, but I am wary about making the hurdles even higher.

---

We need to know that information in order to answer at the appropriate level.

Not always: aren't we supposed to address the question/topic, not just the OP? In a sense it doesn't matter why the OP doubts the claim: because they might doubt it or believe it for the 'wrong' reasons.

Currently an OP isn't required to even understand the claim in question.

Answer 2

Summary: it's hard to fit such questions in with the standards we generally use, which is that a peer-reviewed paper is sufficient to establish a viable answer. Work is needed to review the paper in question, if there's an absence of more recent papers which cite and assess it.

Detail:

The risk here is that we end up with tautological Q&A:

Q: this paper says X, is it true?
A: Yes, this paper says X.

A thorough answer based on just that paper could be done, and might look like a lot like a paper peer-review: but then it's original research. In similar circumstances, I've tried to compare the paper with other contemporary literature on the same subject: that means I'm doing an original synthesis of existing papers, and that's something a lot of our answers do, so I hope that's ok. That way, a paper isn't used only to substantiate itself, and the answer can have decent references.

I think the biggest challenge is when it's about a paper that's just been published.

Part of the problem is that (despite some claims to the contrary) peer-review is not a gold standard. It is, rather, a fallible junk filter.

Once a paper has been about for a few years, cited and examined, then it's possible to tackle the question "is this conclusion from this paper sound?". Otherwise, unless it's an egregious error, there won't be the references around to refute it.

So when such a question is posted, the right thing to do in almost all cases will be to leave it for a year or two, then come back to it to answer it. But that's not great for answerers, and it's pretty bad news for the askers of those questions.

This is a good example: Can phobias be genetic, but created in one generation and not by natural selection?

As it stands, I think the best any answer could do in this circumstance is: "the science, as described in the paper, is / isn't* methodologically sound [reference], and it is / isn't* consistent with known theory [reference] and existing evidence [reference] (* delete as appropriate)." Which isn't really satisfactory, but it might well get a net positive score, and thus would vanish off the list of unanswered questions, though the right answer might not be established for another year or two.

Answer 3

In your example, a good answer would examine the evidence behind (Andersen, 1980). It would make some conclusion about assertion that "Ninety percent of slithy toves gyre more than they gimble".

Or, if the question is about whether or not some particular news story has accurately represented (Andersen, 1980), perhaps a good answer would simply restate what (Anderson, 1980) concludes, and compare it to how the news story described the research.

I agree with EnergyNumbers that the best we could do in the case of asking directly about a published study:

• evaluate the study's methodology
• examine how one study fits in the context of other work covering the subject
  • is this study consistent with other work?
  • is this study apply a more rigorous methodology than other studies?
  • what does the pattern of results concerning this question say (like, do more rigorous studies produce decreasing effect size, suggesting little or no effect?)
  • what does more recent research say?

I agree with ChrisW that a good answer doesn't depend so heavily on acceptance by the OP. Just examine the evidence behind the claim. Tailoring an answer to the specific reason for doubt that an OP has makes the answer less useful for the rest of the internet.