"Checking" an already very reliable source

Question

There is a question about expectation of police killing of African American males. The statements the asker would like verified can be addressed by reference to an article in the Proceedings of the National Academy of Sciences, a prestige, peer-reviewed journal from one of the top scientific organizations in the world. Citing this article in support of the claim would be a slam dunk - we would say "Yes the figures are supported here", quote the article and post it in an answer. The answer would almost certainly receive many upvotes for thorough research and for citing of exceptionally reliable sources, the site would gain prestige for careful research.

The only glitch is that the question already links to that article. But the point is - why does that change anything? If the article would be an excellent answer to a claim found on Youtube, why is it any less reliable in answer to a claim that happens to link to itself?

Nobody on this site is likely to have the expertise to realistically critique an article of this nature - and if they did we wouldn't know that they were telling the truth. The article has already been verified by reviewers much more competent than us. So the question has essentially rendered itself unanswerable, unless we simply allow the article to be a reference for itself. Oddthinking says: "A response should cite analysis that is at least to that level, or stronger.". Since we know that will not happen here, we should either cite the original paper or close the question as unanswerable.

So should we do that, and allow the article to be used in the answer?

Answer 1

Just re-using the 'data' from the question is itself and merely stating, "yep, seems legit" is quite unsatisfactory.

"Skeptcisim is unwarranted" is a statement I consider always 'false' – especially for the purposes of this site.

Now, questioning Einstein on photo-electric effects might seem a bit pretentious, but 'scientific skepticism' means that questioning the science, the methodology, the data, the interpretations and conclusions or how they are presented and reported are all on the menu. Always.

Oddthinking commented below an answer that we look for equally good or stronger evidence, and we might compare these results and findings with previous results, reactions to the current study … and the list goes on.

Specifically, this study — Frank Edwards & Hedwig Leeb & Michael Esposito was mentioned in: "Risk of being killed by police use of force in the United States by age, race–ethnicity, and sex" has an ideal jumpboard as being mentioned in a Nature article:

A pair of high-profile papers published in the past few weeks (1,2) come to seemingly opposite conclusions about the role of racial biases.
— Lynne Peeples: "What the data say about police shootings. How do racial biases play into deadly encounters with the police? Researchers wrestle with incomplete data to reach answers."

And only 1 being our source here and 2 the 'seemingly opposite one'. To me, that means the hunt is on… ¹

If we look further we find a string of articles handling the issue. Or trying to. Within seconds I had a long list of studies, all highlighting different methods, data points, statistics – and conclusions. Some bad, some better, not all the same.

No single one study is perfect, nor can it be. Absolutely no matter in what kind of prestige journal anything is published. The present study is not hammered into a philosopher's stone, nor is it the pinnacle of current scientific consensus.

And here, the study in question may be good, but I would expect at least a detailed explanation:

• why is it considered to be a good source
• why is *this particular paper a good one'
• of the limitations and contradictions from with the study itself or
• in comparison to other studies, whether earlier or later.
• Then a general weighing against the other studies out there, and
• how an answerer would criticise (or 'review as a peer') the current paper, and
• how other authorities have handled this specific issue.
• Did it receive lots of comments,
• was it cited approvingly or in a shady light?

A hint I'd like to share: in social sciences like that one needs to take a thorough look at the foundations of that study, before swallowing a concluding soundbyte about a 'what' without ever getting to know the 'how' of definitions, limitations etc. I want to get to the 'why'. That a source is otherwise 'reliable' is 'nice', but since even the dumbest crackpot may utter a statement that is 'true', but nice sources fail us from time to time, 'nice' is not enough.

Regarding the indeed in general 'good' reliability of this source, we mustn't just swallow everything 'as is' just because it is published there in PNAS. Consider this very recent example:

Dear Editorial Board of the Proceedings of the National Academy of Sciences,
We are writing with deep concerns about a paper recently published in your journal, entitled “Identifying airborne transmission as the dominant route for the spread of COVID-19.”1 The paper made extraordinary claims about routes of transmission, the effectiveness of mask-wearing, and by implication, the ineffectiveness of other non-pharmaceutical interventions. While we agree that mask-wearing plays an important role in slowing the spread of COVID-19, the claims in this study were based on easily falsifiable claims and methodological design flaws. We present only a small selection of the most egregious errors here.[…]

— Meta-Research Innovation Center At Stanford: "Formal Request For The Retraction Of Zhang Et Al., 2020", June 18, 2020. (Signed by a long list of scientists from a wide range of medical fields.)

This is not an isolated 'first' incidence at that paper, but rather a by now common theme with its own Wikipedia page: Why Most Published Research Findings Are False, quoting the title of a most important Ioannidis paper.

To be clear, answering this meta-question sticklishly: I agree that we should "allow the article to be used in the answer?" — but as part of an answer, as that 'the publishing venue is known as being a quite good source' is a valuable 'hint'. Source monitoring is important. And if the paper itself is good, then we cannot be prohibited from using it as well.

In that case now, it looks more like "I am not dignifying this 'skepticism' with a proper answer but remind you to never question authority again. Trust your masters and obey." To be clear: "Yes, that's world class" and 'that's it' is an amplified pure fallacy called "Appeal to Authority". While we do that here on SkepticsSE indeed – and too much for my tastes actually – an answer that simply stops there is not an answer (that is 'of high quality', that is: also not inherently flaggable as 'low quality' or 'not answer') but reads like either lower quality, that while written as an answer, is more like comment, an introduction to an answer, etc.

A single appeal to authority is unwarranted. And if it is the same authority that should be questioned and scrutinised here, we achieved a lot of clicks, perhaps, but failed to promote our cause, surely.

Thus I definitely said that to accept anything on trust, to preclude critical application and development, is a grievous sin; and in order to apply and develop, “simple interpretation” is obviously not enough.

---

¹Curiously, the source2 for that nature article was also in PNAS, has had massive feedback, corrections, letters, and now has been retracted.

Answer 2

I haven't voted to close (but I haven't written an answer either). I believe this type of question can be good for the site and it's readership, even if it is hard to give a good answer in the usual way.

Reasons to keep it:

• We should encourage people to do some research and think critically before/when posting a question. Closing the question because they have found a good source feels counterproductive.

• Being able to find a source is not the same as being able to critically evaluate the source. Both are required to properly challenge a claim.

• Providing a good source in the question does not impact the value of the question in terms of the notability of the claim, or any other measure by which questions are usually judged.

It would be hard to write an answer along the usual lines (here's a top notch source! Wham, bam, done!). But it might be possible to write a good answer which:

• Provides other high-quality sources.
• Explains why the source in question should be considered reliable.
  • Explains (very briefly) how the scientific method works and what a journal paper is.
  • Explains what makes a good journal and why PNAS is considered one.
  • Looks for corrigenda or retractions.
  • Looks at citations of the paper, to see if there is any published criticism from the domain experts.

In this I disagree with Oddthinking's comment that: "A response should cite analysis that is at least to that level, or stronger." Explaining why the given source is a good one would be enough for me to upvote. Though I agree with him that the two "back of the envelope calculation" answers are not useful.

Answer 3

I think using the same source from the question is tacky. It appears to be circular reasoning, even tautology. I favor to close the question. Alternatively, maybe a dubious source saying something similar, then use the good source in an answer. It's a bit manufactured, and I wonder the motivation to do such a thing, but that at least wouldn't look tacky.

I just voted to close that specific question based on this reasoning. I commented:

I’m voting to close this question because the claim made is from a high calibre journal. Skepticism is not warranted. If you need help understanding the statistics involved, try stats.stackexchange.com. If you have questions regarding the political relevancy of this figure, try politics.stackexchange.com

I favor this as a legitimate close reason. I've seen this happen before, but only a few times. I prefer that when it does occur, we add authority to our own claim that the source is reliable and seek the full five close votes.

There is some nuance and subjectivity in the phrase skepticism is not warranted. That implies a trust in the source of the claim and a lack of reason to be skeptical otherwise. Sagan's "extraordinary claims require extraordinary proof" immediately comes to mind. Despite the source and the quality of the science, the claim itself could be so outlandish that I have trouble believing it [for example "Alien life found on Mars!"]. Skepticism would be warranted. Likewise, a claim could also put a common belief on its head. If I happen to hold that common belief, a lone article from a single though highly reputable journal is not enough for me to change my mind, especially of a claim in social sciences. Again, skepticism would be warranted. I would want repeatability and/or other authoritative sources throwing their weight behind the claim first before my skepticism would be resolved. This is mostly why I suggested that the full five close votes be attained if we're going to so this kind of closing. A single mod closing isn't really fair to the community, nor to the mod who will certainly catch a lot of flack for it.

In this particular instance, it wasn't unbelievable to me. I commented saying that over a lifespan of 65 years, the number comes out to less than the 1000 or so people "fatally shot" by police every year, so that seems in line with common sense. I voted to close already. If the simple math in my comment convinces you quickly enough that this isn't really worthy of skepticism, then vote with me. If five votes never accumulate, then the community has spoken.

Answer 4

About peer review

Peer-reviewed articles are not magic.

A team of people, sometimes just one person spend weeks to months (to years) working on an article then typically a handful of grad students will read the paper and escalate to the advisor who gave them the paper to review if something looks very suspicious to them. Some grad students are very thorough and skeptical, others are not.

The primary goal of the peer review is not actually to verify that the paper checks out. It is to select which papers will make it to presentation/print. If the review finds that something in the paper is wrong then it most certainly won't be published, but showing that generally requires much more effort than what reviewers are motivated to invest. Sometimes, showing that a paper is wrong is even more difficult than writing the paper in the first place.

There's also very little to gain from doing so. Good researchers will be thankful for you preventing them to publish a paper that they might have had to retract later, but that is a small and uncertain reward for a large effort. And that's only if the reviewer and reviewee actually get to know the identity of the other side, which is usually not the case. Literally nobody else will care about your service to your field's community if you find a defect in someone else's paper before the publish date.

If you find it after the publish date, not many more will care, and of those who do, not many might actually appreciate the gesture. Depending on the field, there's commonly so little direct criticism of published works going on that if you do that it will easily be construed as a personal aggression. In other words, in serious scientific fields a flawed article will sooner be forgotten than publicly debunked.

About skeptics.se

Trusting a published paper more than a trivial (and sound!) mathematical argument to the point of removing that answer is simply baffling to me. Contrary to what Oddthinking's comment implies, there's no "you must be this reputable to ride" bar on truth.

Why is this site called "skeptics"? The argument of authority is the opposite of skepticism. Trusting the conclusions of an article for the sole reason that it has been published in a prestigious journal is the epitome of the argument of authority.

It is true that sometimes the logic developed in a paper is just too complex (or too remote from actual rigorous thought...) for a layman to critique in this way. This was certainly not the case here. Furthermore, answers that are making an on-topic check with available information may not fully answer the question, but they are valuable nonetheless in developing an intuition on the problem. Isn't the point of an answer not only to answer correctly but also to make the reader understand why and how to arrive at such conclusions themselves?

In that context the "this article said so, question closed" answer is almost completely useless compared to the back-of-the-envelope calculation. We generally consider that pointing at the Bible isn't appropriate proof anymore. Yet the Bible has been peer-review by an uncountable number of the highest scholars of their time, at several dozen of the highest profile international conferences. And it wasn't retracted either.

Answer 5

This touches on a lot of issues that have come up before. I was originally hoping this answer would just serve as a roadmap to some of the previous discussions, but I ended up putting some personal opinions in as well.

Should we permit asking questions challenging accepted science?

Discussed here: If a claim is commonly accepted, does questioning it require a notable counter-claim?

Should we permit asking questions about published scientific articles?

We have had questions here about dodgy pseudoscientific articles published in apparently peer-reviewed (predatory or otherwise) journals. I think we can contribute to showing that they are wrong.

I don't think any source is trusted beyond skepticism (c.f. Lancet MMR autism fraud).

(I personally treat the Cochrane Collaboration articles as the gold standard. But even they have been retracted, to my horror.)

This is also important where the OP hasn't the skills to tell if the source is a quality journal.

Should we be giving an answer which is higher quality than the source references?

Discussed here: May I cite a single article?

Are back of the envelope calculations acceptable?

First discussed here: Math does not need to be cited or sourced which was notable because the Skeptics.SE community rejected one of the Stack Exchange site founder's views!

More definitively discussed here: FAQ: What are theoretical answers?

Discussed over and over since then, ad infinitum. I sometimes wish I had kept a log of how many times we have had back-of-the-envelope calculations posted here that were wrong - especially when they clashed with peer-reviewed analysis. The question that triggered this is an example - two back-of-the-envelope answers were published concluding the opposite results.

Should we permit answers that only link to the original sources?

I think the answer has to be yes, because sometimes we see answers that point out that the original source does NOT make the claim - e.g. when you look at the specific definition that was used, it doesn't match what the OP thought. That seems an appropriate answer to a question.

Should we permit questions about hypothesis that are still in dispute amongst scientists?

In theory, the answer is no.

Where we discuss what's on-topic we suggest going elsewhere for "research-level science".

We also have the "current event" Close Reason, which might well be applied to unresolved science.

In practice, I don't see questions being closed for this reason very often.

Answer 6

When the claim is made by peer-reviewed research in a reputable journal, the question needs to explain the why the asker is skeptical about that research. I think it's just wasting people's time if they don't take any steps to explain or justify their skepticism. They could do this by presenting alternative research (whether peer-reviewed or not). I also think back-of-the-envelope-calculations would justify skepticism when the claim is about frequency of an event.

In this specific case, the only justification for skepticism is that the numbers "seems surprisingly high". This doesn't cut it, so the question should be closed.