13

There is a question about expectation of police killing of African American males. The statements the asker would like verified can be addressed by reference to an article in the Proceedings of the National Academy of Sciences, a prestige, peer-reviewed journal from one of the top scientific organizations in the world. Citing this article in support of the claim would be a slam dunk - we would say "Yes the figures are supported here", quote the article and post it in an answer. The answer would almost certainly receive many upvotes for thorough research and for citing of exceptionally reliable sources, the site would gain prestige for careful research.

The only glitch is that the question already links to that article. But the point is - why does that change anything? If the article would be an excellent answer to a claim found on Youtube, why is it any less reliable in answer to a claim that happens to link to itself?

Nobody on this site is likely to have the expertise to realistically critique an article of this nature - and if they did we wouldn't know that they were telling the truth. The article has already been verified by reviewers much more competent than us. So the question has essentially rendered itself unanswerable, unless we simply allow the article to be a reference for itself. Oddthinking says: "A response should cite analysis that is at least to that level, or stronger.". Since we know that will not happen here, we should either cite the original paper or close the question as unanswerable.

So should we do that, and allow the article to be used in the answer?

| |
  • 1
    I think this question may be a duplicate of this previous one as the answers apply pretty well here too: skeptics.meta.stackexchange.com/q/4366/37236. – Laurel Jun 9 at 15:52
  • 2
    PNAS is not the super reliable source you seem to think it is. Their peer-review process was a bit dodgy until not so long ago, albeit only for papers submitted on a certain track. blogs.sciencemag.org/pipeline/archives/2008/08/28/… ; nature.com/news/scientific-publishing-the-inside-track-1.15424 – Fizz Jun 22 at 7:57
  • 2
    Peer-reviewed journal articles are not necessarily correct. I get the feeling that the general public frequently mistakes peer-review as a guarantee of factual accuracy, but it's not. Don't get me wrong -- a study appearing in a well-regarded, peer-reviewed journal is suggestive of higher-than-otherwise quality. – Nat Jun 22 at 21:56
  • @Nat There are several conversations about this all over the comments of this question, and yes we are aware that peer reviewed journal articles can sometimes be wrong. – DJClayworth Jun 22 at 23:50
20

Just re-using the 'data' from the question is itself and merely stating, "yep, seems legit" is quite unsatisfactory.

"Skeptcisim is unwarranted" is a statement I consider always 'false' – especially for the purposes of this site.

Now, questioning Einstein on photo-electric effects might seem a bit pretentious, but 'scientific skepticism' means that questioning the science, the methodology, the data, the interpretations and conclusions or how they are presented and reported are all on the menu. Always.

Oddthinking commented below an answer that we look for equally good or stronger evidence, and we might compare these results and findings with previous results, reactions to the current study … and the list goes on.

Specifically, this study — Frank Edwards & Hedwig Leeb & Michael Esposito was mentioned in: "Risk of being killed by police use of force in the United States by age, race–ethnicity, and sex" has an ideal jumpboard as being mentioned in a Nature article:

A pair of high-profile papers published in the past few weeks (1,2) come to seemingly opposite conclusions about the role of racial biases.
— Lynne Peeples: "What the data say about police shootings. How do racial biases play into deadly encounters with the police? Researchers wrestle with incomplete data to reach answers."

And only 1 being our source here and 2 the 'seemingly opposite one'. To me, that means the hunt is on… 1

If we look further we find a string of articles handling the issue. Or trying to. Within seconds I had a long list of studies, all highlighting different methods, data points, statistics – and conclusions. Some bad, some better, not all the same.

No single one study is perfect, nor can it be. Absolutely no matter in what kind of prestige journal anything is published. The present study is not hammered into a philosopher's stone, nor is it the pinnacle of current scientific consensus.

And here, the study in question may be good, but I would expect at least a detailed explanation:

  • why is it considered to be a good source
  • why is *this particular paper a good one'
  • of the limitations and contradictions from with the study itself or
  • in comparison to other studies, whether earlier or later.
  • Then a general weighing against the other studies out there, and
  • how an answerer would criticise (or 'review as a peer') the current paper, and
  • how other authorities have handled this specific issue.
  • Did it receive lots of comments,
  • was it cited approvingly or in a shady light?

A hint I'd like to share: in social sciences like that one needs to take a thorough look at the foundations of that study, before swallowing a concluding soundbyte about a 'what' without ever getting to know the 'how' of definitions, limitations etc. I want to get to the 'why'. That a source is otherwise 'reliable' is 'nice', but since even the dumbest crackpot may utter a statement that is 'true', but nice sources fail us from time to time, 'nice' is not enough.

Regarding the indeed in general 'good' reliability of this source, we mustn't just swallow everything 'as is' just because it is published there in PNAS. Consider this very recent example:

Dear Editorial Board of the Proceedings of the National Academy of Sciences,
We are writing with deep concerns about a paper recently published in your journal, entitled “Identifying airborne transmission as the dominant route for the spread of COVID-19.”1 The paper made extraordinary claims about routes of transmission, the effectiveness of mask-wearing, and by implication, the ineffectiveness of other non-pharmaceutical interventions. While we agree that mask-wearing plays an important role in slowing the spread of COVID-19, the claims in this study were based on easily falsifiable claims and methodological design flaws. We present only a small selection of the most egregious errors here.[…]

— Meta-Research Innovation Center At Stanford: "Formal Request For The Retraction Of Zhang Et Al., 2020", June 18, 2020. (Signed by a long list of scientists from a wide range of medical fields.)

This is not an isolated 'first' incidence at that paper, but rather a by now common theme with its own Wikipedia page: Why Most Published Research Findings Are False, quoting the title of a most important Ioannidis paper.

To be clear, answering this meta-question sticklishly: I agree that we should "allow the article to be used in the answer?" — but as part of an answer, as that 'the publishing venue is known as being a quite good source' is a valuable 'hint'. Source monitoring is important. And if the paper itself is good, then we cannot be prohibited from using it as well.

In that case now, it looks more like "I am not dignifying this 'skepticism' with a proper answer but remind you to never question authority again. Trust your masters and obey." To be clear: "Yes, that's world class" and 'that's it' is an amplified pure fallacy called "Appeal to Authority". While we do that here on SkepticsSE indeed – and too much for my tastes actually – an answer that simply stops there is not an answer (that is 'of high quality', that is: also not inherently flaggable as 'low quality' or 'not answer') but reads like either lower quality, that while written as an answer, is more like comment, an introduction to an answer, etc.

A single appeal to authority is unwarranted. And if it is the same authority that should be questioned and scrutinised here, we achieved a lot of clicks, perhaps, but failed to promote our cause, surely.

Thus I definitely said that to accept anything on trust, to preclude critical application and development, is a grievous sin; and in order to apply and develop, “simple interpretation” is obviously not enough.


1Curiously, the source2 for that nature article was also in PNAS, has had massive feedback, corrections, letters, and now has been retracted.

| |
  • As much as I like your idea, DJ's unsatisfactory answer has 30+ upvotes. I saw the number, did a few quick calculations and concluded "it's well within the possible realm". That, plus knowing this community will vote answers like DJ's to the moon while ignoring ones you suggest should br given leads me to want to close. In other words, ask a more interesting question. – fredsbend Jun 9 at 4:55
  • @fredsbend Am not saying that it has to be bad. Overall, although it comes easy to pick out some splinters. A valid outcome of this inquiry might as well praise the piece to bits. I am saying that we should not judge the book by the cover of the journal and that we should have a slightly different approach than Mr Oliver "…and this is true". You know "trust, but verify" – LangLаngС Jun 9 at 6:29
  • 2
    @fredsbend this question has entered HNQ since June 8. I believe you know the voting problem with the HNQ where the majority of the visitors can only upvote (although the answer might not deserve downvote)... – Andrew T. Jun 12 at 12:47
  • 1
    "Just re-using the 'data' from the question is itself and merely stating, "yep, seems legit" is quite unsatisfactory." Indeed: skeptics.stackexchange.com/questions/47583/… And when it comes to Covid studies... NEJM article pulled comes to mind. retractionwatch.com/2020/06/04/… And we did have qs on reput. studies before skeptics.stackexchange.com/questions/39705/… – Fizz Jun 22 at 7:44
  • @Fizz Indeed. This Coronyboogy is a problem, everyone working from 'preprints' and often lousy studies when in print. Do we have a task force that re-reads all these As here to re-check the refs&srcs? Or really for all our stuff? Srsly, I know it's a nightmare, but it seems needed. – LangLаngС Jun 22 at 8:11
  • The other reason I believe that answers just using the cited article should be valid is that the questioner might not know that the cite is from a top class journal. So the question becomes "I saw this claim made in something called Nature - is it true?" and the answer is "Yes, Nature is a world class peer reviewed journal." – DJClayworth Jun 22 at 13:00
  • Where we differ: as part of an A, that is a valuable 'hint', source monitoring is important. In that case now, it looks more like "I am not dignifying this 'skepticism' with a proper answer but remind you to never question authority again. Trust your masters and obey." To be clear: "Yes, that's world class" and 'that's it' is an amplified pure fallacy called "Appeal to Authority". While we do that here indeed (too much actually), an answer that simply stops there is not an answer, but instead an arrogant anti-science aberration. – LangLаngС Jun 22 at 13:12
  • 1
    Appeal to authority has mixed opinions as to its validity. But it becomes necessary when the people having the argument are not capable of handling the complexity of the underlying arguments. Should we do lockdown for Coronoavirus? I'm not able to debate the effectivness of the models that led the experts to say we should, but if it's a choice between Joe Q RadioPundit saying we shouldn't and all the epidemiologists saying we should, that resolves the argument for me, whatever abstract logicians say about me following a fallacy. – DJClayworth Jun 22 at 16:09
  • @DJClayworth: A most unfortunate example, because the lockdown debates are actually about something else that nobody wants to say out loud. – Joshua Jun 24 at 23:22
  • @Joshua I have no idea what you are talking about. – DJClayworth Jun 25 at 12:58
  • @DJClayworth: There are people who benefit from reopening despite the plague and there are people who benefit from keeping the lockdown. The debate lines mostly follow the who-benefits or who-is-closer-to-benefitor rather than a rational breakdown on how dangerous the plague is. – Joshua Jun 25 at 15:03
2

I haven't voted to close (but I haven't written an answer either). I believe this type of question can be good for the site and it's readership, even if it is hard to give a good answer in the usual way.

Reasons to keep it:

  • We should encourage people to do some research and think critically before/when posting a question. Closing the question because they have found a good source feels counterproductive.

  • Being able to find a source is not the same as being able to critically evaluate the source. Both are required to properly challenge a claim.

  • Providing a good source in the question does not impact the value of the question in terms of the notability of the claim, or any other measure by which questions are usually judged.

It would be hard to write an answer along the usual lines (here's a top notch source! Wham, bam, done!). But it might be possible to write a good answer which:

  • Provides other high-quality sources.
  • Explains why the source in question should be considered reliable.
    • Explains (very briefly) how the scientific method works and what a journal paper is.
    • Explains what makes a good journal and why PNAS is considered one.
    • Looks for corrigenda or retractions.
    • Looks at citations of the paper, to see if there is any published criticism from the domain experts.

In this I disagree with Oddthinking's comment that: "A response should cite analysis that is at least to that level, or stronger." Explaining why the given source is a good one would be enough for me to upvote. Though I agree with him that the two "back of the envelope calculation" answers are not useful.

| |
  • 2
    What about DJ's answer? Useful? – fredsbend Jun 8 at 22:30
  • 2
    I think it's much better than the other answers so far, but could be improved by saying not only that the PNAS paper is a good source, but a bit more about why it is a good source. – Jack B Jun 9 at 9:36
2

I think using the same source from the question is tacky. It appears to be circular reasoning, even tautology. I favor to close the question. Alternatively, maybe a dubious source saying something similar, then use the good source in an answer. It's a bit manufactured, and I wonder the motivation to do such a thing, but that at least wouldn't look tacky.

I just voted to close that specific question based on this reasoning. I commented:

I’m voting to close this question because the claim made is from a high calibre journal. Skepticism is not warranted. If you need help understanding the statistics involved, try stats.stackexchange.com. If you have questions regarding the political relevancy of this figure, try politics.stackexchange.com

I favor this as a legitimate close reason. I've seen this happen before, but only a few times. I prefer that when it does occur, we add authority to our own claim that the source is reliable and seek the full five close votes.

There is some nuance and subjectivity in the phrase skepticism is not warranted. That implies a trust in the source of the claim and a lack of reason to be skeptical otherwise. Sagan's "extraordinary claims require extraordinary proof" immediately comes to mind. Despite the source and the quality of the science, the claim itself could be so outlandish that I have trouble believing it [for example "Alien life found on Mars!"]. Skepticism would be warranted. Likewise, a claim could also put a common belief on its head. If I happen to hold that common belief, a lone article from a single though highly reputable journal is not enough for me to change my mind, especially of a claim in social sciences. Again, skepticism would be warranted. I would want repeatability and/or other authoritative sources throwing their weight behind the claim first before my skepticism would be resolved. This is mostly why I suggested that the full five close votes be attained if we're going to so this kind of closing. A single mod closing isn't really fair to the community, nor to the mod who will certainly catch a lot of flack for it.

In this particular instance, it wasn't unbelievable to me. I commented saying that over a lifespan of 65 years, the number comes out to less than the 1000 or so people "fatally shot" by police every year, so that seems in line with common sense. I voted to close already. If the simple math in my comment convinces you quickly enough that this isn't really worthy of skepticism, then vote with me. If five votes never accumulate, then the community has spoken.

| |
  • I would argue that "the claim made is from a high calibre journal" is not quite correct. The claim was made by the TV show, and the high quality journal is a potentially useful source that the asker found but (apparently) was not able to evaluate. – Jack B Jun 8 at 20:46
  • Skepticism isn't warranted for high quality journals? Are we to assume they never make mistakes and retract high profile articles? thelancet.com/journals/lancet/article/PIIS0140-6736(20)31324-6/… – Matt Jun 8 at 20:55
  • 1
    It's not that high quality journals don't make mistakes, it's that this site is not qualified to gainsay them. If you are going to disallow answers referencing high quality journals, what are you going to allow as answers? – DJClayworth Jun 8 at 21:32
  • @Matt In case you weren't pinged, I'd say what DJ said. Retractions from reputable journals are pretty rare. By questioning the conclusions from reputable journals directly, we suggest that we are evaluating science directly, working from expertise in specific fields. It seems to me that the purpose of this site is more along pointers and aggregators, not actually science or even scientific criticism. I'd like to pretend we have that going for us, but we don't. So I guess I might support that aspect being part of the site topic, but I don't know if we're up for handling the quality control. – fredsbend Jun 8 at 21:45
  • Also, there's a major component that I don't have in my answer. Skepticism isn't warranted implies both a trust in the source of the claim and a lack of reason to doubt the claim otherwise. For example, a lone study that violates my common sense, despite source, will not get a close vote from me. – fredsbend Jun 8 at 21:58
  • I've added a significant edit, expanding on when/if "skepticism is warranted". – fredsbend Jun 8 at 22:22
  • 1
    I agree with this approach. The question asker should explain why the are skeptical of the journal's research and conclusions, otherwise the question should be closed. – curiousdannii Jul 16 at 2:23
2

This touches on a lot of issues that have come up before. I was originally hoping this answer would just serve as a roadmap to some of the previous discussions, but I ended up putting some personal opinions in as well.

Should we permit asking questions challenging accepted science?

Discussed here: If a claim is commonly accepted, does questioning it require a notable counter-claim?

Should we permit asking questions about published scientific articles?

We have had questions here about dodgy pseudoscientific articles published in apparently peer-reviewed (predatory or otherwise) journals. I think we can contribute to showing that they are wrong.

I don't think any source is trusted beyond skepticism (c.f. Lancet MMR autism fraud).

(I personally treat the Cochrane Collaboration articles as the gold standard. But even they have been retracted, to my horror.)

This is also important where the OP hasn't the skills to tell if the source is a quality journal.

Should we be giving an answer which is higher quality than the source references?

Discussed here: May I cite a single article?

Are back of the envelope calculations acceptable?

First discussed here: Math does not need to be cited or sourced which was notable because the Skeptics.SE community rejected one of the Stack Exchange site founder's views!

More definitively discussed here: FAQ: What are theoretical answers?

Discussed over and over since then, ad infinitum. I sometimes wish I had kept a log of how many times we have had back-of-the-envelope calculations posted here that were wrong - especially when they clashed with peer-reviewed analysis. The question that triggered this is an example - two back-of-the-envelope answers were published concluding the opposite results.

Should we permit answers that only link to the original sources?

I think the answer has to be yes, because sometimes we see answers that point out that the original source does NOT make the claim - e.g. when you look at the specific definition that was used, it doesn't match what the OP thought. That seems an appropriate answer to a question.

Should we permit questions about hypothesis that are still in dispute amongst scientists?

In theory, the answer is no.

Where we discuss what's on-topic we suggest going elsewhere for "research-level science".

We also have the "current event" Close Reason, which might well be applied to unresolved science.

In practice, I don't see questions being closed for this reason very often.

| |
  • 2
    Your thoughts on this particular question? – fredsbend Jun 9 at 15:12
  • 2
    Sorry, seen too much high-quality garbage from peer-reviewed journals. Seen the forefront of cardiac research doing D grade math in their backing. Seen the utter failure to apply variable elimination. When you get back-of-the-envelope answers that oppose each other is the time to be skeptical of the claim. – Joshua Jun 9 at 15:25
  • 1
    Thanks for the recent clean up on that post. What a mess. – fredsbend Jun 11 at 14:46
  • @Joshua Much as I'm sure you are a smart and honest person, the trouble is that we don't really know that and can't use you as a reference. As soon as we write an answer that says "Peer reviewed journal says X, but someone called Joshua on the internet says Y", we have abandoned the attempt to bring real scientific principles to bear. Nothing to do with whether you are right, but we've turned this from science to an opinion poll. – DJClayworth Jun 22 at 13:26
  • I'm sure you are not this person, but there are right now people out there being paid to write "Environmental science journals are full of mistakes, so why should we believe them that climate change is real?" on every website that will take them. We need to be a site that doesn't accept comments like that. – DJClayworth Jun 22 at 13:26
  • @DJClayworth: On the other hand, what we have right now is a site completely dedicated to argument from authority, whether or not the authority is actually being reasonable. The breaking point for me was the only answer being allowed to stand by the mods was one that referenced a paper that measured pollution of fuel by pounds of fuel used rather than energy extracted from said fuel, and using the right measure changed the effective answer to invert the meaning of the scientific paper, and the OP appeared to be skeptical in the first place because of the political use the paper was put to. – Joshua Jun 22 at 15:28
  • @Joshua I also find it unfortunate that a site devoted to Skepticism only allows for argument from authority, and I've pushed back on that on occasion. But we really need to avoid seeking to challenge expert papers with stuff we worked out for ourselves (because that's what the pseudoscientists do all the time, and try to persuade people their view is just as valid as real experts). – DJClayworth Jun 22 at 16:01
  • 1
    @DJClayworth: If we cannot teach discernment there's no point. – Joshua Jun 22 at 16:24
  • I agree. When I post an answer saying "the original article is published in a peer reviewed journal", I'm absolutely fine if someone comes and posts another answer saying "Here are some other articles that back up the first" or "Here are some sources that cast doubt on the first.". Either of those would be better and I would upvote. – DJClayworth Jun 22 at 18:04
1

About peer review

Peer-reviewed articles are not magic.

A team of people, sometimes just one person spend weeks to months (to years) working on an article then typically a handful of grad students will read the paper and escalate to the advisor who gave them the paper to review if something looks very suspicious to them. Some grad students are very thorough and skeptical, others are not.

The primary goal of the peer review is not actually to verify that the paper checks out. It is to select which papers will make it to presentation/print. If the review finds that something in the paper is wrong then it most certainly won't be published, but showing that generally requires much more effort than what reviewers are motivated to invest. Sometimes, showing that a paper is wrong is even more difficult than writing the paper in the first place.

There's also very little to gain from doing so. Good researchers will be thankful for you preventing them to publish a paper that they might have had to retract later, but that is a small and uncertain reward for a large effort. And that's only if the reviewer and reviewee actually get to know the identity of the other side, which is usually not the case. Literally nobody else will care about your service to your field's community if you find a defect in someone else's paper before the publish date.

If you find it after the publish date, not many more will care, and of those who do, not many might actually appreciate the gesture. Depending on the field, there's commonly so little direct criticism of published works going on that if you do that it will easily be construed as a personal aggression. In other words, in serious scientific fields a flawed article will sooner be forgotten than publicly debunked.

About skeptics.se

Trusting a published paper more than a trivial (and sound!) mathematical argument to the point of removing that answer is simply baffling to me. Contrary to what Oddthinking's comment implies, there's no "you must be this reputable to ride" bar on truth.

Why is this site called "skeptics"? The argument of authority is the opposite of skepticism. Trusting the conclusions of an article for the sole reason that it has been published in a prestigious journal is the epitome of the argument of authority.

It is true that sometimes the logic developed in a paper is just too complex (or too remote from actual rigorous thought...) for a layman to critique in this way. This was certainly not the case here. Furthermore, answers that are making an on-topic check with available information may not fully answer the question, but they are valuable nonetheless in developing an intuition on the problem. Isn't the point of an answer not only to answer correctly but also to make the reader understand why and how to arrive at such conclusions themselves?

In that context the "this article said so, question closed" answer is almost completely useless compared to the back-of-the-envelope calculation. We generally consider that pointing at the Bible isn't appropriate proof anymore. Yet the Bible has been peer-review by an uncountable number of the highest scholars of their time, at several dozen of the highest profile international conferences. And it wasn't retracted either.

| |
  • As has been said repeatedly, it's not that peer reviewed articles are magic, it's just that they are better than random people on the internet. We are random people on the internet. There are thousands of blog posts out there saying "Here's a common sense argument why climate change is wrong", and one of the purposes of this site is to counteract them. – DJClayworth Jul 16 at 3:48
  • @DJClayworth They are statistically better, yes. But that is still an argument of authority. Everyone can critique a published paper, and that critique can be evaluated on its own merit. – Kafein Aug 31 at 10:53
0

When the claim is made by peer-reviewed research in a reputable journal, the question needs to explain the why the asker is skeptical about that research. I think it's just wasting people's time if they don't take any steps to explain or justify their skepticism. They could do this by presenting alternative research (whether peer-reviewed or not). I also think back-of-the-envelope-calculations would justify skepticism when the claim is about frequency of an event.

In this specific case, the only justification for skepticism is that the numbers "seems surprisingly high". This doesn't cut it, so the question should be closed.

| |

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .