Do we apply scientific skepticism at Skeptics.SE?

Question

There was a lot of vigorous debate in the comments on a recent meta-question. A substantial part of it seemed to be arguing about a particular sub-question. This is an attempt to draw out the sub-question.

The Skeptics.SE tour makes a claim:

Skeptics is a question and answer site for applying scientific skepticism.

What does that mean in theory? That has been asked before. I would offer the Wikipedia page on scientific skepticism as a good starting point. It offers several different (consistent?) definitions of scientific skepticism.

In practice, a number of community standards are used to limit the type of answers, including eschewing:

• Anecdotal answers
• Answers based entirely on speculative, logic, mathematics and theoretical models, without empirical evidence
• Original research
• Discussion focussed on the nature or practice of skepticism itself on the main site (although, it is welcome in Meta.)

The question here is whether what we practise matches what we claim. Do we actually apply scientific skepticism on Skeptics.SE?

Related question: From the private beta period, about what the original text should have claimed: Should the question asking instructions be referring to "scientific skepticism"?

Answer 1

I'll use the definition from the top of the Wikipedia article:

Scientific skepticism (also spelled scepticism) is the practice of questioning whether claims are supported by empirical research and have reproducibility, as part of a methodological norm pursuing "the extension of certified knowledge".[1] For example, Robert K. Merton asserts that all ideas must be tested and are subject to rigorous, structured community scrutiny (see Mertonian norms).

---

Questions

People who ask questions on skeptics are almost always, ipso facto (i.e. by the practice of posting questions), being skeptical or practicing skepticism: because they're questioning whether claims are supported.

The ethos of the site encourages (doesn't require) answers to publish the results of reproducible empirical research performed by scientists (i.e. by scientific skeptics): which I think is a sufficiently close match to the first definition of skepticism above, that it's fair to say that questioners are practising scientific skepticism here (or at least engaging with or attempting to read the published results from those who do).

---

Answers

For answers, it's not so clear.

How this site actually works, i.e. what its rules are

When people post answers on this site, they're required to reference any evidence at all as they see fit.

Answerers may choose or may prefer to reference "empirical research which has reproducibility" however that is not a formal requirement. Instead the requirement is only that:

• Evidence

  The answer must be reference some (any) evidence

• Good evidence

  The evidence should be (in the opinion of the person posting the answer) as good as possible

• Better evidence

  The evidence must be better than whatever evidence may have been published together with the claim that's being questioned in the OP (because otherwise "What's the point? It's not an answer."), for example:

  • If there was no evidence published with the claim then any evidence is better than that

  • If the claim wasn't specific and just said, "a recent study has found that [etc.]" then identifying and referencing some specific study/studies is better evidence than that.

  • Some people have posted suggestions or heuristics for how to determine and/or find what, in their opinion, is likely to be the more reliable evidence, for example:

    • How do we know that our references are reliable?

    • May I cite a single article?

• Other users vote

  After any answer is posted, other users of the site (who vote) will read and upvote or downvote depending on whether, in their opinion, the answer is useful.

  Some "attributes of a good answer" are described elsewhere on this meta-site but should IMO include whether:

  • The user's answer is supported by the evidence it references
  • The evidence is good (convincing) and/or, at least, as good as it can be: for example if there's little evidence then the answer may (or should) say so, but must for example provide evidence for its assertion that there's little evidence.

  Users may (I won't say "must" because voting is secret and because there are no formal guidelines about how to vote or what your voting algorithm should be) also choose to upvote or downvote answers based not only on their stand-alone merits but also on their relative merits. For example if there were two answers coming to different conclusions based on seemingly-contradictory evidence, voters might choose to:

  • Upvote both because they're both good (or useful) answers with evidence
  • Upvote only the more convincing one
  • Downvote the less convincing one

  An even better answer would account for all the evidence: e.g. explain why one set of evidence should be discounted, or present a new synthesis (a synthetic theory).

• You can judge for yourself

  See How do I know that the information posted in this site is reliable?

Reason why that's not called scientific skepticism

Some people argue that this site does not allow people's answers to practise 'scientific skepticism': in particular because 'original research' and/or 'theoretical answers' are sometimes not allowed.

• See FAQ: What constitutes original research?
• See FAQ: What are theoretical answers?

Reason why that is called scientific skepticism

On the other hand, one might argue answers meet the second definition of skepticism above, i.e. that "all ideas must be tested and are subject to community scrutiny", because:

• Only published (i.e. scrutinizable and/or scrutinized), not original, evidence is permitted in an answer.

  The scrutiny is often but is not always necessarily narrowly "scientific", because the type of peer review depends on the subject and the type of publication: for example anything published in a newspaper has journalistic/editorial review; anything published by a historian may have been reviewed by other historians; etc.

• Answers themselves are subject to the scrutiny of the 'community' of users on this site.

Answer 2

The question is why is this specifically scientific skepticism, rather than any other sort of skepticism.

Scientific skepticism requires that criticisms of the logical consistency and evidential support for a theory are examined based their own merits (rather than relying on the scrutiny applied by others). If we rely on the scrutiny of others, then we are not applying scientific skepticism, they are, and we are just reporting and contextualizing their efforts. However SkepticsSE does not allow purely logic based criticism or individual research questioning the evidential basis of a claim, and so SkepticsSE does not itself apply scientific skepticism. This is not a criticism of SkepticsSE.

Scientific skepticism requires that we should be able to accept any valid argument against our position. What matters is the validity of the argument, not the source or whether it has been peer-reviewed.

Some definitions of scientific skepticism from Wikipedia

Scientific skeptics believe that empirical investigation of reality leads to the truth, and that the scientific method is best suited to this purpose.

Note this says nothing about providing references.

Scientific skeptics attempt to evaluate claims based on verifiability and falsifiability and discourage accepting claims on faith or anecdotal evidence. Skeptics often focus their criticism on claims they consider to be implausible, dubious or clearly contradictory to generally accepted science. Scientific skeptics do not assert that unusual claims should be automatically rejected out of hand on a priori grounds - rather they argue that claims of paranormal or anomalous phenomena should be critically examined and that extraordinary claims would require extraordinary evidence in their favor before they could be accepted as having validity.

Note it says nothing about providing references.

From a scientific point of view, theories are judged on many criteria, such as falsifiability, Occam's Razor, and explanatory power, as well as the degree to which their predictions match experimental results.[citation needed] Skepticism is part of the scientific method; for instance an experimental result is not regarded as established until it can be shown to be repeatable independently.

Note it says nothing about referencing. Indeed the publication of a paper in a peer-reviewed journal does not mean that it has been shown to be repeatable independently. I know from experience that it can be very difficult to replicate the results in peer-reviewed papers.

Scientists do not regard papers published in peer-reviewed journals as established science. Publication of the work is only the first step towards acceptance, the independent replication and use of the ideas published in the paper is the next step. There are plenty of papers that get published that rarely get cited because the research community discovered them to be of little value or even simply wrong. SkepticsSE on the other hand treats peer-reviewed papers as authoritative (which is a form of appeal to authority), which is not part of scientific method.

SkepticsSE does not allow individual research or back of the envelope calculations or logic/reason based answers, even though all of these are perfectly acceptable in science.

SkepticsSE does not implement scientific skepticism as it does not allow pure reason or individual research. That doesn't mean that there is anything wrong with the form of skepticism that actually is applied. The contributors to SkepticsSE generally don't have the expertise to evaluate individual research or logic based answers, so it is delegated to the peer-reviewed literature. I would agree that is the right thing to do, but it only causes confusion to call that "scientific skepticism", rather than just "skepticism".

I don't think it is a big deal, it is just that saying the site implements scientific skepticism will give the wrong message to any scientist posting here, who would naturally want to use the form of scientific skepticism they are used to in their working lives (and get fed up with being told to change perfectly good (from a scientific skepticism perspective) answers).

Answer 3

Is original research necessary?

There is no doubt that science needs original research and theoretical work. Does scientific skepticism need them too?

"Original research" and "Theoretical answers", in the sense used on this site, are only one of the tools used by scientific skeptics to debunk extraordinary claim and are seldom used for that (e.g. try to count how many Snopes answers only use calculations or self-performed experiments).

I've never seen them used to confirm an extraordinary claim (e.g. Drake's equation is never taken as proof of alien life by skeptics).

In conclusion, no, original research is only one of the possible avenues to apply skepticism, but there are many other ways which don't rely on it.

Definitions

The crux of the problem in the "Reason why not" paragraph of ChrisW's answer is a contention on the definition of "scientific skepticism".

Does allowing only a subset of scientific tools still retains the qualities of scientific skepticism or not? The answers we do give are quite similar in content, argumentation style, and logic to the kind of conclusions found in almost all other skeptics sites on the net and most skeptic authors' books.

I would say that the materials and methods that make up this site are qualitatively very similar to other applied scientific skepticism material and thus, we are applying scientific skepticism in a generally accepted way.

What is an appeal to authority?

An Appeal to Authority is a fallacy with the following form:

• Person A is (claimed to be) an authority on subject S.
• Person A makes claim C about subject S.
• Therefore, C is true.

This fallacy is committed when the person in question is not a legitimate authority on the subject. More formally, if person A is not qualified to make reliable claims in subject S, then the argument will be fallacious.

(source)

There are a lot of arguments about this site presenting "arguments from authority". This is false.

1. Presenting an argument from a legitimate authority is allowed. We only allow reputable references.
2. A requirement of this fallacy is inferring correctness from authority. We don't claim correctness.

I've seen some definitions of this fallacy disallow any authority. In this case, point 2. still protects us from falling in that fallacy.

When a scientist uses a well-known formula, they also rely on some authority to prove it is correct and usable, this doesn't mean they are appealing to authority.

On the other hand, allowing an answerer to report a self-performed experiment is certainly an argument from (personal) authority.

Reliability

The crux is the reliablity of the answers, in particular the ability to estimate it, not the authority, and requesting references does make the answer evidence their reliability. An answer points to a poor quality reference? It's not very reliable. Is an answer pointing to a well established source? It's way more reliable. Is someone reporting a self-invented experiment reliable? Not very, it's anecdotal at best.

Does this prove anything on this site is correct? Nope. We don't claim to be correct.

On the other hand, it does provide a strengthening or weakening of the claims asked about, and as such it qualifies with the aim of scientific skepticism.

Novelty

Do we allow novel material on the site? Yes. Many of our answers do: spot checking, this is the first answer in my feed. It connects evidence to an answer. The evidence was not previously used to answer the question. Novel, useful information is certainly present.

Journalism

Are we a journalistic site? Not in my definition of journalism: news are typically top down and editorial. We are bottom up and community-oriented. We answer questions, so we react to our audience. In fact, we are our audience. Journals do not do that as their core.

Conclusion

What we do is connect questions with answers, questioners with answerers, extraordinary claims with reliable evidence. This is exactly what scientific skepticism is.