How does the StackExchange model work for Skeptical Analysis?

Question

I feel like there might be a fundamental issue with the format of this website.

Overview

The StackExchange model works well for subjects where the question is posed for the purpose of helping the asker, and answers can be deemed "correct" based on how well they help the asker.

On a site like StackOverflow, this works very well and produces "good" answers often because the Asker simply selects an answer that fixes his problem--a relatively objective test. When there are multiple answers that satisfy the "fix the problem" constraint, the one that is easiest for the asker to understand is often picked. This produces highly desirable answers--easy to understand, and with verifiable accuracy. In addition, answers with high user-ratings signify answers which are considered "better" by the programming community in most cases.

Sites like Programmers.SE use the same model for more subjective questions. In many cases, it is difficult for an asker to select a single "correct" answer, as quite often there is none with subjective questions. On these subjective sites, the availability of multiple highly-user-rated answers seems to be the most useful, as they tend to contain well-written explanations of popular opinions on the subject. We see quite often on Programmers.SE many highly-rated yet contradicting answers.

Problem

On Skeptics, however, this model seems to fall apart rather quickly. When a user asks his question, ideally, his motive is to seek the truth about some belief. The selected answer, then, is supposed to be the "most true" out of all the answers. Highly rated answers are also supposed to be those considered to be well researched, and filled with facts.

These criteria seem incredibly hard to test, though.

• An asker of a question cannot ascertain the truth of an answer better than anyone else.
• There is no test for the Truth of an answer.

Even if we remove "truth" from the equation and simply choose "well-researched" as the criteria, it's difficult to determine that either Answer-Acceptance or Answer-Votes correlates to how well researched an answer is. It is certainly quite hard to correlate either of them to truth.

Summary

The basics of the Stack Exchange Model are very simple.

• Answer-Acceptance shows that a the Asker of a question likes an answer best.
• Answer-Voting shows that other users like an answer.
• The usefulness of this Model on a given subject is defined by the usefulness of these two pieces of data.

On Skeptics, with the current configuration of this model, this data doesn't necessarily correlate to the goal of scientific skepticism.

Example

Recently the following question was posed on Skeptics. Do rich companies pay little/no corporate income taxes in the United States? The original question asks both "Do companies pay little to no taxes in the US?" and "Is it possible for a company to pay little to no taxes in the US?"

At the time of its acceptance, the top answer on this question answered neither of these questions. It merely listed a number of ways corporations could reduce their taxes and stated "Yes, it is possible." After an edit, the answer now lists some data about how much money various corporations paid in taxes in recent years, but no comparative scale is given to tell if these are actually "little-to-no taxes."

Yet, regardless of the fact that the question asked is not directly answered, the given answer is well-rated, and the answer is accepted by the asker.

Possible Fixes

I've been thinking of some possible changes which could be made to fix this issue for skeptics--ranging in difficulty-to-implement.

1. Remove the "Accept Answer" option for question askers.
   • Questions are asked here because it is hard to verify the answer. No one user should be able to verify an answer.
2. Organize Answers into two categories, as either Proving or Disproving a belief.
   • We can shift the focus of answering to collecting research, rather than determining truth.
   • Seeking good data will be rewarded regardless of the conclusions of the data, helping remove bias.
3. Increase reputation amounts for asking questions.
   • This attempts to make it so the question-asking population is more likely to have the ideals and purpose of Skeptics at heart.

Conclusion

The StackExchange model doesn't seem to work for the objective task of Skeptical analysis. The current implementation is flawed, but there may be some things we can do about it. I would love to hear other opinions on this as well.

Answer 1

Accepted answers aren't always right even on Stack Overflow. I've downvoted some that were just plain wrong, leaving comments as to how they were wrong. Presumably they were accepted because their advice happened to work in the original poster's exact situation.

The accepted answer gets 15 rep and sits at the top of the answers. The most-voted other answer is right below it, and it's easy to see if the votes suggest that it's better. I don't think wrongly accepted answers are much of a problem. As far as voting goes, we're pretty well stuck with the people we can get, and they will vote as they see fit. We can encourage answers we like with comments and votes, and that's about all we can do (aside from closing/deleting/flagging unsuitable questions and answers).

Any deviation from the standard SE model would require enough justification to make the SE team think it worthwhile, which means that coming up with a proposal that would be useful for multiple sites would be almost essential. As a practical matter, I think this unlikely.

Answer 2

I think this is rather like going on Wikipedia and telling them that an encyclopedia anyone can edit is a bad idea. That may be so, but that's the entire point of the endeavor.

The Area 51 process exists to apply a model that has created one of the best programming Q&A sites in the world and apply it to all sorts of crazy new topics. This is our attempt at applying it to scientific skepticism. If you don't think it works, there are plenty of other websites that attempt different approaches.

Remember, accepted answers are not set in stone. Questioners can change them at any time. I certainly would not hesitate to do so if someone posted a better answer to one of my questions. If you think the accepted answer to a question (or any answer for that matter) is not a good one, all you have to do is write a better answer.

I can't get behind any major changes until some evidence is produced that indicates the status quo is broken, and I have seen no such evidence. The example you cite doesn't appear to indicate that we're doing something wrong. Quite the contrary, it's an example of a how easily an answer on this site can be improved significantly.

^source

Answer 3

There is, I think, a real problem applying the stack exchange model here and I think the issues is highlighted in the question's summary of how the whole SE process works:

The basics of the Stack Exchange Model are very simple.

• Answer-Acceptance shows that a the Asker of a question likes an answer best.
• Answer-Voting shows that other users like an answer.
• The usefulness of this Model on a given subject is defined by the usefulness of these two pieces of data.

The problem is that scientific skepticism isn't about whether you like a particular result but should be about whether that result can be justified by evidence. Answers that people like will frequently not overlap with that objective standard. In fact, they will often be the answers that reflect consensus prejudice of the day, what C H Waddington described as the COnventional Wisdom of the domiNant Group (or COWDUNG for short, not sure where the "U" comes from) in his great book about how mental models help us understand the world, Tools For Thought.

The standard that is most rigorously applied that distinguishes Skeptics from other SE sites is the citation. We don't like answers that don't link to published sources of evidence. But there is a highly significant problem with that rule: selective quotation. The scientific literature is full of perfectly good but contradictory evidence (it is also full of much badly conducted junk that sailed through peer-review unscathed). So it is often very easy to back a particular prejudice with perfectly valid citations. In medical research the counterweight to this is the meta-analysis which tries to objectively assess the weight of evidence and the quality of evidence from many sources. The skeptics approach neither demands nor applies extra weight to meta analyses and the pattern of votes on some questions suggests that a meta analysis carries little weight with the voting community.

As evidence I refer to this question: Is routine screening for breast cancer for asymptomatic women worthwhile? To be fair I may be biased in the interpretation because i wrote the question and the lowest scored answer, but the comment threads, I think, show the problem I'm referring to. My answer summarised the latest Cochrane review (and those reviews are supposed to be the highest standard of evidence in epidemiology or medicine). But the results of the review challenge a couple of decades of clinical orthodoxy. The top voted answer finds a handful of papers that clearly take the conventional position; the second answer summarises the results of the review by the body who triggered the debate by changing their advice (and really baked down and hedged their bets in the face of public sentiment).

Maybe i'm not able to see this clearly because it is my answer, but it seems to me that the voting pattern is more easily explained by voters initial prejudice than by a judgement about the quality of evidence.

So that is my view of the problem. I don't have any clear ideas how to fix it.

Answer 4

From my point of view, the "Accepted Answer" for questions on this site is a bit of an issue, largely because science does allow for things to change and what may be a perfectly valid accepted answer could be completely negated several months later by new data. However, there is also an answer for this built into science already, namely, peer review of existing data and periodic review of older conclusions to ensure they are still valid when new data is presented.

If we were to apply this to the "Accepted Answer" paradigm that is currently in place we could add a sufficient check into place by just adding a new flag to answers, namely, "Flag for Peer Review." If an answer was flagged for peer review, a moderator (or a group of trusted users) could review the question and answer, and if the answer was no longer valid, the "Accepted Answer" flag could be removed. The distribution of reputation is still an issue with the proposal as I don't feel that you should lose reputation for what was once a valid accepted answer; however, by removing the flag we would ensure that new answers could bubble up and replace the previous answer. Furthermore, this would also ensure, that accepted answers are not taken a gospel by someone that is searching for an answer to a question on Google.

Answer 5

I don't see a problem. OPs simply should accept the best answer.

See also: The best posts of Skeptics!