I feel like there might be a fundamental issue with the format of this website.
The StackExchange model works well for subjects where the question is posed for the purpose of helping the asker, and answers can be deemed "correct" based on how well they help the asker.
On a site like StackOverflow, this works very well and produces "good" answers often because the Asker simply selects an answer that fixes his problem--a relatively objective test. When there are multiple answers that satisfy the "fix the problem" constraint, the one that is easiest for the asker to understand is often picked. This produces highly desirable answers--easy to understand, and with verifiable accuracy. In addition, answers with high user-ratings signify answers which are considered "better" by the programming community in most cases.
Sites like Programmers.SE use the same model for more subjective questions. In many cases, it is difficult for an asker to select a single "correct" answer, as quite often there is none with subjective questions. On these subjective sites, the availability of multiple highly-user-rated answers seems to be the most useful, as they tend to contain well-written explanations of popular opinions on the subject. We see quite often on Programmers.SE many highly-rated yet contradicting answers.
On Skeptics, however, this model seems to fall apart rather quickly. When a user asks his question, ideally, his motive is to seek the truth about some belief. The selected answer, then, is supposed to be the "most true" out of all the answers. Highly rated answers are also supposed to be those considered to be well researched, and filled with facts.
These criteria seem incredibly hard to test, though.
- An asker of a question cannot ascertain the truth of an answer better than anyone else.
- There is no test for the Truth of an answer.
Even if we remove "truth" from the equation and simply choose "well-researched" as the criteria, it's difficult to determine that either Answer-Acceptance or Answer-Votes correlates to how well researched an answer is. It is certainly quite hard to correlate either of them to truth.
The basics of the Stack Exchange Model are very simple.
- Answer-Acceptance shows that a the Asker of a question likes an answer best.
- Answer-Voting shows that other users like an answer.
- The usefulness of this Model on a given subject is defined by the usefulness of these two pieces of data.
On Skeptics, with the current configuration of this model, this data doesn't necessarily correlate to the goal of scientific skepticism.
Recently the following question was posed on Skeptics. Do rich companies pay little/no corporate income taxes in the United States? The original question asks both "Do companies pay little to no taxes in the US?" and "Is it possible for a company to pay little to no taxes in the US?"
At the time of its acceptance, the top answer on this question answered neither of these questions. It merely listed a number of ways corporations could reduce their taxes and stated "Yes, it is possible." After an edit, the answer now lists some data about how much money various corporations paid in taxes in recent years, but no comparative scale is given to tell if these are actually "little-to-no taxes."
Yet, regardless of the fact that the question asked is not directly answered, the given answer is well-rated, and the answer is accepted by the asker.
I've been thinking of some possible changes which could be made to fix this issue for skeptics--ranging in difficulty-to-implement.
- Remove the "Accept Answer" option for question askers.
- Questions are asked here because it is hard to verify the answer. No one user should be able to verify an answer.
- Organize Answers into two categories, as either Proving or Disproving a belief.
- We can shift the focus of answering to collecting research, rather than determining truth.
- Seeking good data will be rewarded regardless of the conclusions of the data, helping remove bias.
- Increase reputation amounts for asking questions.
- This attempts to make it so the question-asking population is more likely to have the ideals and purpose of Skeptics at heart.
The StackExchange model doesn't seem to work for the objective task of Skeptical analysis. The current implementation is flawed, but there may be some things we can do about it. I would love to hear other opinions on this as well.