Summary
I did an experiment, using the Skeptics.SE question database, to test a conjecture.
I made a claim in a comment to a newbie about the sorts of questions that work well here. I decided that I should test my own claim against the data, to see if the effect was really as strong and indisputable as my experience suggests.
Here are the results.
My conjecture
Most Skeptics.SE titles are structured as English questions. The first word of an English question can be used to extract some (noisy) data about the type of question.
Group A: True or False Questions
Skeptics.SE is predominantly about testing the truth of claims.
Therefore, we should expect to see more good questions starting with:
- Is, Are, Was, Were, Do, Does, Can, Has, Have, Did,
Group B: Hypothetical Questions
Skeptics.SE is less about hypotheticals than about the real world happenings.
Therefore, we should expect to see fewer good questions starting with:
- Could, Would, Will
Group C: General Knowledge Questions
Skeptics.SE isn't about asking general knowledge questions driven only by curiosity.
Therefore, we should expect to even see fewer good questions starting with:
- Who, What, When, Where, How, Why, Which
Group D: Ethics Questions
Skeptics.SE isn't about ethics and morality. We should expect to see almost no good questions starting with:
- Should
Disclosure
I initially wrote the word lists, above, before looking at the data. After looking at the data, I made some minor additions to the word lists because there were additional words I hadn't thought of. I have marked the additions in bold. I trust the reader will agree they are consistent with the original lists.
Clarifications
The data set is likely to be noisy because:
- People ask questions without understanding the scope.
- People ask questions without the title being structured as a question.
- English is flexible, and the classification by the first word is noisy.
- e.g. "What if..." is likely a hypothetical.
- e.g. "Should I believe that..." is likely a True or False
- Rarely used words will have greater variability in average votes, etc, due to the smaller sample sizes.
The effect should be visible in several ways:
- Coarsely, there should be more questions in the format we expect. That is strongly confounded by the inherent frequency of some words in the English language, and the existence of bad questions, so this measure is of limited value.
- The ratio of closed to unclosed should reflect the expected appropriateness of the first words.
- The average votes per question should reflect the expected appropriateness of the first words.
I don't plan to do any sophisticated statistical analysis. The claim is that this is a bold and strong effect that will be visible through the noise. If it isn't strongly (and indisputably) supported based on the differences in averages, the idea should be discarded.
Method
I wrote a query in the Stack Exchange explorer.
To save effort, the query discards outlier words that only appears once at the start of a question. This is not really justified.
I took the results (28-Nov-13), and manually grouped the rows according to the rules above.
I ignored case. I ignored contractions ("What's" = "What", "Shouldn't" = "Should")
I noted the words that didn't fit the predefined groupings, and discarded them.
For each group, I counted the number of occurrences, the percentage closed and the average score (i.e. upvotes - downvotes).
There may be minor rounding issues in the average votes. I realised too late that I should have included sum total score, not average score, in the query.
Results
Discarded Words
The following words appeared at least twice at the beginning of a question, but were discarded: If, The, In, Evidence, Any, Death, More, Homeopathic, Correlation, New, A, Money, Intelligence, On, For, During
I am satisfied none of those belong in any of the categories.
Group A: True Or False
Number of questions: 3596 Average Score: 11.4 Percentage Closed: 7%
Group B: Hypothetical Questions
Number of questions: 91 Average Score: 10.3 Percentage Closed: 15%
Group C: General Knowledge Questions
Number of questions: 142 Average Score: 10.2 Percentage Closed: 29%
Group D: Ethics Questions
Number of questions: 13 Average Score: 7.5 Percentage Closed: 38%
Discussion
While the average votes are ranked in the predicted order, the difference between them is NOT blatantly obviously significant. I think the data rejects my idea that users vote clearly along my predicted lines.
The percentage closed, however, does follow these lines quite clearly.
I could put all sorts of caveats based on the noisiness of the data, the sample sizes, etc., and explain how more sophisticated statistics might tease out some underlying trends in the voting, but I decline to do so. My original expectation was that there was a new rule that people could use to help judge whether their own questions would do well in the Skeptics.SE environment. The evidence suggests the rule is not very accurate.
The difference between these measures itself is interesting. Does it reflect that the mods (including myself) put more measure into ideas like these about what makes a good question, than the general community does?
