10

Summary

I did an experiment, using the Skeptics.SE question database, to test a conjecture.

I made a claim in a comment to a newbie about the sorts of questions that work well here. I decided that I should test my own claim against the data, to see if the effect was really as strong and indisputable as my experience suggests.

Here are the results.

My conjecture

Most Skeptics.SE titles are structured as English questions. The first word of an English question can be used to extract some (noisy) data about the type of question.

Group A: True or False Questions

Skeptics.SE is predominantly about testing the truth of claims.

Therefore, we should expect to see more good questions starting with:

  • Is, Are, Was, Were, Do, Does, Can, Has, Have, Did,

Group B: Hypothetical Questions

Skeptics.SE is less about hypotheticals than about the real world happenings.

Therefore, we should expect to see fewer good questions starting with:

  • Could, Would, Will

Group C: General Knowledge Questions

Skeptics.SE isn't about asking general knowledge questions driven only by curiosity.

Therefore, we should expect to even see fewer good questions starting with:

  • Who, What, When, Where, How, Why, Which

Group D: Ethics Questions

Skeptics.SE isn't about ethics and morality. We should expect to see almost no good questions starting with:

  • Should

Disclosure

I initially wrote the word lists, above, before looking at the data. After looking at the data, I made some minor additions to the word lists because there were additional words I hadn't thought of. I have marked the additions in bold. I trust the reader will agree they are consistent with the original lists.

Clarifications

  • The data set is likely to be noisy because:

    • People ask questions without understanding the scope.
    • People ask questions without the title being structured as a question.
    • English is flexible, and the classification by the first word is noisy.
      • e.g. "What if..." is likely a hypothetical.
      • e.g. "Should I believe that..." is likely a True or False
    • Rarely used words will have greater variability in average votes, etc, due to the smaller sample sizes.
  • The effect should be visible in several ways:

    • Coarsely, there should be more questions in the format we expect. That is strongly confounded by the inherent frequency of some words in the English language, and the existence of bad questions, so this measure is of limited value.
    • The ratio of closed to unclosed should reflect the expected appropriateness of the first words.
    • The average votes per question should reflect the expected appropriateness of the first words.
  • I don't plan to do any sophisticated statistical analysis. The claim is that this is a bold and strong effect that will be visible through the noise. If it isn't strongly (and indisputably) supported based on the differences in averages, the idea should be discarded.

Method

I wrote a query in the Stack Exchange explorer.

To save effort, the query discards outlier words that only appears once at the start of a question. This is not really justified.

I took the results (28-Nov-13), and manually grouped the rows according to the rules above.

I ignored case. I ignored contractions ("What's" = "What", "Shouldn't" = "Should")

I noted the words that didn't fit the predefined groupings, and discarded them.

For each group, I counted the number of occurrences, the percentage closed and the average score (i.e. upvotes - downvotes).

There may be minor rounding issues in the average votes. I realised too late that I should have included sum total score, not average score, in the query.

Results

Discarded Words

The following words appeared at least twice at the beginning of a question, but were discarded: If, The, In, Evidence, Any, Death, More, Homeopathic, Correlation, New, A, Money, Intelligence, On, For, During

I am satisfied none of those belong in any of the categories.

Group A: True Or False

Number of questions: 3596 Average Score: 11.4 Percentage Closed: 7%

Group B: Hypothetical Questions

Number of questions: 91 Average Score: 10.3 Percentage Closed: 15%

Group C: General Knowledge Questions

Number of questions: 142 Average Score: 10.2 Percentage Closed: 29%

Group D: Ethics Questions

Number of questions: 13 Average Score: 7.5 Percentage Closed: 38%

Discussion

While the average votes are ranked in the predicted order, the difference between them is NOT blatantly obviously significant. I think the data rejects my idea that users vote clearly along my predicted lines.

The percentage closed, however, does follow these lines quite clearly.

I could put all sorts of caveats based on the noisiness of the data, the sample sizes, etc., and explain how more sophisticated statistics might tease out some underlying trends in the voting, but I decline to do so. My original expectation was that there was a new rule that people could use to help judge whether their own questions would do well in the Skeptics.SE environment. The evidence suggests the rule is not very accurate.

The difference between these measures itself is interesting. Does it reflect that the mods (including myself) put more measure into ideas like these about what makes a good question, than the general community does?

  • In hindsight, I should have treated discarded as another category, and found out whether that affected votes. – Oddthinking Nov 28 '13 at 6:45
  • Note: Questions not conforming to the expectations of the community (and of yours truly in particular) tend to get edited into shape. There could be strong confounding factors there. e.g. Questions might be closed BEFORE they are edited. Editors might be more more inclined to edit questions that they find interesting/vote-worthy. Or maybe they leave alone good non-conformist questions, but tend to edit bad non-conformist questions. And now that this has been published, new questions and votes might be influenced by it! – Oddthinking Nov 28 '13 at 6:49
  • Why do you consider "will" hypothetical? "Will milk spoil if left out of the fridge" is exactly the same question as "Does milk spoil if left out of the fridge" – Sklivvz Nov 30 '13 at 16:44
  • @Sklivvz: shrugs Yeah, the data is noisy. The correct way to resolve this would be to take a sampling of questions starting with "Will" and manually classify them into hypothetical or not, and work out whether one is more prevalent (than a control? are such categories be specific enough anyway?) – Oddthinking Nov 30 '13 at 21:56
3

Very cool.

Next step would be an multivariate ANOVA, and if indicated, follow up with post-hoc tests for each of the 6 possible pairings of conditions: (A,B), (A,C), (A,D), (B,C), (B,D), (C,D).

(This might start being a bit too post-hoc, since we didn't commit to a particular analysis prior to the data collection, and we've already seen partial summary of the data, but let's just call this exploratory, then.)

Too bad you didn't use a subset of the data, so that we could do this exploratory work on say 50% of the entire data set, and then confirm any significant findings by replicating the procedure on the remaining 50% of the data.


Some thoughts:

  • That is large enough of a sample to be able to find an effect size of 1.5 votes statistically significant.
  • Your data does not reject your hypothesis that your proposed categorization is reflected in votes.
  • There's likely an age-dependent curve associated with the votes a question has received. Younger questions will have fewer votes. Older questions will have more, but that increase will plateau at some point. (That's a guess.)
  • I agree the hypothesis "the categories are different" hasn't been rejected. But my real hypothesis was "Questions in the General Knowledge category fare so badly, that it should appear as a warning in the FAQ." A 10% hit in votes isn't enough to justify that. Further analysis could be performed, but at that point, my motivation disappears. – Oddthinking Nov 28 '13 at 6:44
  • 1
    Oh, and the age-dependent curve is something that interests me. It is confounded by site-popularity growing over time. If that could be taken into account, it would be great to have a tool that told you how well your question or answer is going (e.g. it is in the top 7 percentile for 2-week-old questions) and where it might end up (e.g. it is predicted it will reach 10 votes within 17 days. It is predicted to have 18 votes at the 6 month mark.) – Oddthinking Nov 28 '13 at 6:53
  • haha okay. I agree with that. (replying to first comment) – user5582 Nov 28 '13 at 6:53

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .