12

Observation: Skeptics.SE has historically had a moderately low ratio of answers-to-questions. [Ref - currently 'Okay', but has been 'Needs-Work']

Hypothesis: Skeptics.SE tend to have well-researched answers which require more effort than other sites, which tend to have multiple suggestions. A relatively low Answers-To-Questions ratio is to be expected, and not a good metric for the health of the Skeptics.SE.

Prediction: With less answers per question, but high-quality questions, we should get more Enlightened badges (10+ votes, first answer) than other StackExchanges web-site. With the alternative explanation - we are just poor at producing answers - we'd expect the answers given to not get 10+ votes.

Method: Get the number of Enlightened badges divided by the number of Questions for each of several StackExchange sites, until bored.

Results:

  • Skeptics: 373 Enlightened Badges of 1877 Questions: 19.9%
  • Gaming: 1038 of 10,963: 9.5%
  • English: 794 of 11296: 7.0%
  • TeX: 746 of 11506: 6.5%
  • Programmers: 905 of 14118: 6.4%
  • Game Developers: 226 of 5068: 4.4%
  • Stack Overflow: 30953 of 2284146 : 1.4%
  • Code Golf: 6 of 458: 1.3%

Conclusion: Skeptics.SE has a far higher rate of Enlightened Answers than other sites. This is consistent with the hypothesis that while Skeptics tends to get fewer answers than other Skeptics.SE sites, the average quality of them is higher.

Known shortcomings:

  • Selection criteria not specified.
  • Only statistical analysis is "inter-ocular impact".
  • Alternate explanations?

UPDATE

In accordance with Sklivvz's request, a pretty new graph - click to enlarge - with a lot more data points - all the ones I could (conveniently) fetch through the API.

A/Q versus Enlightened/Q Graph

Skeptics is ranked #2 in the enlightenment stakes. A surprise result from SciFi there!

Here is the raw data. Each row represents the site's name, the number of answers per question and the number of enlightened answers per question.

 scifi 2.21 0.243
 skeptics 1.675 0.199
 german 2.777 0.176
 cstheory 2.015 0.134
 boardgames 2.805 0.096
 gaming 1.776 0.095
 photo 3.307 0.091
 rpg 3.672 0.09
 astronomy 2.004 0.083
 travel 2.199 0.082
 writers 3.701 0.078
 english 2.843 0.07
 parenting 4.049 0.07
 cooking 3.017 0.069
 gardening 1.868 0.067
 tex 1.813 0.065
 programmers 5.512 0.064
 japanese 2.118 0.057
 bicycles 3.528 0.055
 security 2.834 0.055
 gamedev 2.612 0.045
 productivity 3.76 0.045
 diy 2.183 0.043
 music 2.82 0.041
 math 1.733 0.035
 ux 3.813 0.035
 unix 1.951 0.035
 electronics 2.705 0.035
 pm 4.127 0.032
 fitness 2.199 0.032
 money 2.681 0.03
 dba 2.031 0.027
 quant 2.179 0.026
 philosophy 2.906 0.025
 physics 2.13 0.024
 graphicdesign 2.224 0.024
 homebrew 2.699 0.022
 stats 1.809 0.02
 android 1.629 0.019
 askubuntu 1.626 0.018
 webapps 1.819 0.017
 sqa 3.368 0.017
 gis 2.076 0.016
 codereview 1.92 0.016
 stackoverflow 2.152 0.014
 codegolf 6.902 0.013
 webmasters 1.888 0.012
 apple 1.867 0.012
 mechanics 1.796 0.011
 superuser 1.938 0.01
 judaism 2.235 0.009
 serverfault 2.102 0.008
 onstartups 3.503 0.008
 stackapps 1.304 0.006
 drupal 1.393 0.006
 wordpress 1.455 0.002
 sharepoint 1.615 0.002

Here is the python code that generated the data I used for the graph:

#!/usr/bin/env python

from stackauth import StackAuth
import stackexchange # Initially run against 1.1_4
import time


ALL_SITES = [
    # Grabbed sites from sites.py on py-stackexchange website.
    # Deleted meta sites.
    # Commented out sites that don't work for unknown (versioning?) reason.
    stackexchange.StackOverflow,
    stackexchange.ServerFault,
    stackexchange.SuperUser,
    stackexchange.WebApplications,
    stackexchange.Gaming,
    stackexchange.Webmasters,
    stackexchange.Cooking,
    stackexchange.GameDevelopment,
    stackexchange.Photography,
    stackexchange.StatisticalAnalysis,
    stackexchange.Mathematics,
    stackexchange.HomeImprovement,
    stackexchange.GIS,
    stackexchange.TeXLaTeX,
    stackexchange.AskUbuntu,
    stackexchange.PersonalFinanceandMoney,
    stackexchange.EnglishLanguageandUsage,
    stackexchange.StackApps,
    stackexchange.UserExperience,
    stackexchange.UnixandLinux,
    stackexchange.WordPress,
    stackexchange.TheoreticalComputerScience,
    stackexchange.Apple,
    stackexchange.RoleplayingGames,
    stackexchange.Bicycles,
    stackexchange.Programmers,
    stackexchange.ElectricalEngineering,
    stackexchange.AndroidEnthusiasts,
    stackexchange.OnStartups,
    stackexchange.BoardandCardGames,
    stackexchange.Physics,
    stackexchange.Homebrew,
    stackexchange.ITSecurity,
    stackexchange.Writers,
    #stackexchange.AudioVideoProduction,
    stackexchange.GraphicDesign,
    stackexchange.DatabaseAdministrators,
    stackexchange.ScienceFictionandFantasy,
    stackexchange.CodeReview,
    stackexchange.CodeGolf,
    stackexchange.QuantitativeFinance,
    stackexchange.ProjectManagement,
    stackexchange.Skeptics,
    stackexchange.FitnessandNutrition,
    stackexchange.DrupalAnswers,
    stackexchange.MotorVehicleMaintenanceandRepair,
    stackexchange.Parenting,
    stackexchange.SharePoint,
    stackexchange.MusicalPracticeandPerformance,
    stackexchange.SoftwareQualityAssuranceandTesting,
    stackexchange.JewishLifeandLearning,
    stackexchange.GermanLanguageandUsage,
    stackexchange.JapaneseLanguageandUsage,
    stackexchange.Astronomy,
    stackexchange.Philosophy,
    stackexchange.GardeningandLandscaping,
    stackexchange.Travel,
    stackexchange.PersonalProductivity,
    #stackexchange.Cryptography,
    #stackexchange.Literature,
    #stackexchange.SignalProcessing,
    #stackexchange.FrenchLanguageandUsage,
    #stackexchange.Christianity,
    #stackexchange.Bitcoin,
    ]

stack_auth = StackAuth()

print "(Site, Answer:Question Ratio, Enlightenment:Question Ratio)"

nicknames = []
answer_question_ratios = []
enlightened_question_ratios = []

for site_name in ALL_SITES:
    nickname = str(site_name).split(".")[1]
    site = stackexchange.Site(site_name)
    successful = False
    while not successful:
        try:
            stats = site.stats()

            all_badges = site.all_badges()

            enlightened_badge, = filter(lambda badge: badge.name=="Enlightened", all_badges)

            answer_question_ratio = float(stats.total_answers)/stats.total_questions
            enlightened_question_ratio = float(enlightened_badge.award_count)/stats.total_questions

            print "    (%r, %.3f, %.3f)," % (
                nickname,
                answer_question_ratio,
                enlightened_question_ratio,
                )
            successful = True
        except Exception as e:
            # Probably networking issue. Loop around again
            print e
            time.sleep(20) # Don't hammer the server
  • 2
    Nice! You may want to plot answer-ratio vs enlightened-ratio if you want to prove correlation. Also: can you link to your SQL or code, or was it manually done? – Sklivvz Nov 14 '11 at 7:44
  • 1
    Manually done. If I had used SQL code, I would have included all the StackExchanges. – Oddthinking Nov 14 '11 at 8:38
  • Can you also generate statistics on total-views-per-question vs. total-votes-per-question (downvotes + upvotes)? – Randolf Richardson Nov 14 '11 at 9:05
  • 1
    I am taking these suggestions under advisement. I only did the original data analysis while I was waiting for my real-live-income-producing data analysis to finish running in the background. – Oddthinking Nov 14 '11 at 9:51
  • 1
    @Sklivvz, plot and source-code now added. – Oddthinking Nov 14 '11 at 12:26
  • Fabian has suggested (not in so many words) that we may just be vote-happy. – Oddthinking Nov 14 '11 at 12:43
  • Vote-happy? It seems more difficult to get votes from Skeptics than it does to get votes in other communities, such as SuperUser.com. Perhaps the number of participants is a factor too? – Randolf Richardson Nov 14 '11 at 17:24
  • Chatting to a mod of SciFi, I am not convinced they are in the same class as us, which means this metric isn't doing its job of proving anything. Suggestions welcome. – Oddthinking Nov 14 '11 at 21:37
  • 1
    @RandolfRichardson: Huh. I find it the exact opposite. I participate on sites like stackoverflow and don't get anything like the votes here (see). – Hendy Dec 10 '11 at 20:37
  • @Hendy: That's interesting. Although I'm not on StackOverflow.com (so I wouldn't even attempt to guess at that community's M.O. at this point), I was comparing with SuperUser.com. I suspect there's a strong "individualized" factor based on individual contributions (which may also reflect the personal interests and biases of the voting users) that probably needs to be considered -- larger sample groups are obviously important for this reason, and I wonder if the user bases in these communities might not be significant enough yet for these purposes. – Randolf Richardson Dec 12 '11 at 3:51
3

I see some various possible factors to consider:

Objectivity

From the chart, some of the sites with higher answers/question seem more "relative" compared to this site. I asked a question on parenting.SE and was practically reprimanded for asking for sources! Very different environment.

Opinions = answers on some SE variants; thus many are given.


Possible right answers

On others, it is more obvious that there are multiple ways to solve a given problem (programming, photography, [La]Tex formatting...). Here, rebuttals or confirmations are based on science, and there is often scientific consensus enough that it's not apparent that numerous answers are necessary.


Tendency to find the same sources

Because users sometimes hunt and find the same sources, multiple answers have seemed like tweaks in narrative commentary while repeating the exact same quotes/citations. An example of a time I thought this happened is the question, Do Russians drink as much alcohol as their stereotype suggests?

I add this only to say that sometimes I'm not sure that numerous answers implies health. They sometimes all say pretty much the same thing. (Another example).


User development

When I see deleted answers, they tend to share common characteristics:

  • The user has 1 or 101 reputation (perhaps a bit more)
  • The answer is a few sentences and anecdotal
  • There is a comment from Sklivvz

As these users develop, if they stick around, hopefully the community will continue to be populated with interesting backgrounded people who will answer questions in different ways. For example:

While everyone is held to citation standards, background knowledge/experience plays a huge part in how one approaches the questions, and hopefully a growing diversity of users will increase this ratio.


Moving forward

Some final thoughts:

  • What about tweaking the "reprimand message" for bad answers? I don't follow how long it takes between Sklivvz's usual comment about someone needing sources and deletion. I wonder if the message could be changed to, "Can you try to find a source to support your answer and edit it with a citation?" I guess I wonder what these answerers think -- do they give up? Do they feel justified in their opinion and think they don't need a source? I don't know what happens, but most don't ever seem to get updated and just die. Changing this would help keep more answers alive and increase the ratio. I understand if efforts along these lines are unreasonable due to moderator energy taxation and the validity of the mandate to rtfm (where m = forum rules/FAQ).
  • High-rep users waiting longer to answer questions. I admit that some questions just nag at me and I start digging in. I love researching and putting together comprehensive summaries. I already did this for personal decisions and having a site that encouraged it for other inquiries was like finding heaven. But, if a question exists with a high quality answer, why would a newer user answer it? And if new users don't answer questions because high-reppers have already done it, they aren't able to get the experience of writing them, getting critiqued, editing, and getting the point reward. I admit that this is speculation, but it could be frustrating/discouraging.
  • Living with it. Maybe, as suggested in the question, the high enlightenment badge ratio means that the site is just different. Like I said, maybe it's just a different kind of "health." Questions get answered and answered well. Others don't feel any need to add new ones. It'd be interesting to have another discussion about what SE site "health" means? Users who stick around for longer than a question? Mostly [accepted] answered questions? Views per day? Google pagerank for a given search?
  • "There is a comment from Slivvz" lol, well played – Borror0 Dec 11 '11 at 17:28
  • @Borror0: He is thorough and the site is better for it. Thankless job, that's for sure. – Hendy Dec 12 '11 at 0:52
  • @Hendy to be honest I haven't been adding that many comments lately, as Oddthinking and some high rep users have taken up the job for most part... Furthermore, we really don't delete that many answers -- unless they are unsalvageable. Many users choose to self-delete once they are made aware of the references rule. More often than not, we try to preserve the content if we can. – Sklivvz Dec 13 '11 at 22:52
  • @sklivvz: this was mostly my impression; not an effort in studious counting/research :) And again, I didn't mean that comments from you was a bad thing; I just used you as my token moderator who makes great efforts to constantly inform new users of the protocols. I'm also not sure of how things get deleted, only the typical things I notice about answers that are deleted :) – Hendy Dec 14 '11 at 2:13
0

There is a tendency to criticize any answer that does not completely address the question. Because these are researched and referenced answers getting a complete answer takes a lot of work and research. That takes time. I have several times invested a few hours in researching an answer just to have someone else post the same thing(or very similar too) I was working on.

We could encourage partial answers with positive comments, up votes, and with a statement encouraging people to get involved and share their part of the answer. Though I am not certain this would lead to better answers I would expect it would lead to more.

Update: Does physical exercise control diabetes?- The post partially answers the question and shows relevent information with out fully answering the question but the first 2 comments attack it because it is not fully answers.

Is there any benefit to placing a silver spoon in a water carafe? - Contributing information about possible benefits which the Poster conceded he can not verify but which helps answer why it may be believed. Again not the full answer but could contribute.

Do razors with more blades work better? - Here again we have a response that has pertinent information that does not fully address the question but could definitely contribute to answering the question. It is challenged in comments as not fully answering the question.

There are lots more this was just the first 3 I found on my front page. I am not pointing fingers I think quite a few of us(yes myself included) tend to do it. I am not trying to say that what we did was wrong before but rather a concerted and visible effort to change that impression would go a long way toward encouraging more answers and keep more people engaged in the community. I think this could be a fun and relevant community with great questions and answers. But I get a general feeling of oppression about both questions(especially notability), and answers(mostly completeness and references), that make it intimidating to participate.

  • Can you please give some examples? – Sklivvz Dec 17 '11 at 10:46
  • @Sklivvz - Updated with a few links. – Chad Dec 19 '11 at 14:56
  • in many cases those comments are requests for improvement, not undue criticism. – Sklivvz Dec 19 '11 at 17:27
  • I didn't down vote any of those – Sklivvz Dec 19 '11 at 17:39
  • @Sklivvz - "and thus it does not, all in all, answer the question. – Sklivvz"? - I am not offended by it when it happens to me. I take it as a challenge. And the challenges have improved my answers(In my opinion). But several times I have considered ceasing my involvement over the way they came across(as dismissive of me rather than a constructive criticism of my answer). And because of them I will not even attempt an answer unless I feel I can address it fully. Maybe that is for the better but If I feel that way someone more timid is less likely to get involved in the first place. – Chad Dec 19 '11 at 17:45
  • I see. However, if someone can't take polite and constructive criticism without giving up, then they're probably not cut for this site :-) – Sklivvz Dec 19 '11 at 18:28
  • @Sklivvz - That is the question isnt it? Is this destined to be a site frequented only by those hardy enough not to be intimidated by those who would challenge their point of view? Is that the community we want to have? – Chad Dec 19 '11 at 18:36
  • I dare say, "your answer does not, all in all, answer the question" is not something intimidating or challenging. It's a statement of fact. – Sklivvz Dec 19 '11 at 18:42

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .