8

I've been wondering whether my answers would be more successful if I changed the reading level to a lower level.

If I reduce the complexity of my language, while maintaining the robustness of my answers, will I be rewarded with more upvotes?

4

What an excellent question. I am glad you asked, because I have been wondering about that myself. :-p

I want to run an experiment to measure the effect of reading age against upvotes.

At the time of writing, Skeptics.SE is not available as part of the Stack Exchange Data Explorer, so I hope to return to this when it is.

In the meantime, I would like to share my interim results, so others can suggest improvements or beat me to it.

Method

  • I downloaded the data dump for Skeptics.SE from the SE data explorer. The data is not yet available, so I have to use a proxy until it is available.

  • I chose Bicycles.stackexchange.com as a proxy for Skeptics data, until the Skeptics data is available. I chose Bicycles because the database was small and because it was non-technical, so was unlikely to be confused by code samples. The sample sizes were too small, so I discarded that.

  • I chose SuperUser.stackexchange.com as a proxy for Skeptics data, until the Skeptics data is available. I chose SuperUser because it was the largest database was in the same zip file as the Bicycles data and because it was fairly non-technical, so was unlikely to be confused by code samples.

  • I iterated through all of the answers, and extracted the relevant fields.

  • Concerned that there may be a trend toward either simpler or more complex answers over time, and that that may be confounded with the time an answer was available to vote, I modelled the vote curve, and compensated for the answer's age. I took the answer's raw vote score.

  • I carefully applied the necessary transforms to the Markdown/HTML to cull anchor tags, to convert Unicode references, to map special HTML characters, to remove quoted text as irrelevant and, especially, to remove any quoted code. I converted it to ASCII.

  • I used a sophisticated library to count words. I split on whitespace boundaries.

  • I carefully developed a technique to meaningfully count sentences. I copied a woefully naive bit of code from the 'net that merely counts punctuation, and hence thinks that this counts as six sentences!?!...

  • I decided a naive syllable-counter was sufficient, and copied one from the 'net. It thinks "one" is two syllables, but I don't think the bias matters much, as long as I consistently apply it.

  • I used these metrics to calculate the Flesch-Kincaid Grade Level (FKGL). I didn't attempt to validate it; I know it is naive and easily gamed. However, for this purpose, a simple measure should be sufficient.

  • I divided the answers into buckets based on their (rounded down) FKGL, and computed the mean vote score, and the standard error.

  • I omitted all buckets with less than 20 data points because that's the recommended technique according to my Stats lecturers and textbooks buckets with that few samples looked wrong.

  • I plotted the vote score (and an upper and lower error bar, based on the ± 1 standard error)

  • I modelled it to some or other line of best- fit, and measured, in a statistically rigorous manner, its success at predicting the curve.

Results

Graph of results

Reading this graph

Across the bottom is each "grade" in the readibility scale - the further to to the right, the more complex it is. The actual scale is rather arbitrary (especially over about 10), and I wouldn't read too much into it.

Up the side is the average score (upvotes minus downvotes)that a SuperUser answer got, given its readability. The red line through the middle is the main data you want to read. Answers seem to average between one and two.

Some readability scores are far more common that others, which gives a lot more trust in the reliability of the means. One measure of that reliability is called "standard error (of the mean)". I have drawn two more lines - one above (orange) and one below (blue) the mean. The lines represent the mean + 1 standard error, and the mean - 1 standard error. These lines represent error bars - it gives a feel for how much error there might be due to the samples being too small, and the data being too variable, to get a real feel. If you don't have a feel for it, you can ignore it. Just understand the the data on the edges isn't as trustworthy as the data in the middle.

Conclusion

I have taken a lot of short-cuts here which need revisiting, and I don't have real data from the Skeptics site yet, so it is all very flaky at this point.

It is hard to draw any real conclusions, but it does look like there is some signal in that noise. Not as much as I was hoping for.

It certainly isn't as simple as "the simpler the language, the better." There looks like a sweet spot, around 16 or 17 (whatever that measure might mean), but the curve is gentle on either side.

Worth a more rigorous look, but I am putting it aside for now. I wanted to document how far I got, in case I don't get an opportunity to come back to it for a long time.

12
  • Fascinating. I may need some sleep, but I wasn't sure what the three lines represented. Are they average votes for different sites like Bicycles and SuperUser or are they +1s vs -1s vs mean? Mean of what? The font is kinda tiny. A lot is crossed out. Instead of trying to answer me specifically, you could just edit. I think you are on to some really good stuff and should focus on editing, editing, editing it to clarity.
    – Paul
    May 5 '12 at 11:18
  • Thanks, @Paul. I've added some info on reading the graph and clarifying that I didn't use Bicycles data at all in the end. The font is a little small, but I didn't want people focussing on the actual numbers (which are largely meaningless) but the qualitative shape.
    – Oddthinking Mod
    May 5 '12 at 11:41
  • "Units" was drilled into me from an early age. Starting in high school physics, I think. Without units, you really don't know what you are looking at because anomalies or log-scales or other things can change the plot.
    – Paul
    May 5 '12 at 11:47
  • @Paul: Fair point.
    – Oddthinking Mod
    May 5 '12 at 11:52
  • Note to self: Given I don't have a lot of faith in the real-world meaningfulness of the units of readability, a better solution would be to simply use it as a ranking, and then group them into percentiles of readability. It would make comparing readability between sites harder, but would make the variability of the results more even across the X-axis, and likely much lower than is seen.
    – Oddthinking Mod
    May 5 '12 at 11:55
  • I like the shape of that graph - wonder what that 16/17 peak means. I also wonder if when the Skeptics data is available whether the peak is further to the right for Skeptics than SU...
    – Rory Alsop
    May 5 '12 at 12:10
  • now on m tablet, graph big, readable.
    – Paul
    May 5 '12 at 13:22
  • I think you should twiddle the horizontal (readability) buckets so to smooth the curve. Also - can you github the source?
    – Sklivvz
    May 5 '12 at 14:05
  • I am back on the desktop and the legend is unreadable again....The rendering of that graph must be device dependent somehow... the graph legend is tiny on Chrome under ubuntu 12.04 and the graph legend looks find on a toshiba tablet running android.
    – Paul
    May 6 '12 at 15:14
  • @Paul: It's a PNG file, so its not font rendering, but the browser's overall image rendering. If you click on the image it should open a larger version.
    – Oddthinking Mod
    May 6 '12 at 15:17
  • I think you may be missing a fairly large confounding factor that answer complexity should be related to question complexity.
    – Ryathal
    May 9 '12 at 21:14
  • @Ryathal: there are three different concepts here: conceptual complexity of the question, conceptual complexity of the answer, and language complexity. Often, the question is conceptually over-simplistic, but requires more complex concepts to answer. [citation-needed] However, that doesn't mean that the sentences need to be long or the words need to have lots of syllables. Obviously, if the topic is 'neuropsychology', some sentences are going to have large words...
    – Oddthinking Mod
    May 10 '12 at 0:24
2

Version 2

  • Improvements:

    • Now using the latest Skeptics.SE data, which has just been released.
    • Now using percentiles (more below), instead of absolute buckets. This makes the curve more useful within a site, but makes comparing between sites more complex.
    • Source code now available.
  • Existing Issues:

    • No attempt to remove quoted text or special symbols.
    • URLs probably confuse it. (See below for more discussion.)
    • No attempt to look for changes over time.
    • No attempt to improve sentence counting.
    • Still using Flesch-Kincaid Grade Level, which is only barely fit-for-purpose.
    • No statistical rigour.

Graph

enter image description here

How to read the graph:

The questions were sorted into 36 buckets based on their reading age. (I was aiming for 100 buckets, but that only left 36 or 37 answers in each bucket, and I wasn't happy that that was enough to smooth out the variability. The code I wrote adjusted the number of buckets to ensure there were at least 100 answers in each.)

So, the left-hand side represents the easiest-to-read answers. The right-hand the most difficult-to-read answers. The direct middle represents the median - half the answers are easier and half more difficult.

To give you an idea, here are some (automatically-selected, mid-range) examples:

(Examining these cast some doubts on the whole exercise. The difference isn't particularly stark. I am particularly nervous that URLs are randomising the reading grades.)

Red Line = Reading Grade

Compare the red-line against the right-hand axis. It shows the (arbitrary) reading-grade.

As can be seen, most of the answers are in a reasonable narrow range (say 20-35 FKGL-RG), which only the upper 5-10% pushing up past 40 FKGL-RG.

Green Line = Average Votes

The green line shows the average score for each bucket (upvotes - downvotes, excluding accepted answers and bounties).

Presented this way, there is little evidence of a correlation to suggest reading complexity (at least as measured by this naive system) is associated with voting, with the exception of very simple answers on the far left. Even then, that may just be an random outlier.

(If URLs are pushing up the grade, then this may correspond to answers with few or zero URLs, which makes it a terrible confounding factor.)

Conclusion

Until quoted text, HTML tags and URLs are extracted from the raw text, it is difficult to draw any conclusions at all. However, the lack of any signal here is disappointing - it may not even be worth following up.

Raw Results

Bucket, Median Readability, Mean Score, N, Example Answer Id
1, 13.38, 1.82, 101, 1008
2, 15.37, 4.71, 102, 471
3, 16.76, 6.75, 102, 1140
4, 17.63, 6.07, 102, 5820
5, 18.33, 6.77, 102, 4114
6, 18.95, 9.97, 102, 3695
7, 19.50, 7.26, 102, 5728
8, 20.06, 8.34, 102, 6711
9, 20.55, 7.12, 102, 5582
10, 21.02, 5.70, 102, 4433
11, 21.42, 8.08, 102, 6569
12, 21.89, 8.93, 102, 3063
13, 22.37, 8.08, 101, 787
14, 22.83, 7.57, 102, 3347
15, 23.35, 9.13, 102, 1865
16, 23.80, 11.41, 102, 3436
17, 24.32, 7.26, 102, 6906
18, 24.74, 9.80, 102, 2346
19, 25.28, 8.20, 102, 4476
20, 25.73, 9.19, 102, 2640
21, 26.35, 8.78, 102, 7071
22, 26.92, 8.56, 102, 7546
23, 27.54, 7.83, 102, 3646
24, 28.27, 8.31, 102, 8426
25, 28.99, 7.68, 101, 1171
26, 29.77, 8.21, 102, 2501
27, 30.66, 8.82, 102, 6171
28, 31.82, 8.71, 102, 4830
29, 33.24, 7.67, 102, 3169
30, 34.46, 6.31, 102, 1385
31, 36.29, 8.91, 102, 3328
32, 39.03, 7.57, 102, 4253
33, 42.01, 8.56, 102, 2828
34, 47.19, 5.67, 102, 3578
35, 54.81, 6.74, 102, 3403
36, 74.44, 9.20, 102, 763
5
  • There was a signal in your example links -- the algo found 2 comments as answers... Thus, there are a couple of 404's you may want to correct!
    – Sklivvz
    May 31 '12 at 7:41
  • @Sklivvz: Fixed, but not exactly a signal...
    – Oddthinking Mod
    May 31 '12 at 7:49
  • Could you limit the input text to dictionary words before doing the grade analysis (by ignoring anything not in the dictionary)? This would remove URLs, abbreviations, initialisms for technical things, etc.
    – John Lyon
    May 31 '12 at 23:40
  • Interesting idea, @jozzas. It might filter out too much medical jargon (for example) but worth a try. It might not be simple because [this text should be included](but.this.shouldn't).
    – Oddthinking Mod
    Jun 1 '12 at 0:23
  • 1
    I haven't yet understood the Stack-Exchange-specific changes to Markdown and whether running the text throught the Python implementation of Markdown will give me close enough results.
    – Oddthinking Mod
    Jun 1 '12 at 0:27

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .