9

This answer includes a Python script to parse data. The data source itself is linked. The answer is getting upvoted, presumably because people view it as a helpful contribution.

The relevant help would seem to be deleted answers, which has:

lacking factual references from reputable sources

But this answer does link a data source that at least seems to meet the smell test for reputable.

The FAQ on Meta includes original research.

Answers are original research when they perform non-trivial analysis of available data and present a novel result which requires specialist expertise to review. It is acceptable to provide a collection of evidence, but not to apply non-trivial calculations that require a community of experts to evaluate. (This also includes the use of non-trivial Internet-based tools.)

But that isn't linked from the help, nor is original research mentioned there.

A script seems like a non-trivial calculation to me. But perhaps there is a reason why this should in fact be considered trivial. Note that the calculation itself does not require specialist expertise to review. It's the analysis based on the calculation that is controversial. E.g. this comment:

I just tested a few of the top ones, but creating special rules for catching all/most of the name variants I get 613 Leos (Leo.*), 418 Lukes (Lo?u+(c|k)[aeiou]*s?), 289 Anthonys ((An)?(T|t)h?on(i|y)?), 287 Carls ((C|K)arl.*) (note, this was after removing second hyphenated names using -.*). So I think while Mohammed is definitely a popular name, it is not even close to the most popular name.

The expert analysis being that those patterns are or are not valid ways to identify the same name.

I am more interested in the general issue than in this specific answer, but I think that this answer highlights the issue well.

Is writing code to parse data original research and therefore inappropriate as an answer?

  • 1
    The main problem with that answer is that only verified GfdS' result using their methodology. The doubt was not so much about the implementation of GfdS' method but its design. So the answer added little, in my view, despite the numerous [geek sympathy, I assume] upvotes it had. – Fizz May 12 at 6:51
  • 1
    The top-voted answer to the same question also includes an original-research analysis (without code) similar to the comment you've highlighted (so using a method different from GfdS'), but this top-voted answer also included other contradictory findings from published literature. – Fizz May 12 at 6:55
5

The programmer in me loves this answer.

The skeptics mod hates it. You do indeed need to be an expert (in this case a programmer, and an expert on naming) to ascertain whether the method used is appropriate.

In my humble opinion it is very much original research. Our post notice says as much

This answer is based on original data analysis or non-verifiable data. It is up to the answerer to provide valid, verifiable and potentially replicable evidence. Answers which are wholly based on "original research" are generally downvoted and may be deleted [Emphasis added]

  • 3
    I neither agree with the result of the answer nor with the banner. And I would stress the other part: FAQ says: "non-trivial analysis", "Unverifiable data". This A presents the raw data, how its processed, in a trivial way (sorry @Tahlor , it's not the code, it's the mere method of counting). I guess the banner's applicability in this case is a bit limited? Since we can play with the data, and the inclusion criteria: the FAQ is there to prevent guru results from being presented as facts. I see the A as presenting a transparent tool to see for yourself? – LаngLаngС May 9 at 12:26
  • 6
    @LangLangC Deciding which names can be grouped together and which can't is definitely non-trivial and should be done by linguists or other experts in the field. OPs approach - (overly) generous standardizations specifically for Mohammed and a small number of seemingly arbitrary additional replacements for all other names - seems to be chosen to reach a specific goal; it isn't trivial or unbiased analysis. We can also see how difficult this is by OPs edit, which moved Luis/Louis from outside of the top 10 to second place. – tim May 9 at 12:44
  • 3
    Interestingly, it gives anyone who wants to the tools to generate their own results using whatever interpretation they want (e.g. any true skeptic). It also seems to mirror the results referenced by an external source in another answer (GfdS), even though the 2 were performed independently and their methodology was not provided. Moreover, most of the referenced sources on both the Q and As are hardly written by experts or peer-reviewed. Remind me why it make sense to repress transparency other sources obviously lack? – Tahlor May 9 at 13:49
  • OK. As a tangential illustration and fishing teaching tool I find it still quite useful. Agreed on the OR diagnosis.The rules say that in its current form it's lacking. So how to fix the problem? What is your advice to salvage it? – LаngLаngС May 10 at 6:57
  • @Tahlor It doesn't give just anyone the tools to generate their own results, you have to be proficient at programming in order to use what you've given. If you could get your contribution past a review of experts in linguistics and cultural norms when it comes to naming then I think it would be fair game to use in an answer. As it is it seems to be ever evolving and refined as more data and feedback is gathered (as pointed out by tim above). – Jeff Lambert May 10 at 16:18
  • 1
    I nuked the anwer as OR and highly upvoted as per our policy. The OP does not even dispute that it is OR (they dispute the OR rule). – Sklivvz May 10 at 17:07
  • To be clear, I've always disputed both. I've since made my objections to both more clear. – Tahlor May 10 at 22:43
7

Banning "Original Research"

My impression is the community has (or should) scale back on the ban on "non-trivial" original research—it doesn't seem to be mentioned in the Help section, though there are previous questions that reference some guidelines (e.g. 1, 2).

If such a prohibition is going to be enforced, there's really no excuse not to include it in the official Help section. In fact, burying community standards in a loose conglomeration of FAQs no-where referenced in the Help section, let alone featured, is not particularly sensible. Stack Exchange already has a bad rap for being unwelcoming to new members, but it's crazy that even complying with every stipulation in the official help section is insufficient to prevent a thoughtful, verifiable, and highly-upvoted answer (that was still improving based on feedback) from being swiftly deleted.

Presumably, the intent of any such restriction would be to limit answers that cannot be reasonably evaluated by non-expert members of the community—a good general guideline, but one that can easily be abused or unevenly applied, and lead to the suppression of potentially important information germane to the question. When in doubt, the community should be presented with more information, not less.

Problems with banning code

More generally, I would think there is something very wrong with a community of skeptics that explicitly prohibits referencing primary source data to find answers—a consequence that is almost inescapable by repressing answers that use code to communicate. This is particularly true when a question:

  • Has several external secondary sources that apparently contradict each other
  • No sources directly address the question that could be found in the data with some amount of processing or calculation
  • Is a question an expert would not bother addressing, let alone publishing
  • Expertise in the question domain is limited, if it exists

Providing code perfectly states what assumptions are being made, is reproducible, and is peer reviewable, besides empowering users to find their own answer, features that every answer should exhibit if applicable. I consider it a very dangerous precedent to repress answers that meet these criteria.

Why the referenced answer should not be banned, even under current rules

In the case of the answer you referenced, I would say that tabulation is trivial enough that it would not require a "community of experts" to verify or understand—the only calculation is addition and could be easily be performed manually. We can even break down the relevant sections of the applicable (if potentially apocryphal) rule:

It is up to the answerer to provide valid, verifiable and potentially replicable evidence, and to show convincingly that it is relevant to the question.

Nothing does this better than code.

Every answer should be suitable for review and voting by our community of experts in evaluating evidence.

The results of the code, including the assumptions made, were clearly communicated in both the code and the results table. There was no black-box that non-programmers could not understand. The results even mirrored those published by the GfdS, the official German state-sponsored association for language.

Answers which require non-trivial specialist expertise or are anecdotal in nature are not acceptable due to the nature of this community.

Plenty of commentators had very strong opinions about the assumptions themselves (and importantly, not strictly because there were assumptions), despite the fact these largely common-sense assumptions incidentally mirrored those used by the GfdS (i.e. both methods yielded the same results; notably, the GfdS does not publish their assumptions, while mine were perfectly transparent and reproducible). It's unclear to me whether all of these commentators had "non-trivial specialist expertise" in this field (again, expertise that conflicts with the GfdS), or simply they did not think the question at-hand actually required "non-trivial specialist expertise" to evaluate if the assumptions were valid. But if "non-trivial" should be understood as "high-school knowledge or less" (Sklivvz's comment), it's not abundantly clear to me that "non-expert" members of this community of "experts in evaluating evidence" should be deemed incapable of deciding whether "Mohammed" and "Mohamed", or any other spelling, constitute equivalent names in some sense—that's more than a little insulting.

Even more telling, the accepted answer even advocated their own assumptions regarding what should be included (which incidentally, do not mirror those by the GfdS, nor were there any references to other similar organizations that espouse the proposed approach, nor did the results reflect any published results). The validity of this doubtless "original research" (in the view of these commentators anyway), was never questioned, disputed, or requested to be removed by any moderators.

Conclusion

Moderator intervention should be reserved for the most extreme and dire circumstances. Collectively, answers to questions should consider every angle, even unpopular ones, and trust the skeptics in the community to find the answer that appears to be the most valid. True skeptics should consider all answers, and not stop at the top-voted one (as may be more appropriate in other communities). There really should not be any hard-and-fast rules regarding something like "original research" or providing code, as everything depends on context. There is doubtless a time to ban answers promoting original research (see Daubert standard), but if the answer in the grey area at all, the policy should be to give the power to the community and let the "experts in evaluating evidence" evaluate the answer on merits.

I consider it very dangerous and arrogant for moderators to assume they ought to suppress verifiable information that the community finds helpful—especially in cases when they might personally take issue with the conclusion of the answer presented. Again, in the referenced question, the accepted (and moderator approved) answer performed effectively the same analysis as the answer in dispute, albeit more opaquely and without underlying code; this analysis (which constitutes a non-trivial portion of that answer), has been permitted to stand, without question, while the answer in dispute was deleted. How can we take ourselves seriously with this kind of uneven enforcement?

Without more restraint, we're little better than the dubious sources we claim to refute. We can do better.

Full disclosure: I wrote the answer referenced in the question. Unfortunately, it has been deleted, eliminating much of the context of this answer. I've reposted a (presumably) acceptable answer, that emphasizes the conclusion over the process, the latter of which I errantly assumed skeptics would be more interested in.

  • 3
    If you can get your method passed by your peers (programmers, and experts on naming methodology) then I would agree with you. But this is non-trivial, and we have to just accept that your method is correct. – Jamiec May 9 at 10:11
  • 2
    Surely a significant portion of this community, if not most, are also members of StackOverflow? In this day and age, I don't think this particular code is a huge barrier for most members--perhaps if it were written in Lisp. Similarly, I think most members can decide for themselves whether they think the language decisions are reasonable. Far better to let it stand or sink on its own merits as judged by the community than to repress it, I think. – Tahlor May 9 at 16:28
  • While there are a time and space to discuss our policy, it's frankly inappropriate to do so here. The question is not on whether we should have OR rules but whether this is OR. – Sklivvz May 10 at 17:00
  • 2
    I errantly assumed meta was the time and place -- where should this be discussed? – Tahlor May 10 at 22:15
  • @Sklivvz is saying that you should post another question where you specifically challenge the Original Research policy. So then people can write answers agreeing or disagreeing rather than comments. – Brythan May 11 at 0:35
  • 2
    Oh, OK. I guess I see them inextricably linked. I maintain that code should allowed (and in some cases, encouraged), while the current OR policy is being interpreted to restrict it. If the current OR policy is too narrow to allow for code when appropriate, it should be reconsidered. – Tahlor May 11 at 1:40
  • I also touch on overzealous and uneven moderating, which seems like a somewhat unresolved (and tangentially relevant) concern of the past--can you re-raise this concern without it being considered a duplicate? Or do you just keep commenting on dead questions – Tahlor May 11 at 1:41
  • I would suggest that you look up at other meta questions about the policy and discuss the policy there (or simply vote). This question, I think, should be about the specific question and not the rule. If you have complaints about moderation, then use the contact form and not meta. – Sklivvz May 11 at 7:42
  • 2
    On the other hand though, since you are a new user, I would advise that you get to know the site in more depth before passing judgement on policies and moderation. There are plenty of sites where OR is allowed or even encouraged. This site was built on the opposite premise: no anecdotes or opinion-based answers, only third party resources. A lot of users struggle with this as they expect to be able to write almost anything they want in any textbox as long as it is right. That's not how it works here. – Sklivvz May 11 at 7:49
  • 1
    @Sklivvz: "as long as it is right" -> "as long as they think it is right". My regret is that I haven't kept a careful list of everyone who has posted OR, complained when it is attacked, and then a referenced answer appears that shows their careful calculations were based on false assumptions. – Oddthinking May 13 at 11:21
  • "there's really no excuse not to include it in the official Help section" - false assumption. Perhaps you should ask first? – Oddthinking May 13 at 11:32
  • 1
    I've edited and reflagged, and it was ignored without feedback. I reposted a new answer where every statement was sourced and offered no code, and it was deleted immediately without feedback (this particular answer also led to a ban). This is the heart of my skepticism of the values allegedly espoused here. I still think the fight is worthy (regarding actions taken by moderators, not that particular question, or even necessarily OR), but I'm certainly prepared to move on if the foe is not. – Tahlor May 13 at 18:08
  • 1
    Having saved your answer for later reading and just had time to read it, I agree with everything you say here, Tahlor. Banning trivial use of data from primary sources reduces the site to doing nothing but regurgitating the conclusions of newspapers and other commentators - which are usually exactly the things someone is asking to have fact-checked, as was the case here. That's terrible policy in itself. But what's more, it's policy that's clearly selectively enforced; the accepted answer relied on even more "original research" than yours. [1/3] – Mark Amery Jun 16 at 14:04
  • 1
    [2/3] Worse, the selective enforcement seems to have been clearly politically motivated. Both the "yes" and "no" claim were backed by non-expert, non-peer-reviewed media sources, as you note. Yet yours is the only one characterised as reaching a "novel result" based purely on original research. And in the comments, commenters were clear about objecting to the sources specifically on the basis that they were "a far-right party" and "right-wing tabloids". That's just straight up politically-motivated suppression of an answer based on it reaching a politically incorrect conclusion. – Mark Amery Jun 16 at 14:09
  • 1
    [3/3] If you don't permit using primary sources without corroboration by a "credible" secondary source AND you treat politically incorrect secondary sources as inherently non-credible, then you have a de facto rule that only politically correct conclusions are permitted on the site. That's no longer skepticism; it makes Skeptics into a propaganda vehicle incapable of actually uncovering falsehoods. – Mark Amery Jun 16 at 14:10
1

I have been slow to weigh in here, because I feel like this argument has been thrashed out enough already.

etc.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .