7

I've noticed a lot of answers tend to cite a single study, and from this draw an absolute conclusion that something is or is not true.

I will update this question with examples as I come across them.

In general, I think this approach is harmful and should be discouraged. A single study, despite being peer reviewed has not necessarily been reproduced, could have had errors, may have had a non-optimal sample set or a multitude of factors.

Rather, when there is only a single study in an area should we not encourage answers to say that is our best indication, but that further study is needed?

Edit: I may not have been clear.

I have no problem with answers that rely on a single study per se. Rather the problem is when not enough study has been done in a given area, and answers use the results of those few studies to draw an absolute conclusion. Essentially answering as though there is a consensus when one can not yet exist.

I don't think we need to necessarily up the standards for answers to include more studies (which is not what I meant to communicate), but rather discourage answers which give absolute answers when not enough study has been done to support such a stance.

  • 1
    As a side note, it bothers me how rarely people describe the methodology used by a study. Methodology matters. It impacts the robustness of the study and how much confidence you should have in its findings. It's important information, and thus we should include it in our answers. – Borror0 Mar 31 '12 at 9:22
  • @Borror0 Well said, I completely agree. – Sonny Ordell Mar 31 '12 at 9:24
4

I'm so torn over this issue.

Firstly, you are right.

You are absolutely right that science doesn't pivot around a single study. The meaning of confidence limits and statistical significance means we should expect incorrect results to be published occasionally, and that is only one possible source of error.

Where there is more than one study available, we should list those too. If there are contradictory studies, they should get some visibility too. Well-run meta-analysis should get an even higher priority.

We want our answers to be definitive, and this is how you get definitive answers.

So, yes, so far, I am right behind you.


On the other hand, this raises the bar very, very high. If we make every answer require a full literature review, we are going to quickly run out of people willing to submit an answer.

This site is frequently criticised for demanding too high a standard. I'm happy to defend against that view. By rejecting many of the weak non-answers that are tolerated elsewhere, the quality of the answers we do get is exceptionally good. However, if we raise the bar so high that only scientists actually working in the field can respond, then perhaps we would be better off discarding the Stack Exchange software and becoming some form of a peer-reviewed journal.

(Just pausing a moment, to let myself daydream about that.)

There's also a spectrum of how much evidence we should require, depending on (a) how extraordinary the claim is, and (b) how sure we really need to be.

For example: I would never swallow a cancer-curing drug that had been only tested with the skills of Jamie Hyneman and Adam Savage - their tests are far from rigorous, the idea that a single drug could cure all cancer is outside of our current understanding of the many and varied diseases that fall under that umbrella, and the downsides are potentially fatal.

On the other hand, if Mythbusters says vodka can remove foot odour, then... well, okay, sure, whatever. If I ever becomes an issue, I can always give it a go and see what happens (although antibacterial soap would be a preferred choice). I can provisionally accept that hypothesis until further evidence is provided.


So, I remain torn: I want the definitiveness, but I am not sure we can afford it.

I suspect the compromise answer is to rely on the comments and voting system to ensure that answers that invoke a single study to support extraordinary claims are challenged and better answers are voted up.

I'm going to try to keep more vigilant about this.

  • You and Sonny both make valid points. The logical conclusion from this, then, is to perhaps raise our standards when the nature of the question or answer suggests there will be a lot of noise. – Borror0 Mar 31 '12 at 17:37
  • I just want to clarify something. I'm not talking about raising standards necessarily and have no problem with answers that rely on a single study. I just think such answers should say that further research is needed and not say X is definitely true or false. – Sonny Ordell Apr 1 '12 at 0:39
4

My point on this is slightly different from Oddthinking and Borror0, although my feeling is that we all agree and are trying to describe the same idea from different angles.

Quantity ≠ Quality

First of all, net number of studies is completely, absolutely an irrelevant and basically wrong metric. The number of studies does not tell us anything on the quality, or the appropriateness.

Let's take an answer linking to a single study. Is the study valid? Is it a meta-study of 300 other studies? Or is it a 3-sample study published on a minor journal? Or is it an arXiv pre-print? Or a guy writing a thesis? Or Joe Bloggs' Blog?

All of these things need to be taken into account before allowing or disallowing such an answer.

Are 5 minor studies worth one meta-study?

Absolute truth?

When is it ever appropriate on this site to claim absolute conclusions? It's simply never appropriate! You need exactly ∞ studies to claim certainty. That said, there is a standard of truth in science, in that we call "true" what we accept as temporary best theories. We are asked to describe those, here, and it is appropriate to call them true, in the scientific sense.

We don't always need studies

Think about historical evidence. Most of the time it is found in old books, old newspapers, museum websites, etc. Nobody is going to publish a study on many topics we allow here: What were Einstein's grades?, Did MLK say that?, etc.

Another example are questions about health and hygiene. In many cases, linking to appropriate health ministeries' web sites is more than enough: Do I need to wash my hands after handling chicken?

We allow the use of Wikipedia as reference for simple, high-school level claims. If someone wants to reference the principle of conservation of energy, Wikipedia suffices. Some of the questions we have, only require those: Does this perpetual motion machine work?

The underlying principle is: it depends on the claim. Some claims need strong references, some other multiple references, some other very simple links. It depends.

A different approach

I think the approach should be very different as the community here is a community of experts in evaluating studies.

We expect our community to vote on the answers!

So, people vote, vote, vote. Bury answers that are citing wrong links, and up vote answers with valid references. Pay back the effort of finding studies with reputation!

  • Ah, the beautiful art of iterative agreement. – Borror0 Mar 31 '12 at 20:48
  • Kidding aside, I think you're underestimating the benefits of clearly communicating that some answers do need strong references. I remember coming across upvoted answers where, I feel, the work done was insufficient for the question. I suspect many of our users don't put enough thought into the whether the references are of sufficient quality when voting. – Borror0 Mar 31 '12 at 20:59
  • @Borror0, yes there are some vote fails, just like any other SE site. I think this is a limitation of the model, though. – Sklivvz Mar 31 '12 at 21:14
  • Your point on "truth" is what I want to focus on. My point is somewhat related to consensus. For example, there are decades worth of data and numerous peer reviewed studies showing smoking cigarettes can increase the risk of cancer. There should be little issue as accepting this as "true". When other areas have had significantly less study done, then I feel we should avoid saying something is "true" based on those few studies. I have been seeing answers where something is claimed as "true" when only maybe 2 studies have been done in that area. – Sonny Ordell Apr 1 '12 at 0:45
1

If accuracy is our only concern, you are perfectly right. Peer-review might be the gold standard but it isn't flawless. It doesn't guarantee a representative sample, for one. Therefore, if we want to maximize our accuracy, we should require more than one study to ensure the results are not noise.

If we do move on with that, it won't be costless. As Oddthinking said in his own answer, "this raises the bar very, very high. If we make every answer require a full literature review, we are going to quickly run out of people willing to submit an answer." There's an arbitrage between how high we raise our standards and our capacity to answer questions. The more we require of our users, the fewer users we will have and therefore the slower we will be at answering questions.

Thus, the question is "Is it worth it?"

Frankly, I am skeptical of the necessity of such a rule to be applied to all questions. For most questions on this site, the likelihood of inaccuracy from using one study is quite low. Introducing such a stringent requirement to those answers would be unnecessary friction for minimal gains. In other words, the opportunity cost is greater than the benefits. However, for a certain kind of questions, I think that rule makes a lot of sense. Can you really claim that X does not cause cancer from a single study? Studies of this kind tend to have a lot of noise and therefore the probability for error is greater.

For those reasons, here's what I propose:

When the nature of a topic requires studies who have an high risk of false positive or false negative, we should treat answers referencing only one study the same as unreferenced answers. Ideally, those questions would be answered by a meta-analysis a summary review of the literature will do.

I think this strikes a reasonable compromise.

While increasing our standards for all answers might be a little excessive, I think it can reasonable argued that it is too low for some more complex topics where arriving at a good answer should take a lot of effort. For those questions, I think we must set the bar higher as the result of a bad answer is greater than for the average question.

  • I'm not talking about increasing the standards for all answers. Rather when an answer draws from a single study, that answer should avoid saying something is definitely true or false and only say what the study indicates. When I see an answer that say something is absolutly true linked to only a single study, I'm inclined to downvote it. – Sonny Ordell Apr 1 '12 at 0:40

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .