7

Because this site asks questions that cover a wide variety of scientific backgrounds I would like to discuss our process of answering questions and how likely the information we are providing is wrong, even when we provide references.

I may be completely wrong in my assumption here and I want to approach this as a discussion rather than an accusation, it's just something that made me stop and think.

Consider first this answer to my question about flying sneaky helicopters into someone else's country.

The reason this question received so many votes is because it is 100% accurate and comes from someone who knows what they are talking about because that is in their field of work.

Then, looking at an answer like this one which is correctly referenced, but is based on personal research.

According to our site the answer above is all good and well and the world keeps turning, but how accurate is the second example? Is it possible we are taking the word of one set of experiments and is it even highly likely that there is readily available another set of experiments already published that contradicts the example provided? Something that happens often with science.

I'm trying not to single out the example above, but I assume people will try and draw me on that. Take note that the linked research only goes through to an extract and not the complete paper and the extract refers to "the method of stiletto on capsaicin crystal". What does that mean? Is that the equivalent to adding some chilli powder to your food and claiming it will be well preserved? The issue is the linked paper may not match exactly the real world conditions that are being asked by the OP in the claim.

Of note:

It should be noted that Serrano pepper extract is not the same as pure capsaicin. The Serrano pepper extract contains other substances that may overshadow any destructive effects of capsaicin. It is probable that the Serrano pepper extract contained some small sugars that may have provided some nutritional energy for the bacteria. (Nithin Kesireddy, Jennifer Stock, Melissa Wiesman (Dr. Jennifer Robbins) 2011)

"Does spicy food have antibacterial traits?" maybe the use of sugar or some other ingredients in the food would cause bacteria to thrive regardless of the amount of chilli. The issue is, if this is not your background how can we apply this study to regular food science. I guess a food scientist could tell us for sure, but I'm not sure we've got this right.

I'm concerned that we are actually fueling a debate rather than providing a definitive answer.

What do others think? Am I completely wrong and a single study can provide definitive proof?

6

The problem you describe is why I'm so strongly in favor of detailing the methodology used by any study you cite. It's very easy to overstate the findings of one study, or the level of trust we should have in it. By going in details about the methodology used, it both informs the readers about how rigorous the researchers were and it may warn the author that the study he is citing isn't providing the degree of proof he believed it did.

The whole debate about whether cellphones cause cancer is perhaps the best example of this effect, with many studies having seemingly contradictory results. It's only when you look at sample size, the exact conclusion of the scientists, and at the quality of the methodology that you can have a clear picture.

In other words, being a skeptic is difficult. Logic isn't enough. You need data, data is not enough either. You need to properly analyze it, and never make leaps of logic.

Providing a reference isn't enough, in my opinion. You have to understand it and vulgarize it, too.

4

The reason this question received so many votes is because it is 100% accurate and comes from someone who knows what they are talking about because that is in their field of work.

Of course, we should highly value answers that come from experts in that field. Of course, on the Internet, no-one knows if you are a dog pretending to be an expert in that field, so we expect them to provide references as well - which that helicopter answer did.

(Saying it is "100% accurate" is begging the question though.)

Unfortunately, we can't hope to have experts on every possible field here - our scope is just too broad, and we rely on volunteers with some skills in literature research to answer many questions.

Now the second answer - or indeed any of the answers on the site - may be wrong. As skeptics, we should be prepared to accept that: we should only provisionally accept any theory. But, and I think this is very important, we have the means to check for ourselves. We can see the data that was used. We can get the paper and read it and check that it is appropriate. We can look for others who have cited the paper and see if they repeated the experiment.

I have certainly started to see that happen. Sometimes, the comments aren't "Welcome to Skeptics. Please reference your claims.", they are "I read that reference you cited, and it is inappropriate because..." My heart jumps a little when that happens.

(Aside: StackExchange Inc provides small amounts of funding for established sites to send people to conferences, buy products to review, etc. I have wondered if, when we eventually get to that stage, somehow paying for some of the key journal articles that have been cited might not be a good use of that sort of money.)

Conclusion: Any answer here may be wrong, for may reasons. Providing a mechanism for others to review and reveal mistakes is what makes this site (and science) powerful.

  • I think the aside is a great idea. Apologies for the 100% :) – going Jul 29 '11 at 3:25
1

A single study can't ever provide definitive proof - it's far better to link to well-regarded meta-analyses, if they exist. These generally review any potentially unaccounted-for biases in each of the reviews included.

If somebody posts an answer citing a single study, those that disagree or have contradictory evidence should post their own answer, and outline why and how the original study was flawed. The ability to have multiple answers and the questioner's ability to change the accepted answer is what makes this site so great.

1

It depends on what is available, but I would like people to note the difference between original research and review studies. On pubmed.org for example there is a special link in the search page to filter review studies. I would strongly suggest that people with little expertise in a certain field limit themselves to reading and citing review studies.

Note the link on the right in the screenshot. On cancer there are 295,701 review studies, so no need to look at original research.

screenshot of pubmed.org

0

Am I completely wrong and a single study can provide definitive proof?

We have to bring all our skeptical abilities to bear. Math that most people can follow is one ability. Studies and/or articles that stand up to skeptical scrutiny are others. I think it is very important to be suspicious of "argument by popularity" and "argument by authority".

Anyone can be wrong or at least off-track, and that can include 95% of people in a profession, complete with professors, publications, and grad students.

This wikipedia page is my favorite example.

What I like about stackoverflow.com is it doesn't shut people up for having out-of-the-mainstream views. It only gives them downvotes. Here's an example of what I mean.

BTW: I'm a big fan of Thomas Edison, who loved nothing more than being told by "everyone" that such-and-so was impossible.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .