Does MythBusters count as a credible scientific source?

Question

I see several answers here use MythBusters as a reference, but Adam Savage himself has said:

We will absolutely revisit when we think we screwed up the results. We won't stand by our results—you can't with a data set of one, and two, and five. But we do stand by our methodologies.

Adam Savage at The Last HOPE. July 2008. Event occurs at 4 minutes 37 seconds.

Referenced from Wikipedia: http://en.wikipedia.org/wiki/MythBusters#cite_note-ResultsQuote-19

And the original reference: http://www.youtube.com/watch?v=7425FD3If8w

Answer 1

As pure scientific research, no, the show wouldn't count as a credible source of data. While they are obviously careful and attempt to be thorough in their methods, as Mr. Savage says, their data sets are so small as to be statistically useless. Their experiments would count as anecdotal evidence, as best. There are all kinds of potential experimental errors introduced into the show (if nothing else due to time, budget, etc. constraints). You certainly wouldn't (or at least, I wouldn't) cite them in a scientific paper or something.

However, depending on the situation, they could certainly qualify as a credible source of additional information, particularly when taking a skeptical approach to commonly-held misconceptions. When they take a statement such as "its impossible to do X", or "if you do Y, Z always happens", they merely need to demonstrate one example of that statement being false to debunk it. They do so in the presence of a recording device that provides clear and concrete documentation of their process and results, which makes it pretty strong evidence.

In general, I find it much more reasonable to use their successful experiments as confirmation or a fact, than to use their failed experiments as a refutation of a fact. For example, I would be willing to say "yes, it is possible for a tax cab to be blown over by airplane exhaust", because they've documented exactly that happening on camera. I'd be much less willing to say "no, it is not possible that Archimedes was able to set fire to a boat using just mirrors" based solely on the few times they've attempted it. In that case, a data set of "a few failures" is nowhere near strong enough to base a conclusion on.

In general, they are much like any other source of information. If you believe the show, its cast and crew are honest and credible people, then you will accept the evidence they have on tape as credible. If you believe them to be faking things for television, you will not. And even if you do trust them in general, you still need to factor in the context in which you're hoping to use their evidence, and how thorough you believe each individual experiment was in covering all possible conditions.

Answer 2

The Mythbusters are doing science. They ask questions (the Myths), formulate hypotheses, make predictions, perform experiments, and draw conclusions. Sometimes they revisit myths when new questions present themselves using revised technique to obtain better answers.

The credibility hinges on the quality of the science. This varies considerably and can be poor, particularly when good results require large sample sets which television time limitations prevent. However, to answer many of the questions posed on the show it really doesn't have to be very good science to get results. If the conclusion follows from the data, and the methodology isn't in question, what stops the Mythbusters' from being a credible source?

In other words, the answer is maybe, depending of the quality of their science with respect to the specific question (Myth) under evaluation.

Mythbusters' experiments are set up to answer whether a myth is plausible, busted, or confirmed. In many cases one successful experiment can be enough to declare something possible (if the myth was posited as impossible) and lead to the conclusion obtained.

In other cases Mythbusters' data can be added to larger data sets from other sources or peer-reviewed by different scientists.

Answer 3

I would accept Mythbusters on a case by case basis, with demonstrations that something that is claimed to be impossible can be done having much more weight than attempts to do something that fail.

If we look at the show from the standpoint of mathematical proofs, most of the time they are trying to conduct an experiment that will refute the claims by contradiction. Thus, if the claim is that X can not be preformed but they have a show that demonstrates that it can be done without special effects or trickery then I would accept them as a valid source.

However, I would not accept them for most other claims without additional back evidence. As has been frequently pointed out, they generally don't have large enough sample sets to really prove things one way or the other to the satisfaction of most scientists. This is not a bad thing in and of itself as they can be a good lay source so people can understand a question without having to read what is usually a dry article.

That said though, Adam Savage and Jamie Hyneman are co-authors on at least one paper (PDF) so the case by case basis is likely the best approach to take.