17

I see several answers here use MythBusters as a reference, but Adam Savage himself has said:

We will absolutely revisit when we think we screwed up the results. We won't stand by our results—you can't with a data set of one, and two, and five. But we do stand by our methodologies.

Adam Savage at The Last HOPE. July 2008. Event occurs at 4 minutes 37 seconds.

Referenced from Wikipedia: http://en.wikipedia.org/wiki/MythBusters#cite_note-ResultsQuote-19

And the original reference: http://www.youtube.com/watch?v=7425FD3If8w

  • 2
    Hi, I think this belongs on meta so I've voted to move. I also remember a discussion about Mythbusters on meta some time ago. Can't find it at the moment, though! – John Lyon Aug 29 '12 at 22:14
  • Thanks for moving this, I was hesitant to give an answer that I couldn't really source but didn't know how else to answer :) – KutuluMike Aug 29 '12 at 23:48
  • Thanks for moving this to meta. I didn't realize there was a meta for the skeptic stackexchange. – grieve Aug 30 '12 at 13:46
  • PhD Comics: If TV Science was more like Real Science... – Nick T Aug 30 '12 at 19:32
19

As pure scientific research, no, the show wouldn't count as a credible source of data. While they are obviously careful and attempt to be thorough in their methods, as Mr. Savage says, their data sets are so small as to be statistically useless. Their experiments would count as anecdotal evidence, as best. There are all kinds of potential experimental errors introduced into the show (if nothing else due to time, budget, etc. constraints). You certainly wouldn't (or at least, I wouldn't) cite them in a scientific paper or something.

However, depending on the situation, they could certainly qualify as a credible source of additional information, particularly when taking a skeptical approach to commonly-held misconceptions. When they take a statement such as "its impossible to do X", or "if you do Y, Z always happens", they merely need to demonstrate one example of that statement being false to debunk it. They do so in the presence of a recording device that provides clear and concrete documentation of their process and results, which makes it pretty strong evidence.

In general, I find it much more reasonable to use their successful experiments as confirmation or a fact, than to use their failed experiments as a refutation of a fact. For example, I would be willing to say "yes, it is possible for a tax cab to be blown over by airplane exhaust", because they've documented exactly that happening on camera. I'd be much less willing to say "no, it is not possible that Archimedes was able to set fire to a boat using just mirrors" based solely on the few times they've attempted it. In that case, a data set of "a few failures" is nowhere near strong enough to base a conclusion on.

In general, they are much like any other source of information. If you believe the show, its cast and crew are honest and credible people, then you will accept the evidence they have on tape as credible. If you believe them to be faking things for television, you will not. And even if you do trust them in general, you still need to factor in the context in which you're hoping to use their evidence, and how thorough you believe each individual experiment was in covering all possible conditions.

  • Hmm. No sources, so that is in theory a problem here. OTOH, what source could one possibly provide for this? At most one can form an opinion about whether to accept them as a credible source, which is what Michael has made an attempt to help with. So +1 from me, even though the question arguably has no valid answer on this site. – user3344 Aug 29 '12 at 22:12
  • Good. It does belong on meta. – user3344 Aug 29 '12 at 22:27
  • At the last Comic Con the MythBusters revealed that two episodes this season "are going to be part of puplished papers". [Watch Comic Con Panel] – Oliver_C Aug 30 '12 at 13:47
  • Re "honest and credible people": apparently they lied about an island being uninhabited (see this question on Travel.SE) – mindcorrosive Aug 31 '12 at 6:54
  • lied in the sense that the producers freely admitted it was filmed in Oahu and was simply trying to simulate an uninhabited island (which are kinda hard to find)? – KutuluMike Aug 31 '12 at 13:37
3

The Mythbusters are doing science. They ask questions (the Myths), formulate hypotheses, make predictions, perform experiments, and draw conclusions. Sometimes they revisit myths when new questions present themselves using revised technique to obtain better answers.

The credibility hinges on the quality of the science. This varies considerably and can be poor, particularly when good results require large sample sets which television time limitations prevent. However, to answer many of the questions posed on the show it really doesn't have to be very good science to get results. If the conclusion follows from the data, and the methodology isn't in question, what stops the Mythbusters' from being a credible source?

In other words, the answer is maybe, depending of the quality of their science with respect to the specific question (Myth) under evaluation.

Mythbusters' experiments are set up to answer whether a myth is plausible, busted, or confirmed. In many cases one successful experiment can be enough to declare something possible (if the myth was posited as impossible) and lead to the conclusion obtained.

In other cases Mythbusters' data can be added to larger data sets from other sources or peer-reviewed by different scientists.

  • The problem is also that they don't publish their data or their methodology, so their experiments cannot be peer reviewed or replicated. – Sklivvz Sep 12 '12 at 1:00
  • 2
    @Sklivvz - I disagree, but it does depend upon the myth they are doing. However, they do generally say what they are doing (unless working with some explosive mixtures) and their results are demonstrated as part of the show which is effectively publication of the data and methodology. The biggest issue that most have with the show is the size of the sample set as opposed to publication. – rjzii Sep 12 '12 at 13:04
2

I would accept Mythbusters on a case by case basis, with demonstrations that something that is claimed to be impossible can be done having much more weight than attempts to do something that fail.

If we look at the show from the standpoint of mathematical proofs, most of the time they are trying to conduct an experiment that will refute the claims by contradiction. Thus, if the claim is that X can not be preformed but they have a show that demonstrates that it can be done without special effects or trickery then I would accept them as a valid source.

However, I would not accept them for most other claims without additional back evidence. As has been frequently pointed out, they generally don't have large enough sample sets to really prove things one way or the other to the satisfaction of most scientists. This is not a bad thing in and of itself as they can be a good lay source so people can understand a question without having to read what is usually a dry article.

That said though, Adam Savage and Jamie Hyneman are co-authors on at least one paper (PDF) so the case by case basis is likely the best approach to take.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .