11

I use Media Bias Fact Check very often to check articles and their sources when I find them on social media and generally when I'm reading news or doing "whatever" research. I find it to be reliable. However, I hear from people, including those on the main Skeptics Stack, that it's biased, unscientific, or various forms "unreliable". To be clear, several of the people who disagree with the site simply do so because it disagrees with their (demonstrably biased) opinion, but some people do have their reasons.

I'd like this to be a canonical source for anyone that tries to argue for or against using Media Bias Fact Check to show how reliable sites are when answering or making comments on the non-meta Skeptics site.

For:

Doing my own researching on this, I see numerous fact checking sites use or suggest Media Bias Fact Check for people to do their own fact checking. These seem to all be libraries, but at various universities, so I'd have to assume that they did their own research on the sites to validate the information on their site, even if they are using the same software to build the site. And this is just a sample, not an exhaustive list of sites that recommend Media Bias Fact Check.

I also see it used by MIT, University of Michigan, and others to not only train AI fact checking bots, but also for other uses.

http://en.wikipedia.org/wiki/Media_Bias/Fact_Check

From my use, I see that it generally matches up pretty well with the Ad Fontes Media Media Bias chart.

Against:

The people that say it's unreliable almost invariably use the same 2 articles as their source. Both sites are listed in various places as reliable. Also, both articles are listed on the the Wiki page above.

"Amateur attempts at such tools already exist, and have found plenty of fans. Google “media bias,” and you’ll find Media Bias/Fact Check, run by armchair media analyst Dave Van Zandt. The site’s methodology is simple: Van Zandt and his team rate each outlet from 0 to 10 on the categories of biased wording and headlines, factuality and sourcing, story choices (“does the source report news from both sides”), and political affiliation.

https://www.cjr.org/innovations/measure-media-bias-partisan.php

Media Bias/Fact Check is a widely cited source for news stories and even studies about misinformation, despite the fact that its method is in no way scientific.

https://www.poynter.org/fact-checking/2018/heres-what-to-expect-from-fact-checking-in-2019/

To put a little perspective on this, though, Tamar Wilner is the author of the article on the CJR site and also writes for Poynter about media bias. It could be that her article in January 2018 or a potential personal connection with the Daniel Funke and Alexios Mantzarlis influenced the Poynter article in December of 2018.

There's also Quora, which has varying views on the site, and references the two articles above.

https://www.quora.com/How-trustworthy-is-the-site-called-Media-Bias-Fact-Check

And Just Facts Daily disagrees with their rating on Media Bias Fact Check, even after their rating is reviewed higher.

https://www.justfactsdaily.com/media-bias-fact-check-incompetent-or-dishonest/

More research

As far as being able to determine media bias scientifically, there's been little success in that, so far as I've been able to research. Most of it says that bias is a perception and uses a wide range of words or phrases to create the bias, so saying that not being able to scientifically demonstrate a site as biased denounces a media bias site seems less than accurate.

As an aside, AllSides, Snopes, and others have been subject to similar arguments against them, as seen in this article about Media Bias Fact Check.

https://www.insidesources.com/one-website-sets-classify-news-expose-fake-news/

I also got into length discussions about this topic on another Question here on Skeptics.

So, the question is still: Is Media Bias Fact Check a reliable site?

Edit:

I guess I need to define a few things, since people evidently don't understand what I'm trying to ask here. Unfortunately, this might mean I have a few things wrong about this site, since I thought Skeptics was about getting facts, rather than just opinions.

Media Bias

  • How critical is a source or an author to the political Left or Right.
  • How the wording of an article or source (in general) supports the political Left or Right.
  • Doesn't include which stories the source puts on the "front cover".

Media bias is the bias or perceived bias of journalists and news producers within the mass media in the selection of many events and stories that are reported and how they are covered. The term "media bias" implies a pervasive or widespread bias contravening[*] the standards of journalism, rather than the perspective of an individual journalist or article.

https://en.wikipedia.org/wiki/Media_bias

Contravene: Violate the prohibition or order of (a law, treaty, or code of conduct)

Reliable

Consistently good in quality or performance; able to be trusted.

https://www.lexico.com/en/definition/reliable

1: suitable or fit to be relied on : DEPENDABLE
2: giving the same result on successive trials

https://www.merriam-webster.com/dictionary/reliable

Do I really have to continue on with dictionary definitions of words, or do I need to define what "is" is, too?

What I want

I'm not interested in debating people's personal opinion, I want facts compared to facts. As in "does the Media Bias Fact Check site represent the facts as accurately as possible". I'm not interested in confirmation bias, false dilemmas, casual fallacies, strawman arguments, or other fallacies. I will, however, appeal to an authority if that authority is a real authority on the subject, which means it's not a logical fallacy at that point.

I'm looking for facts, not truths, since "the truth" can be changed by a person's perspective/bias.

Through this, I want to find out if Media Bias Fact Check is an authoritative source themselves. I'm tired of debating with people who don't actually know anything about a subject and want to call them in their BS. If this is the wrong site for this information, I guess that's a major failing of a site that calls itself "skeptics". Sub-edit: The fact that this question has been closed seems to speak for itself.

Edit 2:

This question is about the Media Bias Fact Check site, not the Ad Fontes Media Bias chart. They are two different things on two different sites, run by two different people. Nat's comment and Answer has nothing to do with the Media Bias Fact Check site I'm actually asking about.

17
  • 1
    Strange definition of truth. A bit Orwellian to untie truth from facts. Is "reliable" a fact we can measure, or a truth we can approximate?
    – fredsbend
    Jul 2, 2020 at 23:05
  • 2
    Regarding your 2nd revision edit, this "I'm here for facts, all else be damned" attitude is exhausting. I rarely find dives into that self-claimed creed fruitful. So, why are you asking about MBFC? Because someone said something about it in a comment somewhere else and you disagree? I no longer understand what the issue is; I don't understand what you want in an answer here on meta. I thought you wanted to discuss MBFC as a "reliability tool". If you think you can give some good answers on Skeptics, then do it. If you have criticisms of other answers, you have the comments available to you.
    – fredsbend
    Jul 2, 2020 at 23:23
  • 3
    @fredsbend, I find that having to define the meaning of common words to be exhausting and demeaning to both me and the people asking me to define the words. Yes, I'd like a definitive answer, as best as possible, as to MBFC's reliability, but if all I get are requests for definitions of words, then I guess I'm going to have to muddy the waters with redundant definitions. And if this isn't a place for facts, but rather biased personal opinions, then this isn't the place I want an "answer" from. Take that as you will, because I'm also exhausted of baseless and biased opinions.
    – computercarguy
    Jul 2, 2020 at 23:52
  • 1
    I don't see where anyone currently engaged here hung up on the meaning of common words ... As for wanting a "definitive" answer on MBFC, you seem to have one already: your own. You can put something in the answer space. That would be appropriate, then you'd have something to cite when challenged again on its usefulness. Conflating this particular thing with the site as a whole has me quite confused. Your participation here to date is far too light to glean any understanding of your issues with the policies/culture. There's issues, I can agree. What you think those are I hardly have a guess.
    – fredsbend
    Jul 3, 2020 at 0:34
  • 1
    @fredsbend, I guess you didn't read Kevins "Answer" that nearly entirely asks for definitions of common words. I guess you also missed a comment on my previous version of this Question that also asked for definitions of words.
    – computercarguy
    Jul 6, 2020 at 16:04
  • @fredsbend, and no, an Answer about the Ad Fontes Media Bias chart isn't and answer about Bedia Bias Fact Check, so no, Nat's comment shouldn't have been made into an Answer on this question about Media Bias Fact Check, since they are two different entities.
    – computercarguy
    Jul 6, 2020 at 16:55
  • 2
    I haven't had a chance to look at this again yet, but as I seem to have responded about the "Media Bias Chart" as opposed to "Media Bias Fact Check", I deleted my answer.
    – Nat
    Jul 7, 2020 at 3:44
  • Any direct answer just leads to an infinite regress regarding the bias of its own reporting, so people are being indirect because it's unclear what a direct answer would look like: who do you trust to answer this question fairly? Apparently it's not academics, because they might not be careful. It's not other fact-check sites. Apparently it's not the people here. But what could anyone say that would be more definitive? If you want an overview summarising their entire past performance by a group that you already trust more, the right answer might be that no such thing exists.
    – Robert Mastragostino
    Jul 8, 2020 at 18:45
  • @RobertMastragostino, if you take that stance, then no site, media, news source, or really anything could be credited as being reliable. Starting at well-known, trustworthy, minimally biased sources for information and working from there is how you build a foundation of knowledge. That's what I'm looking for here: building a consensus of what can be relied on for information. That some people keep trying to destroy that consensus is what I'm trying to prevent by asking for facts, rather than sometimes baseless personal opinions.
    – computercarguy
    Jul 8, 2020 at 18:55
  • 2
    Given the title of the CJR article "We can probably measure media bias. But do we want to?" it leaves me with many questions. When looking at any data including opinion pieces why would you not want to understand the biases present in the source?
    – Joe W
    Oct 27, 2021 at 21:28
  • @JoeW, the only time I can think of where you wouldn't want to understand the biases present in something is when it agrees with your own bias and you don't care about anything else. In that case, you like the bias confirmation and probably refuse to believe anything that goes against it. I could give examples of that, but that would probably just be showing my own biases.
    – computercarguy
    Oct 27, 2021 at 22:06
  • 1
    "People that say it's unreliable almost invariably use the same 2 articles as their source" this is factually wrong. You are trying to put an unfounded stigma onto the dissenters.
    – High GPA
    Jun 2 at 19:07
  • @HighGPA, can you provide examples of where people get their information about Media Bias Fact Check being unreliable? All the places I could find eventually sourced those 2 articles, either directly or indirectly by citing sources that eventually did lead to those 2 articles.
    – computercarguy
    Jun 2 at 19:09
  • 1
    @computercarguy Not really. For example, some people doubt the neutrality of MBFC because it falsely labeled a ROC media. Most individuals believe that MBFC is biased for a variety of reasons; those individuals generally do not have COI with MBFC and are not special interests groups. It is OK to say some people believe that MBFC is unbiased; it not OK to say most non-COI individuals believe that MBFC is unbiased.
    – High GPA
    Jun 2 at 19:24
  • 1
    @computercarguy 1) All of the first three answers believe that MBFC is either subjective, or "limited". For example, "there is nothing to suggest that this website has any value beyond the opinion of one Dave Zandt". 2) I only said "non-COI" which is a very neutral word. Since you love to put your words into others' mouths, our discussion ends here.
    – High GPA
    Jun 2 at 20:47

2 Answers 2

Reset to default
5

The concern I have, when reading your question, is that you are focused almost entirely on what various authorities are saying about this site. While appeals to authority are sometimes a useful epistemic shortcut, we need to careful about when and how we employ that shortcut. If you focus too much on what authorities say, you may lose track of the line between objective and subjective arguments (which may have happened here). Instead, I think it may be more fruitful to consider three factors here:

  1. What is the question you are trying to answer?
  2. What is the methodology of the source you are considering using?
  3. To what extent is that methodology predictive of the answer to your question?

Taking these in turn:

  1. The question you are trying to answer - You need to decide, as clearly and unambiguously as possible, what you mean by "media bias." It doesn't matter what other people mean by this term - you are presumably investigating media bias for a specific reason, and your definition of "media bias" should be informed by that reason. Almost everyone exhibits some form of bias, but not all biases are equal. There is a vast difference between a biased choice of stories to cover (which story gets written at all), a biased presentation of those stories (which story gets the front page), a biased reporting of those stories (which facts get emphasized), and outright fabrication of stories. Depending on why you are reading the news, you may care more or less about each of those biases. If you just want to verify specific facts about an event, then you likely don't care whether the news story is on the front page or on page 37, so long as it's true and not misleading. But if you're looking to overhaul your entire media diet, then you might well care about what makes the front page.

  2. The methodology - In this case, it sounds as if Media Bias Fact Check grades sites against a rubric. Do they publish this rubric? How subjective is this rubric? When they apply it in practice, do they make the same decisions which you would've made, or make good arguments for their decisions where they differ from yours? Or do they just spit out numbers, devoid of any context or explanation?

  3. Compare the rubric, and the other answers to questions in step #2, against the definition you came up with in step #1. If they don't match up very closely, then regardless of whether the site is reliable in general, it is not reliable for your purposes, either because it is answering a different question than the one you want to ask, or because it is not answering any sensible question. You need not decide which of those is which; it is enough to conclude that the site is not useful to you.

    On the other hand, if you conclude that the site's methodology would be helpful to you, you can now turn back to the authorities with greater confidence; you no longer care about whether the site's methodology is correct, but instead about whether the site consistently applies that methodology in its ratings. This is a more objective question, for which appeals to authority are more appropriate. And it makes it easier for you to filter out the subjective arguments from authorities who decided on a different definition of "media bias" than you did (in step #1).

8
  • I'm not sure what happened to my previous comment, but I'm not interested in confirmation bias. I'm interested in how accurately the site reports the factual content and how biased other sites are. I want to use the site as a reference point for myself and others, and show people how biased they and their sources are in an impartial way. I'd like to believe my edits since this answer was posted address these questions of yours. Also, the MBFC site itself should answer #2 for you, because yes they do publish their rubric and explain their results for each site on the page for that site.
    – computercarguy
    Jul 7, 2020 at 20:30
  • 2
    @computercarguy: My suggestion would be that you taboo the word "biased" and similar terms such as "positive/negative light." Otherwise, we're just going to argue in circles.
    – Kevin
    Jul 7, 2020 at 21:38
  • Your argument against using a word like "biased" is already a circular argument. It basically says that there's a bias against "those" words, and with a site that has that word in it, it's going to be impossible to avoid using the word. The article you post links to "LessWrong" which uses the word "bias" in the article describing them. I believe you are thinking too hard about the semantics of the question, possible answers, and everything in general, and not enough about actually trying to give an actual answer to the Question. Also, this is why I asked for facts, not personal opinions.
    – computercarguy
    Jul 7, 2020 at 21:47
  • It should be entirely possible to do research on a site and find relevant information about it without letting much bias into the research. That's what good journalists do, and I'm sure any significantly biased Answer is going to be torn to shreds because of it, just like you're trying to do my Question.
    – computercarguy
    Jul 7, 2020 at 21:50
  • @computercarguy: That's the first time someone has accused me of "thinking too hard" in response to a Stack Exchange question. My answer speaks for itself, and I don't think it's necessary to make any further changes to it in light of your edits to the question, nor the comments you've left here. If you dislike this answer, you're free to downvote it and/or write your own.
    – Kevin
    Jul 7, 2020 at 22:07
  • 1
    I have already downvoted it, but because of my RP, it doesn't show. And I was hoping that someone besides me would have a definitive answer. I've already decided that if 2 weeks go by from the original posting and I don't get a relevant Answer, I will give my own answer.
    – computercarguy
    Jul 7, 2020 at 22:16
  • @computercarguy Two weeks is tomorrow. You will answer tomorrow, give or take?
    – fredsbend
    Jul 9, 2020 at 21:24
  • @fredsbend, thanks for the reminder. Yes, I'll try to post an answer tomorrow, unless you or someone else wants to take a stab at it. I've refrained from posting due to wanting to avoid seeming to be biased for MBFC and people assuming I'm caught in a confirmation bias or an appeal to authority fallacy.
    – computercarguy
    Jul 9, 2020 at 22:32
2

I waited 2 weeks before posting this answer to give other's the opportunity to weigh in and do research. Unfortunately that hasn't happened. I didn't want to come off as "of course it's reliable" because of confirmation bias or falling into the appeal to authority fallacy. Since there isn't anyone else that wants to put their foot forward, I will, with lots of research.

TL;DR

Yes, Media Bias/Fact Check is normally reliable. It's not 100% reliable, but nothing is. It's research is done by humans who have biases, even if they are unconscious biases. However, when they are made aware of mistakes, they take steps to address the issue.

Baseline Research

In order to validate if Media Bias/Fact Check (MBFC) is reliable, I need to set some sort of baseline as to what's reliable, what's not reliable, and how I determine reliability. So I needed to research other sites I included in my research. Yes, we're going down the rabbit hole, Dorothy, but I'm cutting it a bit short so this Answer eventually ends.

I've left off research on sites that were clearly biased in their speech patterns. I've also avoided articles behind paywalls, mostly because I don't have access to those articles, either. I've gone through dozens (maybe even over a hundred or more) articles, sources, opinion pieces, and more to get as relevant and unbiased information as I can.

Snopes
After reseaching this site, I find it to be reliable. Not only based on authoritative/reliable sources, but also it's generally accepted by people in general to be reliable.

Ad Fontes Media Bias Chart
I'd rate this also as reliable. It gets updated and has it's methodology published. The research is published, but seems limited in some cases. However, sites with hundred or tens of thousands of articles can't all be checked for biases. Maybe they will in the future, but that's unreasonable today. It's a fairly new chart and has it's critics, though.

AllSides
AllSides also seems to be reliable, however this is a essentially a polling site, so it relies on people's personal opinions, rather than a professional researcher(s). It's also exclusively for the USA.

Many sites (including MBFC) are listed and rated well in this PDF by Unesco:

I could go on, but this seems like it's getting redundant. Many of the articles that I've read include references to multiple fact checking sites, which generally include MBFC, which I've been saving for now, the research of MBFC.

By the way, the reason I did this "baseline research" is so that I could compare if MBFC is as reliable as these other fact checking sites that are considered to be reliable by the consensus of the public and the media in general.

Media Bias/Fact Check
This site publishes their methodology, includes their research on each page (example), and uses other well-known fact checking sites to do their research (some of them listed above). It's also quite often listed in fact checking articles to be used as a source, and listed as an equal to other fact checking sites that have been around for much longer (again, some of them listed above). This site doesn't fact check every article of each site it rates, but again, that's not reasonable due to the relatively short life of MBFC and the fact that it's all done by humans.

MIT used MBFC as a baseline when creating an AI to look for media bias.

The University of Michigan Center for Social Media Responsibility also uses MBFC as a baseline for their "Iffy Quotient", which tracks factual information in Twitter and Facebook.

Against
There's plenty of criticism about MBFC, but mostly from people who think their ratings are wrong. I haven't included that, since they are obviously biased in their own favor. There's even an article written by Just Facts Daily that complains about their rating, which I included in the original Question, so I won't rehash it here.

An article by Tamar Wilner makes a snide remark about the site, calling the founder an "armchair academic" and an "armchair media analyst". On the MBFC, founder Dave Van Zandt calls himself an armchair media analyst with over 20 years experience, so I'm not sure that's such a negative. Also, Wilner's article was written over 2 years ago, so there's plenty Van Zandt and his team could have learned.

https://www.cjr.org/innovations/measure-media-bias-partisan.php

Wiltner writes about media bias and other related information, so should be considered a reliable resource.

  • Tamar Wilner is a Dallas-based freelance journalist and researcher who writes about misinformation, fact-checking, science communication, and all things media.

https://www.cjr.org/author/tamar-wilner

The article asks if people care or whether we should even bother with rating sites, since there is no "baseline". She mentions that AI and other scientific methods to discover bias in articles are struggling and mentions a few sites that are already being used for bias checking.

The problems she mentions about MBFC (and Ad Fontes) in her article seemingly have been since addressed and hints at a bias against non-professional fact checkers, like herself. So basically she's just complaining that they are new sites that don't have a large enough human pool to pull from and their sample sizes are too small. While those are relevant criticisms, they are going to happen with any new resource, which both MBFC and Ad Fontes were at the time of the article.

As I mentioned in my Question, Wilner also writes for the Poynter Institute, which also has a negative reception of MBFC and the article was written later the same year. While I can't confirm anything, it seems likely Wiltner could have a personal influence with the writers of that 2nd article.

Conclusion

While MBFC has it's problems and critics, so do all other fact checking sites. Some of the problems come from "too small" of a sample of a site. That's understandable, since no fact checking site checks 100% of the information on the checked-site. It just means that more time and effort is needed to get a larger sample size, and with continuing checks of the sites, the ratings will become more accurate over time.

And the human factor can be significant, but that can be mitigated with double-checks by multiple people. It can also be mitigated by the rubric used to determine bias and fact checks.

With MBFC generally closely mirroring other fact checking sites that are also found to be reliable (in my Baseline Research above) and the fact that the information is updated moderately regularly, I'd give Media Bias\Fact Check the thumbs up for being a reliable source.

CYA

As much as I've presented here, I'm not infallible and may have missed something relevant. Instead of arguing in the comments, please write your own answer. If there's something wrong with my research, those comments are welcome, but only so I can fix this Answer, rather than add new sections. And because this Q&A is about media bias and reliability, consider your sources before making a claim, as it will be checked.

I've tried to use a wide range of sources without using MBFC to verify their reliability, to avoid circular logic. While I've been able to verify that most of them are reliable through third party sources (not listed in this Answer), if something isn't actually reliable, please let me know so I can address it.

7
  • 1) There is one more article you possibly missed: politifactbias.com/2017/10/…
    – High GPA
    Jun 2 at 20:54
  • 2) "There is nothing to suggest that this website has any value beyond the opinion of one Dave Zandt". Zandt is the president of MBFC. You can check his CV to get a feeling of the political inclination of the affiliated institutions of him.
    – High GPA
    Jun 2 at 20:54
  • 3) MBFC did make factual mistakes on labeling some China/Taiwan-related media, such as Taipei Times. If "reliable" means "consistently good", then it is hard to argue that MBFC is reliable.
    – High GPA
    Jun 2 at 20:57
  • @HighGPA, 1) did you read the first comment on that article? It talks about how biased that site is, so that article isn't credible and mostly just a rant about how 1 site is rated, without reasoning about why it's wrong, just personal attacks about Van Zandt. 2) You just got that quote off the Quora site from a rant post. 3) MBFC does make mistakes, just like everyone else on the planet, they also work to correct those mistakes. You've made several mistakes just on this page. And how have they made mistakes with Taipei Times, etc., specifically?
    – computercarguy
    Jun 2 at 21:38
  • Never mind, you just got that Taipei Times and China statement off the same Quora thread from an answer that was posted 3 years ago. I thought that rang a bell somewhere, so I had to double check.
    – computercarguy
    Jun 2 at 21:57
  • Making mistake is not "consistently good". You are just having a lot of exaggerated claims on your dissenters with little facts. Have you checked the list of institutions affiliated with Zandt?
    – High GPA
    Jun 2 at 22:27
  • @HighGPA, instead of making sweeping claims with no backing, why don't you enlighten me/us? They are your claims so you have the burden of proof. I've supplied plenty of proof already, now it's your turn. Also, In my comment, I asked people not to argue in the comments, but to post their own answer, so why don't you do that?
    – computercarguy
    Jun 2 at 22:31

Not the answer you're looking for? Browse other questions tagged .