# What questions should be asked to help determine whether statistic-based claims are trustworthy?

I've noticed a lot of questions on Skeptics Stack Exchange that are centered around one claim or another derived from a published statistic. We would not be good skeptics if we weren't first skeptical of the nature by which said statistic was calculated.

Is there a good test, set of tests, or list of items to consider about any given statistic or statistical claim? These may be items that routinely crop up and contribute to the "something smells fishy" idea when encountering a "notable claim".

Examples of skeptics.SE questions with statistical claims concerning:

Terrorism

Food

Familial Lines

NOTE: this may almost be more of a Meta- question, but I thought it would likely be fine, here, too. A good answer here would be helpful in addressing many existing and future questions on this site.

• FWIW, I've intended this to be a high-level discussion (e.g. "how is X defined") as opposed to highly-technical (e.g. diving into the mathematics behind the statistics, such as standard deviation values, etc.). In other words: assuming the math was correct, what can be asked to critically verify the integrity&intent of a given statistic? Commented May 11, 2016 at 19:09

There is a wide variety of familiarity and expertise with statistics among the user base here. We tend to defer to experts in the field when qualifying a particular statistic as unbiased or otherwise appropriate.

"Published statistics" can mean many things and we have different ways of evaluating them. These are all types of statistics:

• Raw counts
• A z-score from a single-tailed t-test
• A Bonferroni corrected set of significance tests across multiple hypotheses
• A Bayesian posterior having taken into account subjective belief priors
• There are many more

When presented with a statistical claim I look at the following:

• Is the raw data accurate?
• If the data was collected from what is intended to be a representative sample of a larger population, was the sampling procedure unbiased? Can we expect the results from the sampled population generalize to the population of interest?
• What is the confidence interval on the statistic of interest?
• If statistical tests were done, were these decided upon ahead of time? Was there any exploitation of researcher degrees of freedom?
• If Bayesian analysis was performed, how sensitive is the posterior to the priors? Is there broad agreement amongst experts about what the priors are? Or if not, if the priors are adjusted to reflect the disagreement, does that materially change the posterior?

But, my personal analysis or criticism of evidence doesn't hold any weight on this site, so I don't include it in my answers or comments, although it informs my votes. We look to analysis/criticism that other experts have published.

• I think this is a good answer. At the same time, can you (or someone) address things such as assumptions, definitions, and presuppositions, etc. (or alternatives or complementary ideas)? I'm looking for some higher-level concepts which raise everyone's level of critical thinking. Commented May 16, 2016 at 16:40
• I don't know what you mean. What is there that is stats-related about assumptions/definitions/presuppositions that you want me to address? "Address assumptions" isn't an answerable question... could you be more specific?
– user30557
Commented May 16, 2016 at 17:09
• I suppose I'm looking for a list that more or less encourages logical and abstracted or lateral thinking when encountering "notable claims". The Q&A examples given in the OP in one way or another come down to assessing a definition, assumption, bias, or presupposition. In the "food" one (e.g. "beef vs trees"), the claim is an attempt to operate on the human emotion of "if I sacrifice something personally, it will make me/the world better". In the terrorism example, bias abounds in juxtaposing two hot-button political issues. These sorts of things strike me as being observable "red flags". Commented May 16, 2016 at 22:00
• Okay, that's well outside of statistical analysis, so I think would be better as a separate question.
– user30557
Commented May 16, 2016 at 22:01
• And there certainly may be better Q&A examples already on this site elsewhere. Commented May 16, 2016 at 22:01