What questions should be asked to help determine whether statistic-based claims are trustworthy?

Question

I've noticed a lot of questions on Skeptics Stack Exchange that are centered around one claim or another derived from a published statistic. We would not be good skeptics if we weren't first skeptical of the nature by which said statistic was calculated.

Is there a good test, set of tests, or list of items to consider about any given statistic or statistical claim? These may be items that routinely crop up and contribute to the "something smells fishy" idea when encountering a "notable claim".

Examples of skeptics.SE questions with statistical claims concerning:

Terrorism

Food

Familial Lines

NOTE: this may almost be more of a Meta- question, but I thought it would likely be fine, here, too. A good answer here would be helpful in addressing many existing and future questions on this site.

Answer 1

There is a wide variety of familiarity and expertise with statistics among the user base here. We tend to defer to experts in the field when qualifying a particular statistic as unbiased or otherwise appropriate.

"Published statistics" can mean many things and we have different ways of evaluating them. These are all types of statistics:

• Raw counts
• A z-score from a single-tailed t-test
• A Bonferroni corrected set of significance tests across multiple hypotheses
• A Bayesian posterior having taken into account subjective belief priors
• There are many more

When presented with a statistical claim I look at the following:

• Is the raw data accurate?
• If the data was collected from what is intended to be a representative sample of a larger population, was the sampling procedure unbiased? Can we expect the results from the sampled population generalize to the population of interest?
• What is the confidence interval on the statistic of interest?
• If statistical tests were done, were these decided upon ahead of time? Was there any exploitation of researcher degrees of freedom?
• If Bayesian analysis was performed, how sensitive is the posterior to the priors? Is there broad agreement amongst experts about what the priors are? Or if not, if the priors are adjusted to reflect the disagreement, does that materially change the posterior?

But, my personal analysis or criticism of evidence doesn't hold any weight on this site, so I don't include it in my answers or comments, although it informs my votes. We look to analysis/criticism that other experts have published.