1

Occasionally we get questions about whether a model or estimate proposed by an economist is accurate.

Here are some examples:

Not all questions fit this mold, and not all of these questions are directly about money.

I am not an economist, but it seems to me that economic researchers start out by making some premises/assumptions, build a mathematical model upon those assumptions, plug in some numbers and report the result.

What does it mean to ask if such a model is "true"? How should we address such claims?

2
  • An answer might demonstrate that the economist is wrong by showing an error in the calculations - that is, if the assumptions are correct, the expected outcome is not the one claimed.

  • An answer might demonstrate the economist is wrong by showing an implicit assumption that is false - that is, there is some factor that the economist did not consider. Note: An economist might explicitly state that they are making a simplifying assumption - they are not "wrong" in that case.

  • An answer might demonstrate that newspaper articles/blog articles/memes making the claim are misleading if they take the answer out of the context of the simplifying assumptions. For example, the results of a recent study about the readability of two spaces after a period was very limited - only to monospaced fonts amongst readers used to two spaces - but was widely reported as a general result, which was misleading.

    Similarly, an answer might point out that an explicit simplifying assumption is not accurate in the real world - but again, this does not mean the original economist is wrong. It is only relevant if people are claiming the result is true in the real world, rather than in the simplified world of the model.

  • An answer might support the claim by showing post-publication review supports it - other economists citing the work approvingly.

  • An answer might support the claim by showing it is supported by similar estimates by other economists.

  • If the claim makes a particular future prediction (e.g. the GDP of a country will by X by the end of the year, or a change in law might cause a particular industry to fail), then claim might be supported or refuted by empirical evidence as it becomes available.

1

As a thought, there seems to be some complication from claims about toy models meant only to highlight a single point.

For example, the primary claim asked about in this question is:

Drug regulators’ acceptance of any statistically significant improvement shown in a single randomized trial and lofty drug prices has created a situation where it is now, hypothetically, profitable for a company to run a clinical trials portfolio of chemically inert compounds. While the current cancer drug pipeline is certainly superior to inert drugs, we must rethink market incentives to encourage transformational drug development.

"Low-value approvals and high prices might incentivize ineffective drug development" (2018)

In short, the authors are pointing out that anti-cancer drugs are frequently costly for benefits that may be questionable both in extent and statistical reliability – a situation that the market allows because people who're dying tend to be desperate for any solution, even costly, unpleasant solutions with a low chance of success. The authors seem to be of the opinion that this is an undesirable phenomena, and they argue that it can be somewhat reduced by requiring further testing for regulatory approval.

As the authors point out in their comment's only figure,
           ,
requiring another p<=0.05 test would be expected to reject 95% of false positives that would've otherwise received approval, increasing the break-even cost of an approved product 1/(1-0.95)=20 times from 440-million USD to 8.8-billion USD, assuming various simplifications that they lay out in their "thought experiment".

So, given that they're basically just pointing out a conceptual framework, do we fact-check them as though they were making a real-world claim? And if we do, then what do we fact check? I mean, the paper does reference real-world figures, but only as motivation; it doesn't actually rely on any of those real-world figures to be correct to make its argument – their "Fig. 1" pictured above was meant to highlight the results that they would obtain for different data, as the authors explicitly point out that they're using average/median estimates, basically to show some hypothetical results in their toy model, rather than comment on real drugs.


Then more generally beyond that particular question, how do we address abstract mechanistic claims in general?

I mean, all models are wrong, including the most fundamental laws of physics known today. So merely pointing out that a model isn't perfectly correct is a meaningless exercise; of course all economic models that don't reduce to pure logic are wrong! (And, under incompleteness, we can even say that purely logical claims can be wrong – but that's way off-topic.)

In practice, what we'd do isn't qualify a model as wrong/right, but rather discuss its practicality and merits/flaws. So, would an appropriate tact be to respond to an economics claim by launching into a discussion of economic theory? And if so, what would be the target endpoint: would we be trying to qualify the various merits of an economic idea, or what?


Finally, SE.Skeptics still doesn't have TeX enabled. Above, I managed

increasing the break-even cost of an approved product 1/(1-0.95)=20 times from 440-million USD to 8.8-billion USD

, but beyond such utterly trivial content, it's simply not practical to format math or data here.


Conclusion

I'm not really sure how to address hypothetical economic models as though they were factual claims about the real world, even when they're motivated by real-world scenarios and used as mechanistic explanations for real-world policy advocacy.

That said, I certainly wouldn't object SE.Skeptics expanding its scope to include critical analyses of theoretical frameworks (that's awesome fun!), though it'd be even more important to enable TeX if we'd like to go in that direction.

  • I agree that attacking an economist because the toy model they openly described as a toy model is only a toy model is silly. (We might get upset if someone describes it as a real-world model though, and attack them.) – Oddthinking May 20 '18 at 17:38
  • I am highly dubious about Skeptics.SE users doing their own (original research) economical modelling. Who are we to peer review it? Find an existing economist who has done it and link to them, or ask on Economics.SE. – Oddthinking May 20 '18 at 17:41
  • TeX has a cost - every page is slower if TeX is enabled. This is the cited reason for not implementing it here. – Oddthinking May 20 '18 at 17:41
  • @Oddthinking I do wonder about how to qualify the connection between the toy model and the authors' political advocacy. I mean, their toy model does actually represent a point which a reasonably informed reader could potentially take as due motivation for the authors' political proposition, though I'd tend to suspect that the true strength of the argument would be felt by those who don't understand it; it's those folks who could be described as misled, while the basic arguments themselves aren't necessarily misleading. – Nat May 20 '18 at 18:27
  • @Oddthinking Tangentially, I was also a bit unsure about how to qualify the recent claims from Trump's tweets. The issue there was that, technically, Trump wasn't actually making any claims himself, but merely pointing out others' claims, speaking of them hypothetically; so, technically, Trump wasn't wrong about them. Though, I'd tend to interpret such tweets as having implicitly intended to mislead readers, despite not being literally untrue. Many political arguments broadcast to a wide audience seem to have this twisted construction where they're meant to be misinterpreted. – Nat May 20 '18 at 18:32
  • @Oddthinking The obnoxious thing is that I can't even cast too many stones on this issue because most folks can't seem to follow my trains of logic on anything I consider to be non-trivial despite my best efforts. Likewise, I found the example economic question to be very simplistic and straightforward; glancing at the abstract for a second was plenty for me to get what they were talking about. So, is it then reasonable to accuse them of deception? I'd be disinclined to do so at all if not for Twitter, but then it seems like the audience on Twitter tends to be more gullible, so... I dunno. – Nat May 20 '18 at 18:38
  • 2
    I mean, members of the intended audience for a Nature article should really understand the comment's significance with little effort, so I wouldn't find that comment to be at all misleading in that context. And then presumably many of the Twitter users who commented/shared it also understood it well enough. But, what about the wider audience that might read it and take it to be true outside of the toy-model context? And then that gets into questions about how to deal with conveying statistical information to a general population that doesn't get statistics, etc., and it's so messy. – Nat May 20 '18 at 18:40

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .