A bit of pushback against the empirical tide

There has naturally been a bit of pushback against empiricist triumphalism in econ. Here are a couple of blog posts that I think represent the pushback fairly well, and probably represent some of the things that are being said at seminars and the like.

First, Ryan Decker has a post about how the results of natural experiments give you only limited information about policy choices:

[T]he “credibility revolution”...which in my view has dramatically elevated the value and usefulness of the profession, typically produces results that are local to the data used. Often it's reasonable to assume that the "real world" is approximately linear locally, which is why this research agenda is so useful and successful. But...the usefulness of such results declines as the policies motivated by them get further from the specific dataset with which the results were derived. The only way around this is to make assumptions about the linearity of the “real world”[.] (emphasis mine)

Great point. For example, suppose one city hikes minimum wages from $10 to $11, and careful econometric analysis shows that the effect on employment was tiny. We can probably assume that going to $11.50 wouldn't be a lot worse. But how about $13? How about $15? By the time we try to push our luck all the way to $50, we're almost certainly going to be outside of the model's domain of applicability.

I have not seen economists spend much time thinking about domains of applicability (what physicists usually call "scope conditions"). But it's an important topic to think about.

Ryan doesn't say it, but his post also shows one reason why natural experiments are still not as good as lab experiments. With lab experiments you can retest and retest a hypothesis over a wide set of different conditions. This allows you to effectively test whole theories. Of course, at some point your ability to build ever bigger particle colliders will fail, so you can never verify that you have The Final Theory of Everything. But you can get a really good sense of whether a theory is reliable for any practical application.

Not so in econ. You have to take natural experiments as they come. You can test hypotheses locally, but you usually can't test whole theories. There are exceptions, especially in micro, where for example you can test out auction theories over a huge range of auction situations. But in terms of policy-relevant theories, you're usually stuck with only a small epsilon-sized ball of knowledge, and no one tells you how large epsilon is.

This, I think, is why economists talk about "theory vs. data", whereas you almost never hear lab scientists frame it as a conflict. In econ policy-making or policy-recommending, you're often left with a choice of A) extending a local empirical result with a simple linear theory and hoping it holds, or B) buying into a complicated nonlinear theory that sounds plausible but which hasn't really been tested in the relevant domain. That choice is really what the "theory vs. data" argument is all about.

Anyway, the second blog post is Kevin Grier on Instrumental Variables. Grier basically says IV sucks and you shouldn't use it, because people can always easily question your identification assumptions:

First of all, no matter what you may have read or been taught, identification is always and everywhere an ASSUMPTION. You cannot prove your IV is valid...

I pretty much refuse to let my grad students go on the market with an IV in the job market paper. No way, no how. Even the 80 year old deadwoods in the back of the seminar room at your job talk know how to argue about the validity of your instruments. It's one of the easiest ways to lose control of your seminar. 

We've had really good luck placing students who used Diff in diff (in diff), propensity score matching, synthetic control, and even regression discontinuity. All of these approaches have their own problems, but they are like little grains of sand compared to the boulder-sized issues in IV.

He's absolutely right about the seminar thing. Every IV seminar degenerates into hand-waving about whether the instrument is valid. He doesn't mention the problem of weak instruments, either, which is a big problem that has been recognized for decades.

Now, Kevin is being hyperbolic when he categorically rejects IV as a technique. If you find a great instrument, it's really no different than regression discontinuity. And when you find a really good instrument, even the "deadwoods" in the back of the room are going to recognize it.

As for IV's weakness in the job market, that's probably somewhat due to the fact that it's been eclipsed by other methods that have not been around as long as IV. If and when people overuse those methods, it's highly probable that people will start making a lot of noise about their limitations. And as Ed Leamer reminds us, there will always be holes to poke.

Anyway, these posts both make good points, though Kevin's is a little over-the-top. Any research trend will have a pushback. In a later, more pompous/wanky post, I'll try to think about how this will affect the overall trend toward empiricism in econ... (Update: Here you go!)