The pool player analogy is silly

In a lot of debates about economic methodology, someone will bring up Milton Friedman's "pool player" analogy. The pool player analogy was part of Milton Friedman's rationale for modeling the behavior of economic agents (consumers, firms, etc.) as the optimization of some objective function. Unfortunately, the analogy is A) not that good in the first place, and B) frequently misapplied to make excuses for models that don't match data.

Here's the original analogy:

Consider the problem of predicting the shots made by an expert billiard player. It seems not at all unreasonable that excellent predictions would be yielded by the hypothesis that the billiard player made his shots as if he knew the complicated mathematical formulas that would give the optimum directions of travel, could estimate accurately by eye the angles, etc., describing the location of the balls, could make lightning calculations from the formulas, and could then make the balls travel in the direction indicated by the formulas. Our confidence in this hypothesis is not based on the belief that billiard players, even expert ones, can or do go through the process described; it derives rather from the belief that, unless in some way or other they were capable of reaching essentially the same result, they would not in fact be expert billiard players.  

It is only a short step from these examples to the economic hypothesis that under a wide range of circumstances individual firm behave as if they were seeking rationally to maximize their expected returns (generally if misleadingly called “profits”) 16 and had full knowledge of the data needed to succeed in this attempt; as if, that is, they knew the relevant cost and demand functions, calculated marginal cost and marginal revenue from all actions open to them, and pushed each line of action to the point at which the relevant marginal cost and marginal revenue were equal. Now, of course, businessmen do not actually and literally solve the system of simultaneous equations in terms of which the mathematical economist finds it convenient to express this hypothesis, any more than leaves or billiard players explicitly go through complicated mathematical calculations or falling bodies decide to create a vacuum. The billiard player, if asked how he decides where to hit the ball, may say that he “just figures it out” but then also rubs a rabbit’s foot just to make sure; and the businessman may well say that he prices at average cost, with of course some minor deviations when the market makes it necessary. The one statement is about as helpful as the other, and neither is a relevant test of the associated hypothesis.

Actually, I've always thought that this is kind of a bad analogy, even if it's used the way Friedman intended. Using physics equations to explain pool is either too much work, or not enough.

Suppose the pool player is so perfect that he makes all his shots. In that case, using physics equations to predict what he does is a pointless waste of time and effort. All you need is a map of the pockets. Now you know where the balls go. No equations required! Actually, even that's too much...since in most pool games it doesn't matter which balls go in which pockets, you don't even need a map, you just need to know one fact: he gets them all in. It's a trivial optimization problem.

But if really good pool players made 100% of their shots, there wouldn't be pool tournaments. It would be no fun, because whoever went first would always win. But in fact, there are pool tournaments. So expert pool players do, in fact, miss. They don't quite optimize. So if you want to predict which pool player wins a tournament, or why they miss a shot, you need more than just a simple balls-in-pockets optimization model. And you probably need more than physics - you could use psychology to predict strategic mistakes, biology to predict how arms and hands slightly wobble, and complex physics to predict how small random non-homogeneities in the table and air will cause random deviations from an intended path. 

The point is, if you use an optimization model to represent the behavior of someone who doesn't actually optimize, you're going to get incorrect results.

Of course, the pool player analogy wasn't Friedman's whole argument - the next paragraph is critical:

Confidence in the maximization-of-returns hypothesis is justified by evidence of a very different character. This evidence is in part similar to that adduced on behalf of the billiard-player hypothesis - unless the behavior of businessmen in some way or other approximated behavior consistent with the maximization of returns, it seems unlikely that they would remain in business for long. Let the apparent immediate determinant of business behavior be anything at all - habitual reaction, random chance, or whatnot. Whenever this determinant happens to lead to behavior consistent with rational and informed maximization of returns, the business will prosper and acquire resources with which to expand; whenever it does not, the business will tend to lose resources and can be kept in existence only by the addition of resources from outside. The process of “natural selection” thus helps to validate the hypothesis - or, rather, given natural selection, acceptance of the hypothesis can be based largely on the judgment that it summarizes appropriately the conditions for survival. 

That turns out to just be wrong. There are plenty of theoretical ways that non-profit-maximizing agents can stay around forever. Also, there are always new people and new companies being born and entering the system - there's a sucker born every minute, so as long as they drop out at some finite rate, there's some homeostatic equilibrium with a nonzero amount of suckers present. And finally, this argument obviously doesn't work for consumers, who don't die if they make bad decisions.

So Friedman's analogy was not a great one even on its own terms. Sometimes consumers, firms, and other agents don't perfectly optimize. Sometimes that's important. So you might want to model the ways in which they don't perfectly optimize.

But actually, everything in this post up to now has been a relatively minor point. There's a much bigger reason why the pool player analogy is bad, especially when it comes to macro - it gets chronically misused.

In pool, we know the game, so we know what's being optimized - it's "balls in pockets". But in the economy, we don't know the objective function - even if people optimize, we don't automatically know what they optimize. Studying the economy is more like studying a pool player when you have no idea how pool works.

In economic modeling, people often just assume an objective function for one agent or another, throw that into a larger model, and then look only at some subset of the model's overall implications. But that's throwing away data. For example, many models have consumer preferences that lead to a consumption Euler equation, but the model-makers don't bother to test if the Euler equation correctly describes the real relationship between interest rates and consumption. They don't even care.

If you point this out, they'll often bring up the pool player analogy. "Who cares if the Euler equation matches the data?", they'll say. "All we care about is whether the overall model matches those features of the data that we designed it to match."

This is obviously throwing away a ton of data. And in doing so, it dramatically lowers the empirical bar that a model has to clear. You're essentially tossing a ton of broken, wrong structural assumptions into a model and then calibrating (or estimating) the parameters to match a fairly small set of things, then declaring victory. But because you've got the structure wrong, the model will fail and fail and fail as soon as you take it out of sample, or as soon as you apply it to any data other than the few things it was calibrated to match.

Use broken pieces, and you get a broken machine.

This kind of model-making isn't really like assuming an expert player makes all his shots. It's more like watching an amateur pool player until you he makes three shots in a row, and then concluding he's an expert.

Dani Rodrik, when he talks about these issues, says that unrealistic assumptions are only bad if they're "critical" assumptions - that is, if changing them would change the model substantially. It's OK to have non-critical assumptions that are unrealistic, just like a car will still run fine even if the cup-holder is cracked. That sounds good. In principle I agree. But in practice, how the heck do you know in advance which assumptions are critical? You'd have to go check them all, by introducing alternatives for each and every one (actually every combination of assumptions, since model features tend to interact). No one is actually going to do that. It's a non-starter. 

The real solution, as I see it, is not to put any confidence in models with broken pieces. The dream of having a structural model of the macroeconomy - one that we can trust to be invariant to policy regime changes, one that we can confidently apply to new situations - is a good dream, it's just a long way off. We don't understand most of the structure yet. If you ask me, I think macroeconomists should probably focus their efforts on getting solid, reliable models of each piece of that structure - figure out how consumer behavior really works, figure out how investment costs really work, etc. That's what "macro-focused micro" is really about, I think.

So let's put Friedman's pool player analogy to rest.


Updates

Chris House (who was the first person to ever introduce me to the pool player analogy) has a response to this post. But as far as I can tell, he merely restates Friedman's (flawed) logic without addressing the main points of my post.