Sunday, March 16, 2014

Once Again, the Models are Not Reality

We continue to act as if models are capable of predicting the future of complex systems.

This time it's economics:

The OECD economists looked at their own work forecasting the direction of the world economy over the last several years and admitted: “GDP growth was overestimated on average across 2007-12, reflecting not only errors at the height of the financial crisis but also errors in the subsequent recovery.”

One reason is that they over-estimated the positives of government regulations. Another is that assumptions changed (e.g., oil prices).

Of course, it is just as much a problem of user error because the forecast numbers are assumed to have greater predictive power than they have. Or something:

In an autobiographical essay published 20 years ago, the left-leaning economist Kenneth Arrow recalled entering the Army as a statistician and weather specialist during World War II. “Some of my colleagues had the responsibility of preparing long-range weather forecasts, i.e., for the following month,” Arrow wrote. “The statisticians among us subjected these forecasts to verification and found they differed in no way from chance.”

Alarmed, Arrow and his colleagues tried to bring this important discovery to the attention of the commanding officer. At last the word came down from a high-ranking aide.

“The Commanding General is well aware that the forecasts are no good,” the aide said haughtily. “However, he needs them for planning purposes.”

What is it with numbers that make people assume that a number with two digits after the decimal point means the formula is as precise as the number?

Attach another number to the first number--like a standard deviation--and, oh my God!--it might as well be a fifth Gospel.

Even more frightening is that numbers can at least get people to act as if they are precise even if they know better. If just for planning purposes, of course.