Models Versus Reality

This post of Jonah's at The Corner, regarding the relative value of computer models (in this context, as applied to Climategate), reminded me of an experience I had as a young engineer.

One of my first jobs was doing flight test analysis on the AMRAAM missile. My group would take data from actual missile firings and determine what had gone wrong (or right) in the test. Another group in the same building was responsible for maintaining the AMRAAM computer model that was used for simulations and pre-mission testing.

For one set of flight tests, I ran across an anomaly in the way the missile was behaving in a particular flight environment. We saw the same behavior repeated in several tests, and after checking the old wind tunnel reports from when the missile was first designed, we figured out what was causing the problem and wrote a report about it. A year or so later, as a result of that work, the missile software was modified to prevent the problem from occurring.

But a week or so after we submitted the report, the manager of the simulation group stormed into my cube and slammed a copy of it down on my desk. "This analysis is all wrong!" he exclaimed.

"What's wrong about it?" I asked. He pointed at a graph taken from the missile's internal sensors showing the anomalous behavoir. "That! That's all wrong!"

Nonplussed, I just stared at him. "That's from a sensor in the missile, from flight test. It happened several times. What's wrong with it?"

"It doesn't do that in the sim!" he yelled.

So there you have it: to a modeling nerd, when the simulation disagrees with reality, reality is at fault...