Monday, January 10, 2005

Teaching to the Test

The military isn't releasing precise attacks-per-day statistics although they are willing to give rough numbers and rough trend ideas. I've used these myself (a September 2004 post on Effectiveness used the numbers). But like body counts, they've become too hot to be useful.

This press conference explores some of the issues in keeping the number quiet:

Q Thank you. Tom Lasseter, Knight Ridder. I was wondering if you could compare daily contact numbers today versus daily contact numbers pre-Ramadan, and if you could also tell us perhaps why those numbers are not made available now, as they were before. There was a time when during briefings there were daily contact numbers given and even trend lines given, sort of, you know, this week versus a week, a month ago; this month versus last month. I think that we're all -- you know, when we hear that attacks have gone down since Ramadan, it would be helpful if we could have numbers to compare to other time
periods.


GEN. METZ: One of the reasons I think it's less and less fruitful to look at too much of the data in trend lines is because of the changing situation. And that data that was collected a year ago has a different context -- is in a different context than the current data. Obviously, we prepared for and knew that there was going to be a spike in the Ramadan time frame, and you couple that with the activities around Fallujah, we did have that increase.


The other thing we have done is we have looked at periods like pre-sovereigncy, post-sovereigncy and begun to break those up. We can probably show and try to normalize that data, but really to me it has less and less value, based on the continued changing of the situation. But we have seen -- we did see the rise in Ramadan, we have seen the tapering off. But as I said in my opening, I am fully anticipating, as the corps commander, a surge as we go into these last three weeks before the election. And that surge would be in the neighborhood of, let's say, 70-some attacks a day may surge to 80, 85. That's the kind of magnitude of numbers of attacks across the country we're talking about.


Q If I could, again, I understand that there was a spike during Ramadan, but what I'm asking is, you know, if you're looking at this graph of Ramadan now, how do the numbers today compare to numbers in the week before Ramadan, for instance? If you take what happened during Ramadan as being something separate from the normal trend line, if you looked at pre-Ramadan and post-Ramadan, how does that line up?


GEN. METZ: Let me do it this way. Pre-April was a level, April was obviously a time that there were lots of attacks because we not only had the fight in Fallujah and the parts of former regime areas and throughout the provinces in the south, so that the April level was an elevated level. That stayed rather constant through April-May and into sovereigncy. After sovereigncy, we saw some tapering off, so the summer was a little lower. August was obviously a spike, with Muqtada al-Sadr's militia fights in August, tapered off again after August to a level about the April-May level, then not as much a spike as we saw historically from the previous Ramadan in '03, but still a spike in Ramadan, and then now a tapering off again. But again, I'm anticipating that spike to come back as we move into the elections.

And later:

Q Eric Eckorn (ph), New York Times. You mentioned something like 70 attacks a day. Does that include every attack you learn about on Iraqi police or National Guard or government officials in cars or in stations or whatever? And secondly, over the last month or two, has the mix changed dramatically, as it seems to from the news that we see, away from attacks on American forces and toward attacks on Iraqi forces and government officials?


GEN. METZ: I would tell you that the attacks that we record and put in a big database are still a majority against the coalition, although we, as you mentioned, we have seen an ever-increasing focus of the enemy on Iraqi security forces and intimidation of or murder of senior leaders. The recent example is of the governor here in Baghdad, and other officials. The 70 is a metric of everything that we're -- that's available to us -- attacks on us, attacks on Iraqi security forces, on civilians and infrastructure. So what I personally watch is about a seven-day average because that gives me kind of the time frame that I'm interested in as it moves. And then there's a database that goes as far back as -- and then it becomes a little less meaningful because of the changing situation. So we look at those attacks, and they are in total at about the 70 level over the past week, over the past seven- day window. And of that, we're in the 50s against the coalition.


Q But if you measured attacks by their severity and effect, I mean, we see every day 10, 15, 20 ING or police killed, some days no Americans killed. So are you in that list sort of comparing a shot fired at a convoy versus a car bomb?


GEN. METZ: See, that is the reason I don't want to put too much stock and weight into the data that we have been collecting for, to my knowledge, probably about a year, because you're absolutely right: The small arms fire against a convoy that drove by has very little to do with the major vehicle-borne suicide delivered vehicle that killed 20 or 30 Iraqi citizens. I mean, you're absolutely right. And that's why we shouldn't put too much stock in too much of the data. We need to use it for what it's valuable for.


As long as soldiers and Marines reported the numbers quietly with no worries about how they were used and the military analyzed them for internal purposes, these numbers were fine. But once the press seized on them and gave them significance without necessarily giving them context, the numbers became useless. The danger became that everyone would teach to the test, so to speak. Soldiers would report knowing that increases would look bad. Staff would analyze knowing the press was looking for upticks. Insurgents might even attack just to get the numbers up knowing the press would protray it as insurgent progress. Leaders might then make decisions based on numbers that weren't telling us anything about the war but were telling us about the importance of the number. Add to this the problem that in April 2004, the basis for the numbers changed making my chart prior to April worthless to the numbers starting in April. Add that the types of attacks change over time as we correct weaknesses, and long-term trends are not trends at all. How do you compare rifle and RPG attacks to IEDs or to attacks on civilians or to assassination attempts? How do you fit in contacts intitiated by coalition forces as opposed to those initiated by the enemy? The number needs perspective but once it was publicized it took on a value separate from analysis.

So I won't be able to update my chart. That's fine. The number got too hot to be useful for public consumption.