Flawed methodology, flawed results. Worst of the Lancet.

I feel quite confident that anyone questioning the number of violent deaths reported in the recent Lancet article Mortality after the 2003 invasion of Iraq: a cross-sectional cluster sample survey will be said by some (those for whom a bigger number is better, perhaps?) to be merely partisan. Pointing out to these people that Les Roberts, who "instigated the study and assisted with the analysis and interpretation of the data and the writing of the manuscript" is blatantly partisan himself, publicly says that he is against the war, and campaigned as a Democrat for a seat in the House of Representatives before pulling out in May of this year, is unlikely to get past that.

Pointing out that 600,000 violent deaths is absurd on its face, being roughly twice the combined death tolls at Hiroshima, Nagasaki, and Tokyo, is unlikely to get you anywhere. Noting that attributing any increase in deaths to the invasion of Iraq, as opposed to the actions of fanatics and insurgents, is suspect, is unlikely to be well-received. In fact, you might just as well resign yourself to the fact that you're talking to a true believer in the boundless evil of the American military, and accept that there is no way that you will ever convince them that the true death toll is anything less than whatever the highest number they've ever seen propounded by "authority" is, or that blame accrues anywhere other than to their preferred targets for the deaths that did occur.

That said, let's take a closer look at the methodology used and the results obtained by this survey, and see if we can't come to some conclusions regarding the validity of each.

First, let's look at the methodology used. This was a face-to-face survey, which, of course, required that the researchers identify the people they would actually visit. They took a list of the 18 governorates within Iraq, and allocated 50 "clusters" across these areas on the basis of population - the governorate with the largest population (Baghdad) was allocated 12 clusters, less populous ones got proportionately fewer (the two smallest were apparently not sampled at all due to "miscommunications"). This accomplishes geographic and administrative-unit stratification, and is accepted practice.

Within each governorate there are smaller administrative units. Dividing the size of each smaller unit by the total population of the governorate gives the proportion of the population of that governorate in each unit. Units were selected randomly, but with more populous units having a greater probability of being selected. This causes heavily populated areas to be sampled more often than rural ones; it's a trade-off between representativeness of the sample across regions of different population density and representativeness for the population as a whole. They could have stratified again based on population density, which would have given them more insight into death rates in the less densely populated areas, but at the cost of less accurate estimates for the densely populated ones. They chose door "B". That's fine - as long as they accounted for that in their statistics. As far as I can see, though, they did not. In fact, the words "urban" and "rural" don't occur at all, except in the title of a referenced article (Mortality and use of health services surveys in rural Zaire) which is cited in support of the idea that they have actually undercounted deaths.

Next, they randomly chose a "main street" within each selected administrative unit from a list of all main streets. It isn't clear what the definition of a "main street" is, but one criterion appears to be that it had to be crossed by residential streets. What impact this had on selection isn't entirely clear, but certainly it rules out any areas in which there are no crossing residential streets. By doing so it further biases selection towards urban areas, but to a degree dependent on the relative proportion of crossing side streets to main streets in an area. Small towns and rural areas might completely lack main streets by this definition, for all I know.

The next step in the process was to randomly choose a crossing residential street from among all those crossing the selected main street. Again, it isn't clear exactly what impact this has, but again, there appears to be an implicit condition applied - that the crossing residential street have at least 40 residences on it. Again, because smaller towns and rural areas are less likely to have any qualifying crossing streets by this criterion, there appears to be a bias toward heavily populated areas.

Because between 30 and 50 percent of Iraqis live outside of urban areas, this cumulative bias could have a significant impact on the results of the survey if mortality rates differed significantly as a function of the urban/rural divide.

The final step was to number the residences on the selected residential street and randomly select a starting home, then proceed to the next available house until 40 homes within each "cluster" had been sampled. All studies make compromises between collecting the best possible data and reality, but this one is a heck of a compromise. If you are looking at deaths due to car bombs, for example, people living near one another have a much higher chance of having been similarly impacted - half the people in the neighborhood could have lost a family member to a single bomb. Select one unlucky cluster, and you might wind up with an estimate that is far too high. Miss the unlucky clusters altogether, and you'll wind up with an estimate that is too low. But that's just adding error. As long as you account for this appropriately when you do your statistics, expanding the reported confidence interval, you're fine.

Note here that the researchers don't say whether they have done this. They never talk about deaths within a cluster being correlated, except across their "14-month" (really, one 14-month and two 13-month) time periods used to look at change in mortality rates over time:

The SE for mortality rates were calculated with robust variance estimation that took into account the correlation between rates of death within the same cluster over time.[14]

But that's a different issue. We're talking about accounting for correlated deaths that occurred at the same time. Did they account for that?

Now here's where we get to the most serious problem with the study. Two survey teams went house to house, collecting data, under the supervision of "field manager" (and co-author) Riyadh Lafta. The authors report that Lafta made "decisions on sampling sites" - which means that the choice of homes for the survey wasn't as random as it would have been under ideal circumstances. It isn't clear how broad this power was, but it at least extends to the "responsibility and authority to change to an alternate location if they perceived the level of insecurity or risk to be unacceptable". Certainly there should be no expectation that the interviewers would expose themselves to gunfire, but the vagueness of this provision causes concern. No explanation of how the alternative sites were chosen is provided; one example is provided:

In Wassit, insecurity caused the team to choose the next nearest population area, in accordance with the study protocol.

but "in accordance with" may mean simply that choosing the next nearest population area was allowable under the protocol, not that it was mandated by it, and it doesn't tell us how the residences within that area were chosen at all.

How often was Lafta forced to use this discretion? We don't know, because the authors don't say. They should, because it is a crucial factor in verifying the validity of the study. If the interview teams stuck to surveying randomly selected households in almost all cases, then the sample of households, at least, was nearly as random as the methodology for selecting them allowed. On the other hand, if the interview teams were forced to find new areas to survey with any degree of regularity, it is possible that they selected areas (intentionally or unintentionally) that looked like promising candidates for previous violence.

We don't know how often this discretion was used, but there is a troubling reference to "extreme insecurity during this survey" - does this mean that Lafta was forced to use discretion in choosing survey areas with great frequency? If so, this raises grave doubts about the survey.

A few other relevant notes on the methodology used. Presumably, the interview teams were previously known to and selected by Riyadh Latah. No attempt to separate combatant deaths from non-combatant deaths was made (out of fear, apparently). Probing to establish the details of deaths (e.g. who was responsible) took place at the end of the interview. Death certificates were requested and provided in 92% of cases.

1,849 households completed the survey, accounting for 12,801 people. Less than 1% of households refused to participate, a surprisingly low number. Whether this is due to cultural norms or some other factor, we don't know.

All right, at this point, what do we know? The sample was almost certainly skewed heavily toward urban areas, though no breakdown is provided along these lines. The "field manager", Riyadh Lafta, had discretion to deviate from the randomly designated sample at any time; we have no idea how often he did so, though there is a hint that he did it frequently. Moreover, we know that the integrity of the survey depended entirely on the integrity of this man and his interview teams. It also depends entirely on the integrity of those who compiled, cleaned, and analyzed the data. Their impartiality is not beyond question. For all we know, Lafta is a Ba'ath Party member.

And what did this survey produce? We know it produced an extremely (unreasonably, in my opinion) large estimate of deaths due to violence since March, 2003. It also shows the number of deaths accelerating - each successive 13-14 month period is more violent than the last.

But most of all, it produced some really crazy numbers. Like this:

From June of 2005 to June of 2006, the survey recorded 30 deaths due to car bombs. Simply scaling that up to the size of the population of slightly over 27 million equates to about 64,000 deaths from car bombs during this period nationwide.

Now, from Iraq Body Count, we know that the average number of people killed in a car bombing (assuming that anyone was killed at all, which is not always the case, and using their "reported maximum killed") is about 5.38 - we can estimate the 95% confidence interval using bootstrapping techniques at 4.57 to 6.27. So we know we need to have about 11,800 deadly car bombs nationwide to get to 64,000 deaths. At least 11,800 total car bombs, because not every car bomb can be assumed to kill people, and some would be targeted against non-Iraqis, and so would not count.

Were there that many car bombs during that time frame? Well, no. Not even close.

In the first four months of this year, insurgents detonated 284 car bombs against civilian, police or military targets, according to U.S. military statistics.

That's 71 per month, which gets you 923 over the 13 month period. So we're off by a factor of at least 12 to 13.

Now, it's not like these are kidnappings, or even shootings, which might go unnoticed or unreported. These are cars blowing up in populated areas. To even suggest that 92% of them go unnoticed is frankly ridiculous. So we have one data point that suggests that the Lancet study is flat wrong.

Of course, it could be that they just happened to select a cluster in which a huge car bomb went off - that was a possibility I pointed to earlier. The researchers should have noted that episodes of violence within clusters were strongly associated, and expanded their confidence interval accordingly if that's what happened, but it's a possible explanation that requires no more than incompetence on their parts - as opposed to mendacity.

So what about air strikes? The researchers report that 78,000 people have been killed as a result of air strikes alone. Once your eyes stop rolling around your head, note two things. First, airstrikes occurring after the hottest portion of the war (after March, 2003) killed about 18.39 (9.7-26.17) people each, provided that they killed anyone at all. By my count, and using a somewhat liberal definition of what constitutes an "air strike", Iraq Body Count records 137 of them since March 20, 2003. But to account for the number killed (according to these researchers), we need more than 4,000 of them. Now sure, the argument can be made that Iraq Body Count misses some deadly air strikes. But more than 95% of them? This doesn't just strain credibility, it forcibly rips its limbs from their sockets.

So the Lancet study is deeply flawed, overcounting violent deaths from air strikes and car bombs by a factor of maybe 15-20. But the largest portion of deaths comes from gunshots. 56% of all violent deaths - about 350,000 people - are estimated to have died from gunfire. IBC counts 2,330 incidents of death in which "gunfire" or "gunshots" are listed as the first cause - killing an average of 2.96 people each. That's 6,901 deaths from gunshots.

Now, it's true that, compared to airstrikes or car bombs, deaths by gunshot are relatively likely go unreported (though, as the researchers tell us, they were shown death certificates for 92% of deaths, so the death was presumably reported to someone). But 98.1% of them? Does anyone really believe that?

The authors go to great pains to point out that population-based methods like theirs routinely count more deaths than passive surveillance does:

Data from passive surveillance are rarely complete, even in stable circumstances, and are even less complete during conflict, when access is restricted and fatal events could be intentionally hidden. Aside from Bosnia,[21] we can find no conflict situation where passive surveillance recorded more than 20% of the deaths measured by population-based methods. In several outbreaks, disease and death recorded by facility-based methods underestimated events by a factor of ten or more when compared with population based estimates.[11,22–25] Between 1960 and 1990, newspaper accounts of political deaths in Guatemala correctly reported over 50% of deaths in years of low violence but less than 5% in years of highest violence.[26]

But Iraq is not Guatemala, or even Bosnia. Car bombs and airstrikes are not "intentionally hidden", nor are they the result of pandemic disease.

No, undercounting by passive measures is unlikely to account for these discrepancies - yes, they undercount, but no, not by a factor of 15:1, not for car bombs and airstrikes, and not by 50:1 for gunshots. The most reasonable way to account for the extreme discrepancies between the numbers reported in the Lancet and those arrived at by common sense and alternative calculations is this - the Lancet researchers' results were skewed by the choice of "alternative" sites by Riyadh Lafta, by methodological bias toward urban areas that were heavily impacted by violence, or both.

You may disagree. Maybe there is something missing from my thinking that makes these events far more likely to go unreported in Iraq than in Bosnia. But there is a relatively simple way to determine who is correct. The Lancet researchers report that subjects were able to produce death certificates for 92% of deaths. That means there is no need for population-based surveys. Instead, we can simply ask those issuing death certificates for a simple count.

Care to fund that study, MIT?