Thursday, October 12, 2006

Lies, Damned Lies, and Lancet

The Brit journal Lancet has staged another pre-election hit on the administration, claiming excess deaths of 650,000 Iraqis since liberation. Not surprisingly given Lancet's record, the methodology of the study is junk, it's authors partisan, and it carefully avoids a perfectly straightforward way of establishing the true facts.


The invasion of Iraq has cost the lives of 655,000 Iraqis, most of them killed as a direct result of violence, according to a disputed new study.
The figure — 13 times higher than the gloomiest of previous estimates —was compiled by random sampling of households across Iraq. It concluded that 601,000 people had died in the country’s unrelenting violence, and the rest from medical conditions and diseases whose treatment has been neglected as a result of the disruption caused by the conflict.
Here's what's wrong with this.


The survey estimates the increase in the death rate since the invasion with that in the
14 months pre-invasion (comment 39). However any balanced view of effect of the invasion on the Iraqi people would contrast their mortality after the invasion with that under a much larger slice of Saddam's rule.

By choosing 14 months, the researcher eliminates his gassing and other slaughters of the Kurds, the slaughter of the Shias post Gulf war, and the millions that died in the Iraq's attacks on Kuwait and Iran. I'd guess Saddam killed people at the rate of 2 million every 10 years, and if that's so, there are no excess deaths.

Sample Error

The authors play the classic game of feeding garbage data into nice statistical analysis software and claiming that the garbage out is sanctified by science.

To reach this result, they polled 1800 households, who reported 547 deaths since liberation and 82 during the last 14 months os Saddam. I'll accept these numbers as correct, though plenty won't, since Iraq is in the bottom quartile of world trustworthiness, with a
CPI of just 2.2.

Now the core of any statistical survey is the integrity of its sampling. For example if you were to run a survey of people's attitude to the Internet using an on-line questionnaire, you'd fail to sample the Web haters and get invalid results.

Nobody would do anything quite so daft, but bias creeps to the most carefully constructed surveys. For example phone surveys often miss the folks who work late hours, which may be why many voter polls contain a higher percentage of lefties than exist in the population. You can correct for this because the last election gives an exact measure of the number of Democrat and Republican voters, so you ask people how they voted and give more weight to the Republicans. Well, you should do that...

But that correction needs something objective like election results. In Iraq the problem is that violence is highly localized and unknown.

The problem with sampling death in Iraq is that risk varies hugely by area. It's commonplace that the North and South are relatively safe and Baghdad and environs are dangerous. But for a statistical study you need to know exactly how risk of mortality varies by area. And nobody knows - here's a BBC correspondent yesterday on driving in Baghdad:
There is no sensation like driving around Baghdad. Outside you can see what looks like the normality of an Arab street — kebab sellers fanning the coals, a glimpse of a garish display outside a wedding goods shop, a pile of fridges taking up the pavement. But a glimpse through a tattered curtain of a car is all there can be. It is considered far too dangerous for a westerner to be seen on the road.

In a city where dozens of bodies are found every morning and not a day passes without apparently random explosions in the streets, any car journey feels like another spin of the roulette wheel.
So even he doesn't know where is safe and where is not. If the man on the spot doesn't know where the violence is, the researchers and Lancet for sure don't. So they have no way of correcting for geographical bias and their numbers are valueless.

FYI, here are the death rates for Iraq from Lancet compared with the death rates given in the CIA Factbooks for Iraq, the UK and the US. (The Factbooks use internationally validated data, not figures pulled from the air by creepy Valerie Plame types).

SourceCountry2006 Estimated Death Rate per 1,000
CIA FactbookIraq5.37
CIA FactbookUS8.26
CIA FactbookUK10.13

The CIA numbers make sense because the Iraqi population is young (median age 19.7), so less die each year than the older Brit population (median age 39.3) or the US (median age 36.5).

Hence the Lancet numbers are way out of line - probably because they oversampled Sunnis who under Saddam were the ruling elite and so more likely to be killers but since liberation are more likely to be victims.

Partisan Authors

Lancet sprang a similar October surprise before the 2004 elections, so we know their motivation. They used a paper from the same team, and comment 39 here alerts us to the background of one of its members:
Les Roberts (epidemiologist) is most prominent in the news for his study estimating that 100,000 Iraqi civilians had been killed in the Iraq war, at a time when official U.S. government estimates were much lower.

Roberts campaigned for office in 2006, running in the Democratic primary for the U.S. House of Representatives seat of the 24th Congressional District in Chenango County, NY. He withdrew from the running on May 17 and endorsed the remaining Democratic candidate in the race.

Since then, some have discussed Les Roberts' possible candidacy for the state senate seat in his district.

What The Researchers Chose Not To Do

Death is one of the most easily recorded statistics - it leaves dead bodies that need to be shipped to mortuaries and then buried. Even suicide bombers are rarely vaporized - London's July 7 bombers famously donated their heads to science.

The researchers used what they must have known was unverifiable sampling to undertake the risky task of contacting and interviewing 1800 households. But why not just interview the head clerks of 1800 mortuaries? Now Saddam has gone, there are no secret mass graves to worry about. And if an average mortuary handles 10 bodies a week - 500 a year - then even at Lancet's daft death rates you'd only need 700 mortuaries to handle the entire Iraqi death rate.

Plus mortuaries will mostly be on the phone (so don't risk researchers being kidnapped), and you can cross check their data with hospital records and death certificates.

In my opinion the fact that the researchers chose not to follow this safer, cheaper and quicker route using mortuary records confirms them, and Lancet, as charlatans. And our MSM as incompetents for not spotting these obvious flaws.

None of which is exactly new news...