Claims of HIV Vaccine Success are Premature

The excitement is palpable — the vaccine that nobody thought would work “appeared to lower the rate of HIV infection by 31.2 percent compared to placebo ”, according to the press release, although printing all three significant figures sets my inner sceptic on edge.

The BBC story improved things marginally, reporting a rounded percentage:

“Scientists announced last month that a combination of vaccines gave a 31% level of protection in trials among 16,000 heterosexuals aged 18-30.
Doubts had been raised about whether the finding was significant.
But new data published at a conference in Paris indicates that, while small scale, the findings are robust and statistically significant.”

Exciting, but while statistically significant sounds very scientific and reliable (the results were published in the New England Journal of Medicine (NEJM) no less), the journalists should have read the report itself. The figures reveal a little sleight of hand.

The study randomised over 16 000 people to either the vaccine program or a placebo, with none of the participants knowing what they had (a double blind trial — the best sort). The randomisation here is key, as the study was rather underpowered and was only likely to produce a marginal result at best, and any deviation from this randomisation may have introduced biases that were hard to spot.

The researchers actually carried out three statistical tests on the data from the trial: intention to treat (ITT), per-protocol and modified intention to treat (mITT) analyses. The first two look at those participants who were enrolled (ITT) or completed the treatments (per-protocol), and so preserve the randomisation of patients. These both failed to show a statistically significant benefit from the vaccine.

The mITT process removed several people from the analysis, both breaking the randomisation and producing a statistically significant result. This might have been useful if the vaccine had a clear benefit, but the published benefit was not 31.2% exactly. Rather the confidence interval for the benefit (the range in which the researchers were confident that the true benefit figure lay) was 1%-52%, with the lower bound only staying positive because of the arbitrary choice of 95% confidence intervals chosen by statisticians over the years.

With the two tests that avoided the possibility of bias getting ignored by the press (since they didn't make the press release), and the remaining result showing that the vaccine could have had no actual benefit, the publicity seems a little unjustified.