Claims of HIV Vaccine Success are Premature

HIV vaccine
The excitement is palpable — the vaccine that nobody thought would work “appeared to lower the rate of HIV infection by 31.2 percent compared to placebo ”, according to the press release, although printing all three significant figures sets my inner sceptic on edge.

The BBC story improved things marginally, reporting a rounded percentage:
“Scientists announced last month that a combination of vaccines gave a 31% level of protection in trials among 16,000 heterosexuals aged 18-30.
Doubts had been raised about whether the finding was significant.
But new data published at a conference in Paris indicates that, while small scale, the findings are robust and statistically significant.”
Exciting, but while statistically significant sounds very scientific and reliable (the results were published in the New England Journal of Medicine (NEJM) no less), the journalists should have read the report itself. The figures reveal a little sleight of hand.

The study randomised over 16 000 people to either the vaccine program or a placebo, with none of the participants knowing what they had (a double blind trial — the best sort). The randomisation here is key, as the study was rather underpowered and was only likely to produce a marginal result at best, and any deviation from this randomisation may have introduced biases that were hard to spot.

The researchers actually carried out three statistical tests on the data from the trial: intention to treat (ITT), per-protocol and modified intention to treat (mITT) analyses. The first two look at those participants who were enrolled (ITT) or completed the treatments (per-protocol), and so preserve the randomisation of patients. These both failed to show a statistically significant benefit from the vaccine.

The mITT process removed several people from the analysis, both breaking the randomisation and producing a statistically significant result. This might have been useful if the vaccine had a clear benefit, but the published benefit was not 31.2% exactly. Rather the confidence interval for the benefit (the range in which the researchers were confident that the true benefit figure lay) was 1%-52%, with the lower bound only staying positive because of the arbitrary choice of 95% confidence intervals chosen by statisticians over the years.

With the two tests that avoided the possibility of bias getting ignored by the press (since they didn't make the press release), and the remaining result showing that the vaccine could have had no actual benefit, the publicity seems a little unjustified.

Read more!


Has Teacher Pay Really Improved Under Labour?

With the Chancellor of the Exchequer, Alistair Darling, signalling a public sector pay squeeze, and the ever more political and activist head of the Audit Commission quango, Steve Bundred, recommending a pay freeze as “a pain-free way of cutting public spending, it is worth looking at the figures to see how much teachers have really benefited from government largess during the boom years.

The answer, in case you don't want to read to the bottom to see the graph, is not at all!

Bundred wrote in the Observer last Sunday:
“let's dismiss the notion that spending on health and education will be protected. There are good reasons why they won't and shouldn't. One is that, at a time when inflation is likely to be between 2% and 3%, a pain-free way of cutting public spending would be to freeze public sector pay, or at least impose severe pay restraint. This is especially true if real wages in the private sector are still falling.”
adding a political stance with:
“Health and education will not be immune from pay restraint, partly for reasons of fairness to others, … and also because ministers will correctly assume that as public sector workers have done well over the past decade, they will tolerate some modest real reduction in earnings.”
This is misleading in two ways.

Wages Are Not Falling

First, although pay growth has slowed, wages are not falling. As Ken Mulkearn of Incomes Data Services wrote in the Guardian, the reported negative private sector pay awards are skewed largely by the loss of bonuses in banking:
“The data for April 2009, using figures not seasonally adjusted and excluding bonuses, shows earnings growth of 2.5% in the private sector and 3.3% in the public sector, consistent with IDS research on pay settlements. In the private sector, the official figures show manufacturing (where most freezes are) at 1% and private services at 2.9%.”

Teachers Have Not Seen Pay Rise

Second, teachers have not done well out of the last decade, despite repeated claims from ministers and the uncritical acceptance of this factoid in the media. The graph shows an index of how (sixth form) teacher pay, which has been largely pegged to school teacher rates, has increased compared to the All Items Retail Prices Index. I start at 2001 as that is the earliest data I can find from the teacher union websites.Clearly, our pay moved ahead of inflation for the first couple of years, but since 2004 there has been steady slide. Not bad, but we've hardly “done well over the past decade”

What? Ministers are Being Deceitful?

Now, I don't mind joining in on a general belt-tightening, but at least I would like my pain to be recognised — I can't bear to have the millionaire ministers looking down their patrician noses at me, feeding me lies about my own pay and telling me that I should be grateful to have had it so good.

Read more!