2008-08-13

Sats Results - Not a Good Test

The Press Association manages to make much of a one percent change in Key Stage 3 figures:
TESTS ATTACKED AS STANDARDS PLUMMET
Teachers' leaders have warned that too much importance is placed on national school tests as figures revealed a drop in reading standards.

Almost one in three 14-year-olds are failing to reach the reading level expected of their age group, according to Government figures.
...
Dr Mary Bousted, general secretary of the Association of Teachers and Lecturers (ATL), said: "The Key Stage 3 tests are an irrelevance. No-one will be interested in the results when young people apply for a job."
Plummet? Physics teachers train their students to assess the level of uncertainty in presenting data, but the government press release makes much of a 1% drop in one measure. The sample size is huge, for sure, but the tests are different and there is an element of judgment in deciding the grade boundaries. Is Ed Balls really saying that these tests are so precise, year on year? Is he aware of the reliability of the tests? Can he tell us what they actually measure for that matter?

At least the ATL has called it right. An irrelevence, except that their side effects are not. This, from the Guardian puts the issue well:
For all age cohorts, the system of assessment is now the major inhibitor of much-needed curriculum innovation.
But Sats are also suffering from a more general public policy problem. Even if they were ever a reliable indicator of performance, over time they've tended to become merely a guide to schools' willingness and ability to teach to the test.

This is a classic example of Goodhart's law - that a measure of performance is no longer a reliable indicator once it becomes a target.
Why does everyone but ministers and managers know this?

No comments: