Martin Thrupp: National Standards comparisons impossible

Across the six Rains schools being studied, judgments against the national standards were being affected by many sources of variation at national, regional, school and classroom level. Photo / Thinkstock

The Government is again releasing National Standards data later this month in a new standardised format that will seem to make it easier to compare schools.

In this respect the Government is playing catch-up to the newspapers which published the data in standardised formats last year, although all have so far resisted ranking the schools from best to worst in National Standards league tables.

The published data appears straight-forward enough - percentages of children "above", "at" "below" or "well below" the standard for their year group in each school. But there is nothing standard about what underlies the tidy rows of figures. Schools' approaches to making judgments against the National Standards are so idiosyncratic and wide-ranging that it is impossible to accurately compare achievement between any two schools, let alone "apples with apples" comparisons across more than 2000 New Zealand primary and intermediate schools.

The extreme variability in processes underlying National Standards judgments is well illustrated by the latest report of the Research, Analysis and Insight into National Standards (Rains) project. Across the six Rains schools being studied, judgments against the national standards were being affected by many sources of variation at national, regional, school and classroom level.

For instance, schools are on different trajectories around the National Standards related to their diverse contexts and past practices. There are differences in approaches to National Standards categories such as "well below", differences in matching the categories to curriculum levels and differences in the rigour of the data sent to the Ministry. Other school-level variations in approaches to the National Standards include how much schools rely on formal assessment tools compared to other evidence in making their judgments, their choices of tests or other assessment tools, and the very specific details of the procedures used by schools for assessment and moderation.

It is quite wrong to expect to compare any school's pattern of achievement against National Standards with that of any other school, even for schools with relatively similar intake characteristics, such as in the same decile. There are simply too many sources of variation, leaving each school grappling with the National Standards in ways that preclude fair comparison.

The numerous sources of variation that underlie schools' judgments also mean that any claim of overall improvement or decline in the achievement of New Zealand children against the National Standards will be quite spurious.

The Ministry is pinning its future hopes of making judgments against the National Standards more comparable on an online Progress and Consistency Tool that every school will be required to use. But if the PaCT is intended mainly as a form of national moderation (ie informing other assessment processes rather than itself becoming the assessment tool for making judgments), then it can be expected to be an expensive failure. This is because while it might address some of the sources of variation, particularly the subjectivity of individual teacher perspectives, there are many others it would not be able to get any purchase on.

Another problem with the PaCT is that it will overlook the causes of variability in judgments and deal only with (some of) the symptoms. By failing to recognise the underlying causes of variation, it is likely to allow the Government to ignore the impact of contextual inequalities between schools, for instance, the effects of diverse and unequal intakes and communities, school locations, staffing and other resources.

National Standards may be a government aspiration but they are not national and they never will be while there is so much potential for local variation. It is almost comical - if it weren't so serious - that data representing such variation is being put into the public domain for comparative purposes when there are so many differences between schools in what it actually represents.

Martin Thrupp is Professor of Education at the University of Waikato and leads the Rains project on National Standards.