Martin Thrupp: Global rankings best viewed at arm's length

ECD's chief education spokesman, Andreas Schleicher

Opinion

The Programme for International Student Assessment (Pisa) is a large-scale survey by the OECD every three years.

Professor Martin Thrupp is head of Te Whiringa, school of educational leadership and policy at the University of Waikato.

The Programme for International Student Assessment (Pisa) is a large-scale survey by the OECD every three years. As a way of comparing and ranking the educational performance of secondary school students in more than 70 countries and regions, it gets a great deal of international attention and speculation as to what the results might mean.

In New Zealand the rankings are certain to be seized on as evidence from different political perspectives.

While they might reflect some shift in our education system since the results released in 2013, here are 10 reasons to take any improvement or decline in the rankings with a grain of salt:

1. Pisa provides only a narrow base on which to judge education systems.

2. There have been serious concerns raised about the statistical approaches underlying the rankings.

3. The technical papers that allow researchers to unpick the rankings are only released the year after the main results, by which time the media storm over the rankings has long passed.

4. The Pisa questions are intended to work for students in all participating countries but this is likely easier said than done. India is one country that refuses to take part because of the "socio-cultural disconnect" between its students and the Pisa questions.

5. The results are based on a sample of students and there is evidence that only a select group is involved in some settings. In the case of Shanghai, which topped the league tables last time, more than a quarter of students were not represented.

6. The OECD's attempts to use Pisa data for subsequent analyses (for instance its "Pisa in focus" monthly series) reveal simplistic assumptions.

7. Successes of countries that rank highly are hard to replicate elsewhere. Before Finland topped the 2003 Pisa rankings, the Finns believed their education system was only average. But Finland's advantages were based on decades of social history and politics, not something that can be easily copied.

8. Another example of a country ranking that is hard to replicate: Shanghai educators point to the coherence of their education system. But students in Shanghai also typically do many hours of homework each night in a way that would not be welcomed by most New Zealand families.

9. Governments that worry too much about falling in the rankings often go into "Pisa shock". This typically leads to knee-jerk bad policy decisions such as increasing testing where there is a lot of it already.10. Governments often cherry-pick Pisa findings to justify policy.

More generally, the OECD seems to be keen on opening up public education to the for-profit global education industry.

In the case of New Zealand, the OECD's chief education spokesman, Andreas Schleicher, allowed a video clip supporting the "Investing in Educational Success" policy to appear on Hekia Parata's National Party website in 2014.

Far from being non-partisan, this was endorsing the Government's announcement of new education spending in an election year.

The present Government and future ones would do better to focus on accounts that get closer to the day-to-day work of New Zealand schools.

At the same time New Zealand educational research needs to be allowed to ask hard questions rather than being framed up by policymakers, as has often become the case in New Zealand in recent years.

There are numerous problems caused by education policies of the present Government but we don't need Pisa rankings to tell us that.