Karen Dobric: Top-level exam just right for talented

A large shift has taken place in the standards set for the secondary school Scholarship award. The more demanding system now in place reflects the large monetary awards given.

The pass rates seem to have changed for Scholarship, but that is because the standards for 2003 and last year were different. The standard was at level 3 (on the National Qualifications framework) in 2003 and at level 4 last year.

From 1990 to 2003 Scholarships were given to the top 3 to 4 per cent of students who sat the Bursary examination. But New Zealand Scholarship, the correct title for the new qualification, is designed for only the most talented students in year 13.

There seems to be a real issue about the different pass rates between 2003 and NZ Scholarship last year. Yet that difference does not mean there is a problem. Each award had different standards because the levels are different.

What is problematic is the expectation of the public and politicians that the pass rates would be the same or similar. This is an unfortunate situation and comes from a lack of public knowledge about assessment standards.

An important point to understand is that what is called standards-based assessment uses standards or criteria to describe what students should be able to do for each grade. This is also called criterion-referencing.

Grades are given by comparing each student's work with the criteria for that grade. If the main reason for assessment is to find out what students have learned, and are able to do, criterion-referenced assessment is effective.

The old system of norm-referenced assessment was based on percentage marks fitting the normal curve of distribution (the so-called bell curve) that could be statistically changed by scaling, which is using mathematical equations to shift the marks.

That means that the standard can change from year to year, but the pass mark could stay the same or similar. This has been a popular tool because it looks like the pass mark is consistent, and because students can be easily listed from first to last (and some people think that is important).

Unfortunately, all it really can do is compare students with other students by putting them in first-to-last order.

So when Bursary was being used it was possible to set the pass mark more easily through scaling, and make it the same from year to year. It was easy to choose the top 3 to 4 per cent of students who got the scholarship award, and those marks could be scaled as well.

If the top students in one subject got 78 per cent in the exam and in another subject the top students got 88 per cent, for example, they could be scaled so that top students in both subjects got 97 per cent or any other desirable mark. Just because the pass mark is the same from year to year does not prove the standards are the same.

Because of scaling the standards may be very different from year to year even though the pass mark or top mark is the same.

But when standards-based assessment is used, students are being assessed against real criteria. Also, it means that the marking can be justified and explained and not just changed by an equation to make it appear consistent (the same from year to year).

The new Scholarship standard focuses on a student's high-level understanding in each of three subjects. It asks for a superior quality of work so the top performers in each subject can be identified. The possible grades are "not achieved", "scholarship" grade, or "scholarship with outstanding performance" grade. Each grade has standards that are stated in each subject.

One issue that has been raised is the difference in achievement patterns between Scholarship subjects. Yet universities have no wish to keep exactly the same pass rates between subjects from year to year. They try to keep the same academic standards in each subject from year to year.

In university subjects as different as French, physics, music and philosophy, pass rates might not be the same. And if no student reaches the standard set for a grade in a particular subject, the grade is not given. But the standard stays stable.

NCEA levels 1, 2 and 3 have been designed to maintain the same academic standards from year to year; so has Scholarship.

There might be many different reasons for the issues that have come up in Scholarship last year.

One is that most of the people setting them would not have the experience of setting and marking an exam like this at level 4 because the last separate exam for scholarship was in 1989 (since then only the level 3 Bursary exam has been used).

Another reason is that new subjects have been added to levels 3 and 4 since the last separate exams were set - for example media studies, computing and Korean.

Also, schools are not used to getting students to sit separate exams at level 4, and many teachers today were not teaching back in 1989, so preparation of students might have been a challenge.

So the whole situation is new for the exam writers and markers, for schools, teachers and students. A combination of these reasons could easily help to make the pattern of results for level 4 last year substantially different from the pattern for level 3 in 2003.

However, that does not make the results "wrong". Even apart from these possible reasons, both the level 3 and the level 4 results might be thoroughly appropriate for each level. It is difficult to argue that they can - or should - be compared at all because they are, after all, different levels.

This is the only time that this particular situation will occur, and it seems certain the reasons that could have led to some of the issues will be carefully looked at by the Qualifications Authority to ensure the patterns of results this year will be justifiable.

I hope our talented students continue to aim for and achieve high standards with this new level 4 Scholarship qualification.

* Karen Dobric, a senior lecturer in the Manukau Institute of Technology's centre for educational development, has written a doctoral thesis on the NCEA's adoption. The article represents her own view.