But can the bureaucrats put the NCEA back together again?

Maclean's College head of mathematics Rory Barrett, left, and retired education professor Warwick Elley have been raising red flags about NCEA since 1991. Pictures / Martin Sykes, Richard Robinson

The NCEA argument in our house goes like this: "I don't need to pass my NCEA maths exams."

"Yes you do."

"No I don't. I've already got numeracy [14 Level One credits required for university entry]."

"Don't you want to get more? If you passed the exams it would show you have more maths skills."

"But I've already passed. I'd be better off doing study on other subjects."

"But you're not very numerate."

"Yes I am - I've got enough credits."

Despite my exasperation, I have to acknowledge the logic of my daughter's argument which, while horrifying, is consistent. There's a tradition of students doing the minimum to get a pass. But I fear the National Certificate of Educational Achievement introduced in 2002 to replace School Certificate and other secondary school qualifications, has unintentionally institutionalised doing just enough.

The bigger concern is that learning (in this case maths) is taking a back seat to assessment - the result being an NCEA market where students are trained to get attainment credits and teachers spend too much time assessing rather than teaching.

The concerned parental voice is largely absent from NCEA research but student and teacher views have been canvassed and echo such worries - about student motivation, fairness, variability of results, over-assessment and the fragmentation of subjects.

The latest member survey of the Post Primary Teachers' Association (PPTA) shows a third of teachers don't approve of NCEA - up from 10 per cent disapproving in a previous survey. Another third support NCEA, while the rest are ambivalent.

Many argue NCEA's problems are being worked on. But a growing body of research shows rips in the NCEA fabric which raise the disturbing question: is NCEA too torn to mend?

* Notwithstanding its early promise, we have to conclude that the hopes for the NCEA as an assessment system that provides better support for lifelong learning are "not yet achieved". - Shaping our futures: Meeting secondary students' learning needs in a time of evolving qualifications, New Zealand Council for Educational Research, 2005.

* There is evidence that the 80-credit requirement encourages a minimalist approach by students. Many students agreed that it was hard to be motivated to do more than the minimum 80 credits and many indicated there is little motivation to aim for Merit or Excellence when these credits carried no extra value. - The Impact of the NCEA on Student Motivation, College of Education and School of Psychology, Victoria University, June 2006.

* This [the implementation of standards-based assessment] has produced a degree of variability in the results of external assessment that is beyond public and professional tolerances. In parallel, the external moderation of internal assessment by schools was felt by many to be insufficiently robust to ensure national consistency. - Report on the Performance of the New Zealand Qualifications Authority in the Delivery of Secondary School Qualifications, State Services Commission, August 2005.

* There is urgent work to be done in regard to the moderation system. If it doesn't have credibility with teachers, it will certainly not have credibility with students or with the general public. - Teachers talk about NCEA, Post Primary Teachers Association, March 2005.

When Rory Barrett, head of maths at Macleans College in Auckland, said to the Herald on Sunday in July: "If I wanted to, I could get my cat through just about any internal unit standard," he was both amused and taken aback by the reaction.

Barrett, who is at pains to point out he's not speaking for his school but as a maths teacher of 32 years, found on his class wall the next day a picture of a cat with a list of the NCEA standards it had passed.

But there was also teacher outrage. "One emailed saying how dare I imply people are cheating the system. Another said I had set back the course of mathematics teaching by five years."

At the heart of Barrett's cat comment and the reason my daughter can get 14 Level 1 maths credits without sitting an exam is the difference between unit and achievement standards. Unit standards are all or nothing credits that students either pass or fail - achieve or not achieve in NCEA jargon - and are entirely internally assessed by schools.

Achievement standards have four possible grades - not achieved, achieved, merit and excellence. Some are assessed internally and some by external exams.

But, in one of the great mysteries of NCEA, schools can mix and match unit and achievement standards as they see fit. The result is some students get their numeracy credits entirely by internal assessment, others entirely by external exams, and many by a mix of the two. In NCEA terms the process is lauded as providing multiple pathways to learning - the idea being that if students achieve the same standard it doesn't matter how they get there.

But the problem is there is nothing standard about the process at all - internally assessed credits are easier to pass than external exams, and unit standards are easier to pass than achievement standards.

Teachers know it: "I don't like unit standards, I think they're for the cabbage classes!" (Teachers talk about NCEA.) And students know it: "Internals are way easier because you are tested after you learn the topic." (The Impact of the NCEA on Student Motivation.)

Parents know it too. My daughter gained two maths credits last week by sitting a practice test the day before the real test. Both tests had identical types of questions (but with different answers) and students who failed were given the opportunity to re-sit a third test.

That leads to an unanswerable question. Of the 37 per cent (21,887 students) who gained all of their Level 1 numeracy credits from internally assessed standards in 2005, how many are actually numerate?

Barrett sees the internal/external assessment debate as the tip of the iceberg. "When credit acquisition becomes all-important and there are many paths to these credits, many of little worth, then the integrity of subjects is lost. The possession of credits is meaningless."

He's critical also of how too many standards has fragmented external exams. "There is so little in NCEA of any mathematical worth and it's so disjointed that provided you know how to teach the right tricks, you can get students to pass and get merits and know virtually nothing."

Concerns like this have led to about 40 schools adopting the Cambridge exams. At Macleans College students have the option of both, but Barrett says, "There is far more in the Cambridge syllabus than there is the NCEA syllabus. It is far more advanced and far more demanding. You've actually got to be able to do some mathematics, not just learn a few tricks."

The criticisms don't stop here. Marking processes also mystify and de-motivate students: "What they saw was an illogical and unfair system where it is possible to fail certain achievement standards despite passing merit or excellence questions." (The Impact of the NCEA on Student Motivation.)

The bizarre anomaly arises out of the nature of standards-based assessment. Students either achieve or they don't - there is no in-between and no allowance for making careless errors, even if everything else is right.

Barrett is scathing about the purist ideology and its effect on scholarship students. He's seen papers where students received an achieved grade, but under a traditional marking system would have got 80 per cent. "They were perfectly good at maths but they made small mistakes and were given no credit for what they had done."

There are signs of a softening of the purist stance. Barrett predicts this year in scholarship exams part marks will be allowed, saying if that happens many are going to ask why the same can't happen in NCEA Levels 1, 2, 3.

For standards-based purists that creates a problem akin to sacrilege. The moment the marking regime is relaxed, the system can no longer, strictly speaking, be called standards-based. With part marks students aren't actually attaining the standard; it's like making several valiant attempts at 2m in the high jump, never quite clearing the bar, but nonetheless being given good marks for effort.

One who understands the machinations of NCEA better than most is retired education professor Warwick Elley. He's been raising red flags about NCEA since 1991 when it was announced the new regime was coming. Despite his research being quoted in questions to parliament, for 15 years he's been largely ignored.

Lately he's been driven to writing poetry: "I've a story that's brutal and candid/Of the rise of the god, Unit Standard/It was dreamed up one day/By our NZQA/With a promise I'd call underhanded."

Why does he persist? "I was frequently frustrated and I frequently throw up my hands and say it's not my problem. But I'm interested in quality education. I still work in developing countries trying to improve educational quality as much as I can and I don't like to see backwards steps."

Elley's lifelong career in education, including the development of PAT tests in the late 60s, has spanned both standards-based and norm-based assessment. The former, which is what NCEA uses, is where students are judged to have passed or failed the required standard regardless of how others did. Norm-based assessment, which we had before NCEA, is where what grade a student gets depends on how the student compares with the rest of their age group.

Elley's argument is straightforward. Standards-based assessment works fine when you have a clearly defined domain to test. Examples include the Arvidson Spelling Lists, which identified a set of commonly used 3000 words and arranged them in seven levels of difficulty. But for subjects like English, science, maths, economics and history there are no clear, one-dimensional ladders for students to climb, making it impossible to define clear standards.

What you get is vague, open-ended descriptions open to a wide range of interpretations. To achieve in history, for example, students are expected to "accurately identify historical facts and ideas". To get merit they accurately identify "a range" of historical facts and ideas. To get excellence they identify "a wide range".

It's also vague in English - "Show [convincing/perceptive] understanding of text". In subjects where there are networks of knowledge, few black and whites and lots of shades of grey, standards-based assessment simply doesn't work.

"In order to describe what a student knows or can do in history for example, you have to be able to describe so many aspects of their knowledge that it is just not worth doing," says Elley.

His criticism has focused on NCEA results - the discrepancies between internal and external assessments and variations from year to year in subjects. Elley presents research which demonstrates that year-to-year variations greater than 5 per cent show something is amiss. Successive years of students don't vary that much; one group is only ever brighter or dumber than the previous by about 3 per cent.

What causes the discrepancies is the difficulty of the exam. In the past, such variations were smoothed out by scaling, but with NCEA, where everyone is supposedly working to attain the same standard, the smoothing-out doesn't happen. Variations are exacerbated because of the vague standards which examiners interpret in wildly different ways.

That's resulted in some yo-yo results - the most spectacular being in 2004 when students attained scholarship in a stinker of a calculus exam with just 18 per cent.

It was after that debacle that the State Services Commission made the recommendation "to define and bring forward normative boundaries to function as a safety net for the four grades of Non-Achieved, Achieved, Achieved with Merit and Achieved with Excellence for the 2005 examination cycle." It also recommended the pass rate for Level 1 should be set at 70 per cent and that pass rates for Level 2 and 3 should be settled urgently.

Elley says normative boundaries have helped a little with extreme variations - 5 out of 140 mainstream standards varied by 20 per cent or more compared with 11 the previous year. But 87 standards showed 5 per cent or more variation and 40 were inconsistent by 10 per cent or more. "This is indefensible," he says.

The State Services recommendations represent an ideological crisis for NCEA. Suddenly the standards-based regime is beginning to look like it's getting the norm referenced treatment. With an expected 70 per cent pass rate, it looks a bit like School C, which allowed a 60 per cent pass rate.

New NZQA man is sure problems can be sorted

Bali Haque is an unashamed NCEA fan. "There are wonderful things going on in schools which are using unit standards, achievement standards and the pathways to give students real, genuine success," says the former principal of Pakuranga College and now deputy chief executive of the New Zealand Qualifications Authority.

"If people are going to knock NCEA, then it's really dangerous for our schools and our country, because we can't actually afford to simply be saying to students we're not going to recognise your learning."

Yes, the system is going through some changes, but they're not critical. "I don't think anybody should expect major shifts in policy. NCEA is bedding in. It's successful. We're getting good feedback across a whole range of stakeholders."

But students who sit their exams next month are going to see little difference from last year. Except that for the first time, "not achieved" will appear on result slips for those who don't make the grade - but not on the student's record of learning. The problem with marking - where it's possible for students to not achieve if they make errors on achieved questions, but pass the merit and excellence questions - is still being addressed. More work is needed too on the unintended consequence of NCEA de-motivating students. "It would be a mistake to jump in and make policy changes before we fully understand the details of the problems," says Haque.

He is adamant that while exam marking is now using "profiles of expected performance" relating the current year's results to previous years, it's not a form of norm-based referencing.

"It's not norm-based because where we find results are not sitting around our expectations, we will ask why. If we find the question is appropriate, the marking is appropriate and everything fits, we will leave that alone because it's really really important that we get an idea about what the real level of achievement is."

Haque acknowledges NCEA is at a crossroads.

"The sense I'm getting is that we have reached a bit of a tipping point. Very few would would want to go back.

"I think there is a consensus building that last year we got it right ... Well, we got it better."

But it doesn't take a maths genius to work out that with a third of teachers against NCEA, a third for it and a third ambivalent, the future is in teachers' hands. And with the burdensome assessment workload being teachers' number one concern, reducing the number of standards to assess is logical.

He says one of the problems and strengths of NCEA is that the process is so transparent. Everybody can see the results and the research.

"It's all out there, it's all open."

But isn't it like The Emperor's New Clothes - NZQA isn't seeing the obvious? That no one wants to say standards-based assessment for academic subjects doesn't work?

"I would say standards-based assessment does work, can work and we are working towards making sure some of the issues thrown up in its implementation are addressed."