"I was just over it and started applying for jobs outside teaching, thinking, this is not me."
His first class were "interesting kids". Three are dead and one is in prison.
"Then this dynamic 4ft 10in principal came in and saw something in me that I didn't see at the time," he said.
"She made me a senior teacher, doing stuff in my professional development, and got me hooked into teaching. One of those critical moments in your life, I suppose."
His was principal at Freyberg Community School, with two campuses and a special needs unit, in West Auckland when he was invited to apply for his current job 13 years ago.
At Hereworth his role had become "more of a CEO", with the deputy principal "driving the educational aspect".
"A lot of my work now is on marketing - brand promotion and co-ordination."
One of Mr Scrymgeour's first initiatives was to lower the school's perimeter hedge by half, so it wasn't "a big barrier", and actively encourage visitors.
"The idea is to get people in the gate - just come and see what is going on. We want people to understand what is actually going on, even if they don't choose to come and join us."
From 2008 to 2010 he received media attention about "letting boys be boys".
"Not being macho, but being respectful and growing up to be a good member of society. It's not that different for a girl, there are just a few things we need to do."
He said the sit-down-shut-up-and-do-this approach to teaching seemed to suit girls' education better than boys'.
"I'm generalising, but boys generally want to make and watch things move."
So Hereworth programmes include movement.
"There will be an academic theme through the day but they will be doing some physicality and creative stuff - hands-on things - so that every boy has all needs taken care of, not just the academic ones, because otherwise they get frustrated.
"That is the sort of stuff I can do here," he said.
"I tried to do it in the state sector but couldn't get the resourcing, because schools all had to be the same."
He wanted to resign in 2011 after the end of his second five-year contract but because of the global financial crisis the school was not doing well.
The roll had dropped to less than 200 but this year it started at more than 220 "and we will grow to 230-plus, which is where we want to be".
When the economy slumped, the number of boarders almost halved "which was a huge loss for the school".
Boarding was not as popular as it once was and it was not always a matter of accommodating students from other regions, he said.
A family returned from overseas and decided to board their son even though they lived close.
"The kids could probably wave to the parents from the boarding house window. They just thought it was the best thing for them and the kid was quite happy. It's just horses for courses, I suppose."
With the school's books back in black, he leaves at the end of term one.
He is adamant the ongoing legal dispute with sacked teacher Emma Fox is unrelated to his departure.
She is seeking almost $250,000 from the school after claiming unfair dismissal in 2010.
The Employment Relations Authority ruled against her but she has appealed the decision in the Employment Court, with a decision imminent.
After a "24/7" job, Mr Scrymgeour lives on site, is keen to take a few months off and has "a couple of options" overseas, but remains committed to primary schooling.
"We don't have the social issues - we are not dealing with the girl thing or have hormones flying around, alcohol and drugs sort-of-stuff.
"I am completely supportive of my high school colleagues but I think in this age, these formative years, we can set up their education."
