"I remember the first time we won, we were walking through Wellington Airport Monday morning after the election, people just stood and clapped," Key told the Weekend Herald. "In the second term it was the odd wave, in the car and back to the office.
"The shininess and the newness that you have when you come in your first term, that's going to come off any Government and any leader so it's much more immediate in terms of being back to business."
He is a little regretful at the latest couple of incidents over the shirt and the Beckham conversation.
"From time to time I might push a little bit too hard and I have got to be a bit more careful."
But essentially he sees it as the media's problem, not one that comes between him and the public. He hasn't changed the way he behaves.
"These stories have always been there from time to time. Actually they are an example of where the media is generally out of sync with the public.
"The public talk colloquially, the public's grammar's not perfect. They kid around and I don't think they overly mark me down for that. They just see me as a normal guy.
"I came in as John Key and I'm going out as John Key. The media or our opponents will try and portray that as being too casual. I don't agree with that.
"You are not going to change me and if you do, it will look like a fraud, it will be a fraud."
The defensiveness continues with his challenge to show him an example of where he had been required to be incredibly serious and wasn't.
"I always am. Frankly, I work 19 hours a day pretty much and six-and-a-half days a week. Within those days is a huge range of things I'm doing, a massive range."
With 30-odd speeches a week and countless briefings on a huge range of subjects, it was little wonder he did not recall everything that was said.
He is referring to the fiasco over the spy agency GCSB which told him in September its surveillance of internet mogul Kim Dotcom in January had been unlawful and how it was unable to give him quick answers in preparation for Question Time about the number of briefings he had had.
"I ended up having to do a bit of bloody forensic analysis myself so I called (GCSB boss Ian Fletcher) in later on and said: 'look, I just don't think you guys have served me well. I've ended having to do all this work and you guys should be able to provide me with those answers'.
"And I said: 'you've really let me down and you need to go away and think about it'."
He said that conversation was what caused the GCSB to "rip the place apart" and that is when they found a note about a briefing he had had in February.
That's when the "brain fade" he had attributed to GCSB was pinned on him and he copped opposition flak for being too relaxed in his oversight of the spy agencies.
"I know it's sport but realistically I'm pretty confident I get the vast, overwhelming bulk of things right."
He is convinced the Government is on track when it comes to the issues that matter more to voters, more elective surgery, national standards, and not withstanding the spike in unemployment this week, the economy.
"I had (Reserve Bank Governor) Graeme Wheeler in my office (on Wednesday) and I said: 'Okay, tell me what you could do if you were me'.
"In the end it is the same prescription; there is no secret, magic bullets out there. He said, 'carry on what you are doing', basically."
Key's relaxed character translates to his leadership style as Prime Minister - it is not hands-on in the way that characterised his predecessor, Helen Clark.
He is said to give his ministers a lot of freedom and is very relaxed with them, right up to the time he needs to be ruthless, as one insider put it.
Like a soft parent, he doesn't do a lot of reprimanding of ministers, so that when it does happen, it carries a lot of force.
Key is emphatic that he will fight the 2014 election, dismissing claims by commentators that he has somehow lost his mojo. But that doesn't stop him talking about legacies.
"I want to leave New Zealand in better shape than I found it. I know the job of Prime Minister is not forever and I'm going to do the best I can every day to make that difference.
"If I got hit by a bus this afternoon, my answer would be I have."
So if he got hit by a bus this afternoon, who would replace him?
"I had historically always thought it would be Simon Power, but he obviously left."
He agrees that Bill English, Steven Joyce and Judith Collins would put up their hands - "at least".
Key acknowledges that a stronger Opposition has made things tougher for the Government this term than the first term when Labour had come out of nine years in Government.
"They are more focused and so they should be. If you want to win you have got to put pressure on."
But he reserves his highest praise for Greens co-leader Russel Norman, not Labour's David Shearer.
"If you want my view, the politician of the year will be Russel Norman by quite some margin. Russel Norman is eating David Shearer's lunch."
The types of issues the Government is confronting in its second term are much more complex than its first.
For example, in its first term it promised to increase the number of frontline police officers. That's about recruitment and not hard to implement.
The second term promise is to reduce crime with specific targets for groups of offenders. That is far more challenging to the justice sector.
Key says there are three types of issues he has to deal with.
The first are those that just happen on your watch, such as the Christchurch earthquake or the application by a Chinese company to buy Crafar farms.
And for all the opposition to the approval, he is convinced Labour would have dealt with Crafar the same way if it happened under its watch.
"Shearer wouldn't have been putting up a member's bill to ban overseas sales (or farmland) or putting a flag on a bloody farm."
The second type of issues are part of the Government's agenda, such as the sale of up to 49 per cent of Mighty River Power.
Despite the opposition, National campaigned on it and Key believes National would do itself more damage if it did nothing.
"It's better to do what you think is right and hopefully (voters) like the prescription. But you can't be scared of your own shadow."
The last type of issues are "your own self-inflicted mistakes".
"Yep, we have a few of those but given the huge number of issues we deal with every day, week after week, month after month, do we get that right more often than we get it wrong?
"I reckon we get it about right."
Second-term blues
Foreign Affairs restructuring
SkyCity Convention Centre-pokies deal
John Banks' election donations
ACC/Bronwyn Pullar/Nick Smith's resignation
Retreat on class sizes budget
Crafar farms sale approved
Maori challenge to SOE sales
Dotcom raids
GCSB unlawful surveillance
MSD kiosks privacy breach