He had a point. People passionate about politics regard the other side as not just wrong or misguided, but morally deficient and, quite possibly, criminally so.
That's how they explain the policies they disagree with. And the perceived moral failings of their political opponents motivates them politically.
The senior minister who accused me of being as, I recall, an "associate of drug dealers, tax cheats, and gang members" held me, I believe, in some regard.
We had always got on well. We did so right through our parliamentary careers and now at social events are positively chummy. He didn't for a minute believe that I was a crook. But the accusation at the time served its purpose. It deflected attention away from parliamentary questions that were proving politically sticky.
His accusation put me on the back foot, and allowed him a breather.
I didn't feel so bad once I realised he hadn't meant anything untoward - it was just parliamentary tactics.
And so we had Labour leader David Shearer accusing Prime Minister John Key of not telling the truth, and Key responding that Shearer should "put up, or shut up".
Both men are telling the truth. I have absolutely no doubt that Key could not remember a particular image on a particular slide in an otherwise run-of-the-mill presentation on the day-to-day work of the Government Communications Security Bureau.
I am also certain that Shearer is telling the truth. He genuinely believed Key subsequently made a quip about the slide in an off-the-cuff speech and that the quip was filmed. It turns out that there was no evidence. And Key, being careful, accepted that he may have made such a quip but couldn't remember. I believe him.
Being Prime Minister is the busiest and the most intense job in the country. Prime Ministers have to remember a lot. But it's impossible to remember the stuff that at the time carries no special significance.
I have known Shearer for 25 years and he, like Key, is a man of the highest integrity. We are lucky in this country to have men of such calibre as our leaders. Perhaps if Shearer has a fault it is of being a little too trusting. His sources, as good as they appeared to him to be, turned out to have been mistaken.
So what are we to make of it all? Well, in one regard it doesn't matter.
The public don't care for the "I said, he said" debate. But in another way it matters.
Key was thrown off course. He was having to defend himself and to defend himself as being truthful. That's the bit that lodges in people's minds. The question has been raised.
No one looks good explaining they were telling the truth. It starts to rub away at the shine of office.
It's unfair but that's how it is. Politicians throw dirt around for a simple reason: it works.
It enables one party to gain at the expense of another. But there's no doubt that it also diminishes our view of politics and high office.
In that respect, we all become the losers.