Joubert looked at me, I looked at him, and we both laughed.
That conversation took place three days before Joubert made his return to the international scene in the England v Wales match on March 12.
His previous Test match, also at Twickenham, had been the Scotland v Australia World Cup quarter-final, with its dramatic finale and subsequent furore.
On Monday of this week, Joubert was named on the panel of officials for rugby's return to the Olympics in Rio.
The following day he was on a flight to New Zealand for Super Rugby duties, beginning in Christchurch on Friday.
He had been open and accessible when in London for the Six Nations Championship but did not want his comments to distract from the match itself. Finally, this week, with the Olympic roster confirmed, an appointment about which he admits to "being thrilled", he felt able to go on the record about events last October.
As many will tell you, from fellow refs, to players, even to the Scots, Joubert is affable, amenable, level-headed, highly regarded and accomplished at what he does, a rugby man to his very core, without ego and without edge, generous and popular, too, no matter that it was his decision to award a contentious penalty in the last minute of that bristling, edge-of-seat World Cup quarter-final that denied the Scots a chance of a famous victory.
The goal was kicked and a nation sank to its knees in despair. Scotland captain Greig Laidlaw looked a broken man. There was more. At the final whistle only seconds later, Joubert dispensed with the customary handshakes and headed straight down the tunnel, disappearing from sight. That act in itself angered many.
Gavin Hastings hurled a mouthful of abuse in Joubert's direction. So, too, Matt Dawson. They were not alone. The following day, World Rugby issued a statement stating that the wrong decision had been made.
It should have been a scrum, not a penalty. That is by-the-by. It was the abrupt exit from the pitch that drew so much criticism. There is a simple question to be asked. Why? "In my head was a desire to avoid any possible unseemly confrontation that would mar what had been a wonderful occasion," Joubert said.
"I had it in my mind somewhere that there had been an incident between the official and the England coaches [Andy Farrell and Graham Rowntree, for which they were later fined] in their match against Australia and I just didn't want any of that to happen, not because I don't understand the emotions of the moment for players and coaches, their desire for answers to questions, but just because I did not want that to become another possible incident. That was my thinking, not for myself but for the situation."
Reasonable, plausible, if not entirely exonerating Joubert. There is another question to ask. Before I can, Joubert poses it himself. "In hindsight, would I have reconsidered that decision?" he says. "Absolutely."
But he had no chance to reconsider that decision. Nor to explain it. For referees are not allowed to speak on any matter without approval. If anyone had reason to be enraged by the outcome of that match, it was Vern Cotter, the Scotland head coach.
Instead, it was Cotter who took it upon himself to ring Joubert a couple of days later, with the critical storm raging all around, and tell him he understood the complexities of the decision in real time.
It was sport and everyone just has to accept what happens and get on with it. There was no anger, no spite, no desire for an inquest, just one bloke to another putting it in perspective.
They all met again in Dublin on the final weekend of the Six Nations, Joubert running the line for the Ireland v Scotland game. "I had a coffee with Vern before the game and a beer with Greig [Laidlaw] afterwards," said Joubert this week.
"They were very natural and typical interactions which I enjoyed." Joubert is like that himself, a man reared at the side of a rugby field, watching his dad, Des, referee matches, absorbing the atmosphere, revelling in the nature of the sport.
The 38-year-old from Natal is still like that. He loves the tradition of the Six Nations, loves the fact that there is still a Friday-night dinner for referees and officials, that there are caps and presentations after matches, that there is fierce tribal rivalry, inflamed emotions, but still, he insists, respect for the referee.
Joubert is a companionable and rounded figure. When officials arrive in Durban to referee, he insists on hosting them, taking them on trips up-country to the historical battlefields of Rorke's Drift and Blood River, explaining the history of his country to them.
This is not a man to be reviled, as he was in some quarters. Joubert has no objection to people questioning decisions. Mistakes will happen. More is the pity that he was not allowed to articulate his views at the time and then this would not have festered.
Joubert is a decent bloke. That is the nub of all of this.
The return to Twickenham was not without its stresses. "I was definitely aware of the increased scrutiny," said Joubert. "But I cherish these big occasions and once the game kicked off I felt calm and assured back in the arena." Back where he most belongs, no matter the noise and clamour.
