The man stood down from Parliament after Speaker Trevor Mallard's claims about rape has spoken out.
Newstalk ZB and the Herald have decided not to publish the parliamentary staff member's name to protect his family.
He was stood down by the closed shop Parliamentary Service last week, which is exempt from the Official Information Act and will not have to release documents over the alleged incident.
Referring last week to the alleged assaults, Mallard said: "We're talking about serious sexual assault. Well that, for me, that's rape."
In a two-hour sit-down discussion in his home, the devastated man said: "The accusation of rape has put me in a very dark place.
"I was driving to Parliament the day after the bullying and harassment report on the place was delivered and heard on the radio that a 'rapist' could be stalking the corridors and it disturbed me greatly," he said.
However early that afternoon he realised he was the so-called "rapist" when he was summoned into the office of the Parliamentary Service boss Rafael Gonzalez-Montero to be stood down.
A colleague at the centre of an unsubstantiated complaint against him three years earlier had come forward again after complainants were urged to do so by the Speaker.
"At no time was I spoken to by the review's head Debbie Francis which I thought I would have been considering an alleged incident had been investigated and was found to be without merit.
"It's ironic that the review was about bullying and harassment. I feel I've been bullied out of Parliament and harassed within it, particularly by the Speaker's claim," the teary-eyed man said.
He said his family was dumbfounded, they couldn't believe he could be accused of sexual misconduct.
"Arriving home after being stood down I was numb. I sat stunned thinking this can't be happening to me," he said.
The complaint was ruled to be unsubstantiated last year, laid two years after the incident happened.
The man said it resulted from working alongside a colleague at Parliament when a clipboard was lost.
"We searched for the clipboard which was important and with great relief we finally found it. She gave me a high five but being a little old-fashioned I hugged her back, that was honestly all there was to it," the man said.
Two years later he said she laid a complaint and both of them were interviewed. In a written decision after the investigation last year, her claim that he hugged her from behind, pushing his groin into her, was found to be unsubstantiated and no further action was warranted.
However after the call from Speaker Mallard last week, the woman, who the man said he'd had a few sharp exchanges with since the hug, asked for the complaint to be reconsidered.
Immediately after that he was sent packing from Parliament, with Mallard summoning the media to declare: "I don't want to cut across any employment or possible police investigation, but I am satisfied that the Parliamentary Service has removed the threat to the safety of women working in the Parliamentary complex."
The Speaker understood the same man was responsible for the two other claims of serious sexual assault. He later added one of the key dangers is no longer in the building.
The man said he's dumbfounded but the same woman was involved in one of the other complaints. He said he passed a comment about another woman's hair looking nice, with the original complainant telling her he was looking at her breasts.
The third complaint came following a platonic friendship he had with another colleague, who on one occasion came around to his house with her son for a cup of tea with his wife. He says he kissed her on the cheek once as he was farewelling her and he suspects she was put up to the complaint by someone else.
After talking to the man, NewstalkZB saw the finding of the investigation against him, a finding that would usually be kept under wraps by the unimpeachable Parliamentary Service. The finding bore out everything the man had claimed and found the claim against him was unsubstantiated.
An experienced defamation lawyer, Hugh Rennie QC, was asked whether the man had been defamed but wouldn't personalise it to the Speaker, preferring to make his comments in a general sense.
Rennie said the statement that a "threat" has been removed is an allegation of continuing risk which is unjustified on the facts stated. There is no evidence stated of a continuing threat and to suggest there is, and that the response needed is to be barred from the workplace, is defamatory.
He also said: "The association of these statements with a general allegation that rape has occurred at Parliament is defamatory of the person stood down as it implies that person is a past rapist. Even if the claims made against him were correct, none of the matters alleged was rape.
"There is a public interest in receiving correct information but none in receiving incorrect information. The serious allegations that have been made against this man - not named but readily identifiable in his workplace - are defamatory. On the facts stated it is very unlikely that a defence that this was done in the public interest would succeed," Rennie said.
The distraught man said: "I never thought I would ever find myself in this situation, it's not who I am, I'm thoroughly devastated. I would like to be able to return to work to clear my name and I expect, at the very least an apology from the Speaker for labelling me as a rapist which I most certainly am not.
"Surely he must have known the background to the complaints and if he did, his comment is slanderous as I'm sure many in Parliament now know I'm the one who has been stood down.
"I have been married for many years and have throughout been monogamous."
The head of the Parliamentary Service, Gonzalez-Montero, is in meetings until later in the day and was unavailable for comment.