Under the three strikes law, brought in under the previous National government, he was sentenced to seven years.
Previous judges have called the sentence "grossly disproportionate" to the gravity of the offence but they were reluctantly bound by the law.
The Human Rights Commission was at the Supreme Court as an intervener, there to offer greater perspective on the issues.
Its lawyer, Andrew Butler, told the court the commission supported Fitzgerald's appeal and cited the Bill of Rights Act.
"Avoiding the imposition of seven years' imprisonment on Mr Fitzgerald will do no damage, no harm to the underlying purposes of the 2010 amendment act.
"Indeed, it will likely reduce the public contempt it might otherwise be held in. In this country, we like people to get a fair crack at the whip, not a whipping."
Fitzgerald first became known to mental health services aged 15.
He has remained on the radar ever since, and has a history of offending.
Most of these offences sit at the lighter end of indecent assault, including his third conviction.
Psychologists say while he has some ability to determine right from wrong, his ability to fully comprehend the three strikes rules would have been limited.
Butler urged the court to immediately release Fitzgerald.
"Mr Fitzgerald should not stay a minute more in prison, he needs help, that he is not getting it would appear, in his current environment."
Butler said while the offence was awful for the victims, it didn't qualify as serious violent offending.
Outside the court, Fitzgerald's lawyer Kevin Preston said he should never have been subject to the three strikes rule because the courts could have chosen not to convict.
"Taking into account the gross disproportionality, the breach of his human rights, and the section 9 of the New Zealand Bill of Rights, the courts should interpret the legislation to give affect to that."
He said the Supreme Court should discharge him without conviction, release him from prison, and get him the help he needed via mental health services.
Preston said the lower courts were wrong to interpret the law the way they did.
Fitzgerald has already served four years of his sentence, and Preston said that had taken a toll.
"It's very, very sad," Preston said.
Madeleine Laracy, for the Crown, argued that the law was clear.
"The maximum sentence must be imposed, there is no exception there and nor is there any qualifying proviso to it in the nature of the manifest injustice safety valve," she said.
In 2018, Labour's plan to repeal the law was derailed by its coalition partner New Zealand First, who weren't prepared to offer their support.
Now, Labour has doubled down on its party's commitment to getting rid of the law, which it says is leading to "absurd results".
