Coley left but was followed by the females and confronted outside a convenience store - he tried to calm the situation and leave but was surrounded by the group, including Palmer.
One of the women punched Coley then Palmer punched him in the head "with considerable force", today's Court of Appeal decision said.
"Mr Coley did not see it coming. He had his hands by his sides at the time. He fell back, striking his head. Mr Palmer fled the scene, and when apprehended he claimed to have acted in self-defence."
Several days later, Coley died from an acute subdural haemorrhage.
In August Palmer was sentenced to 22 months in prison at the Invercargill High Court but appealed the decision, stating he ought to have been sentenced to home detention.
Today, Justices Wild, Miller and Winelmann released their decision to uphold the sentence and to correct Palmer's release conditions to include any treatment or counselling programmes ordered by his probation officer and to not associate with any people named by that officer.
Palmer's lawyer Fiona Guy-Kidd submitted that during sentencing, Justice Nicholas Davidson overlooked the fact that home detention served the purposes of denunciation and deterrence, failed to mention the need for rehabilitation and emphasised her client was "extremely young" at the time of the offending.
Guy-Kidd said imprisonment would be disproportionately severe for Palmer because the facility was far from his home and so would get few family visits and that the judge placed too much emphasis on deterrence.
"Finally, she submitted that the sentence was plainly wrong," the Court of Appeal decision said.
However, Justices Wild, Miller and Winelmann did not accept sentencing judge "made the mistake" of assuming home detention didn't denunciate and deter and "nor did the judge overlook rehabilitation".
They also did not accept that the sentence was plainly excessive because the offending was "highly culpable".
"Mr Coley was not aggressive; on the contrary, he sought to escape and to avoid confrontation. He was pursued, surrounded and attacked. Mr Palmer's punch took him by surprise, and death ensued."
They also said offending of this kind was "far too common" and so denunciation and general deterrence were very important considerations and while custodial sentences were not the "best rehabilitative option for a young person" it was short.
Palmer's release is likely to be June next year.
The judges ruled Palmer's drug use on the night of the attack, his three bail breaches and assault three days before the manslaughter also counted against a sentence of home detention.
