Telefoni said Lee flashed a gang signal, and jurors were shown video footage of Lee gesturing with his pinky and thumb out.
The two groups were walking towards one another slowly and as they approached, Telefoni punched Lee in the head before the group onslaught started.
Riki Wiremu Ngamoki and Paul Simon Tuliloa have also pleaded not guilty to murdering Lee, who was killed on March 5 last year.
A fourth man, who has name suppression, stomped on Lee's face at least six times and allegedly removed a shank hidden in his beanie to stab Lee.
Telefoni, 20, said Lee and an inmate assisting Lee were Mongrel Mob members.
Under cross-examination today, he said he had no knowledge of one-punch manslaughter cases.
He was asked if he knew that when somebody falls over on concrete and hits their head, they might die.
He was asked if he saw Lee hit his head on the concrete floor when he punched him in the prison yard.
And he was asked if the idea was to ensure Lee was unconscious by the time he hit the floor.
Each time, the answer was "no".
"Well, how hard did you want to hit him?" prosecutor David Johnstone asked.
"Just so he couldn't get up."
"You punched him hard because you were hoping that might minimise the risk to yourself?"
Telefoni said this was correct.
Telefoni, Ngamoki and Tuliloa have admitted injuring Lee with intent to injure.
Footage from Paremoremo played to jurors showed one attacker leaping into the air multiple times before landing feet first on Lee.
At one point a vivid red patch appeared on Lee's torso but after this, Lee was stabbed at least three more times.
The trial before Justice Simon Moore and the jury continues.