University of Auckland epidemiologist professor Rod Jackson said the worst may be yet to come.
It is "too soon to relax", although the country is nearing its peak, Jackson said.
He said the push for change is "politicking" and not many businesses want to remove vaccine passes at present.
He told Morning Report that looking around the world other countries did not go straight up and down their peaks and New Zealand would at risk of "yo-yoing around" if vaccine passes and other public health interventions were removed too soon.
Vaccine passes should be retained until it was clear that the Omicron outbreak was just about over.
"We're at the top at the moment. It makes absolutely no sense to remove any effective public health measures when we're still at the top.
"It's crazy. I think it's political nonsense to be pushing to take them away now."
He said more than one million New Zealanders still need to get their booster. As well, the unvaccinated are twice as likely to catch Covid-19, three times as likely to transmit it as fully boosted people and five times more likely to be in hospital.
"We're not over it yet ... those relatively small numbers of people, when you do all of those multiplications, they are sufficient to overwhelm our health system."
He referred to what was happening in the UK and parts of Australia where there are rising case numbers.
"I know there's huge pressure to take away the vaccine passes but I think it's a mistake."
He said it was business who forced the government to introduce vaccine mandates and he did not believe they were hugely in favour of taking them away now.
"I think this is politicking."
It did not make sense to change the traffic light setting in the next few days either.
"We've got more people in hospital today than we've ever had. We've got more deaths than we've ever had.
"It just doesn't make any sense to be relaxing public health measures that have proven to be incredibly effective at the peak of an outbreak."
PM wants to get setting 'right for future'
Prime Minister Jacinda Ardern told Morning Report that the traffic light system must be "no more restrictive" than needed and mandates won't be as necessary after the first Omicron wave.
Cabinet is meeting today to review vaccine mandates, vaccine passports and the traffic light system, though any decisions will not be announced until Wednesday.
The changes will mark the biggest domestic shake up to Covid-19 restrictions since Omicron arrived on Aotearoa's shores.
"We know that in the future we're likely to have additional waves of Omicron... We're already seeing that in other countries," Ardern said.
"So let's make sure we get the Covid protection framework, that traffic light system, right for the future.
"We want it to be no more restrictive than it needs to be, so if there are areas we can pare it back, we will."
She said with a highly vaccinated population the government believed mandates and vaccine passes would no longer be as necessary once the Omicron outbreak had peaked.