Bridges called Ardern a "part-time Prime Minister" and said she could have made the trip during Parliament's recent three-week recess.
But Robertson said Bridges' comments were "desperate and disrespectful".
"Tokelauans are New Zealanders and I notice that Simon Bridges says that the PM should be focusing on everyday New Zealanders," Robertson said.
"Is he telling the Tokelauan community that they're not everyday New Zealanders?"
Robertson took exception to Bridges' "part-time" reference.
"The Prime Minister is extraordinarily hard-working, as in fact every Prime Minister of New Zealand is. There's no such thing as a part-time Prime Minister. That is a desperate comment from someone polling 6 per cent.
"I do wonder if there's a bit of a sexist overtone in that."
Asked if Bridges' comments were about Ardern being a mother as well as Prime Minister, Robertson said: "I do think that's an inference some people would take from that."
Bridges said his comments were about holding the Government to account.
"Mr Robertson is very quick to infer sexism but it's actually about Labour failing to deliver on its promises."
Robertson would not comment about whether Ardern should have gone to Tokelau during the recess, saying he did not have those details.
"But these visits have to be organised in particular ways."
During Question Time today, Robertson distanced the Government from Deputy Prime Minister Winston Peters' comments backing Te Kawerau ā Maki in the land dispute.
This morning Peters doubled down, telling Newshub that the protesters opposing the housing development at the site were "imposters".
Robertson said that was Peters' view.
"What he was saying is that there are many people who are at Ihumātao who are not mana whenua. Equally there are people at Ihumātao who are mana whenua," Robertson told the House.
The Government initially said it would not intervene and supported Te Kawerau ā Maki, but after protests escalated, Ardern brokered a pause in the housing development while talks continued to find a solution.
Peters' stance was backed up by NZ First MP and Government Minister Shane Jones, who did not resile from comments he made to Waatea News that the protesters - led by Save Our Unique Landscape (Soul) - were not credible because they wore yoga pants.
"A lot of those people are only interested in iwi politics when it accords with their far-left views," Jones told reporters before Question Time today.
"I don't see why they should be given any legitimacy at all within the Treaty settlement process. They should all pack up their freedom camping gear and go home."
Jones suggested that the Greens, who have joined the protests, should also back Te Kawerau ā Maki.
"The Soul issue is quite different from settling the Treaty claim. This is a claim already voted on by the Green Party, They supported the Te Kawerau ā Maki claim.
"We should stand up for the people who tried to show some leadership and not surrender the interests that they had endeavoured to secure for their people to a bunch of people who just jumped on a political carousel."