While the Vodafone support person was able to remotely access and change settings in an effort to fix the problem, the capacity for the back door to be exploited by a rogue company employee was concerning.
He said the access was managed through sending a line of code to the modem which then reset it and allowed access by entering a generic password.
But the IT expert, who has experience working on IT security with intelligence agencies, said he was concerned to find Vodafone had its own access to his modem.
When he contacted the company, he said he was told: "We just made this hole so we can get in."
The expert said it could potentially allow a rogue company employee to create their own wifi point of access or a virtual private network - either of which could allow remote access to the computers connected to the modem.
"It looks like you're protected but they have remote access and it's very hidden."
The security expert spent $300 on new equipment to block Vodafone's access, which he says the company reimbursed him for, and raised his concerns with its security team.
He was told the company was looking at installing an "opt-out" setting which would allow customers to block Vodafone's remote access.
But he said he was concerned such a setting could allow access as a default, meaning unwitting customers were still exposed.
Vodafone and Spark did not answer questions about whether customers were told that the back door access existed.
A spokeswoman said the "service" was seen as "a really positive part of our service offering".
"This is a service to take pain points out of technical conversation with our customers to optimise their experience."
She said any employee who manipulated settings to gain access beyond the modem was hacking - not only banned in employment agreements but a crime.
A Spark spokesman said customers were not told of the company's ability to access their modems remotely until they asked for help.
"We only access the modem remotely when we are asked to by the customer."
The Spark employee was able to check to see if the modem worked, change settings, reboot the modem or set up Wifi.
"We will only do this once the agent has got verbal approval from the customer."
She said the company believed it was standard practice across the internet industry in New Zealand.
"The agent has very limited access and is not able to view any sensitive or personal information."
A Slingshot spokesman said the company did not have direct remote access in terms of changing settings, but was able to push updates of software or preconfigured settings to a customer's modem.
The company alerted customers with a line in its terms and conditions saying it would "reserve the right to occasionally manage your modem".
Voyager owner Seeby Woodhouse said his company had remote access to the modems it sold to help customers.
"There is a potential security risk but there is a security risk in having people configure their own modems."
He said the increased threat from ransomware - which hijacked computers - and other online threats meant ISPs would likely seek greater access over time to improve security.
And while he said customers rang regularly to ask that techs remotely access their modems, the ability to do so wasn't flagged at the point of sale. He said there was likely value in doing so.
Waikato University associate professor Ryan Ko - director of the New Zealand Institute for Security and Crime Science - said internet providers were already able to view anything sent to and from people's computers across the internet.
He said the danger around remote access would come from a disgruntled worker at an internet provider and it was "high value targets" rather than the average user who would be at risk.
"The whole thing exists on the fact you trust your ISP to keep their security up to speed. It all depends on trust."