The owner of the Mortgage Centre in Rotorua, Graeme Leigh, said local lenders had already been relaxing the restrictions for borrowers with less than 20 per cent equity.
But throwing the restrictions out altogether would be "good news" for Rotorua first home buyers.
"For us here locally, it's very, very good. I got a bit of flak right at the beginning ... for saying I'd seen my first home buyers disappear, [but] that actually happened."
However, as banks became more comfortable granting loans, those buyers had already begun to re-emerge, he said.
"They'd [banks] began to tell us just quietly, 'We're opening the doors a wee bit for the LVRs'.
"It's all good news for a change."
According to the latest QV figures, the average house value in Rotorua reached $274,984 last month - up 2.4 per cent from a year ago.
The Reserve Bank introduced loan-to-value restrictions in October last year, limiting banks' lending to borrowers with less than a 20 per cent deposit to 10 per cent of new lending.
Mr Spencer said without those restrictions annual house price inflation might be 2.5 per cent higher.
Labour housing spokesman Phil Twyford called on the Government to "fine tune" the restrictions by applying them only to Auckland and Canterbury.
"Why should regional New Zealand continue to be punished for the Government's failure to fix the Auckland housing crisis and get the Canterbury rebuild happening?"
ASB economist Christina Leung said the Reserve Bank would not remove the restrictions unless it was confident the housing market was slowing on a sustained basis and that interest rates would sufficiently contain house prices.
Westpac chief economist Dominick Stephens said it was possible the restrictions' timetable could change if the housing market took off again.
"But in our view that is unlikely to happen, with interest rates rising as they are."
The Rotorua Daily Post will be investigating home affordability in our region and ways for entry-level buyers to take their first steps on the property ladder. Watch out for our coverage in the weeks ahead.