Housing supply conditions had also started to improve and in Auckland progress was being made in freeing up the supply of buildable land and improving the consent process.
The government announced another round of land in Auckland to be targeted for building houses on this week.
Spencer said the Reserve Bank believed the lending restrictions were achieving their purpose.
"The financial system is less vulnerable to an adverse housing shock and banks are now less exposed to potential credit losses as the interest rate cycle turns upwards."
At this stage he said the earliest date for beginning to remove them was likely to be late in the year. But warned that before it removed the restrictions the bank wanted to be confident that the housing market was responding to interest rate increases and that immigration pressures would not cause a resurgence of house price pressures.
ASB economist Christina Leung said the Reserve Bank's timing was in line with its expectations that the restrictions would most likely be removed over a window from late 2014 through to mid-2015.
"The RBNZ will not remove the restrictions unless it is confident that the housing market is slowing on a sustained basis and that interest rates will be sufficient to continue containing house prices, as well as confirmation that net migration is slowing."
Leung said the Reserve Bank could ease its way out of the restrictions by lifting the 10 per cent "speed limit".
However there was already scope for banks to increase their restricted lending given the share of new lending came in at 5.6 per cent for the first six months from October well under the 10 per cent limit, she said.
Read the full speech here:
Dominick Stephens, chief economist for Westpac, said it had been speculating on the future of the loan restrictions and today's speech had given a very direct answer to questions about the limits.
Stephens said it was possible the timetable for restrictions could change if the housing market took off again.
"But in our view that is unlikely to happen, with interest rates rising as they are."
Stephens said Spencer's speech also indicated the Reserve Bank was looking at a less steep increase in the official cash rate than it had indicated in March due to the high exchange rate.
Spencer said floating mortgages could be 7 to 8 per cent in two years' time, closer to their average of the past 20 years.
Some economists had predicted floating rates would be that high by the end of this year.
Spencer said the extent and timing of interest rate increases would depend on a number of uncertain variables, in particular the exchange rate and housing market pressures.
