He said the Government was not dealing with the "fundamental issue", which was that the increasing cost of getting a tertiary education was driving the rise in student borrowing.
"We have to look at the fact that course fees have gone up significantly, and that has been one of the big drivers in the increased student loan borrowing. I'm not going to make up policy on the spot, but, clearly, reducing fees is a priority for us."
Tertiary Education, Skills and Employment Minister Steven Joyce said the savings Mr Hipkins referred to as "tinkering" were much more significant than that.
"The net cash cost of the scheme in the last year was down to $400 million -- that is cash out, less repayments. In 2009/10, it was $771 million.
"The net cost is dropping substantially... If we can get this overseas-based borrower stuff going, I can see us getting to a point where there is very little more going out than what is coming back in."
Mr Joyce said officials had not yet estimated when that point might be.
New Zealand universities have in recent years increased domestic student fees by the maximum allowable 4 per cent.
This has caused significant anger among some students, with the University of Auckland forced to lock down its annual fee-setting meeting after violent protests.
In last year's Budget the Government announced the annual maximum fee movement would be 3 per cent.
In 2014, the Government's share of the full cost of each student's tertiary education was 71 per cent, rising to about 82 per cent after the subsidy provided by interest-free borrowing is accounted for.
Mr Joyce said that attempting to bring down fees through further Government subsidisation would be unfair on those who chose to enter the workforce rather than attend university.
"I think that would increase inequity and move money away from lower-income people to higher-income people. Making some sort of contribution -- and it's only a 30 per cent contribution -- to university study, I think most people would see as reasonable, provided it's not too onerous in terms of the time taken to pay it off."
However, Mr Hipkins said bringing down tuition fees would reduce student debt and give more people access to tertiary education, particularly those from poorer backgrounds.
"The Government takes more tax if the workforce is more highly educated. We have just got to be willing to make the investment upfront, in order to reap the dividends further downstream."
Germany has recently abolished tuition fees. Asked if Labour would consider that, Mr Hipkins said "a range of options" were being considered. "We will certainly be looking at ways to bring down the cost of tertiary education."