"We are talking about people that have been contacted by IRD and are refusing to act and then effectively deliberately defaulting," Mr Joyce said.
It is unclear if that assurance will help quell student protests this year.
Last year protest erupted just days after Budget 2012 when changes were announced increasing the student loan repayment rate from 10 cents to 12 cents in the dollar, removing the voluntary repayment bonus and removing eligibility for student allowances for postgraduate study.
Students in Auckland, Wellington and smaller centres took to the streets, tipping over rubbish bins, and lighting small fires to protest the changes.
Around 400 University of Auckland students blockaded Symonds Street.
Included in Budget changes today, repayment thresholds would be increased for overseas-based student loan holders, and fixed repayment obligations would be introduced.
A borrower with a $20,000 loan balance would pay it off in 15 years, rather than in 35 years under the old rules.
The interest they would pay would halve from $19,940 over 35-years, to $8425 over 15 years.
Mr Joyce has been organising for a private debt collection agency in Australia to help find defaulting borrowers.
"This tracing activity involves a group of 10,000 borrowers, believed to be in Australia, who have made little or no effort to repay their debt, or even make contact with Inland Revenue," Mr Joyce said.
While the Government had already reined in $60 million from overseas student loan borrowers, according to Revenue Minister Peter Dunne, Mr Joyce said they needed to "up the ante".
The cost to the taxpayer had already dropped from 48 cents in the dollar in 2009, to 39 cents in the dollar by March his year.
Mr Dunne said 14 per cent of overseas-based borrowers did not meet the interest charged on their loans.
Labour's tertiary education spokeswoman Megan Woods feared the move was a continuation of the moves in the last Budget.
"Particularly the reduction of the repayment holiday from three years to one year - it doesn't allow New Zealanders that freedom to go away in that important part of career development overseas and then come back with all these skills.
"This is a really punitive approach."
New rules for overseas borrowers:
Loan Balance - Amount to be paid per year
$1000 or less: The whole balance
$1000-$15,000: $1,000
$15,000 - $30,000: $2,000
$30,000 - $45,000: $3,000
$45,000 - $60,000: $4,000
greater than $60,000: $5,000
READ MORE -
John Armstrong: Steady-as-she-goes budget
Meridian next on the block
Research and Development gets $130m funding boost.
All state house tenancies to be reviewed
Government allocates $188.6m for welfare reforms.
ACC levy cuts to cancel out fiscal drag
$94m to make KiwiRail freight business self-sustaining
Govt to work with councils on housing planning and consents
Govt to cut student allowance and loan spending
Families will be paid to care for disabled adult relatives
Govt considers WoF for rental housing
$19m to establish first charter schools