"And I think you kind of got to take a bit the rough with the smooth. And we do that over here. We do deport the odd person, but after a while we say 'you are fundamentally a New Zealander and you take the good with the bad'."
Deportees are under no obligation to engage with police when they arrive in New Zealand.
Justice Minister Amy Adams announced an information sharing agreement with Australia on deported criminals last month, said the next step was to change the law so deportees were subject to the same conditions they would be had their sentence been served in New Zealand.
Today, Ms Adams said supervision orders could be put in place for the most serious offenders, and the law would soon be changed to cover all deportees.
"Corrections will make the assessment as to whether she meets that criteria [to be termed a very serious offender]."
Colt's case comes only days after Mr Key met counterpart Malcolm Turnbull in Auckland to protest a new law that has seen non-Australians deported if they incur a jail sentence of 12 months or more.
Labour leader Andrew Little said the Colt deportation - if confirmed - was a "classic example" of what New Zealand was facing because of Australia's tough new law.
"The [New Zealand] Government knew about the law change at the time...the Government has done nothing about it until now.
"It is very late, it is a classic thing for this government - waiting for an outcry before they try and do anything. They are just negligent, time and again."
Colt was part of a 40-strong group that lived in squalid conditions near Canberra.
In 2012, authorities took away 12 children from the group and later testing revealed that only one had parents that were not related.
The children, who were underfed, hardly went to school and lacked basic skills such as how to shower and clean their teeth, later told of suffering sexual abuse.
A judge ordered Colt's five children aged under 16 to be put in foster care.
He noted the case appeared to be one of intergenerational incest and Colt may well have been a victim of incest herself. According to previous reports, Colt's mother's parents were brother and sister.
Her mother and her husband moved from New Zealand to Australia in the 1970s.
In 2013, Colt was sentenced to a year in jail after she tried to organise one of her sons, aged 15, to help her abduct his brother.
No one in the family has faced any incest-related charges.