Last year the previous Government unveiled plans to add 1800 prison beds at a cost of $1 billion, with more double bunking in Ngawha Prison, a new 245-bed block in Mt Eden Prison, and the new 1500-bed prison.
Justice Minister Andrew Little said it was his "strong preference" not to build a new prison, which he called a symbol of the "abject failure of our criminal justice system".
Corrections Minister Kelvin Davis echoed this sentiment, adding that construction work had yet to begin.
"I'm looking at all options to reduce the prison muster, so that it doesn't end up being built. Officials are being sent away to work out what will have an immediate impact.
"We'll rule out the stuff that won't make New Zealand safer."
Labour wants to lower the prison population by 30 per cent in 15 years, a target Little described as "ambitious".
Little said he had no plans to revisit the bail laws, switching the focus to crime prevention, prisoner rehabilitation, and rolling out more therapeutic courts, which can divert offenders away from jail and into treatment if they plead guilty.
Bill English warned that the mushrooming prison muster could become a critical issue.
"I think where you're going to find a lack of stability is the Government not knowing what it's going to do about the big pressure coming on the Corrections portfolio.
"They'll get overwhelmed by what's happening if they don't get smart on that."
Corrections national commissioner Rachel Leota said the department currently could hold an extra 400 prisoners, which should rise to 600 by the end of the year.
Davis said some innovative thinking could be used to rehabilitate women prisoners, whose needs and motives are different to men.
"Men normally do things because we're a bit stupid. Women normally commit crime to protect others, their families, their children."
He added that he would put an end to privately-run prisons.
"There will be no more private prisons while I'm the minister. With the current situation with Auckland South [run by Serco], we're locked in a long-term contract, and it will be costly to break that at this stage. So we'll just have to see how that pans out."
The Serco contract ends in 2040.
Last week Justice Minister Andrew Little announced the contentious three strikes law would be scrapped in 2018, prompting criticism Labour was going soft on crime.
Describing it as a "silly" law, Little said that "throwing people into prison for longer and longer just isn't working".
Labour would replace it with its own plan to "reduce serious offending rates".
The three strikes law was passed in 2010 and applies to 40 serious sexual or violent offences.
The first-strike conviction results in a normal sentence and a warning, the second in a sentence without parole and a final warning, and the third in the maximum sentence for that offence without parole - though parole eligibility can be granted if a judge deems the sentence manifestly unjust.
The latest Ministry of Justice figures up to July 2017 show that 8503 offenders were on a first warning, 216 were on a final warning, and two have been convicted of a third strike.
