Finance Minister Steven Joyce said the comments were inappropriate and Ngaro had apologised to English, deputy Prime Minister Paula Bennett and Joyce.
"He got a bit carried away. It's not the way we operate. We work all the time with providers who have different political views. It's important you do that and we certainly don't look to take an approach where a provider would get penalised for their political views. It's fine to disagree with people politically but to make any suggestion it might impact on your relationship with government, that's where it's overstepping the mark."
He put Ngaro's comments down to a rookie mistake and frustration with the way National's policies were being portrayed.
"But he realises what he said was over stepping the mark. He's very disappointed in himself."
Ngaro has since put out a statement saying his comments were "a bit naïve, poorly worded and I absolutely regret what I said."
Willie Jackson did not comment but Labour leader Andrew Little said it was a sign National was desperate.
"It looks like an implied threat to any organisation contracting with the Government. It's totally out of order and after nine years of this Government, New Zealanders deserve a lot better."
Ngaro was also reported as saying there was tension within the Salvation Army between the Church and the social and policy arms and claimed Labour was "using its constituents for political fodder" by highlighting the homelessness debate.
Ngaro was appointed to Cabinet last December after Prime Minister Bill English took over from John Key.
In his statement Ngaro said he believed the Government was doing a lot to tackle the challenges in housing. "I am hugely positive and proud of this work, and this was what I was trying to impart.
"The Government doesn't think it has all the answers to tackling challenges in housing and social support, which is why we work so closely with a range of fantastic community organisations.
I'll continue to engage with our partners and ensure we are working on real solutions to help New Zealanders who need assistance with housing."
Jackson's advocacy for charter schools has put him at odds with Labour policy which is opposed to the schools. Little said he was not aware MUMA had applied for a second charter school, but said Jackson had stood down from his "private interests" after becoming a Labour candidate.
Labour has said it will review the schools if in Government, and will retain those that meet criteria such as having fully qualified teachers as 'special character' schools.
