The legal threat came after Mr Haynes said Dr Smith had behaved like a bully at a Fish and Game Council meeting on July 18.
He said the minister described the council as a "rabid NGO [non-governmental organisation]" and threatened to limit its role if it did not "tone down" its criticism of irrigation and agriculture - a key part of National's economic agenda.
His description of Dr Smith's behaviour was backed by several other attendees.
The minister's criticism stemmed from a series of Fish and Game billboards which showed pictures of dead fish and dirty rivers and said: "Irrigation: How much ... is too much?"
Prime Minister John Key said yesterday that Dr Smith would be "silly" to take legal action.
Labour leader David Cunliffe compared the minister's alleged threats towards Fish and Game to Government's decision to cut funding to the Problem Gambling Foundation after it had openly criticised National's policies.
Mr Cunliffe said this morning: "To even suggest to a statutory body that following their statutory mandate would run into conflict with the government and it could result in funding change - I think that is abhorrent."
He said Dr Smith's behaviour reflected an increasingly arrogant government.
"This week we have seen that culture of creeping arrogance get worse. We have four MPs in trouble.
"Minister Smith with his attempt to muzzle an agency for doing its statutory function. I think it's the third strike and he should be out.
"I'm calling on the Prime Minister to let that minister go from his portfolio as Conservation Minister. After Pullar and Ruataniwha this is one strike too many.
"We've seen Gerry Brownlee with his airport problems, we've seen Ms [Claudette] Hauiti with her put-it-on-the-plastic, and we've seen Jonathan Coleman apparently being untruthful about when he was briefed on the FBI."
Watch: Anne Gibson interviews Nick Smith
Dr Smith resigned as ACC Minister in 2012 after helping his friend Bronwyn Pullar with an ACC claim. He was also accused of gagging his conservation department last year over its concerns about the proposed Ruataniwha dam in the Hawkes Bay.
Mr Key planned to meet Fish and Game this afternoon and reaffirmed that there were no plans to limit the council's powers.
He stood by his minister, saying official notes from the meeting showed he did not threaten to change the organisation's statutory role.
The notes were released by Dr Smith's office yesterday and appeared to show that Dr Smith had questioned Fish and Game's advocacy work.
The notes said: "Big issue - F&G need to work out what they want to be: a statutory body - legislation and a relationship with Government or an NGO?? Statutory monopoly!!"
Dr Smith said Mr Haynes' public comments contradicted a blog he wrote three days after the meeting.
In the blog on Fish and Game's website, Mr Haynes said Dr Smith had "delivered his opinions and thoughts as professionally as ever" at the meeting.
Mr Haynes also said in the blog that the minister had "chided" Fish and Game for challenging dairy's polluting of lakes and rivers and described environmental NGOs as "rabid".
Read more: Nick Smith's legal threat over bullying claim