Mr Haynes' account was backed by three other people at the meeting.
Dr Smith rejected these claims, saying a Department of Conservation official had taken a record of the meeting which did not match the accusations of political interference.
He told Radio New Zealand he wanted Fish and Game to engage more with agriculture and irrigation in order to get the best outcomes for freshwater quality.
"While it is absolutely right for them to advocate for freshwater, I do think they sometimes got into the space of being anti New Zealand's most important industry, that being the dairy industry."
The minister released the DOC official's notes this morning. One of the bullet points in the notes said: "F&G need to work out what they want to be: a statutory body [with] legislation and a relationship with Government, or an NGO?"
Dr Smith's office confirmed he was considering legal action against Mr Haynes, though no further details were given.
Watch: Anne Gibson interviews Nick Smith
The Fish and Game Council was established under the Conservation Act, but it is an independent organisation funded by fees from hunting and fishing licenses. It reports to Parliament and has formal statutory obligations to develop national policy on managing and conserving sports fishing and hunting resources.
Mr Johnson said the minister had implied at the meeting that the organisation would be restructured if it did not tone down its stance on water quality.
"He said that he's worried that Fish and Game is losing its way, that Fish and Game struggles with being a Government statutory body and instead is being a rabid NGO," he said.
"So he made it very clear to everyone there that he wants to see us toning down our advocacy for clean water."
United Future leader Peter Dunne said Fish and Game should be left alone.
"Issues like the impact of dairy intensification, or major new irrigation schemes on water quality are obvious matters of interest to Fish and Game and the interests they are obliged by law to represent and advocate for.
"To threaten - directly or obliquely - to curtail their role because they perform it well, is unacceptable in any circumstances," he said.
Labour's conservation spokeswoman Ruth Dyson said Dr Smith had "yet again" overstepped his mark as a minister.
She pointed to previous cases of claims of political interference by Dr Smith, such as his lobbying for a friend's ACC claim while he was ACC Minister, and allegations that he gagged DOC over its concerns about the impact of a new dam in the Hawkes Bay.
