"If all the Defence Force family ... threw us their party vote, we'd probably be at 15 or 16 percent."
Mark also said in his speech that the $25,000 in Government funding No Duff received last year was "courtesy of me".
"So I'd like to think you recognise that, and I'd like to think you recognise the responsibility that comes with it."
Mark said the comments about funding and about New Zealand First were "quite separate" and had been spliced together in a way that lacked context.
"I didn't say vote for New Zealand First ... If you listen to everything I said in that 25 minutes, I think I was being pretty candid and up front that the prospects of me being in the job were not high [because of low polling]."
During Question Time, Ardern backed Mark's assertion that he was not making threats.
But she said he should not have been talking about New Zealand First at an event he attended as the Minister.
"Of course I have shared with him my view that when speaking in a ministerial capacity, references of a party political nature should be left at the door."
Mark said he would be more careful in the future.
On it's Facebook page, No Duff said it was an apolitical organisation.
"We hold no view on the Minister's recent comments, but we appreciate his willingness to speak frankly and with candour.
"The Minister is an important member of the Veterans community. He is one of us."
National Party leader Simon Bridges said that Mark was using his ministerial heft to make threats to members of a charity.
He compared this to Regional Economic Development Minister and NZ First MP Shane Jones, who has faced criticism for being in a ministerial meeting about funding for a project in which he had declared a conflict of interest.
"The NZ First Ministers are learning something this week. They can do anything and Jacinda Ardern won't act," Bridges said.
Last week co-founder of Animal Evac NZ Steve Glassey accused Agriculture Minister Damien O'Connor of threatening to cut Government funding, which O'Connor has denied.
"We have had veiled threats from officials and even a minister that if we continue to draw attention to [Ministry of Primary Industries'] deficiencies, our chances of getting funding will be affected," Glassey told a parliamentary select committee.
He accused O'Connor of saying to him: "You can't go riding us and then come to us for funding."
But O'Connor denied this, saying that Glassey was always negative about the great work MPI did.
"Steve will always extrapolate things out. I said it's really important to be positive when we're trying to negotiate a better deal with him,:" O'Connor said.
"I think Steve going around criticising MPI staff at every single opportunity when everyone is doing their best is not a very productive way forward."
Ardern backed O'Connor, Jones and Mark during Question Time.
