Foreign Minister Murray McCully dismissed his allegations, saying that they were "utterly baseless".
Mr Little made the comments outside Parliament's debating chamber, and was therefore not covered by legal privilege.
Mr Little said he had received a letter from the Hagaman's lawyer this morning. He was now "considering the issue and taking advice".
He added: "I will not be restrained from undertaking my constitutional role of calling the government of the day to account."
The Auditor-General's office said today it had not yet decided whether to open an investigation into the case.
On April 18, Labour called for the Auditor General to look into Scenic Hotels Group's contract to manage the Matavai resort after it was discovered that Mr Hagaman, gave a $101,000 donation to National the month before the contract was announced in 2014.
Mr Little said at that time that the close timing of the donation to the awarding of the contract "stinks to high heaven".
Mr Hagaman and wife Lani Hagaman said that after seeking advice, their solicitor had notified the Labour leader today that defamation proceedings would ensue "unless the position can be properly retrieved".
Lani Hagaman said their lawyers had confirmed their view that Mr Little's allegations were "false and defamatory".
"We are asking for a full retraction and apology because no one should be verbally attacked and denigrated because they believe in democracy and the right to make their own unsolicited political choice on who they want to give a donation to," Lani Hagaman said.
"The decision to make the donation was completely unsolicited and was Earl's personal decision and nothing to do with the Scenic Hotel business."
Lani Hagaman criticised Mr Little's use of his Parliamentary role to attack their business.
She said her and Earl did not "come from power or privilege" and had strived to give others a livelihood and provide support to regional economies "where other big hotel chains won't invest".
"The position Andrew Little holds is one of power and privilege. It should be a privilege to be elected into Parliament and work hard for the people of New Zealand, rather than to cast unjustifiable slurs on people because they have made a donation to the party of his prime political opponent."
The Matavai is owned by the Niue Tourism Property Trust on behalf of the Government of Niue, which owned the resort before then.
That arrangement was put in place in 2011 to ensure oversight of the aid investment New Zealand was putting in.
Foreign Affairs Minister Murray McCully appoints the trustees who are Ross Ardern (NZ's High Commissioner to Niue and father of Labour MP Jacinda Ardern), Ministry of Foreign Affairs deputy secretary Jonathan Kings and former High Commissioner Mark Blumsky, who was formerly a National MP and now lives in Niue.