Mr Mallard said he was tempted to invoke a "Arkell vs Pressdram" response - referring to a British case concerning the Private Eye magazine.
In the 1971 case, when told by the plaintiff's lawyer that his client's attitude to damages would "be governed by the nature of your reply", the magazine replied: "We would be grateful if you would inform us what his attitude to damages would be were he to learn that the nature of our reply is as follows: F*** off."
The letter from Ms Collins' lawyer gave a deadline of this Thursday.
Mr Little said he had also received a letter and was speaking to a lawyer, but that he "wasn't losing sleep over it".
"Basically it says please apologise the way we've asked you to do before, and if you don't we will proceed," he said.
"Subject to talking further with the lawyer, the letter hasn't changed my view that I don't think an apology is required."
Ms Collins has continued to deny allegations from the Opposition that she or her office was responsible for leaking an email sent to her by former National Party president Michelle Boag.
The email gave details about Bronwyn Pullar, the ACC claimant who went to the media last month after being mistakenly sent the details of about 7000 ACC clients.
Privacy Commissioner Marie Shroff is investigating how the email got to the media, and Ms Collins has told Parliament she will resign if the leak is found to have come from someone in her charge.