Mr Horan said there was no restriction on calls made on parliamentary phones, but Mr Peters called that comment "just staggering".
He said it was improper use of the phone, especially when there was a free 0800 number to place bets.
Mr Peters said he had acted on other information about Mr Horan which justified dumping him even if he was not charged over money allegedly taken from his mother's accounts.
"I stand by my decision based on the evidence that I had and I'm confident that I have acted in the interests of Parliament, the nation and my party and I do not resile from that."
Forensic accountants are reported to be examining Olwen Horan's bank statements for evidence to back up allegations by the MP's brother Mana Ormsby that Mr Horan took their mother's money to buy takeaways, hire DVDs and to fund his gambling.
The Herald understands lawyers acting for Mrs Horan's estate will decide whether to a lay a complaint with police in about a week.
But Maori Party co-leader Tariana Turia said Mr Peters had jumped the gun by expelling Mr Horan before the allegations were proved.
Green Party co-leader Russel Norman agreed. "I do think we need to respect natural justice ... I don't think kicking people out just on the basis of some allegation is a just outcome."
Mr Horan returned to Parliament yesterday where he intends staying on as an independent MP.
Prime Minister John Key said that as a list MP thrown out of his party, Mr Horan had no mandate to stay, but he did not say he should leave.
While Mr Horan has acknowledged he does bet on horses, he yesterday said claims his gambling was out control were untrue.
Problem Gambling Foundation chief executive Graeme Ramsey said that while Mr Horan's calls to the TAB "could raise question marks", they were not evidence of a problem.