Dastyari is considering a legal response to the incident, which may have infringed race discrimination laws.
Islamophobia and racism were getting worse in Australia on both the left and right of politics, he said.
"I worry about all the people out there that have to put up with all this kind of abuse who don't have the structures that someone like I'm lucky enough to have."
Prime Minister Malcolm Turnbull said there was no place for racial vilification in Australia.
"That is because our society is built on a foundation of mutual respect. It should have zero tolerance for racist abuse like this," he told Seven.
Opposition Leader Bill Shorten said he was disgusted by the "ugly" incident.
"What is this country coming to when you cannot go out for a meal without being abused by racist idiots," he told Seven.
The men from the right wing group Patriot Blue filmed the incident and posted it on their Facebook page.
When he called them racists, they replied with: "What race is Islam?"
In a Q&A session at the Footscray pub with fellow Labor MP Tim Watts, Dastyari said he was regularly subjected to similar incidents because he was a Muslim.
"All of this is the rise of the radical right in this country, it is the rise of One Nation right," Dastyari said.
"These are people who feel incredibly empowered because of what Pauline Hanson has done for them. You dance so far to the right that it gives those a little bit further out a sense of entitlement."
One Nation rejected Dastyari's portrayal of the incident.
"Sam Dastyari gets heckled because he's a wanker," Pauline Hanson's chief of staff James Ashby told AAP. "And not because he's a Muslim."
One of the men involved says the Labor MP gives as good as he gets.
"Sam's a strong bloke he's got a thick skin, he's been in politics for a long time, " Neil Erikson told ABC radio today.
"He called us rednecks, which is a racist term in fact, so look, he gives as good as he gets. I think he's playing the victim a little bit."
The member of the far-right group was wearing a high-vis Toll Group shirt in the video. The transport operator is investigating whether any of the men work for the company. A Toll spokesperson told AAP the actions in the video "in no way reflect our beliefs and values as a company".
Erikson told Melbourne radio 3AW he wasn't employed by Toll and wore the shirt to his "other job".
He was convicted in 2014 after pleading guilty for stalking a Melbourne rabbi. He was sentenced to a 12-month community correction order, involving 150 hours of unpaid work.
He's appealing a 2017 conviction for beheading a dummy outside a council office in protest at a proposal to build a mosque in Bendigo.
Erikson said he had the right to free speech. "If he wants to go for Toll and try and hurt me financially he can go for it ... I believe that everyone has a right to freedom of speech," he told ABC radio.
Speaking to The Today Show, Dastyari left things on a lighter note. "They went to a bar and didn't even stay for a drink. What kind of patriots are they?"
- AAP
