The clip has since gone viral on social media, and several have pointed out the abhorrent nature of his comment. It prompted a charity that acts for the travelling community to speak out against him.
Carr had referenced his special contained "career-ending" jokes, and the backlash he sparked may have been more than he bargained for.
"It's estimated that around 75 per cent of Europe's gypsy and Roma population were murdered in the Holocaust," a viewer wrote.
"Take Jimmy Carr off the air and cancel all his projects immediately."
Another said: "Absolutely abhorrent. I know we joke about it but we have a serious issue regarding offensiveness in comedy."
An anti-fascism and anti-racism campaign group Hope Not Hate called the joke "a shame" in a tweet.
"Comedy is an amazing tool for progressive change and it's such a shame that Jimmy Carr decided to use his platform to celebrate the murder of one of the most marginalised groups in society."
And Olivia Marks-Woldman OBE, who is the CEO of the Holocaust Memorial Day Trust said: "We are absolutely appalled at Jimmy Carr's comment about persecution … and horrified that gales of laughter followed his remarks."
Charity The Traveller Movement called the comments "disturbing".
Carr addressed the incident in a live show recently, according to reports.
The Mirror published that Carr had a heckler at his how who yelled out: "Are we going to talk about the Holocaust?"
The comedian then replied: "We are going to talk about cancel culture, the whole thing. We are going to talk about f***ing everything people. Relax."
It is not the first time Carr has been in hot water over an offensive joke.
In 2016 Carr shocked Comedy Central's audience during the Roast of Rob Lowe when he made a joke about 9/11.
And in 2015, the British comedian sparked complaints to communications regulator Ofcom when he made a joke about dwarves.
He said: "I tried to write the shortest joke possible. So, I wrote a two-word joke which was: 'Dwarf shortage'. It's just so I could pack more jokes into the show."
He then looked directly at the camera and added: "If you're a dwarf and you're offended by that, grow up."
Before he launched into the controversial gag, he told presenters: "Don't play my stuff on your show, Ofcom will get involved."