He said: "Once I realised that that was the way this character was thought of, I just didn't want to participate in it any more. It just didn't feel right."
Azaria, who is Jewish, added he could understand the negative reaction to the character as he would be equally offended by a character that played up Jewish stereotypes.
He said: "I started thinking, if that character were the only representation of Jewish people in American culture for 20 years, which was the case with Apu, I might not love that."
Hank's fellow cast member Harry Shearer - who voices characters such as Mr Burns and Ned Flanders - questioned whether it mattered as the job of an actor is to inhabit a variety of characters.
The 76-year-old "Spinal Tap" star said: "I have a very simple belief about acting. The job of the actor is to play someone who they're not. That's the gig. That's the job description.
"I'm not a rich nuclear power plant owner. I'm not a Bible-believing Christian that lives next to Homer. I'm not any of those people."