Robbie asked Felix what he was born as, and while Felix obliged - many viewers felt the question was "out of order".
"Robbie, what the hell are you playing at asking a trans guy what his name used to be? Shameful asking him to dead name himself in front of thousands of people and on prime time TV.... you're in my bad books buddy," one viewer tweeted.
"Robbie proving what a c**k he truly is! Complete inappropriate question to ask a transgender their dead name,' another fan raged.
"Robbie Williams asking a trans man all about his life BEFORE transitioning whilst he sits with his feet up on the desk is so gross and disrespectful. #XFactor is dead and finished," a third viewer vented.
Another user wrote: "Trans lad comes out to audition on The X Factor, first question Robbie Williams asks is what his name was when he was born. F**ks sake. Dead names aren't people's business."
However, other viewers flocked to the father-of-three's defence, claiming Felix was open about his backstory.
"Oh leave Robbie alone. He will have been told to ask so that they could promote the backstory of Felix," one fan wrote.
Another agreed: "If Felix wants to be open and bring up his choices in his audition, there's nothing wrong with Robbie asking questions, #xfactor ps. His voice was beautiful."
MailOnline has contacted representatives for Robbie and The X Factor for comment.
Felix sang Kodaline's emotional hit All I Want for his audition, explaining that he picked the song because it's the first tune he ever performed live aged 13.
The contestant left the judges blown away, with Louis Tomlinson gushing: "As a recording artist the tone you have is amazing."
Simon Cowell added: "You have a great voice, and even when you messed up the notes because I think you became emotional it made me like you more."
Felix explained: "It's difficult because if I played my voice back in February I had a falsetto that has now disappeared.
Robbie reassured him: "Felix, don't worry about it. You killed the vocal, your journey so far in life is fascinating and I want to see where the rest of it takes you."