One viewer, Catherine Schythe, described the new system as "horrible" and "unfair" on Twitter.
Melissa Edwards said it was "cruel".
"Much prefer the old way of selecting top five," she tweeted.
Yolanda agreed, writing: "I don't like this 'swap' business. #cruel", while Bonnie posted: "This is a painful way of making selections. Gotta be a better way! Putting me off watching!."
The pressure of the challenge certainly got to several of the young men.
Tears were not only shed by performers who missed out but also guys who had been successful.
"Boys. Cry less. Sing better," tweeted Mark.
Five performers - Michael Duchesne, Jimmy Davis, Andrew Lambru, Cyrus Villanueva and Big T - on Sunday made it through to the next stage.
The heartbreaking performance of the night was from 15-year-old Isaiah Firebrace who, despite immense talent, forgot the words to Tracy's Chapman's Fast Car.
If Isaak created controversy with his selections, UK singer James Blunt is in for some trouble on Monday.
Blunt has already filled four of his five seats with the first four singers that performed in the over-25 age group. There are seven more contestants to sing.