This to the obvious embarrassment of the remaining members of the boyband who were present with Cowell on stage. The sense of crisis was unmistakable.
It was an uncharacteristic loss of control from Cowell, who is generally punctilious about the image he presents to the world.
One executive at a rival broadcaster said he felt that last weekend was Cowell's "Michael Barrymore moment" - a fall from grace of someone who previously could do no wrong.
"He was obviously drunk," said a fellow guest.
Another said: "Simon has believed his own publicity for a few years now - that he's a genius. It's a huge blow for him to fail, and then he can't fix it."
Anyone who doubts there is big trouble at The X Factor should note that Rita Ora and Nick Grimshaw - his two latest additions to the judging panel - didn't bother to show up.
Perhaps they already assume it is pointless keeping Cowell sweet.
Neither thinks they'll be asked to return next year after the bloodbath of the current series.
And Cheryl Fernandez-Versini, who counts the TV mogul as a friend despite their explosive fights in the past, came without her husband, whose name was written out on a place card at her table. According to a fellow guest, she looked "stricken" with anxiety.
Only a few hours before the awards, the overnight viewing figures of the revamped X Factor were released. They made shocking reading.
For The X Factor had been surpassed on Sunday night by its television antithesis, the Antiques Roadshow.
And on Saturday, the epic live showcase night attracted a mere 5.6 million viewers - the worst rating since the show began in 2004.
To make the defeat yet more sour, its BBC rival Strictly Come Dancing pulled in an impressive 9.4 million viewers.
The days of The X Factor grabbing over 19 million viewers and uniting the nation in a debate over the relative merits of Stacey Solomon and Leona Lewis are long gone.
There is even talk that the show may be axed.
"If you get viewers of around five million the sums no longer stack up," said one TV executive. "It doesn't make sense."
Officially, Cowell is said to be "determined" to fix The X Factor.
He has already issued a public mea culpa for the shambolic rejigged judges' houses episode and was seen in the studio on Saturday, after the cameras had stopped rolling, apologising to the contestants for the new gladiatorial format.
Cowell is so unnerved that he has been making uncharacteristic slip-ups and diversions - launching into bizarre asides about his dogs Squiddly and Diddly, even saying how surprised he would be if they spoke to him on Saturday night.
It seems he feels his age is catching up with him. One source close to the show claimed that he worries that sitting close to his much younger judges makes him look older, adding: "He hates how he looks on screen."
Unofficially, the word used is "panic."
A source close to Cowell tells me: "Simon keeps on calling the producers all through the night to try to work out what to do to save it. They are in a terrible spin."
Indeed, behind the scenes, a game of pass the buck is being played which is far more manic and entertaining than anything on TV.
Executives at Cowell's company Syco will not accept the plummeting figures recorded officially by the Broadcasters Audience Research Board - which is surely a measure of how desperate things have become.
One told me this week that ten million viewers tuned in. This figure, it turned out, was arrived at by the company by adding a few million estimated viewers via computers and mobile devices.
Respected industry analyst and former ITV executive Steve Hewlett said: "Without question they had a grim week. Whether it's the start of something very serious we'll have to wait and see."
So how did we get here - and what will Simon Cowell do to try to save the show?
For some years, The X Factor ratings have been going down. The high was in 2010, when an average of 14.1 million people tuned in each week.
By 2014, viewers had fallen to 8.6 million. From 2011 to 2013, Cowell was absent, launching The X Factor in America, where it failed and was axed.
His return in 2014, however, did not bring with it an upswing in ratings - hence a shake-up of the judges and format this year.
Cheryl Fernandez-Versini has been keen to take the credit for some of the changes, saying she counselled Cowell to get rid of the themed weeks and revealing that they would talk for hours on the phone in the middle of the night, brainstorming ideas.
So what has gone wrong? It could be that viewers are getting bored of the carefully manufactured drama. Certainly there is widespread cynicism about the sob stories wheeled out by contestants in the early stages.
Viewers are increasingly annoyed, too, by the large volume of adverts.
Many also complain of the show's patent insincerity - the theme on Saturday was "This Is Me", but contestants were shown being told what their song choices were. So much for the individuality of the performer.
The recent failure of the winners to break into the wider music industry is another factor.
Although Cowell rather bravely says that "four or five" of the current crop have the potential to sell records, last year's winner Ben Haenow has failed to cut it and is clearly on his way back to his native Croydon.
Who is interested in 2012 winner James Arthur any more, or 2010 champion Matt Cardle?
Another issue is the revised format - it is one of the truisms of television that if the viewer is aware of a format, the format is no good.
Given the endless changes to it, viewers can't help but detect a whiff of desperation.
One idea was to have the judges hold up paddles with marks on them - a direct imitation of the Strictly judging panel.
Cowell was keen enough to propose the idea to ITV, but it was vetoed by the entertainment team led by Director of Entertainment and Comedy, Elaine Beddell.
Another bizarre scheme was to introduce a fifth "legend" judge. Cowell got this idea after Sir Tom Jones was axed by rivals at The Voice.
I'm told Sir Tom agreed to do it and was among several legends lined up - before this, too, was vetoed by executives at ITV.
That caused some bad feeling as Sir Tom had apparently gone so far as to go into rehearsals before being let down at the last minute.
It's said there has even been a three-hour meeting on the issue of whether to change the colour of the floor on the stage.
To be tinkering with formats during the actual run of a show certainly speaks of a degree of chaos.
Cowell, meanwhile, is taking care to be very complimentary about ITV, which is under contract to make a further series in 2016.
What is on everyone's minds is what will happen next spring, when the broadcaster must renegotiate for 2017 and beyond.
Sources at ITV were this week making positive noises about bidding again for The X Factor. It remains their biggest entertainment hit and their biggest Saturday night show.
However, one well-placed TV boss told me there is a belief that ITV will want to pay far less in future (Syco are on a £140 million deal covering three years), and that some conversations have already started with Sky about whether they would like The X Factor in their schedules from 2017.
Sources at Syco don't deny that such negotiations are "possible" and a TV executive told me: "Even if The X Factor loses half its viewers, Sky would still think Christmas had come early if they managed to poach it."
Which leads us to the nightmare possibility keeping Cowell awake at night - which is that The X Factor might be replaced by its great rival The Voice, which ITV are keen to poach from the BBC. The loss of face for him would be huge.
Indeed, one source close to Cowell said he feels the UK has "run dry" for him and explains that, following the sad death of his mother Julie after a long illness earlier this year, he is ready to pick up the reins of his old life in Los Angeles.
Hence the surprising announcement two weeks ago that he is planning to be on the panel of America's Got Talent. I am told that the show, usually filmed in New York, is going to be made in Los Angeles this year. He will then, aged 57, be making Britain's Got Talent, America's Got Talent and The X Factor.
It's a schedule as busy as any he has faced.
He is a phenomenally driven individual, and has often talked about how much he hates to lose, which does help to explain the extraordinary decision to do more.
On a personal level, now happily settled with American socialite Lauren Silverman and son Eric, a move to Los Angeles makes sense.
I'm told Lauren is looking forward to being based in Los Angeles again as it will make it easier for her to see her son by former husband Andrew. They share custody and she has been crossing the Atlantic on a weekly basis to see him.
Yet Cowell's colleagues insist he hasn't given up on the show. Simon tells friends he is "determined but not down" about the challenges. But can he truly hope to turn the tide?
As expert Steve Hewlett notes: "It's a very strong show which still generates huge amounts of interest but it is an ageing format. And it's not ageing particularly well."
- Daily Mail
