The 44-year-old's messy marriage break-up, swift rebound with model Yarbrough — almost a decade his junior — and embarrassing public trashing of his co-host Georgie Gardner have seen an ongoing slump in the show's female viewership.
That demographic, which once made up a significant chunk of Today's ratings, "basically now think he is an a** hole", media analyst Steve Allen told The Sunday Telegraph.
"And who could blame them really," Allen said.
"Here you have his ex-wife who supported him and raised his kids and she's left out in the cold while he is off with his younger girlfriend in various exotic locations on holidays every other week.
"For Nine to think that there is any other reason viewers are off the show is, to me, just complete denial."
The show's ratings dive began in September 2016 at the same time Stefanovic was first pictured with his new girlfriend.
Matters weren't helped when he was caught by an Uber driver slagging off Ms Gardner in the back of an Uber with his brother Peter Stefanovic.
And the damaging defection of Lisa Wilkinson to Network Ten, apparently over the much lower salary she was receiving compared to Stefanovic, also hurt.
Since then, he has failed to shake a cloud of negativity that continued to impact viewer numbers and saw Today finish last week with its worst figures in 12 years.
"Given that TV is measured 24/7 and ratings are delivered the next day it does have the effect of harming a network's yield as they have to potentially top up campaigns with no revenue increase," Moonlighting media analyst Ian Warner told the newspaper.
The show's rival, Channel Seven's Sunrise, recorded an average metropolitan audience of 268,000 last week, compared to Today's 199,000.
The report today cite "top-level sources" saying the embattled star is "gone" and will be told this week by Nine's chief executive Hugh Marks that he will be replaced in 2019.
Frontrunners to fill his chair alongside the popular Gardner are David Campbell, who currently fronts Today Extra, and regular fill-in and Sydney radio presenter Ben Fordham.
Channel 9 last week issued strong denials that Stefanovic's position was untenable, but conceded the ratings slump was concerning.
His long-time manager Sharon Finnigan also "denied her star client was a dead-man walking", The Sunday Telegraph said, although Stefanovic himself declined to comment.
Stefanovic was once one of the network's most popular faces, with ambitions to move across to a coveted full-time role on 60 Minutes.
He has steered a number of stand-alone shows, including the disastrous panel show The Verdict and up-and-down This Time Next Year.
Attempts at rehabilitating his image with glossy magazine images and more female-focused editorials on air have failed.