US election: How did the political polls get it so wrong, again?

President Donald Trump pumps his fist after speaking in the East Room of the White House. Photo / Evan Vucci, AP

As the saying sort of goes: "To lose one election may be regarded as a misfortune; to lose both looks like carelessness."

It's an idea that might well be playing on the minds of Democrats as political wonks calculate the narrow path to victory that remains for Joe Biden following the US election result.

With just a handful of states remaining, the 2020 results clearly show that the upset of 2016 in which Donald Trump romped home against Hillary Clinton, was no aberration.

As the UK Daily Telegraph put it: "Trumpism has been vindicated."

Democratic presidential candidate Joe Biden with his wife Jill Biden, speaking to supporters in Wilmington, Delaware. Photo / Andrew Harnik, AP

Election polls ahead of 2020 showed Joe Biden winning, some of them by a landslide with his lead the largest any candidate had seen since Bill Clinton in 1996.

He was widely predicted to be around 10 percentage points up, however results so far show Biden has taken around 51 per cent of the popular vote while Trump is on 49 per cent.

At the time of writing, Biden's path to victory looks possible, but precarious, coming down to a "razor thin" margin of votes in "blue wall" that Trump managed to secure in 2016.

So what happened? Biden's lead was larger than Clinton had in every battleground state and he was predicted to win the popular vote by eight percentage points, according to FiveThirtyEight. Indeed the site still claims he is favoured to win, with odds of 89 in 100.

However, that's at odds with other polls, such as one by the ABC-Washington Post which showed the Democrat having a 17-point lead in Wisconsin – only to see the state come down to the wire with less than 30,000 votes separating the rivals with 98 per cent of votes counted.

Municipal workers extract Luzerne County ballots from their envelopes in Wilkes-Barre, Pennsylvania. Photo / Mary Altaffer, AP

Meanwhile a Quinnipac poll suggested Biden had a five-point lead over Trump in Florida and four point lead in Ohio. Instead Trump won both states, by three and eight points respectively.

So what is going on? And why can't the media and polling companies seem to get it right?

One theory favoured by conservative commentators is the "shy Trump" voter, that argues the public are unwilling to out themselves as fans of the Donald in a move that could see them ostracised socially.

Fox election analyst Steve Hilton subscribes to this and told the Mail Online that "because of the incredible degree of hate that's been directed to President Trump and his supporters by nearly all the media, you've got this situation … where people didn't necessarily want to admit to pollsters who they were supporting because it was socially embarrassing to do so".

Fox host Tucker Carlson agreed, saying the media needed to admit to the fact they'd been getting it wrong and being afraid to express one's voting preferences was a big problem.

"The most basic question it raises is, is this a free country?" he asked during election coverage.

"If you are afraid to express a legitimate political view in public, if you believe you could be fired for it or banished from polite society is it a free society? And the answer of course is no it's not."

"You don't want to live in a country where you don't feel free to tell others who you're voting for."

But despite the neatness of this theory and underdog factor it speaks to, other commentators put the failures down to changing voter preferences and a simple margin of error.

The reality, critics argue, is genuine chunks of undecided voters in battleground states and perhaps failure to adjust polls for education levels which can be a key indicator of voting preferences.

"This obsession with finding hidden pockets of support for Trump is part of a larger phenomenon we've observed for a while now: Trump is down in the polls, and has been for months, but if you ask Americans if they think he will win, many still say yes. And to be clear, Trump can win. We think he has a nearly 1 in 4 shot of doing so in our forecast, but that doesn't mean he's not an underdog," Geoffrey Skelley wrote for FiveThirtyEight in the run up to the 2020 vote.

President Donald Trump speaking from the White House is projected on a television at a restaurant in Tokyo. Photo / Kiichiro Sato, AP

"This is similar to what we said in the lead-up to the election in 2016. Then, Trump was an ordinary, average polling error away from winning the election, in which if he beat his polls by just a few points in just a couple of states he could win — and in fact, that's exactly what happened."

A detailed study into the performance of the 2016 polls by the American Association for Public Opinion Research found that they were "generally correct and accurate by historical standards."

However they still failed to accurately gauge Trump support, which the authors put down to real change in voting behaviour in the final weeks of the campaign, a lack of adjustment for college education levels and some evidence of a shy-Trump effect – however other factors did not back this up.