It must be even more surreal for the castle's real-life inhabitants, the 8th Earl and Countess of Carnarvon. Not only do they have to relinquish their home for weeks on end to an army of film crew, but they then get to watch it inhabited by actors on TV each week.
Not that they're complaining. Since the show started, visitor numbers have rocketed from 300 people a day to 1300. All of which helps when you have a 50-bedroom castle and a 400ha estate to maintain.
Highclere has been a home for more than 1000 years but the current building was designed in 1842 by Sir Charles Barry, who also built the Houses of Parliament.
Described by our guide as "Georgian with Victorian icing", it is a solid, imposing structure with a flurry of intricately adorned turrets and a striking Italianate tower.
A tour takes you through many of the rooms that feature regularly in the series. We start in the mahogany-panelled library, an impressive space lined with more than 5000 books, the oldest dating back to 1538. From here, a concealed door leads into the music room, an opulent haven lined with Italian 16th-century gold embroideries.
Look up and there's an exquisite baroque ceiling painted by Francis Hayman, one of the founders of the Royal Academy. In the corner sits a mahogany desk and chair designed for Napoleon.
Fans will recognise the drawing room as the place where the Crawleys gather before dinner. Lined with lavish green damask silk and featuring an ornate crystal chandelier, it reflects the late Victorian fascination with French rococo design.
Perhaps the most impressive space is the saloon, a medieval-style hall in the heart of the house surrounded by a spectacular 15m-high vaulted gallery. Anchored by an impressive stone fireplace, its walls are lined with brightly painted heraldic shields and covered in 17th-century Spanish leather.
The stunning oak staircase that ascends into the house's central tower took almost a year to carve and install. Leading off the gallery upstairs are the bedrooms of Lady Cora, the late Lady Sybil and Lady Edith (the scenes in Lady Mary's bedroom are shot in a studio) and sharp-eyed viewers will recognise the hallway down which in the first series Kemal Pamuk was dragged after his untimely demise.
Interestingly, many of the show's storylines were inspired by real-life events. Highclere was used as a hospital during World War I and the 6th Earl also married a wealthy American heiress.
Even if you've never seen the show, there are still reasons to visit Highclere. In 1922, the 5th Earl of Carnarvon with archaeologist Howard Carter discovered the tomb of the Egyptian boy pharaoh Tutankhamun. Descend into the castle's cellars and you'll find an Egyptian exhibition filled with artefacts from the dig and a mock-up of the tomb.
For Kiwi Downton devotees, the big news is that the fifth series premieres on Prime on Thursday.
No clues yet as to plotlines but Gareth Neame managing director of Carnival Films, maker of the series, said it will deliver "all the usual highs and lows, romance, drama and comedy played out by some of the most iconic characters on television". I can't wait.
Rob McFarland travelled to Highclere with assistance from Visit England.
