The source added: "At worst it was extremely offensive. At the very least it was stupid."
The images show Hollywood posing for photographs in the Nazi uniform alongside his wife Alexandra, 53, and a friend.
The baker's choice of outfit is somewhat surprising considering his ancestor's links to the British military in the Second World War.
In 2015, while taking part in the BBC ancestry show Who Do You Think You Are?, it emerged that Hollywood's grandfather Norman Harman served with the 99th AA regiment Royal Artillery in Britain, in Tunisia and Italy from 1940 to 1945.
But Hollywood is not the first public figure to be involved in a Nazi uniform row.
In August, two serving policemen and a Ministry of Defence employee were pictured dressing up as Nazi soldiers for a Second World War re-enactment.
MoD staffer Aaron Buckingham, 50, assumed the character of a Nazi corporal.
While Sergeant Andy Dunlop, 48, of Devon and Cornwall Police and Hampshire Police Constable Simon Merritt, 43, took on the role of German soldiers.
In 2005, Prince Harry was pictured wearing a Nazi uniform to a costume party.
Hollywood is the only remaining member of the old Bake Off team after the move to Channel 4 and it was mixed reviews for his new colleagues Noel Fielding, Sandi Toksvig and Prue Leith, who replaced the much-loved Mel Giedroyc, Sue Perkins and Marry Berry.
There is no suggestion that Paul or anyone else at the event has any sympathy with the Nazi Party.