The poll, by online clothing firm Banana Moon, found that 53 per cent of Brits don't have a specific dress code at work, and many workplaces now allow "casual dress".
The survey found that more than one-in-three felt "judged" by what they wore and colleagues and bosses criticised their choice of clothes.
A total of 39 per cent said they would rather they had a set dress code at work or a uniform so they wouldn't have to stress about what to wear.
Vicky Charles, a copywriter from Wiltshire, said she experienced "clothing Nazis" at a previous firm.
She said: "The dress code was 'smart casual', which was rather unclear.
"My boss kept coming up with specific details that would be emailed around: men couldn't wear shorts and women couldn't wear leggings.
"Because the dress code was so vague, it could be interpreted in a lot of different ways, but when upper management saw specific examples of what they didn't want us to wear they sent an email - so something that was okay one day would be out of the question the next week.
"We ended up with really weird and backwards rules."
Banana Moon Clothing boss Alex Grace said: "Work uniforms have changed dramatically over the years.
"Whereas it was once expected that employees would wear a suit or formal dress from 9 to 5, the lines are now blurring between what we wear inside and outside the office.
"While casual dress codes can be more comfortable for employees, it's still important for employers to define what appropriate office dress looks like.
"Putting dress code guidelines or a standard uniform in place can be beneficial in reducing confusion in the workplace over what is and isn't acceptable, and can lead to happier employees in the long run."