Respondents also said being treated badly and poor workplace communication were obstacles to output.
Ernst & Young partner Braden Dickson said most workers wanted to do a good job.
While many admitted to spending time on social media, the biggest time wasters were unnecessary emails and having to wait on other people, he said.
Wairarapa Chamber of Commerce chief executive Stephanie Gundersen-Reid said she hadn't received any feedback from local employers that staff were wasting time with frivolous emails or on social media.
However, Taratahi Agricultural Training Centre spokeswoman Natalie Bowie said the school was currently putting together a social media policy for its employees "in response to a changing world".
The policy was for staff who used social media as a part of their role, not in response to work time being wasted, Ms Bowie said.
Engineering, Printing and Manufacturing Union director of organising Rachel Mackintosh said productivity was most affected by workplace relationships.
Respectful relationships, where workers felt secure in their jobs and had unpaid work recognised by their boss, were closely linked to a business' success.
The amount of time workers wasted or spent on social media was hard to measure, Mr Mackintosh said.
"We think that it's more complex. If workers are feeling demotivated they are likely to be distracted by social media," he said.
"If workers are motivated and working hard and feeling valued and they are part of a productive and constructive working relationship, any time they do spend on social media is likely to be minimised."
Victoria University senior human resource management lecturer Geoff Plimmer said workplace communication needed to be improved both ways.
Employers had a right to ban social media use in the workplace, but Dr Plimmer questioned whether that would solve productivity problems.
Under-skilled managers were more of a problem - particularly when it came to rewarding good performance and dealing with poor performance.
- additional reporting Ben Chapman-Smith, APNZ