Respondents also said being treated badly and poor communication in the workplace were obstacles to output.
Ernst & Young partner Braden Dickson said the reality was most workers went to work every day wanting to do a good job.
While many admitted to spending some time on social media, the biggest time wasters were unnecessary emails and having to wait on other people, he said.
"This is not feet-on-table time wasting. People are saying 'help me use my skills, communicate with me, tell me when I'm doing something good or bad in my job'."
Dickson said New Zealand's aging workforce and drain of skilled workers heading overseas meant the country needed an urgent "conversation" about productivity.
The report said organisations had worked hard over the past year to find productivity gains - through cost cutting, redundancies, removing process duplication and increasing automation - but only so much could be "squeezed" from such initiatives.
"New Zealand has already felt the net productivity gains as a result of companies tightening the screws to adapt to changing market conditions," the report said.
"In future, these tried and true programmes will only produce diminishing returns."
Furthermore, there was evidence that job security and workers' confidence in the market had become a major issue as a result of these measures.
Only 23 per cent of workers said they felt secure in their current jobs, which was important because job security went hand-in-hand with productivity, the report said.
Big corporations and government agencies needed new thinking and "game changing interventions around more radical uses of technology and automation".
"It's time for New Zealand organisations to get better clarity around the work that genuinely needs to be done by full-time employees, and the work that can be done better, cheaper or faster by technology or by other people, through contracting, outsourcing or offshoring," the report said.
Employers should work hard to give workers the right training for their roles, provide good work-life balance policies, and help people feel more secure in their roles.
Government, boards, company executives, and managers should have a clear view of what productivity potential existed in their workplace and how to achieve it.
Dickson said most respondents felt their company lacked any clear structure for measuring productivity.
Businesses also needed to have clear definitions of individual roles so people knew what their job was and how their skills were being used.
"If we want to grow our economy, we're going to have to address productivity systematically and in a constructive way," Dickson said.