"They come hoping to make Kiwi friends, find jobs and improve their English, but often find them stuck with people from their own communities, no jobs and no chance to practice English at all," Sekikawa said.
"They go home disappointed and disillusioned, and we really need to have an in-depth look as to whether this is giving New Zealand a bad name."
Sekikawa said working holiday visa holders were also being exploited, but often do not complain because they knew they would be here only for a limited period.
AUT University Business School PhD candidate Danae Anderson who has done research on exploitative conditions of migrant workers here said this visa category has not been one to have had negative media reporting - so there has been little focus to see if it was working or not.
"Because the visa is short-term and no permanent jobs can be accepted limits the likelihood that the work gained will be decent jobs," Anderson said.
"Of course some workers will be exploited, but the limited capacity of the labour inspectorate to investigate means that the onus will be on the visa holder to report."
Anderson said that for many, the time and effort taken to report and follow through will not be worth it.
"It's really the temporary nature of the visa category and work that can be obtained that starts to feed workers into precarious, dirty and potentially dangerous work," she added.
Utako Neda, 29, came to Auckland from Sendai in October last year as a working holidaymaker hoping to connect with New Zealanders and get Kiwi work experience.
"But the only jobs I could find were as a waitress in a Japanese restaurant and a part-time job with a local Japanese television station," Neda said.
After a year, she has made just two Kiwi friends - both of whom speak Japanese.
Neda, who was a TV assistant director in Japan, is now on a student visa and learning English at a private training establishment.
Another working holidaymaker, Anna Masumoto, 28, is also finding it difficult to make local friends.
"I came here thinking I will make many New Zealand friends, but I am really not sure if Kiwis are interested in making friends with Japanese," said Masumoto, a nail artist.
Immigration New Zealand area manager Marcelle Foley said a 2013 Ministry of Business, Innovation and Employment study found working holidaymakers tend to take a positive view of New Zealand.
The top three source countries for working holidaymakers were India, Britain and China and Japan ranked ninth on the list.
"Surveys indicate that working holidaymakers have a positive net impact on the economy as they spend more money than they earn," Foley said.
"Working holidaymakers are a valuable source of labour for employers, particularly in sectors such as hospitality and agriculture."
Foley said their positive view may generate benefits in the long term, translating into a preference for New Zealand in future travel or business.