More than two dozen 10m² buildings have recently appeared on one Freshco orchard block, despite Hastings District Council not receiving, nor granting resource consent.
Neighbours say the developments, both large scale and small, will devalue the neighbourhood and their homes, create noise, and even inspire criminal activity.
A meeting called by residents to discuss the issue was held on October 14, and drew a crowd of 60.
Newly elected Hawke's Bay Regional Councillor Jerf van Beek, who lives in the area, chaired the meeting, and Heretaunga Ward councillor Ann Redstone attended.
Residents' spokesman Robert Griffiths told Hawke's Bay Today after the meeting "there doesn't appear to be anything we can do".
His property backs on to that owned by RJ Flowers.
He says the problem isn't RSE workers - they are a much-needed resource during the fruit harvest season, he believes, and one orchards can't do without.
"We're not concerned about what they're doing, it is where they're doing it.
"It's ridiculous building these sorts of buildings on good productive land."
Griffiths said crime goes up during the fruit harvest season, and while it is "not necessarily" RSE workers, "crims use that as a mask".
Inspector Rob Sum said police do not consider seasonal work to be a significant driver of criminal activity.
Redstone, who doesn't live in the area, attended the meeting to "explain the process" to residents.
She said accommodation was needed in Hawke's Bay because "we are desperately short of pickers".
She said the consent for the RJ Flowers development has only just been lodged and is "not guaranteed".
It would initially be for around 26 people, with the possibility of housing up to 96 people in the future, she said.
And "rigid rules" would be in place, including a curfew of 8.30pm, as well as "wrap-around care".
Griffiths questioned the reality of 96 people complying with an 8.30pm curfew, particularly in the summer.
Over the past three months, Trotter Rd resident Stephan Roess has watched each one of the roughly 26 10m² cabins begin to dot his neighbouring property.
He says it is like an "extra village".
He said they already have to deal with spray drift, helicopters flying above, frost fans and boy racers.
"What more? It is a lot to deal with and I just think it is enough."
Hastings District Council group manager planning and regulatory services John O'Shaughnessy said a resource consent had been lodged for seasonal worker accommodation for 96 people at 117 Twyford Rd, but the council needed more information before consulting the community about it.
He said resource consent will be required for FreshCo's 10m² buildings on Trotter Rd, and council has not yet received an application for this accommodation.
Council staff have met three representatives of the community and have received two letters outlining their concerns about the proposal, O'Shaughnessy said.
"There are numerous small accommodation facilities for seasonal workers located in orchards throughout the Plains Zone.
"As far as council is aware these have not created any problems or generated any complaints."
Freshco did not respond to repeated requests for comment by Hawke's Bay Today.
RJ Flowers referred questions to Hawke's Bay Labour Governance Group chairman Garry Jones.
Jones said Immigration Minister Iain Lees-Galloway provided the industry with several challenges, including to ensure the industry wasn't competing with urban-based housing.
"That's why, across New Zealand, but certainly in Hawke's Bay there has been a strong push to move those workers into purpose-built accommodation."
He said there is about $35m worth of developments occurring across the region, and looking to house 1750 RSE workers.
Last season, there were 4400 RSE workers in Hawke's Bay. In 2020 it is hoped 5360 will work in the region.
An outcry over a lack of housing in Hawke's Bay has prompted Hastings District Council to add a variation to its proposed District Plan, that would allow more seasonal worker accommodation facilities to be built.
It is currently open for further submissions and will close at 5pm on October 29.
Mayor Sandra Hazlehurst said all new developments would be required to comply with noise limits and the buildings would be relocatable to enable industrial use of the site once the accommodation facilities were no longer required.
Limits around this would also ensure minimal encroachment on quality growing soils.