"They often declined because they didn't like the neighbour, or they didn't think the fence was good enough and there is no repercussion for people who decline houses.
"I'm sorry, but that sort of thing has to stop."
Mrs Bennett said people often went onto the waiting list because they wanted a particular house.
"They've seen it over the years. They want that house so they will decline until that house becomes available."
That might be all right when the list was very short, but when there were thousands on it, they were potentially taking opportunities away from other people.
However, the biggest reason given for declining a house was because tenants thought the community was unsafe — 18 per cent; outside the preferred area — 12 per cent; and distance from essential services — 11 per cent.
Mrs Bennett's spokesman said she knew of a case in which a family was living out of a car and who declined a house because it was on a hill and they had wanted a flat section.
At present, there are about 4,500 on the waiting list, but with movement in and out of the list, the number in any single year is a lot higher.
Of the 4,500 on the list, 1,200 are already in a state house which is deemed unsuitable for their needs.
Mrs Bennett also revealed that since the Government began reviewing state house tenancies, 264 tenants who had been paying market rents had moved into the private sector and 12 had bought their own house.
Housing Minister Nick Smith told the same conference workshop that the Government was set to beef up the powers to prosecute private landlords in some circumstances without the need for tenants to go the Tenancy Tribunal.
It could apply to tenants with major problems such as those in boarding houses who might not have the capacity to take a tribunal case.
He was also looking at ending the monopoly of councils to issues building consents, as is done in Australia where certain engineering firms for examples could be licensed to grant consents, with commensurate liability.
About 700 delegates have registered at the conference, the first held since National won a third term in September last year.