But everyone else was losing out, he said.
"It's all very well to say tradesmen might not have that much money themselves, but it hits people who are on low salaries and have their taxes deducted at source and can do nothing," Milne said.
"It really does annoy me - it's the nudge-nudge, wink-wink stuff. A campaign that says we no longer find this acceptable will make it difficult for tradesmen to walk up and say you can either pay this much in cash or more by card. I think you could wipe it out reasonably quickly."
The campaign should model itself around the drink driving campaign, he said.
"My advice to anyone offered that sort of deal is to refuse," Milne said.
"I don't see what's so bloody clever about a tradesman saying you can pay me cash and I won't pay tax on it. Bugger them."
Master Builders chief executive Warwick Quinn said under-the-table deals were not only illegal, but they also put customers in a tight spot.
"If there's a dispute - and a scary number of jobs end up disputed - there's no remedy," Mr Quinn said.
"And it's just for the sake of trying to save $5 an hour. Especially at this time of the year, the work is so scarce out there - you have to make sure the Is are dotted and the Ts crossed.
"We say all jobs must have a written contract. If there's a problem, how do you resolve it with no contract and no money theoretically changing hands? It gets quite messy."
Revenue Minister Peter Dunne said the Inland Revenue Department was increasingly targeting tradesmen and small businesses in its campaign against tax avoidance, which had recovered $170 million in 18 months.
"It's very hard to be precise about the size of the black economy, but we think it's substantial," Mr Dunne said.
"I've heard some quite fanciful estimates - like 20 per cent of the economy. It's very difficult to quantify but it's significant by any measure ... We're trying to crack down where we can."
The tax avoidance measures were expected to raise $750 million over four years - "which is a big sum, and it's probably only a proportion of what's out there", he said.
"It will have made a big dent in it, but after four years I would be very surprised if the Government at the time didn't want to reinvest in the area."
But stricter enforcement needed to be accompanied by changes in community attitudes, he said.
"Part of the problem has been the New Zealand psyche to put one over the IRD with 'mates rates' or nod-and-wink jobs. That's fine for those people, but the revenue lost is made up by everyone else - we all end up paying a little bit more," Mr Dunne said.
CASH ECONOMY
When tradesmen offer you15 per cent off they avoid ...
* 15 per cent GST for labour.
* 28 per cent company tax.
...and could be getting...
* 15 per cent GST exempt for materials.
Tax avoidance measures:
* $170 million of black market activity has been cracked down on in 18 months.
* $750 million is what it is expected to raise over four years.
* $5.70 for every dollar invested is being recouped
The size of the black economy
* At least $750 million.
* As high as 20 per cent of GDP, or $40 billion, according to a "fanciful" estimate.
