New Zealand had been "so blessed" with freshwater resources that it had been "a bit slack" about managing it, he said.
"Collectively we have got a job to step that up, both from a water quality and also a water quantity perspective."
Much of the blame for declining water quality in New Zealand has been pointed at intensive farming, which has contributed to increases in nitrate and E.Coli levels in New Zealand's waterways.
Smith, however, said blaming farmers alone was unfair. New Zealand needed to avoid the trap of dividing rural and urban Kiwis if it was to address its water quality problems, he said.
He cited the Ministry for the Environment's reporting on freshwater, which showed that the average E.coli level in urban areas was 400 parts per 100 ml. In farmland, the average E.Coli level was 180 parts per 100ml, in forestry land it was 50 parts and in national parks it was 20 parts.
"I give those figures because it is too easy for urban New Zealanders to point the finger at rural New Zealand and say this is all your problem," Smith said.
"My message to them is have a look in the mirror."
If people in cities were going to demand that farmers needed to improve their environmental record, they had to "confront the fact" that old, leaky sewerage pipes or diffuse pollution from urban areas also contributed to poor water quality.
Smith conceded that the relative area of urban waterways that needed improvement was small compared to rural waterways affecting by farming.
"But if New Zealand is to tackle this, in my view it has to be done in a fair way. And both need to do the investments and costs for improvement."
In February, the Government set of goal of making all rivers safe to swim in by 2040.
That means all waterways deeper than 40cm will have to meet the safe standard for E.coli contamination 80 per cent of the time.
The Government has denied that it weakened the definition of a "swimmable" river in the new policy, though an independent assessment by Niwa recently confirmed that the standard was "less stringent" than the one it replaced.
Smith said today that regional councils which had criticised the freshwater policy as "too soft" could set their own, more ambitious targets.
During a question-and-answer session, a member of the audience challenged Smith on whether his water quality standards had been understood by the general public.
Smith said there had been an "extra level of politicisation" around freshwater standards because it was election year.
Earlier, he said that freshwater management was one of the political issues that made him nervous, because it was complicated and hard to discuss in soundbites.