Hughes detailed a list of breaches of the State Service Code of Conduct, including potentially illegal recordings of insurance claimants, public sector employees accessing the New Zealand Transport Agency database for TCIL and spying on the Green Party and iwi.
Separately, police released a report which found 16 officers over 15 years had given information to TCIL, and another four had moonlighted for private investigation companies.
New Zealand Institute of Professional Investigators Ron McQuilter, who is managing director of private investigation firm Paragon, said he did not have concerns about the conduct of the wider private investigation industry.
"We know the rules, and the rules are pretty transparent really. The issue in the report is really the conduct of Thompson and Clark," McQuilter said.
"Most PIs, we know the rules. The findings of the report was really that government departments need to tighten up in how they instruct PIs and basically not let the tail wag the dog," he said.
Hughes apologised to people whose privacy had been breached by state servants or their contractors.
He revealed that he had laid a complaint with police today over TCIL potentially unlawfully recording closed meetings of insurance claimants discussing legal action against Southern Response.
Hughes also found that two former Ministry for Primary Industries employees engaged in secondary employment with TCIL and had accessed the NZTA database on behalf of TCIL.
Hughes also looked at the Ministry of Business, Innovation and Employment and found that its service "as a whole" breached the Code of Conduct by failing to maintain an appropriate level of objectivity and impartiality.
"MBIE's management of its regulatory responsibilities in the petroleum and minerals areas through the creation of Operation Exploration, showed evidence of poor regulatory practice," Hughes said.
He was scathing in his criticism of government agencies' actions against New Zealanders exercising their democratic rights through TCIL reporting on the activities of groups such as Greenpeace, the Green Party, the Mana Movement and some Northland East Coast and Taranaki iwi groups, which he said were treated as a "security threat".
"This is an affront to democracy, and government agencies should have challenged TCIL's definition and treatment of issue-motivated groups," he said.
"I am clear that it is never acceptable to classify a person or group of people as a security threat just because they lawfully exercise their democratic rights, or use that as justification for gathering information."
Hughes said he had issued new standards which agencies would need to comply with by April 30 next year.
Hughes said that in addition to laying a complaint with police around the potential unlawful recording, he had also lodged a formal complaint about TCIL's conduct with the Private Security Personnel Licensing Authority.
TCIL had been removed from the Government Procurement Panel.
Green Party co-leader Marama Davidson said the report showed government departments colluded with private security firms working for oil and gas companies to spy on New Zealanders.
Thompson and Clark director Gavin Clark said it would consider all aspects of the report but he accepted the findings that some processes around how and when some work was carried out could have been more stringent.
"An internal review of this is already underway but we maintain that there is a legitimate place for the work that we do, to help agencies keep their people safe," he said.
The police review identified 16 instances of information being given to Thompson and Clark.
It also found four instances of police officers undertaking unauthorised secondary employment with private security companies.
But it found no evidence that police engaged, tasked or directed any external security consultants to undertake surveillance (whether lawfully or unlawfully) on behalf of police, and there was no evidence of criminal or corrupt behaviour by police staff.
What sparked the inquiry?
In early 2018, questions arose about whether Southern Response Earthquake Services had used Thompson and Clark Investigations Ltd to surveil insurance claimants.
Who was reviewed?
131 government agencies, subsidiaries and crown research institutes. Police conducted their own review for legislative reasons.
What was looked at?
How and why government agencies used external security consultants, whether they were used for surveillance and the relationships between government employees and agencies with Thompson and Clark.
What has happened?
A police complaint has been laid over the actions of a Thompson and Clark contractor in recording closed meetings of insurance claimants; a Serious Fraud Office probe is ongoing into the actions of two former Ministry for Primary Industries employees in accessing the NZTA database; Thompson and Clark has lost its place in government procurement; State Services Commissioner Peter Hughes has issued new standards to improve transparency and consistency across government agencies.
