To meet the threshold, the Solicitor General's prosecution guidelines say there needs to be enough credible evidence for there to a "reasonable prospect" a jury will convict.
However because many sexual assault cases revolved around the issue of consent, they often boiled down to "he said, she said" and weren't strong enough to put before the court.
Anderson said it was a frustration to police.
"Investigators work tirelessly, passionately," he said. "They get disappointed like any other human when they've toiled away at a certain case and the evidence has been there, or they believe the evidence has been there, and the outcome isn't what people expect."
His view was that many of the cases were not suited to New Zealand's "bruising" adversarial justice system.
"Me personally, I think the system isn't serving our victims well. That's my own view in terms of years of investigating these cases and thinking about this quite deeply."
Anderson suggested a shift away from the adversarial approach for sex cases entirely, to an inquisitorial model might help.
Or - and while he knew defence lawyers would be aghast - he said juries could be replaced with specialist judges, or panels of judges.
"You could tinker around the edges of the system - there could be other things - but would that change biases brought by individual jury members? Probably not."
Convenor of the New Zealand Law Society criminal law committee Steve Bonnar, QC, said most defence lawyers would be strongly resistant to the removal of juries in sexual assault trials.
"Trial by jury is a significant right and an established right and I don't think you'd likely remove that in any case, let alone in cases where people can go to jail for 20 years," Bonnar said.
"We like to think judges are more objective than juries but the reality is that when judges do it day in day out, they're human too ... and there's a risk that judges can lose impartiality as well."
Read more: How thousands of reported rapes disappeared
Bonnar said while he accepted there was a problem with the under-reporting of sexual violence, he said it was important to maintain a standard of evidential sufficiency.
"The other side is that I would say most experienced defence lawyers have dealt with cases where they've been convinced it was a false complaint - and they may be relatively rare - but we do need a system that is robust enough to protect against those things."
Yesterday, Parliamentary Under-Secretary to the Minister of Justice Jan Logie said she expected to take recommendations later this year for a reform package on sexual violence.
Where to get help
To contact police, call 111
Rape Crisis NZ: 0800 883300 or for a local branch detail, visit http://www.rapecrisisnz.org.nz/