The men have name suppression. Their allegations including unnecessary and prolonged use of manual restraints and solitary confinement.
Dr Ellis told the court this morning that the process of restraint and putting the men in seclusion could turn violent.
One man allegedly suffered a fractured rib in an August 2003 incident, and in May 2014 fractured a finger while being restrained.
There was a lack of natural justice, Dr Ellis said, citing the example of one of the men being ordered into seclusion after allegedly assaulting a staff member.
"You say if someone punches a staff member, that staff member has to afford natural justice before taking any action?" Justice Rebecca Ellis asked.
"They should get someone else involved ... it could have been self-defence, you don't know," Dr Ellis replied. "You shouldn't have the alleged victim determine what's going to happen."
The five defendants - the Attorney-General, the District Inspector, the Mental Health Review Tribunal and Waitemata DHB and Capital & Coast DHB - all deny the claims and will present their case later in the hearing.
The men all have intellectual disability issues, and two are autistic. They were all committed as special patients after coming before the courts on violence charges around 15 years ago, when they were found unfit to plead.
Two of the men remain in secure care, with another living in a residential rehabilitation service home.
Dr Ellis outlined other concerns about the men's treatment today, including that they lacked access to therapy, and that rights such as the ability to make phone calls were removed as a form of punishment.
He said patients in forensic mental health facilities had the right to have sex, and should be able to arrange visits from partners or sex workers. The question of whether sex between patients should be allowed was more complex, he said, but sex education should be provided.
The men were treated differently because of their intellectual disability, Ellis said, and were detained indefinitely, with their special patient status reviewed on an ad hoc basis, without criteria for terminating continued detention being provided to the men.
The length of detention as a special patient is disproportionate to the gravity of alleged offending, Ellis said -- it exceeds the maximum penalty for the charge had the men been fit to plead.
A number of European countries took into account the length of imprisonment special patients would have been subject to, had they been fit to plead.
Ellis questioned why the Attorney-General had ordered one of the men to be detained, saying the judicial, and not the executive, branch should have ordered the detention in such a case: "Effectively, it's Guantanamo Bay -- the executive detains indefinitely."
The legal action was arranged with the help of their litigation guardian, Colin Burgering, who works for the Justice Action Group.
A litigation guardian is authorised to conduct proceedings in the name of, or on behalf of, an incapacitated person or minor.
The case is set to take six weeks.