What else it should deprive them of is not so black and white.
Some would argue prisoners getting an education inside is unfair - why should they earn qualifications when those on the outside struggle to pay their way through study?
Yet wouldn't we rather prisoners use their time inside to get qualifications and advance themselves in the hope that makes them less likely to reoffend once on the outside?
Because, aside from a handful of exceptions, all prisoners will eventually be released.
Imprisonment is punitive. But it must also have a rehabilitative aspect, otherwise what's the point?
Until 2010, only prisoners serving terms of more than three years were prohibited from voting.
That law changed in 2010 to cover all prisoners.
The High Court yesterday issued a "declaration of inconsistency" after a complaint from five prisoners who argued the extension of the ban infringed the right to vote in the Bill of Rights Act.
In his decision, Justice Paul Heath said the purpose of his declaration was to draw the public's attention to the fact Parliament had enacted legislation inconsistent with a fundamental right.
Our country's Bill of Rights applies to all. If a low-level prisoner is sufficiently interested and engaged in politics that they want to have their say, I say let them.
They are still part of our society, whether you like it or not.
The declaration by the High Court does not change the law.
But it's a strong statement from the judiciary that adds significant weight to the argument of those prisoners campaigning for change.
We now wait to see what, if anything, the response will be from Parliament.