"It may mean putting them in some nice modern building which no doubt satisfies the building inspector but is simply not home to them."
The judge, who has been president of the High Court's family division since 2013, told a conference of local authority care leaders that social workers should recognise when patients preferred to be happy than safe.
He added that some professionals want to "rescue" patients from "unsatisfactory, unsuitable, rather squalid" surroundings which in some cases they have "literally been born in and lived the whole of their lives".
He said: "Merely demonstrating that if you let that person go on living in that house there is a foreseeable and appreciable risk that one day a neighbour or carer will come in and find them with a broken neck at the bottom of the stairs - is that sufficient justification for making them leave, if it is going to make them thoroughly miserable?"
"It is no good just saying most people would prefer to live longer in nice new accommodation without breaking their neck; some people would not," he told members of the Association of Directors of Adult Social Services.
Sir James, who is due to retire next year, said he had felt "personal outrage" at the "inhumanity" of older couples being separated and put in different care homes.
He said he believed that most people would prefer to stay with their partner instead of being moved to live somewhere "safer".
There is evidence to suggest that a disproportionate number of people die a short time after being moved to a care home.
Figures from an LSE study carried out in 2011 showed that elderly people have a 55 per cent chance of surviving their first year in a care home. The likelihood then rises to 70 per cent for the second year and falls after that. The average length of survival was 15 months.
Sir James added: "We do know that people die of a broken heart. I have read of cases where one person died and then the other dies a couple of days later. How long do people last if they are uprooted? A very short time."
Broken heart syndrome, or stress cardiomyopathy, is a real condition caused by shock such as bereavement or separation in which the left ventricle changes shape, weakening the heart muscle.
It is normally survivable but can be fatal in elderly people or those with a pre-existing heart condition.
A 2014 study published in the journal JAMA Internal Medicine found that the proportion of people who had a heart attack or a stroke in the month after a loved-one died was double that of a control group who had not been bereaved.
Every year cases are reported of elderly couples who die within minutes of each other. Earlier this year Leicester couple Wilf and Vera Russell died four minutes apart after 71 years of marriage.
Caroline Abrahams, charity director at Age UK, said: "In some cases partners may have different care needs but except in rare situations couples who want to stay together should be allowed to so.
"When councils arrange care for an older person they have a legal duty to ensure it meets their needs. They have to take into account their psychological needs, as well as how any arrangements will respect their right to family life under the Human Rights Act."
What to look for when choosing a care home
Pay attention to detail. Are residents' fingernails clean, are their clothes stained, do they look underweight, do their clothes match? If there is a garden, is there anyone out in it?
Ask questions. What's the home's policy of anti-psychotic drugs and sleeping medication? How often do residents have a bath or shower? Do they go out everyday and what are the resources for this? If there is a garden, is there anyone out in it?
Talk to other families. Ask for your details to be passed on for personal references.