The letter - signed by 127 senior figures including Sir Al Aynsley-Green, the former Children's Commissioner for England, Lord Layard, director of the Well-Being Programme at the London School of Economics, Dr David Whitebread, senior lecturer in psychology of education at Cambridge University, and Catherine Prisk, director of Play England - suggests children should be allowed to start formal education later to give them more time to develop.
A spokesman for Michael Gove, the Education Secretary, said the signatories were "misguided", suggesting they advocated dumbing down.
"These people represent the powerful and badly misguided lobby who are responsible for the devaluation of exams and the culture of low expectations in state schools," the spokesman said.
"We need a system that aims to prepare pupils to solve hard problems in calculus or be a poet or engineer - a system freed from the grip of those who bleat bogus pop-psychology about 'self-image', which is an excuse for not teaching poor children how to add up."
By law, children must be in school by the age of 5, although nearly all are enrolled in reception classes at 4.
The letter says children who enter school at 6 or 7 - in line with Scandinavian education systems - "consistently achieve better educational results as well as higher levels of wellbeing".
It would mean delaying formal schooling for up to two years, experts suggesting they should instead have play-based activities with no formal literacy and numeracy requirements.
"The continued focus on an early start to formal learning is likely to cause profound damage to the self-image and learning dispositions of a generation of children," the letter says.
The letter is circulated by the Save Childhood Movement, which is launching the "Too Much, Too Soon" campaign tomorrow. It will push for a series of reforms, including a new "developmentally appropriate", play-based framework for nurseries and schools, covering children from 3 to 7.
Wendy Ellyatt, the founding director of the movement, said: "Despite the fact 90 per cent of countries prioritise social and emotional learning and start formal schooling at 6 or 7, in England we seem grimly determined to cling on to the erroneous belief that starting sooner means better results later.
"There is nothing wrong with seeking high educational standards ... but there is surely something very wrong indeed if this comes at the cost of natural development."