In the firing line is London's Walden House, which is currently leased to Westminster Council and is home to 40 families and a total of 141 people, including eight with disabilities.
At the moment, the social housing costs £600 a month for a two-bedroom apartment.
If the billionaire's plan proceeds, by 2023 the block would be replaced by a mix of more up-market dwellings, some social housing and businesses, The Mirror reports.
Residents are able to apply for new homes, but many occupants — some of whom have lived at Walden House for four decades — worry they could end up on the street due to long waiting lists for public housing.
A group of residents who would be affected by the redevelopment have penned a Change.org petition against it, attracting more than 100,000 signatures.
In the letter, the residents claim demolishing the building will "destroy our community and uproot us from our homes".
"The Duke of Westminster owns more land than the Queen. He is the richest landlord in the UK with an estimated wealth of £9 billion, which at the age of 28 makes him the richest person under 30 in the world. He does not need to do this to us," the letter reads.
"We are a community that have been living in this area for over 40 years. Young families who have children that attend the local church school, elderly residents that should not be worried about losing their homes but rather be enjoying their retirement, single parents, and families that have been living here for generations.
"We are a community, we live amongst friends and family and now face the possibility of having our community fragmented and destroyed."
The petition argues residents will not be able to afford alternative accommodation in the area, which means they are "facing homelessness and the break up of our community".
"There is nothing wrong with these beautiful buildings — Grosvenor just want to make more money. This is simply about putting profit before people," it claims.
"This is valuable land, and we are seen as getting in the way of building high-rise luxury investment flats."
Both the Duke and Westminster Council claim the demolition has been pushed back from 2021 to 2023 to allow residents more time to resettle, and the council also told the Mirror citizens would be "offered suitable alternative council accommodation within Westminster".
But that hasn't stopped the project from being slammed by signatories of the petition, who have described it as "disgusting" and "morally filthy".
Who is Hugh Grosvenor?
Widely considered to be one of England's most eligible bachelors, the 28-year-old instantly became a multi-billionaire after inheriting his father's title and wealth in 2016 over his older sisters Lady Tamara van Cutsem and Lady Edwina Grosvenor due to England's primogeniture custom that favours male heirs.
He previously worked as a marketing representative for a company that turned coffee grounds into biofuels, and his family have close ties with the royals — his father was friends with Prince Charles and the Duke is a godfather to Prince George.
According to Forbes, "the Grosvenors have owned some 300 acres of Mayfair and Belgravia for generations" — meaning they control some of the choicest real estate in London.
