A new special housing area for up to 90 apartments is approved - but the tenants will potentially have to tolerate years of a quarry being filled up at their back door.