"There's a very serious allegation made against this man of privilege and entitlement. I'm working with agencies to tackle sexual violence and misogyny."
Senior councillor Darryl Smalley said the city's reputation was being tarnished by the association.
"I think the vast majority of people in York feel exactly the same in protecting the reputation of the city ...
"York is fantastic, we have got a lot to be proud of and one thing we don't want is to be so burdened with this link to Prince Andrew.
"It is important for us as councillors and city leaders that we raise these concerns, just as servicemen and veterans did in their letter to the Queen.
"It's our duty really to make clear that we are very unhappy that despite him not using titles such as 'His Royal Highness' he still somehow retains the title of Duke of York. It seems strange to a lot of us here in York."
The Duke of York will keep his Royal Highness honorific, but will not use it.
Buckingham Palace said in a statement, "With the Queen's approval and agreement, the Duke of York's military affiliations and Royal patronages have been returned to the Queen.