In a combative statement in Downing Street, she admitted that the two sides were still a "long way apart" and at an "impasse".
She warned the leaders were "making a mistake" if they thought she would back down on her red lines, and said failure to compromise would poison relations completely.
"Throughout this process I have treated the EU with nothing other than respect. The UK expects the same. A good relationship at the end of this process depends on it," she said.
"It is not acceptable to reject the other side's proposals without an explanation."
The pound fell sharply as May delivered the angry retort, as markets assessed that the chances of "no deal" have risen. She was condemned by Remainers and Labour - but Tory Brexiteers said she was right to stand her ground against Brussels.
The premier did offer something of an olive branch by unilaterally guaranteeing the rights of around three million EU citizens already living in the UK.
The high-stakes clashes with the EU have threatened to destabilise May further ahead of an explosive Tory conference in just 10 days' time, where she could face an all-out mutiny from Eurosceptics over the Chequers plan.
Support for May's stance in the Cabinet is said to be weakening, with Home Secretary Sajid Javid and Aid Secretary Penny Mordaunt among those thought to be edging towards backing looser Canada-style links with the EU.
There are signs that the EU recognises that the jibes could have gone too far, with EU commission president Jean-Claude Juncker saying the two sides have to be "careful" about engaging with each other "like two loving hedgehogs".
Irish PM Leo Varadkar also stressed that he still believed there will be a deal, and suggested that elements of May's Chequers plan could be included in a future trade package.
In a statement designed to shore up her own political position as well as rebuking the EU, May accused the EU of offering the UK two "unacceptable" options.
She said one was join the EEA and accept free movement and all EU rules without having a say in making them.
The other option was a looser Canada-style free trade agreement - but allowing Northern Ireland to stay within the bloc's customs jurisdiction.
On the latter she said: "It is something I will never agree to - indeed, in my judgement it is something no British Prime Minister would ever agree to. If the EU believe I will, they are making a fundamental mistake.
"Anything which fails to respect the referendum or which effectively divides our country in two would be a bad deal and I have always said no deal is better than a bad deal."
Earlier, Brexit Secretary Dominic Raab lashed out at "dark mutterings" in Brussels about "Northern Ireland being the price Britain will have to pay" for leaving the bloc - a reference to demands for the province to stay under EU customs rules after Brexit.
"That is not going to happen," he told the BBC's Politics Live. "Not with this PM, and not with this parliament."
Raab accused the EU of "yanking up the handbrake" and leaving a "serious question mark" about whether they were taking the talks seriously.
He branded EU council president Donald Tusk "unstatesmanlike" for tweeting a picture of himself choosing cakes with May and the message, "Sorry, no cherries".
"We've been rebuffed on our plans without any coherent explanation as to why, there's these sort of rather dogmatic pleas to the single market unity but our plans were very carefully crafted around that," he said.
He added: "We're going to hold our nerve, stay calm and keep negotiating in good faith.
"We've revved up the motor of these negotiations, I've been out there a lot more frequently to get motoring, to make progress and the EU have just yanked up the handbrake and for the negotiations to go forward they're going to have to take their hand off the handbrake."
He said May was "a very stoic, resilient character".
The British response reflected anger from both Brexiteers and Remainers at the sabre-rattling behaviour from leaders in Salzburg yesterday.
The DUP, which is propping May up in power, welcomed her assurances on the Northern Ireland border issue, saying she was standing up for the UK's interests.
Nigel Dodds, the party's Westminster leader, said: "You only need checks between Northern Ireland the rest of the United Kingdom if you are having different regimes operating and different rules being imposed on Northern Ireland compared to the rest of the United Kingdom, that is unacceptable, the Prime Minister has made that clear today."
Tory MP Jacob Rees-Mogg, who chairs the Brexiteer European Research Group, (ERG) praised the PM's "strong and forthright" speech - but urged her to revert to pushing for a Canada-style deal.
He said: "The EU is not acting in good faith. This makes the Prime Minister's task harder and she is right to remind them that no deal is better than a bad deal.
"The Prime Minister is also right to protect the rights of people from EU member states living in this country regardless of whether there is a deal.
"Likewise her determination not to erect barriers between Northern Ireland and the Irish Republic is sensible and something many have previously suggested.
"However, there is still no reason to suppose that Chequers can work either for the UK or the EU.
"It is time for the Government to start putting forward as its plan a Canada-style free trade agreement for the whole of the UK."
Scottish First Minister Nicola Sturgeon described May on the Brexit negotiations as "dreadful" as she branded the Chequers proposals a "dead duck".
Sturgeon said a no deal or "no detail" Brexit was not acceptable following the Prime Minister's update.
She said: "The Prime Minister's statement was dreadful. The EU's view of the Chequers plan was bluntly and clearly expressed in Salzburg yesterday, but should not have come as a surprise to the UK Government, given that it was not a new position.
"Clearly, the Prime Minister has not been listening - Chequers is a dead duck but if her tactic now is to try and double down on those proposals and then seek to blame the EU for a no deal outcome, then she will do huge damage to all of those she is supposed to serve.
"What the latest development shows more clearly than ever is that the only remotely workable way for Brexit to happen is for the UK to stay in the single market and customs union."
