Led by Michael Hooper, Australia controlled the breakdown battle in the second half and Jones suggested that New Zealander Doleman was trying to "even it up".
"I think when you play against 14 men the referee has a significant impact on the game and you've got to be good enough to understand what that is and we weren't good enough to understand what that is," Jones said. "And therefore we paid the price.
"He evens it up. He helps the team with the red card. You look at the history of the game, whenever you get a red card the referee evens it up. It's social reciprocality, it happens, that's normal and we've got to be good enough to handle it.
"That happens in every game of rugby I've seen. The team gets a red card and the opposition gets evened up, because they're nice blokes, referees. I'm not criticising the referees, I'm not using it as an excuse, that's the reality of rugby."
Hill and Swain had been engaged in a running battle before the red card, which also resulted in the England lock being sin-binned for yanking his hair, which Australia head coach Dave Rennie suggested may have been part of a premeditated strategy.
"I'm not sure if it was a team plan, but there was certainly provocation there," Rennie said. "Not just in that situation but also earlier in the game. We'll have a decent look at the footage and work out how we're going to appeal that. We'll have decent look at the card. We'll be seeking clarity around it."
England have now lost five of their seven matches this year under Jones who received the full backing from the Rugby Football Union after another two-win Six Nations campaign.
Jones retains confidence that England are still on the right track as they prepare for the second in a three-Test series in Brisbane where the Wallabies have won their last 10 matches dating back to 2016.
"I'm disappointed that we've lost," Jones said. "The results aren't good enough. I accept that and that's my responsibility. We'll work hard to turn it around.
"We've got a committed group of players, a committed coaching group. We started the game well, we put ourselves in a position to win the game but we just weren't quite good enough on the day to do that."
England had more entries into the Wallabies 22 but failed to come away with points, aside from Ellis Genge's try, until the late rally when the results was already settled. And Jones says their lack of clinical edge is what separated the teams.
"We had opportunities to put pressure on them and we just let them off at various stages," Jones said. "There were a couple of times when we were on their line. There were opportunities to score but we didn't score and, conversely, when they had opportunities to score they scored straight away.
"We led 14-9 and in the first half, we probably should have been ahead by a number of points and we weren't. And those sorts of things, they come back to bite you."
