They left about 7.30pm to get food but a "huge argument" broke out in the car, Graves told APNZ.
"Keisha was trying to jump out of the car while I was driving. I was swerving around the road and trying to pull her back into the car. Her clothes ripped. We caused a big scene and police turned up, and they have put me up for assault."
As they waited for police to arrive, the pair sat apart. Other people turned up to support Castle-Hughes and Graves said he left.
He was arrested and spent the rest of the night behind bars until he appeared in the Auckland District Court yesterday and was remanded on bail.
He was banned from contacting or trying to contact his now ex-girlfriend but as he spoke to APNZ outside court, he received a text message that he said was from her.
It said: "How do I drop charges".
Castle-Hughes' agent, Gail Cowan, did not return the Herald's calls yesterday and the actress could not be contacted.
Graves declined to say what the fight was over, saying it was personal, but added: "The police have made it bigger than what it was."
He hoped Castle-Hughes would tell her side of the story to police. "It's still pretty much in the air ... I'm just hoping that by her coming forward and saying [what happened], it will clear things up on the court side."
Graves said he and Castle-Hughes had a history of breaking up and getting back together and this wasn't the largest disagreement they'd had.
"The fact of the matter was that it was public and I had to physically pull her back from trying to jump out of a moving vehicle, which caused a commotion.
"We have had bigger fights."
Graves said he had just moved to Auckland from Tauranga to try to give their relationship another chance. "We have had our ups and downs, like any relationship. I suppose that's between us."
He confirmed they had split following his arrest and doubted they had a future together.
It is not the first time the pair have had a run-in with police. In March last year, Castle-Hughes was taken away in handcuffs from a central Auckland nightclub following a fight.
Graves reportedly came to the defence of his girlfriend when another patron made a derogatory remark about her role in Whale Rider. Neither was charged.
In 2010, Castle-Hughes split with Bradley Hull, the father of her 4-year-old daughter, after seven years together.
