Cave, his music teacher, arranged for him and another student to travel to his bach on Waiheke one weekend in 1982, the man told the court.
"Being a country boy, I had no money, no income, absolutely no way of getting a ferry to Waiheke Island," he said. "I have the assumption that we were led there."
The day he arrived at the bach at Waiheke's Onetangi Beach, the two students were "plied with alcohol" and he fell asleep.
"I woke up, didn't know where I am or where they are. Don't know what's going on. I walk into the bedroom and … that's when I see Leonard Cave naked."
The man alleged that Cave was touching the other student, who was on the floor. Cave seemed surprised when he entered the room, he said.
"It ended immediately. He was caught in the act".
The man said he then "somehow became part of … what was happening" and ended up lying naked on top of Cave.
He and the other student left the bach the next day and never spoke about the incident with each other or Cave.
Asked by Crown prosecutor Jacob Barry why he reported it to police in 2020, he said: "Because it suddenly dawned on me that he had done this to other people."
Under cross-examination, defence lawyer Warren Pyke challenged the man's recollection of events.
The man agreed that his memory of the incident was "very hazy", but said that he remembered the alleged abuse very clearly: "I know what I saw."
Pyke challenged the man on why he joined Cave on the bed.
"I was drunk," he responded. "I don't know how it happened. So I can't answer that."
The man rejected the defence's suggestion that he voluntarily got into the bed with Cave. He said Cave had got him drunk and coerced him.
"He got me drunk. That's how he got me on the bed. He didn't drag me on the bed…. He led me there."
The other former student involved in the alleged incident also gave evidence today.
He said he had a "vivid" memory lasting between 30 seconds to a minute of Cave abusing him in the bedroom of the bach.
"I one hundred per cent remember him putting his hand down my pants."
Pyke asked whether he ever recalled being on the floor of the bedroom and being touched by Cave - as alleged by the other student. He said he had no recollection of being on the floor or being touched by Cave in this way.
The trial continues this afternoon.