Rape Prevention Education executive director Debbi Tolhill said there were "lots" of reasons victims did not report.
"People often don't feel like they will be believed," she said.
"They might be scared of the person who attacked them - they might just be scared.
"Maybe it's because it's a family member and they might be reluctant, thinking of the impact on the rest of the family if they told - that's a reality."
Tolhill said in many cases the victim simply blamed themselves, and felt a sense of shame or embarrassment.
"It might be someone they are in a relationship with, a close family friend and these things make it much harder for victims to report what has happened," she said.
"I think there can also be some concerns about talking to police because once you go to the police, a lot of what happens next is out of your hands.
"Also, it can be very traumatic for people getting through a court case, re-traumatising even."
Tolhill said when someone was subject to sexual violence or abuse, the thought of having to speak about it in great detail, and repeatedly, to strangers could be a terrifying thought.
The thought of having to go through it again in a courtroom before a judge and jury was even more difficult.
"These people are already feeling extremely vulnerable, and they are being asked to talk about quite intimate things," she said.
"They are often challenged and questioned (by the defence) and often the victim feels like they are seen as the instigator - they are questioned about being drunk, they are questioned on the way they were dressed.
"Of course, the cause of rape is not any of that, it is because someone has raped you."
Tolhill said in recent years police have "made great inroads" into making the reporting process easier and less traumatic for sex victims.
"But there's always more to be done," she said.
Tolhill said most victims disclose first to a friend or family member, and that person's response was crucial.
"Often the person doesn't know how to respond, but you need to keep calm and acknowledge then for what they have told you, give them assurance that you believe them," she urged.
"That's one of the biggest things concerns for victims, that people won't believe them or they will get bad reaction.
"Then they become very, very reluctant to go on and seek further help."
Tolhill said anyone affected by sexual violence should, even if they never went to police, seek counselling.
"People deal with it in their own way... and people have to do it when they are ready and it might not be immediately after the assault.
"It may not be a month later or even a year later, but when they are ready there is always help out there."
Sexual violence - why we need to talk about it
Kathryn McPhillips of the sexual abuse survivors support service HELP said the social taboos around the issue had a "massive" impact on victims.
"To get over trauma is to talk about it," she said.
"The existence or rape myth also stops people from discussing it.
"Victims are so vulnerable when they talk about it because it's so harmful, they need to know that the person they are speaking to is going to be gentle, respectful and won't blame them.
"But you can't guarantee that's the response they are going to get, so they don't speak up."
McPhillips said society as whole had a responsibility when it came from removing blame and shame from victims.
"Somebody else has done it to you - you have not done this to yourself," she said.
Challenging rape culture was also crucial, particularly around issues that demean or degrade women in society.
"Sexually loaded jokes in the work place, for example, where people write things off as 'boys being boys' - you need to stand up to that," McPhillips said.
"Just getting fluent in seeing that kind of thing and challenging it, not aggressively, but saying that it's not okay.
"There are a whole bunch of us in this sector trying to change this - but we can't do it on our own.
"We need everybody to be on board."
HELP has been working hard to push that message, particularly to younger people.
They set up the website Dear Em, which aims to educate young people about sexual harm as well as inspiring and empowering them to "look after your mind, body and spirit through the good and bad times".
"It's full of tools for young people about how to get over hard times, ways that they can manage, how to help your friends - and sexual assault is part of that," McPhillips said.
If you're in danger NOW:
• Phone the police on 111 or ask neighbours of friends to ring for you
• Run outside and head for where there are other people
• Scream for help so that your neighbours can hear you
• If you are being abused, remember it's not your fault. Violence is never okay.
Where to go for help or more information:
• The Harbour, for those affected by harmful sexual behaviour
• Help Auckland 24/7 helpline 09 623 1700
• Rape Prevention Education
• Wellington Help 24/7 crisisline 04 801 6655, push 0
• Male Survivors of Sexual Abuse
• Women's Refuge: Free national crisis line operates 24/7 - 0800 REFUGE or 0800 733 843 www.womensrefuge.org.nz
• Shine, free national helpline 9am- 11pm every day - 0508 744 633 www.2shine.org.nz
• Shakti: Providing specialist cultural services for African, Asian and Middle Eastern women and their children. Crisis line 24/7 0800 742 584
• White Ribbon: Aiming to eliminate men's violence towards women, focusing this year on sexual violence and the issue of consent. www.whiteribbon.org.nz
