Time said one person wrote to him saying "go and kill yourself and I hope your daughter is raped".
He said as much as that hurt, he was not taking it personally.
"When people are emotional or upset about something they say things that they really don't mean and that's how I've taken it."
Time said he thought of the barbershop as a place for men to get together and talk about some of the things they bottled up.
He said it could help prevent serious issues like men's suicide.
"I probably shouldn't be doing what I'm doing because having women come into the store to get men's haircuts would help our revenue but my vision outweighs that. It's not about the money for me."
But speaking on Newstalk ZB's Friday Forum, Consumer NZ chief executive Sue Chetwin said the idea of rejecting a woman from a barbershop was madness.
"The fact that he thinks that men might go in there for counselling or to chat about boys things, he's living in the past, he needs to move into the real world."
Chetwin said going to a barber was different to something like a women's only gym.
"Going to get your hair cut at the barber, it's going to take about 15 minutes.
"There are woman barbers so I think that sort of negates that and also if you think that you're going in there to talk meaningfully about your life then women actually do exist in life and they're around all sorts of places."
Also speaking on the panel commentator Mark Unsworth said he thought it was a silly move to refuse the woman.
"If there are people in there who are going to get upset that there's a female sitting in another chair when they're having a haircut, then it's a bit strange to me."
However, Unsworth said the abuse Time received was disproportionate to the offence caused.