"We realised women just have a harder time in business, and especially in tech, because there's a magnifying glass on them."
And the difference in treatment they received was drastic and instantaneous, with emailed queries that would normally go unanswered for days attracting a swift - and polite - response.
"It was like night and day," Dwyer told Smart Company, detailing how the (mostly male) developers and graphic designers they'd enlisted to help build the app were initially rude, evasive and condescending when asked for an update on the progress of their work.
"It would take me days to get a response, but Keith could not only get a response and a status update, but also be asked if he wanted anything else or if there was anything else that Keith needed help with."
When Keith was at the keyboard, "suddenly everyone was dropping everything to make sure that they were responding and keeping him happy ... It was very odd."
In one case, she said, a developer started an email with "Okay, girls ..." while another had tried to sabotage the project by deleting content after Gazin refused to go on a date with him.
Contractors always referring to Keith by name, while many did not bother to use the women's names in email correspondence.
The pair even gave Keith a tech-bro personality, describing him in a blurb as a "dude's dude" who had played football in college, was devoted to his wife and he couldn't wait to be a dad. "He doesn't really understand Kate and I," Gazin wrote. "But he's been happy to help us with our project before we find husbands."
They even created a fake Twitter profile for Keith, describing him as Witchsy's Business Director.
It is not the first time that an online gender switch has highlighted a startling change of email etiquette.
Back in March, resume writer Martin Schneider accidentally used his colleague Nicole Hallberg's signature block, and the pair decided to swap for a couple of weeks to see what would happen.
Scheider said in a podcast about the experiment the client suddenly became "rude, dismissive, ignoring my questions, telling me his methods were the industry standards (they weren't) and I couldn't understand the terms he used (I could)".
Hallberg described her side of the equation as being "like getting a vacation".
"Everyone thought I was a lot smarter, instantly," she said.
"It's like I just woke up and was better at my job - I didn't have to prove anything, I didn't have to argue with clients, I didn't have to deal with the mansplaining and little digs and the little subtle sexist comments. No one was calling me 'sweetie' any more, that was great."