Almost half - 46 per cent - of all Leave voters said this is the case, with some 28 per cent of Leave backers suspecting pro-EU involvement of the secret services.
On June 20, voter Kenneth Priestley made a Freedom of Information request on whatdotheyknow.com, asking why the pencils aren't replaced with black pens to avoid potential fraud.
Jordan Lawrence, representing the Electoral Commission, responded: "Pencils have been used partly for historic and partly for practical reasons."
Though pencils are provided within polling stations, there is no legal obligation to vote with them and using your own pen will mean ballot papers will still be counted.
The Electoral Reform Society echoed the warning, telling a voter on Twitter that "you can bring your own pen/marker/sharpie".
Voting rules are relatively relaxed and mean that people can theoretically vote with a tick rather than a cross if they wanted to.
Others have suggested that people should press extra hard when drawing their cross to ensure that it is still visible even if it is erased.
The Electoral Commission has welcomed people bringing a pen and said that observers watch counts to ensure that they are kept secure.
"By tradition, pencils are available in polling booths for voters to mark their ballot papers," a spokesperson said. "If a voter wishes to bring their own pen and use that, it's fine."
A video posted on one woman's Facebook page shows a policewoman taking down notes after she apparently offered her pen to voters.
"I'm lending a pen...I have never told them or asked them which way they are voting," Jacqui Jackson said.
"I was asked to come here and hold dogs today and offer people the use of a pen."
- Additional reporting: NZ Herald
