"It is frustrating," said Mr Grant. "I had to spend the whole day taking it down."
The scoreboard cost about $8000, including installation, and was funded by a grant.
The union is insured, with a $1000 excess, and it was hoped the board could be repaired.
Mr Grant said he wasn't certain whether the scoreboard would be fixed in time for this rugby season, which was on the brink of getting under way.
"It's more about the amount of vandalism that's being done," said Mr Grant.
"Now it's the scoreboard and there has been a little bit of graffiti up in the grandstand.
"We and the district council need to get together and maybe do more about putting cameras in and lighting in that area," he said. "It's just ongoing."
In other attacks, doors had been broken and a window was smashed.
On those occasions alarms were thought to have activated, scaring the vandals off, because nothing was taken.
Senior Sergeant Warwick Burr said Masterton police wanted to hear from anyone with information about Sunday's incident.
People can call Masterton police on 06 370 0378 or anonymously phone Crimestoppers on 0800 555 111.