Fitzpatrick said interpretations around scrum laws were a particular bugbear.
"The amount of time we waste at scrums," he said. "I'd like it to be more black and white so there's not so many different interpretations."
Sir Clive Woodward was on the receiving end of the most infamous scrum penalty of them all when, in the 2003 World Cup final, a dominant England scrum was inexplicably penalised by Andre Watson in the final minute, allowing Australia to draw level.
All ended well on that occasion but Woodward echoed Fitzpatrick's call for clarity.
"When there is two positive coaches and a good referee there is not a lot wrong with the laws, but if scrums are going to go down, down, down and teams are going to take an age at the lineouts it kills the game.
"With my job at the British Olympic Association I'd talk to athletes and coaches and they like rugby but they don't understand it. You know it as a coach. The referee will blow his whistle and you'll say, 'What was that for?' If the coach doesn't know the public doesn't so we've got to simplify the game. You don't have to change a lot but we've got to make it a lot clearer how the game is refereed."
Both believe there needs to be better pathways for young referees who leave the ranks early but want to retain ties to the sport.