But Prime Minister Jacinda Ardern and National Party leader Simon Bridges supported the trip, saying it promoted trade just before the CPTPP trade deal kicks in at the end of the year.
Mallard said he would have preferred to watch the game on TV at home.
"Long-range air travel has no romance for me. I think about 1984 I got sick of catching planes ... Because I couldn't get there directly, I had to go via Hong Kong. It took me 20 hours, leaving midnight on Thursday.
"If someone offered me a trip to Japan where I was on the ground for 48 hours over a weekend and it was free and had no trade benefits, I'd say, 'Thank you but no. I'd rather watch it on TV.'"
Seymour said Mallard and Brownlee should repay the money.
"They're taking the mickey out of the taxpayer. If you think this was really about diplomacy, then you probably think that Gerry and Trevor are running on for the ABs against England next week.
"Of course it's just a jolly. It's a huge waste of taxpayers' money, but it's also an insult to say this was about diplomacy. I mean, come on. When you go all the way to Japan and there happens to be an All Blacks game on at the same time - the NZ public are not stupid.
"To say this was some sort of a hardship, if he had gone economy class, I might believe him."
Mallard shot back, referring to Seymour's stint on Dancing With The Stars.
"There are dozens of NZ firms who thought they could get a major benefit from bringing their contacts in.
"I'm prepared to go with New Zealand companies' judgement rather than the judgement of members of parliament who prefer Dancing With The Stars to attending here."
Though former Speaker David Carter said he would not have gone on this trip to Japan, Mallard said Speakers had lead trade missions before.
"Some countries regard the Speaker - because the Speaker sits just behind the Governor-General and the Prime Minister on the order of precedence - as a fairly important person, and sometimes that results in doors being opened that wouldn't otherwise be opened," Mallard said.
"If I can help NZ companies, I'm happy to do that."
Brownlee not only rejected repaying the money, but even suggested that a minister should have accompanied them on the trip.
"There was also an announcement made there by the ambassador about the enhanced partnership between New Zealand and Japan, which has been announced by the Foreign [Affairs] Minister.
"Very surprising to me that there was actually no minister there to do that."
NZ First MP Shane Jones said it was "churlish and petty" to attack Mallard and Brownlee for the trip.
"Lay off them ... Trade and rugby, I've discovered, are never that far apart. One of the greatest trade in human capital are rugby players overseas."
Ardern said she had no issue with Mallard and Brownlee going to an All Blacks game as long as the trip promoted New Zealand's interests.
"My expectation would always be that the primary purpose is deriving greater, wider support for New Zealand's proposition abroad. My understanding from what I've seen of the programme is that it aimed to that as well."
In a statement today, the Speaker's office said Mallard and Brownlee met with around 200 business and political leaders as well as officials involved in RWC2019.
"These included the Japanese Speaker, former New Zealand PM Scholars, key contacts with NZ businesses in Japan, senior politicians, and members of the NZ Parliamentarian Friendship League.
"The main feature of the trip was a reception at the embassy to which NZ exporters invited their Japanese contacts."
