He said while other access rights would be suspended, the Herald journalists would not be prevented from using their office, provided access was "properly facilitated."
Dr Smith said he stood by his decision to take action against the entire office, rather than just the journalist involved.
"This was not the decision of one member of the Press Gallery, but rather a decision of the New Zealand Herald.
"As I have said, I am very disappointed to have to take action affecting long standing members of the Press Gallery, but the seriousness of the breach left me with no option."
News of the suspension caused an outcry among other media outlets and commentators.
The Media Freedom Committee, the Engineering, Printing and Manufacturing Union and the Parliamentary Press Gallery have each expressed their concern about the ban.
The Speaker suspended the accreditation of the New Zealand Herald's parliamentary journalists for 10 days from Monday morning.
It was a sanction after the newspaper's website published a photo of guards struggling with a man trying to jump from the Public Gallery into the Debating Chamber.
The photo was taken by political editor Audrey Young on a mobile phone. Filming the Public Gallery is against Parliament's rules.
The Media Freedom Committee's secretary Tim Pankhurst said it was "an over reaction at a critical time when an election is looming".
He said Standing Orders needed to allow for exceptional circumstances.
"If there was a brawl in the House - and that's within the realms of possibility - the media is forbidden to show that," Mr Pankhurst said. "Similarly, picturing this week's disturbance in the Gallery is forbidden.
"That is despite it being a news event of considerable interest that disrupted Parliament and led to a spat between the Leader of the Opposition and the Prime Minister."
Mr Pankhurst said Dr Smith is generally regarded as an able Speaker, but "he might be well advised to hop off his high horse in this instance".
"The media are the eyes of the public and Dr Smith also serves the public. He is our employee."
The EPMU - which represents journalists - said while it was a breach of the Standing Orders which governed Parliament it was an overreaction to ban all of the Herald's Parliamentary journalists.
"It risks having a chilling effect on political journalism. Greater restrictions have already been placed on parliamentary press gallery journalists in the past three years, including restricting their movement around the parliamentary complex."
The union called on the Speaker to take into account the importance of a free news media to the democratic process.
The Parliamentary Press Gallery also condemned the decision as "extraordinary," saying it set an "extremely concerning precedent."
Press Gallery chair, Jane Patterson, said journalists were accredited individually and had to take responsibility for their own actions.
"To suspend a whole office on the basis of one person's actions is unfair and disproportionate."
She said banning an entire news organisation from Parliament in the lead-up to a general election could deprive the Herald's readership of political coverage at a critical point, "a clear disruption of the media's critical role in the democratic process."
Greg Treadwell, acting journalism curriculum leader at AUT University, said New Zealanders should be deeply concerned by Dr Smith's actions.
He said it set an "incredibly worrying precedent".
"What will the punishment now be if a more serious breach of Parliamentary rules occurs? Excluded for months at a time? Years?" Mr Treadwell said.
"I would call on the Speaker to think carefully about his reaction and, if possible, significantly alter his ruling."
Rival newspaper The Dominion Post has also criticised the decision as "outrageous", with editor Bernadette Courtney saying she will be making a complaint to Dr Smith.
