Prison authorities said nine prisoners escaped following a shootout with the Army.
Two days ago, 10 warders were taken hostage at the prison.
Mr Chaudhry and his 17 fellow ministers, the last of the hostages taken at gunpoint on May 19, were released from Suva's parliamentary complex at 5 pm yesterday.
Soon afterwards, the military handed executive power to the new President, Ratu Josefa Iloilo, chosen at yesterday's meeting of the Great Council of Chiefs.
Ratu Iloilo was Speight's choice for President.
The military-appointed interim Government is expected to be dissolved today, with Ratu Iloilo announcing his interim Government lineup.
Last night, Speight and his supporters remained in the parliamentary compound.
The events are a victory for the rebels, who have been granted total amnesty after deposing the 1999 democratic Government led by Mr Chaudhry.
But last night Mr Chaudhry was determined that he would continue to have a leadership role in Fiji.
"I have a vision for Fiji," he said. "Unfortunately, we were not allowed to fulfil that - we were deposed after 12 months. I will continue to work for the people."
Mr Chaudhry told the Herald that his political future in Fiji was for "the people to decide."
He said he had been beaten up on the second night after the takeover but felt no animosity towards Speight. "I have no ill-feeling towards anybody. I think I am a very forgiving person."
As part of a traditional Fijian forgiveness ceremony, Mr Chaudhry and the hostages drank kava with Speight and some of his supporters shortly before their release.
Then, as the hostages climbed into two Red Cross trucks that had come to Parliament to take them home, the two men hugged.
The rebels are expected to hand back their weapons to the military today and will be able to return to their former positions with the Army.
It is not clear what role Speight will play in the new Government. He has said, however, that he would consider the position of Prime Minister if that was the will of the people.
Last night, an Air NZ flight into Fiji was cancelled after security personnel decided the situation had become too volatile. Spokesman Alastair Carthew said a flight between Auckland and Los Angeles had overflown its stopover in Fiji.
In Wellington, the Minister of Foreign Affairs, Phil Goff, said his advice that New Zealanders should avoid Fiji stands, "and stands stronger than before."
The 800 or 900 New Zealand tourists there had chosen to ignore the advice, and there would not be much sympathy for them if they were charged for leaving if an evacuation were necessary.
Mr Goff said the restoration of Parliament and the Government would not be on the agenda of those making decisions in Fiji.
If the Administration was technocratic and perhaps representative, New Zealand would be right behind it, but that was unlikely.
"What I fear will happen is that a lot of ultranationalists and Speight supporters will end up in the lineup, and we will find dealing with those people extraordinarily difficult.
"We won't want to deal with people who not only lack legitimacy, but are repugnant to us by virtue of the actions they have taken."
New Zealand would be cool and distant, but he was not advocating withdrawing its official representation in Suva.
The Government would look at ending military cooperation and would reduce aid.
More Fiji coup coverage
Main players in the Fiji coup
The hostages
Under seige: map of the Parliament complex
Fiji facts and figures
Images of the coup - a daily record
George Speight: "I’m certainly not mad."
