Telecom's upgrade nightmare

By PETER GRIFFIN

Telecom is counting the cost of a fault that knocked out 3000 of its 5000 payphones last week.

Adapting the phones to credit-card use triggered the problem but Telecom will go ahead with the change over the next few months.

Engineers from Telecom contractor Downer Connect have been working night and day to visit and repair each payphone. The phones failed when a software upgrade corrupted their flash memory.

About 3000 of the payphone network of 5000 were corrupted. It still has to fix about 900.

As the crisis subsides the question of who is liable for the fault has yet to be answered.

Telecom chief executive Theresa Gattung said the loss of the payphones had cost the company a few hundred thousand dollars a day in lost revenue when the bulk of phones were out.

Telecom had also booked advertising to say credit cards could be used in payphones. Those plans were thrown into disarray.

The software triggering the fault was provided by French equipment vendor Ascom Monetel, which also provides the phones.

Telecom's head of enterprise services, Bruce Buddicom, said Telecom had a service-level guarantee with Ascom Monetel built into its contract that covered both software and hardware.

The priority had been getting the payphones fixed and the question of responsibility had yet to be discussed.

"We've been putting in some big hours, working with France at night. It's something we'll be exploring with them when we get the issue resolved."

In theory the software upgrade of Telecom's payphone network should have taken one night and involved a 45-minute data download to each phone from a central server.

The upgrade was designed to make each phone compatible with ANZ's E-gate payment system, which verifies credit-card transactions.

Telecom's card-based payphones accepted credit cards back in 2001, but without verification built into the process, Buddicom said, cases of fraud quickly mounted.

The corrupting software upgrade commanded each phone to dial back into the management system to receive the update via a bank of modems. Buddicom said the system became overloaded as the phones constantly tried to connect to the server through the modems.

"Each phone was repeatedly calling in to try to get the new software. It just locked up our communication path between the management system and the phones."

In the end Telecom had to shut down the central management system. It set up a new bank of modems and engineers manually connected the payphones to re-install the bug-free software.

Buddicom said Telecom would be more cautious with its next software upgrade. It would be managed by region so a national failure would be avoided.

"We'll modify the software, engage in some aggressive pre-testing and do it again. At an absolute maximum it will be a matter of months before we get the credit-card functionality out there," he added.

Telecom's payphone nightmare came as electricity failures paralysed North American cities and a virulent Blaster computer worm infected hundreds of thousands of computers around the world.

A report on the Australian Parliament website written by analyst Adam Cobb has recently drawn controversy for the level of detail it goes into about threats to Australia's critical infrastructure.

Although written in 1998, the report lists critical nodes in Australia's communications networks, down to key telephone exchanges, and outlines the consequences of their being brought down.

Cobb said a handful of key exchanges or "choke-points" linked east and west Australia with terrestrial communications. Attacking those "critical nodes" could sever terrestrial communication across the continent.

"With the exchanges gone, those remaining systems would be overwhelmed by the demands of regular telephonic and data traffic that daily cross the continent," wrote Cobb.

He went on to point out that in early 1998 both Queensland and Auckland were hit with blackouts as parts of the electricity network collapsed.

"As the Auckland crisis proved, contemporary cities quickly grind to a halt when electricity, telecommunications and financial networks are out of action."

No publicly available reports published recently that outline potential threats to New Zealand infrastructure are believed to exist.

The Centre for Critical Infrastructure Protection is the Government body that evaluates such threats. Its manager, Jay Garden, was not available for comment.