The Waikato health board was forced to shut down all of its PCs to tackle a Conficker attack.
At 2am on a Thursday last December computer technicians working on a software upgrade at Waikato Hospital realised the District Health Board's IT network was under attack.
One of the world's most prolific computer menaces, the Conficker virus, had wormed its way deep into the Waikato DHB's computer systems.
Left to propagate, Conficker could have hijacked the board's powerful IT resources, using them to steal personal information or to pump out millions of spam messages.
To avoid losing control of its IT infrastructure, the DHB was forced to shut down its entire network of 3500 PCs so it could purge the virus. The move caused major headaches for the organisation's 5800 staff and disruption to health services which cover 360,000 people and a tenth of the country.
As well as losing access to the vital computing resources they relied on to do their jobs efficiently, DHB staff also found themselves in an information void as the crisis unfolded. Without computers, the board's management had lost its most effective means of communicating with staff: email.
As doctors and nurses reverted to paper-based work processes and fax machines began to run hot, managers were relying on phone calls, texts and messages conveyed through the media to keep staff and patients updated.
The plea through the media was that patients should stay away from the main hospital in Hamilton and the DHB's smaller outlying facilities unless treatment was essential.
It took two days of round-the-clock work before most IT services were back up. It was almost three-and-a-half days before all the DHB's computers could be safely turned on again. But the fallout from the Conficker attack has lasted much longer.
The crisis sparked several investigations, including one by Audit New Zealand. Its report was presented to the DHB's board last week and highlighted a raft of IT security and policy failings that together enabled Conficker to sneak into one of the country's major computer networks.
What is the Conficker virus?
One of the most virulent and successful pieces of malware (malicious software) to be seen on the internet for several years, the Conficker "worm" now resides in millions of computers around the world. It gives its criminal masters access to a powerful covert global network of computing power.
In its latest internet Security Threat Report, published this week and summarising malicious online activity during 2009, IT security company Symantec says strong growth in Conficker activity was one of last year's most notable features.
Symantec's managing director for the Pacific region, Craig Scroggie, says the worm exploits IT network security weaknesses which haven't been "patched" (fixed through software updates) to embed itself in large organisations' computer systems.
Once inside a network, worms such as Conficker can monitor the key strokes of computer users and are able to recognise potentially valuable snippets of information when they are typed in, such as bank account login numbers and passwords.
The malware can then transmit these details back to computers controlled by hackers who on-sell the information through a sophisticated web-based black market.
Scroggie says the aim of Conficker's criminal masters is to infiltrate an organisation and remain inside its systems undetected for as long as possible.
"The longer they are there the deeper they are able to go within the network, enabling them to get a rich amount of information out of the network to sell through the underground economy."
Viruses such as Conficker can also set themselves up within corporate networks to act as mass emailers of spam messages. They can hijack a system's resources and use them to send out millions of spam messages each day, all under the remote control of anonymous criminals who are likely to be based thousands of miles away.
How did it infect the Waikato DHB?
Investigations after the incident established Conficker found its way into the organisation's network through a fairly simple chain of events.
A USB thumb drive infected with the virus was inserted into a computer housed in a parking attendant's booth at a Waikato Hospital car park. That computer did not have anti-virus protection and had not been patched, according to the Audit NZ report.
The PC was connected to the DHB's IT network, which was also insufficiently protected against viruses, meaning Conficker was able to take hold across the wider network.
Why was the DHB not protected against a known virus?
Reports into the Conficker attack point to a series of failings at the DHB which combined to open the door to the virus.
These failings included patching of PCs not being up-to-date and the organisation's anti-virus software from international security company CA (previously Computer Associates) being inadequate.
The DHB had been in the process of patching its unprotected computers when the attack hit.
Other problems investigators identified included the widespread, and potentially dangerous, practice across the DHB of staff using USB drives to swap files between computers.
A large number of computers not controlled by the DHB's information systems department were also connected to its network, introducing another security flaw.
The use of "weak" - or easily guessed - passwords across the network was also widespread.
One of the problems the DHB suffered from - a delay in deploying security patches across its network - is a problem Scroggie says is common among larger organisations. While it is a simple process for home PC users to install security updates from Microsoft as soon as they are made available, large organisations often need to test the software updates to ensure they do not interfere with their existing complex programs.
"When Microsoft issues a serious security update and rushes it out urgently, there's a pretty good reason to be having a look at why they've issued it and working out how to get it into your system quickly," says Scroggie.
"It's the organisations that aren't able to react quickly that are generally the ones that have been impacted [by viruses like Conficker]."
What fallout has there been?
The DHB said last week that one outcome of the Conficker attack was that in February, two months after the crisis, CA issued a global update of its anti-virus software aimed at ensuring organisations with the new patch would be protected against similar Conficker threats.
That fix appears to have come too late to ensure CA retains the DHB as a customer of its security software after the board's bruising experience with Conficker.
The board has rejigged its information services capital budget for the 2009-10 financial year to allow it to buy replacement anti-virus software, and it seems unlikely CA will be on its list of potential suppliers.
The board has also outlined a number of changes to IT security policies and practices aimed at preventing a repeat of the virus attack.
Although the organisation-wide shut-down of its entire computer network threw the Waikato health system into upheaval for at least two days, the district health board's chief executive, Craig Climo, said last week that the decision to turn its computers off until they had been purged of the virus meant "Conficker didn't get us, we got it".
Climo says the decision to shut down the network helped to speed up the recovery from the virus.
"We have heard of a major New Zealand site where recovery [from a Conficker attack] took five weeks."
