A market watchdog has released a damning report on cyberattacks that hit the NZX over August and September last year, forcing it offline for several trading days, plus an earlier, volume-related glitch that forced it offline during April 2020.
The Financial Markets Authority said the New Zealand Stock Exchange had been caught short on technology and skilled staff - and that the DDoS attack was forseeable but not planned for.
It also found what it called "cultural" problems, including being secretive about problems and failing to appreciate their effect on market participants.
"NZX rarely accepts fault, and is not upfront and open when things go wrong," the report said.
The FMA added that despite several steps taken by the exchange to beef up its security holes since September (see below) "there are some critical gaps remaining."
The report comes amid a rash of online attacks at a time when our government has failed to follow Australia's move to ramp up funding cybersecurity funding. The Reserve Bank is bracing for a review of its recent data breach, which followed internal warnings over underspending that were ignored.
"NZX rarely accepts fault, and is not upfront and open when things go wrong," the FMA report said. Pictured: FMA chief executive Rob Everett.
The FMA's review of NZX technology issues, released this morning, has found the stock exchange failed to meet its licensed market operator obligations under the Financial Markets Conduct Act "due to insufficient technology resources."
Scope of the problems
The FMA began a targeted review of NZX's technology after it suffered trading volume-related system issues and outages in April 2020. The scope of the review was expanded following DDoS (Distributed Denial of Service) attacks on NZX in August 2020.
The FMA also had concerns that NZX's trading system was unable to trade securities at zero or negative yields. The volume-related issues and DDoS event repeatedly halted or disrupted market activity.
Report's key findings
Overall, the FMA review found the NZX did not have adequate technology capability across its people, processes and platform to comply with market operator obligations and especially in the context of hosting critical infrastructure
The NZX's systems did not meet regulatory requirements or expectations for fair, orderly and transparent markets, the regulator found.
In respect of NZX's trading volume-related issues, the FMA review concluded fundamental tools and practices were either lacking, insufficiently robust or not fully utilised.
NZX aware of limitations, not not accept responsibility
NZX was aware of the capacity limitations of its core back end processing system, particularly as daily trading volumes had increased in the last three years, the FMA said.
FMA chief executive Rob Everett said market participants gave feedback that NZX did not accept responsibility for known systemic issues and was slow to act:
"The feedback from market participants mirrors our own observations and is a major concern that needs to be addressed by the NZX board and executive, Everett said.
"The failure to properly consider the broader ecosystem in which the exchange operates, and to fully engage with industry feedback and concerns, were contributing factors to the volume-related issues."
Insufficient crisis planning
The FMA review found NZX's crisis management planning and procedures were basic.
While the NZX said the DDoS attack (where automated bots overwhelmed its servers) was on a huge scale and unforeseeable, the regulator disagreed, saying, "A DDoS attack was foreseeable."
The FMA review said an attack of sufficient magnitude to take down servers - and with them, the NZX's market announcement platform - was at least possible and should have been planned for.
The watchdog noted that Crown agency CERT NZ had released a warning about escalating DDoS attacks on Asia-Pacific targets in November 2019
NZX self-rated its IT security profile at a basic maturity level, indicating that a number of best practices had not been adopted.
Actions required
NZX is required to develop a formal action plan to address the issues raised by the FMA.
The market regulator has met with the NZX Board to discuss its findings and received assurances that the NZX Board takes responsibility for making the necessary investment and to address the issues highlighted in the report.
Earlier (see below), NZX warned that bolstering its defences could lead to costs that have to be passed on to clients.
Today, the FMA said more work was need on hardware upgrades, upgrades of "out-of-date" software.
It also found cultural and planning issues, and found staffing gaps.
The FMA report said NZX had a "small" inhouse IT team - appropriate for a normal small-to-medium business, but not one running critical infrastructure.
It was consumed by day-to-day tasks and small incremental upgrades, lacking the capacity to address areas such as performance monitoring, continuous version management of software, failover planning and risk management.
The FMA said next steps need to include recruiting a chief risk officer, a head of network architecture and a head of IT security.
The exchange's chief information officer, David Godfrey, quit on September 28, the day after a daylight savings blunder that came on top of the earlier DDoS attack and clearing outages.
No reason was given for his departure. An NZX spokesman said Godfrey's abrupt exit - before recruitment for a successor had begun - was not related to the various IT problems.
'Not upfront and open'
The FMA report also said, "We consider there are internal cultural factors that have contributed to NZX's failure to have adequate technological resources.
The regulator saw, "a failure to fully understand and manage its interdependencies with the wider ecosystem that has been detrimental to NZX's strategic planning, issue identification, and appropriate crisis response and resolution."
The exchange's failure to accept fault and be upfront about it, meant downstream costs for market participants when things went wrong.
The FMA recommended hiring a relationship manager and resolving service-level agreement issues to "restore trust."
"The detailed and critical feedback received from participants is a major concern and needs to be considered and addressed," the report said.
There was, "the real risk of distrustful and tense relationships at a time when growing trust and confidence in our capital markets is crucial."
Toothless watchdog
Sanctions for a breach of NZX's statutory obligations are limited, Everett told the Herald.
Fines were off the table.
The FMA could, in theory limit or revoke the NZX's trading licence.
"But removing the license of the country's only licenced stock exchange is a very big call. So effectively limited to calling on the exchange to take action."
Regardless of the limitations of his powers, the FMA boss said the NZX had made progress and that as long as it followed all of his agency's recommendations, "I'm confident they'll emerge in decent shape."
The parties had yet to agree on a timeline for the NZX to get itself into shape. Talks were going on that front.
The FMA will publicly report on NZX's progress in its annual NZX Obligations Review, to be released in June 2021.
NZX responds
NZX chief executive Mark Peterson said in statement soon after the FMA report was released, "NZX accepts that it did not meet the high standards it sets for itself in key areas of technology resources. We also agree that improvements are required and we are committed to delivering these improvements via an action plan that will be agreed with the FMA. We will work constructively with the FMA through that process and engage closely with the broader capital markets technology ecosystem."
Security upgrade costs could be passed-on
In a December 21 update, NZX said it will continue to bolster its IT and cybersecurity systems over the coming months - and that related costs are "likely" to be passed on to its clients.
This comes after another year that has seen several hot local IPO prospects, including Laybuy and Aroa Biosurgery, ultimately opting to list across the Tasman.
The exchange said: "NZX accepts that it did not meet its own high standards in certain areas of its technology systems," after suffering a sustained cyberattack over August and September, and problems with its clearing system earlier in the year.
In a statement, the exchange did not put a figure on the ongoing security upgrade, but did offer that "there is no impact on the FY2020 earnings guidance".
In a December 2 update, NZX said it expected ebitda for its 2020 financial year (which coincides with the calendar year) to be "around the top of the guidance range of $30 million to $33.5 million".
The exchange won't comment on any impact to its FY2021 guidance until it delivers its FY2020 full-year report on February 17.
Reviews carried out by EY and local security outfit InPhySec had already seen several steps taken to tighten security.
But the exchange said it was still in the process of agreeing a formal action with the FMA. Once it had done so, it would be in a position to detail costs.
"NZX recognises the need for further technology investment in 2021, particularly in the markets businesses, in order to enhance the stability and resilience of its technology framework," the exchange said.
"This includes enhancing the Securities IT team and cybersecurity counter-measures, with related pricing to market participants to be considered. NZX is well advanced, in conjunction with market ecosystem participants, for a major upgrade to its core trading system around the end of March 2021," it added.
"The board has not yet considered the consequences on pricing for NZX services, but some cost recovery process is likely."
The NZX also wants to implement a series of changes recommended by its new Technology sub-committee, created in November, including better crisis management, better communications "with the ecosystem" and "bolstering NZX's IT organisational structure with some specific specialist skill sets".
Although no costs were revealed for the IT and cyber-security upgrades in train, the NZX gave a reference point for its last major upgrade, saying: "NZX initiated its technology infrastructure modernisation programme in 2017, with $12m invested over the four-year period to 2020, in projects that focused on clearing, infrastructure and trading system improvements, modernisation, and capacity improvements."
A spokesman said NZX has shared the full EY and InPhySec reports with law enforcement authorities and regulators, but would not be making them public because of security concerns, in line with GCSB advice.
A broad-brush summary released on December 4 offered no detail on various big-picture questions around the DDoS attack including whether the attacker was politically or commercially motivated, where they were located or what ransom if any, they demanded to stop smothering the exchange with automated bot attacks.
But GCSB director-general Andrew Hampton did say his agency believed the perpetrator was a criminal gang rather than a bad state actor.
Hampton noted that although his organisation had assisted the exchange -for part of the spy agency's brief is to protect economic security by shielding top companies and exporters - a DDoS attack only smothers a website with an over-load of connection requests, forcing it offline. There is not any risk that data will be stolen.
They’re doing so well competing with other exchanges, I’m sure increasing the price will only help them be even more competitive. 😬
Although scant detail was offered in the December 4 summary of the EY and InPhySec reports, the exchange did say: "InPhySec said the severity of the cyber-attacks went well beyond anything previously seen or that could have been reasonably forecast - the volume, sophistication and persistence of the attacks were unprecedented in a New Zealand context, and are amongst the most severe we are aware of to have been experienced internationally. It said the attacks fundamentally changed expectations about this sort of attack for the industry."
It said NZX had been "assisted in managing the attacks by being well advanced with a significant network upgrade started in 2019". Work on this upgrade with Spark, "created a 'match-fit' team that meant NZX was able to respond quickly and effectively".
The decision "to engage Akamai, a leading global cybersecurity company, was also highlighted as being central to NZX responding to the threats", in the independent reports, according to the exchange's summary.
Content network delivery specialist Akamai last made headlines in NZ for its at-times rocky partnership with Spark during the 2019 Rugby World Cup.
The GCSB was also roped in to assist.
During the DDoS attack, NZX emphasised that only its website, not its trading systems, were under assault. However, it had to suspend trading for the first few days of the cyber-attack because, with its site down, continuous disclosure obligations were not being met.
The exchange switched to alternative ways to get information to market participants as the DDoS attack ground on.
On September 18, after the dust had settled, NZX launched an alternative site for market announcements, which could be accessed in the event its main site was taken offline by another DDoS attack - aping a tactic adopted years ago by MetService.
