“I firmly believe that another change of leader at this time is not what Optus needs or what our customers need. The disruption and uncertainty could actually set back the transformation underway and create further risks,” he said.
The crisis stemmed from a botched firewall upgrade at Regency Park exchange in South Australia, where technicians failed to redirect traffic before locking down a critical “session border gateway”, according to Optus’ submission to the inquiry.
Optus’ Paul O’Sullivan, former chairman, John Arthur, chairman, and Stephen Rue, CEO, at the Triple Zero service outage hearing at Parliament House in Canberra on Monday.Credit: Alex Ellinghausen
The submission reveals Nokia contractors used an outdated procedure document and incorrectly rated the change as “zero risk”, bypassing crucial engineering reviews.
It states that Nokia personnel selected an “outdated Method of Procedure (MOP) document” for the change. This outdated document “did not include making changes… to ‘offload’ traffic from the Session Border Gateway” before applying the lock.
The work was also brought forward 24 hours without proper authorisation.
Automated alarms for general call failures were generated within minutes of the outage starting. Those alerts, at 12.20am on September 18, were dismissed as routine maintenance activity, while five customer calls reporting Triple Zero failures between 10.13am and 2.57pm went unescalated – a “serious process issue” that Optus acknowledged was unacceptable.
Loading
The outage wasn’t identified until 1.17pm when South Australian Ambulance Service called Optus directly. Services were restored by 2.34pm, but the damage was done.
Efforts to conduct welfare checks on affected customers were then delayed because the technical fault meant that standard Call Detail Records (CDRs) “showed no unsuccessful calls”.
Engineers had to devise new methods to identify affected customers, with the first 100 identified by 7.50pm – more than five hours after services were restored. A further 524 customers were identified the next morning, followed by seven more days later.
Welfare checks didn’t commence until 8.34pm on September 18, with Optus learning of the first fatality at 8.43pm. By Friday afternoon, three deaths had been reported, rising to four by Saturday.
The company has appointed former Sydney Water chief executive Dr Kerry Schott to lead an independent review, and engaged consulting firm Kearney to oversee network management improvements. The inquiry continues.
More to come.
Get news and reviews on technology, gadgets and gaming in our Technology newsletter every Friday. Sign up here.
