Regulator's Assessment of M1's Service Outages in Recent Years

Ms Lee Li Lian asked the Minister for Communications and Information with regard to M1's outages in October 2013 and February 2014 and IDA's assessment that M1 has "taken all reasonable measures" to test the Radio Network Controller (RNC) software upgrade before implementing it and that there has been "no undue delay" to the service restoration (a) what are all the reasonable measures that had been undertaken by M1; (b) whether IDA's assessment of the investigation into the outages is publicly available online; (c) whether there is any roll-back plan and whether a roll-back has been effected; and (d) if so, how long did it take to make the decision to do the roll-back.

Madam, IDA has completed its investigation into the two M1 service disruptions which occurred on 16 October 2013 and 4 February 2014. Summaries of the cases, the investigation outcomes and IDA's determinations were published on IDA's website on 5 January 2015.

Mdm Speaker, allow me to explain to the House what happened in both incidents. The first service disruption occurred on 16 October 2013. New software was being installed onto one of M1's core network elements, called RNC. Upon installation, M1 observed abnormalities in the network and immediately started investigations. After carrying out initial diagnostics with its vendors, M1 could not identify the cause. Hence, M1 decided to uninstall the new software in the RNC and to "roll-back" to the original version. This step is a common industry practice when trying to isolate a problem that could arise from new software installation. The "roll-back" was done about two hours from the time of the disruption. However, I understand that the problem persisted and so, logically, M1 started to troubleshoot the other network components associated with the RNC. M1 subsequently established that there was a software bug in the Media Gateway and the disruption was subsequently resolved within an hour thereafter.

Page: 27

IDA assessed that the preventive measures undertaken by M1 for this incident were reasonable. Firstly, M1 had carried out thorough testing of the software, including interoperability and compatibility, prior to the installation. Secondly, M1 successfully attempted the same software upgrade on another RNC in September 2013. Thirdly, the test cases adopted by M1 were extensive and comprehensive as M1 combined different test scenarios into each test case. This assessment was corroborated by IDA's third-party external consultant, whose advice IDA had sought for this case. Mdm Speaker, on service restoration which I elaborated upon earlier, IDA also assessed that there was no undue delay by M1 in troubleshooting, isolating and restoring services.

For the 4 February 2014 incident, investigations started as soon as M1 received feedback from some subscribers that they were unable to make calls and send messages. M1 found a software bug which caused the mobile site switches to perform in an unstable manner. This, in turn, led to congestion at the various network layers. As this incident was not triggered by any network changes or recent software installation, roll-back was not necessary. To rectify the situation, M1 re-initialised the switches, as well as other critical network elements, and services were fully restored thereafter.

In its assessment, IDA also found that there was "no undue delay" in service restoration during the 4 February 2014 incident. M1 promptly activated its vendors upon discovery of the incident. Given that neither M1 nor its vendor encountered similar issues previously, there was no workaround solution readily available. This meant that M1 needed time to locate the cause of the incident before deciding on the correct restoration action. It should also be noted that until the disruption on 4 February 2014, the same set of mobile site switches were deployed in many overseas operators' networks and were also deployed in M1's network for several years without incident.

Madam, both my Ministry and IDA fully appreciate the critical nature of mobile services and the importance that people place on these services for their everyday communication. This is why I have asked IDA to continue to ensure that our telco networks function optimally. I would like to assure the House that we remain committed to strengthening the resiliency of these networks, together with the operators. We expect operators to make sound technical decisions when designing and operating their networks. At the same time, we do not assess their culpability with perfect 100% hindsight. That would not be fair. Instead, we look at whether the decisions taken at the time were reasonable, based on the information available then to the decision-makers.

While no network service is immune to failure, telcos can definitely improve the way they update and assist their customers when disruptions occur. We should expect better service, and I have told IDA that M1's public communications during the incidents should have been better managed. Operators must provide early and timely updates to its

Page: 28

customers when outages occur.

Ms Lee Li Lian (Punggol East): Madam, it is good to know that a roll-back plan was in place and had been activated. One supplementary question: does the Minister think that two hours is a good timeframe?

I think, under the circumstances, M1 did the best that they could, given the information they had. There is no standard in this world for us to judge whether two hours were adequate or inadequate. In hindsight, looking at all the actions they had taken, IDA and the Ministry are quite satisfied that two hours were a reasonable time for them.