Reinforcing our commitment to network resilience
Posted on April 4, 2016
13 min read
Earlier today I spoke on Telstra’s behalf at the CommsDay Summit in Sydney. I took the opportunity to address the recent mobile network challenges we’ve faced and the steps we’re taking to minimise the chance of similar issues happening again.
You can read my full speech below.
I’d like to begin by acknowledging the traditional owners of the land we meet on, the Gadigal people of the Eora nation. I pay my respects to their elders, both past and present. I also acknowledge Minister Fifield and other distinguished guests.
I know you may have been expecting to see Tony Warren this morning, but I asked him if he would give up his spot so that I could talk to you in person about the recent challenges we’ve faced. He wanted me to let you know that he is looking forward to discussing the policy and regulatory settings in the Australian mobile industry at your next conference.
As an industry, the bond we have with our customers is simple. In a connected world, our customers expect always-on services. As all of you know, events of the past six weeks have impacted Telstra’s ability to meet those expectations, and led to questions about the state of our mobile network.
It is an unfamiliar situation that we find ourselves in, but one we are confident we can work through.
At Telstra, we take our responsibility to connect people very seriously. Our mobile network supports more than 16.9 million customers and carries around 70 million voice calls every single day – more than all other Australian mobile networks combined. We know that our customers rely on us and that’s why we were so disappointed by what has happened.
Many of you are our Telstra Wholesale customers, and we are committed to keeping you informed about what happened, why it happened, and what we’re doing to fix it. As Chief Operations Officer, it’s my personal commitment to ensure that is done to your satisfaction.
Some of you are also our competitors. Now, every company will go through challenging times (and I know some of you in this room have as well) but I believe that our industry is much better off, and our customers are much better off, if we are willing to face those challenges with courage, transparency and honesty.
This is the simple reason I am here, and there can be no more appropriate context in which to share our difficulties than at the CommsDay Summit. Regardless of your proximity to these issues, there are learnings to be shared.
So what happened?
Our initial review has confirmed the incidents were not related, although two of the disruptions were due to delays in processing the registration of mobiles devices.
Telstra’s network is highly duplicated and designed for the reliability you have come to expect. Whilst at no time did it suffer a system-wide failure, each of these events did impact varying numbers of our customers and we are working to ensure this does not happen again. It is important to note that our network is now stable and operating as it should.
Let me take you through what happened during the service interruption on 9 February.
On the morning of the 9th technical staff were investigating a fault with one of the signalling nodes used to manage 3G and 4G wireless data sessions and voice calls in the mobile network. With evidence of increasing degradation on the health of the node and potential service risk, the issue was escalated and a decision was taken to isolate the node from the network – a standard operational procedure in such an event.
That happened at around 12:30pm, but unfortunately due to processes not being followed properly the subsequent node restart initiated incorrectly. This meant that around 15% of all mobile devices connected through this node needed to re-register when establishing a new voice call or data session.
The mass re-registration of these mobile devices then overloaded the other mobile signalling nodes, impacting approximately 15% of our customers directly and some more at times during the event where they were unable to establish new voice calls or data sessions.
As soon as we identified what had occurred we worked to address the fault and take action to bring customers back online as quickly possible. In doing so we prioritised voice services over data services. Most impacted devices were able to establish new data sessions by 1pm with some residual customers still impacted. The network was stabilised and all services restored at around 2:30pm.
I’ll now turn to the service interruption that occurred on 17 March.
Just before 6pm some customers nationally were sporadically unable to make 2G, 3G and 4G voice calls or establish a mobile data session.
Calls between Telstra mobiles were failing intermittently with voice call volumes dropping by approximately 50%. Calls to fixed line telephony services were not impacted if the mobile was connected to the network. SMS services were largely unaffected, although some delivery delays occurred. Data services were affected where customers may have been unable to establish a data session, or existing data sessions may have been disconnected.
Service restoration commenced from 7pm through limiting the volume of 4G signalling for devices reconnecting with the network, and configuration changes were made in the mobile network to speed up recovery. These changes reinstated network stability.
Although the user experience was similar, the issue is different to the disruption that occurred in early February.
The problem was caused when a significant number of customers – initially international roaming customers, and then domestic customers as well – were unexpectedly disconnected from the network. When they all attempted to reconnect at the same time, which happens automatically, we saw a period of overload in the database used to register devices.
The ability of mobile networks to deal with these mass re-registration events is not unique. Our industry experts have already told us that this is a global challenge faced by many in the industry.
Finally, in relation to the service interruption of 22 March.
Some Telstra mobile, IP Telephony (TIPT) and NBN voice customers may have been unable to make or receive calls intermittently between 11:30am and 12:50pm, primarily in Victoria and Tasmania. This incident effected around 3% of our customers and services were restored by around 5.30pm.
What are we doing to stop this happening again?
We are committed to getting to the bottom of these incidents and are taking all necessary steps to minimise the risk of it happening again. Our customers expect nothing less.
We are well into a thorough review of the network. I am leading this review and it involves our own specialist teams as well as external experts from around the world. We have already progressed short- and medium-term actions to improve resilience and robustness in the mobile network.
Changes have been implemented to increase the capacity and path diversity of critical signalling channels and a temporary layer of traffic management protection has been added to minimise the impact of events like the ones we saw on 9 February and 17 March. Within a few days we expect to augment capacity in a key platform (Home Location Register – Front End aka HLR-FE) that manages our customers’ subscription data.
In conjunction with our global partners Ericsson, Cisco and Juniper we have assembled a team of internal and external engineering experts to do an end-to-end review of our network. While this work is underway, Telstra Operations has a heightened awareness plan including Executive-level review of any changes planned for the mobile and core IP networks.
Our network is resilient and we are determined to get the best advice from around the world to help ensure that it stays that way. Our focus is on ensuring our network is the best available and rebuilding our customers’ trust by meeting their expectations every day.
As a way of saying sorry, we have offered two free data days for all our mobile customers, whether they’ve been impacted by the incidents or not. We are extremely pleased with the performance of our network during these two days.
During the Free Data Day in February, the network reached weekday peak traffic levels by 8am and it climbed rapidly from there. Over the course of the day we had twice as much traffic as we would normally see on a Sunday, which typically sees as much traffic as a weekday. Before long, we had had our highest single day of traffic ever.
By midnight our customers had downloaded 1,841 terabytes of data. That’s the equivalent of around 2.3 million movies, or 5.1 million episodes of Game of Thrones.
Yesterday we reached the peak network traffic level of the previous Free Data Day by 8am. By 4pm we had already surpassed the record set in February. By the end of Sunday, our customers had download 2,686 terabytes of data, which is three times the amount of data downloaded on a normal weekday and 46% more than the amount downloaded on Free Data Day in February.
And some of our customers really do have a big appetite for data. One customer in particular took to Reddit to say that he had used the first Free Data Day to download more than 400 GB. On Sunday he set about breaking his own record. Throughout the morning and afternoon he was downloading at speeds of more than 100Mbps, and by then end of the day he had downloaded an incredible 994 GB.
We were pleased with the way the network performed, given this tsunami of data. To ensure fairness for all customers accessing the network, traffic balancing mechanisms tuned and optimised the mobile network during the day.
Like any of our global peers there will always be issues that arise in such a large and complex technology environment. We understand there is a heightened degree of interest in relation to our network performance at the moment. Social media is a very immediate, if not occasionally over-enthusiastic, indicator of network performance.
But you only need to look at the results of our Free Data Days to see what our network is capable of. These two days of customers sending and receiving huge volumes of data across our network have given us a glimpse of the future. Customers have bigger and bigger expectations of what mobile technology and connectivity will offer- and we are determined to meet, if not exceed, those expectations.
That kind of network performance only comes through constant investment.
What does this say about Australia’s mobile market?
While we are acutely aware of the seriousness of these incidents, I’d also like to make a broader point about the health of Australia’s mobile market.
Over recent weeks we have seen our competitors trying to take advantage of the challenges Telstra has faced. That’s a natural reaction in a healthy, competitive market. Customers will choose between carriers to get the best option for their needs.
Strong competition in a healthy market is enabling more mobile connectivity than ever before. In 2014 nearly one-third (29 per cent) of adult Australians were mobile-only phone users—with no fixed-line telephone at home. We have one of the highest smartphone penetration rates in the world (6th), ahead of the UK, Germany, the US and Japan.
The percentage of Australia’s population with mobile coverage is higher than markets that have much greater population density and much smaller geographies. Coverage in the UK reaches 98% and New Zealand 97% . Telstra’s mobile network covers more than 2.4 million square kilometres. That means we reach 99.3% of Australia’s population. Telstra can offer 4G to 96% of the population, and we are continually investing to ensure we reach 99% by June next year.
Recent analysis by CommsDay showed “Australia’s mobile carriers have achieved one of the highest proportion of 4G/LTE subscribers to total mobile base in the world, significantly above the global average – and with the ratio climbing strongly year-on-year.”
The strength of Australia’s mobile network market is underpinned by the presence of globally significant telecommunications companies that have the capacity to make significant investments.
Telstra has continually chosen to invest in our network and over the three years to June 2017 we expect investment in our mobile network to total more than $5 billion, and we are on track to meet this. We are determined to offer the best possible products to our customers.
Meanwhile, Optus last year reported spending over $2 billion on its mobile network over three years, with an additional $1.8 billion reportedly planned for this year. And at the recent 1800 MHz spectrum auction, Optus spent $196 million, Telstra $191 million, Vodafone $68 million and TPG – not traditionally known as a major player in mobiles – spent $88 million, flagging its intentions to move into the mobile or perhaps fixed wireless market.
I point to our competitors’ investments without commentary on individual situations, but to highlight the aggregate investment commitment.
Australia is a market with a good track record of investment with plenty of potential to continue to do so. In fact in 2014 the Australian mobile market recorded an investment per person figure of $90.19 – the 7th largest across the 31 OECD countries for which data was available. This impressive figure doesn’t even account for the large amounts being committed through spectrum auctions.
Strong competition means Australian consumers are paying globally-competitive prices. The ITU’s Measuring the Information Society Report 2015 ranked 182 countries for the cost of mobile services as a percentage of Gross National Income. Australia was the 9th most affordable, behind only Macao, Hong Kong, Singapore, Denmark, Qatar, Norway, UAE, and Luxembourg.
Customers nation-wide are hungry to take advantage of the investments we have been making. Figures on data usage across the Telstra network show strong growth. In recent years, the amount of traffic on our mobile network has been increasing by around 50% per annum.
Australia’s mobile companies and policy makers have consistently been at the forefront of technological change, standardisation and adoption. Telstra was the first to bring 4G technology to Australia in 2011. Just five years later we announced a plan to deploy our first 5G trial network with Ericsson in 2018 on the Gold Coast.
What does all this add up to? According to research commissioned by the AMTA and carried out by Deloitte, this impressive take up of mobile technology delivered a $43 billion boost to Australia’s GDP in 2015 .
Australia has one of the best mobile eco-systems in the world. It is a dynamic and competitive market that is delivering real benefits for customers and the economy. Australia’s smart policy approach and steady regulatory settings have successfully encouraged the creation of a resilient market, characterised by competition on both coverage and price.
We know that we live in a very competitive market, and we know that we have work to do to re-establish Telstra’s reputation and relationships with our customers. This means not only delivering the best network, but also dealing with the huge anticipated growth in data, leading the development of new technologies, and supporting innovation in new areas of our business.
That work is already underway and we are committed to meeting that challenge. Thank you for the opportunity to provide these insights and wish you well.