Insights Blog The Importance of Software Testing in Telecom


The Importance of Software Testing in Telecom

We rarely think of telecom as having complex software issues; we think of it as either my connection works (and we’re happy) or it doesn’t (and we’re annoyed).  However, telecom has become more complicated, problems tend to be large when they happen, and public failures can lead to pricey class-action lawsuits and bad press.

While cost vs. quality are the two big factors, quality can mean more than just the strength of one’s network.  Phone customers are a fickle bunch, and once there’s a reason to switch carriers, whether for money or for better service, it is easy to do.  Perhaps a better question here should be, “What are telecom’s risk factors?”

Smart phones have become practical personal tools that we use constantly.  On average, we touch our phones 2,617 times a day during our 80 daily interactions.  A survey says that 40% of people would rather lose their voice for a day than their cell phone.  As a result of cell phone intimacy, there can often be feelings of betrayal after service outages, billing errors, and of course data breaches.

Outages are a very real warning that your trust in your carrier may be misplaced.  Reliability is now a concern moving forward.   Here are some examples of telecom outages:

  • On March 5, 2018, Telekom Romania’s network crashed and was down for several hours; users were compensated for the full day and received other benefits too
  • In 2016, Telstra’s bug-caused outage knocked 10% of their customers offline
  • In 1990, a misplaced BREAK in nested code caused 9 hours of cascading failures knocking switches offline, blocking 50% of U.S. calls and causing $60M in lost revenue for AT&T
  • A nationwide (U.S.) 5-hour outage prevented over 12,000 911 emergency calls in March of 2017
  • Royal Bank of Canada experienced online and mobile access problems on January 3, 2018; a glitch caused similar problems with Nationwide bank customers in the UK in December of 2017

Well, accidents happen.  Hardware fails.  Things catch fire.  Call it fate or Murphy’s Law, sometimes a bad luck event occurs.  But trust can be more fragile when billing is incorrect.  That’s when thoughts of conspiracy and price gouging begin, as it may seem less like an understandable accident, and more somewhere between sloppy and by design.  Thoughts of fraud or arbitrage may occur.  Testing billing accuracy is called revenue assurance, and it is almost exclusively tied to telecom.

Historically, there has been difficulty with telecom billing accuracy.  A major telecom overbilled 11,000 customers by up to several thousand dollars due to a software bug back in 2006.  In 2010, Verizon was sued for overbilling 15M customer for several years due to software errors, after $90M more was billed than should have been billed.

On February 28, 2018, Airtel had to pay a Bangalore customer Rs 5,000 after overbilling them at Rs 9,979 despite a 1,999 cap on the account.  CenturyLink was sued in a class-action suit in 2017 for the wide discrepancy between quoted rates and charged rates.  Of course, underbilling can also occur, such as when Bell Atlantic failed to bill about 400,000 AT&T customers in their January 1998 bills, although customers probably weren’t upset about that.

Lastly, there is the “breach” (pun intended) of trust, where personal information gets released.  This includes GTE Corporation of accidentally publishing directories for telemarketers that included 50,000 unlisted residential phone numbers with addresses and a critical T-Mobile bug that allowed people to hijack users’ accounts.

Large data breaches include: 14M Verizon customers (thanks to 3rd party NICE’s misconfigured cloud-based AWS S3 bucket), India’s Reliance Jio’s 100M record breach, Malaysia’s recent 46.2M customer breach (multiple carriers, plus some medical records), Swisscom’s .8M person breach, Bell Canada’s .1M breach this January (after last May’s 1.9M), UK Telecom 3’s 6M, and TalkTalk’s 21,000.

There is an easy solution to all of these problems: better software testing at the carrier level.  Network testing can catch software bugs that might take down a network or switch.  Revenue assurance can keep billing honest.  Cyber and penetration testing can mitigate specific security risks and highlight where changes are needed.

But that’s not all.  Telecom has become a combination of cell phone, land line, streaming video and internet service.  Many different types of devices must be proven to be able to connect and work wired and wirelessly, often through in-the-wild managed crowd testing.

The push to increase data speed and access means that new technologies like NFV and 5G will seek mainstream implementation.  And all of these will require heavy testing to ensure that the theoretical solutions work once they become a reality.