AT&T said it has made changes to prevent what happened in February from occurring again.
“We fell short of the standards that we hold ourselves to, and we regret that we failed to meet the expectations of our customers and the public safety community,” the company said in a statement.
The FCC said the outage blocked more than 92 million calls as voice calling and mobile data services were disrupted for 125 million devices for several hours on a Thursday.
At least 25,000 calls to emergency services were blocked, although some of those may have been people calling 911 just to test the service. The outage also disrupted the communications of first responders, as it cut off service to the devices operated by users of the AT&T-operated First Responder Network Authority, known as FirstNet.
The report also offered new details on what caused the outage. Early that morning, AT&T implemented a network change that included some improperly configured equipment. This caused the entire network to enter “protect mode,” which revoked service for all of the company’s wireless customers.
It took AT&T close to two hours to roll back the network change and at least 12 hours to fully restore service because the company was overwhelmed by people trying to re-register their devices, according to the report.
The FCC identified several errors on the part of AT&T that led to the “extended scope and duration” of the outage, including technical problems that were compounded by a lack of adherence to best practices.
“This ‘sunny day’ outage highlights the need for network operators to adhere to their internal procedures and industry best practices when implementing network changes, to implement sufficient network controls to mitigate configuration errors so they do not escalate and disrupt network operations, and to have appropriate systems and procedures in place with adequate capacity to facilitate prompt recovery from large-scale outages,” the FCC wrote.
