“On September 5th, Twitter experienced the loss of its Sacramento (SMF) datacenter region due to extreme weather. The unprecedented event resulted in the total shutdown of physical equipment in SMF,” Carrie Fernandez, the company’s vice president of engineering, said in an internal message to Twitter engineers on Friday.
Major tech companies usually have multiple data centers, in part to ensure their service can stay online if one center fails; this is known as redundancy.
As a result of the outage in Sacramento, Twitter is in a “non-redundant state,” according to Fernandez’s Friday memo. She explained that Twitter’s data centers in Atlanta and Portland are still operational but warned, “If we lose one of those remaining datacenters, we may not be able to serve traffic to all Twitter’s users.”
The memo goes on to prohibit non-critical updates to Twitter’s product until the company can fully restore its Sacramento data center services. “All production changes, including deployments and releases to mobile platforms, are blocked with the exception of those changes required to address service continuity or other urgent operational needs,” Fernandez wrote.
The restrictions highlight the apparent fragility of some of Twitter’s most fundamental systems, a problem Peiter “Mudge” Zatko, Twitter’s former head of security who turned whistleblower, had raised in a disclosure sent to lawmakers and government agencies in July.
In his whistleblower disclosure, first reported by CNN and The Washington Post, Zatko warned that Twitter had “insufficient data center redundancy” that raised the risk of a brief service outage or even the prospect of Twitter going offline for good.
“Even a temporary but overlapping outage of a small number of datacenters would likely result in the service [Twitter] going offline for weeks, months, or permanently,” according to Zatko’s whistleblower disclosure. (Twitter has criticized Zatko and broadly defended itself against the allegations, saying the disclosure paints a “false narrative” of the company.)
News of the data center outage comes a day before Zatko is due to testify before the Senate Judiciary Committee.
In a statement about the Sacramento outage, a Twitter spokesperson told CNN, “There have been no disruptions impacting the ability for people to access and use Twitter at this time. Our teams remain equipped with the tools and resources they need to ship updates and will continue working to provide a seamless Twitter experience.”
Data centers need “reliable water, power, humidity controls and refrigeration to live,” said retired brigadier general Greg Touhill, who served as the US government’s chief information security officer in 2016 and 2017.
“You want redundancy, not duplication, of your data locations to enhance your cyber resilience so you can take a punch from a natural disaster [or other event] that can take down a single piece of equipment or data center,” Touhill, who now heads the CERT Division at Carnegie Mellon University’s Software Engineering Institute, told CNN.
