Facebook and the apps under its umbrella, including Instagram and WhatsApp, were inaccessible for hours on Monday. The outage hamstrung the communications of billions of people, businesses, and other organizations.
Though Facebook is not typically thought of as a critical business service, we observed small numbers of companies using Workplace by Facebook and monitoring Facebook as part of their third-party risk management efforts within Bitsight data sets. While this represents a relatively small percentage of businesses overall, it goes to show that third-party risk extends to a wide variety of business partners—some of which you might not expect.
The outage also highlights potential risk associated with other large-scale service providers. “This was a large, correlated failure in a system that was designed to be resilient,” said Ethan Geil, Senior Director, Data and Research at Bitsight. “It makes the possibility of extended disruptions at other large infrastructure networks seem more imaginable."
What Happened, Exactly?
Facebook said that the outage was the result of a configuration error on its backbone routers. According to an article on Facebook Engineering:
Our engineering teams have learned that configuration changes on the backbone routers that coordinate network traffic between our data centers caused issues that interrupted this communication. This disruption to network traffic had a cascading effect on the way our data centers communicate, bringing our services to a halt.
Our services are now back online and we’re actively working to fully return them to regular operations. We want to make clear at this time we believe the root cause of this outage was a faulty configuration change. We also have no evidence that user data was compromised as a result of this downtime.
A post on Krebs on Security offered a bit more information:
At approximately 11:39 a.m. ET today (15:39 UTC), someone at Facebook caused an update to be made to the company’s Border Gateway Protocol (BGP) records. BGP is a mechanism by which Internet service providers of the world share information about which providers are responsible for routing Internet traffic to which specific groups of Internet addresses.
In simpler terms, sometime this morning Facebook took away the map telling the world’s computers how to find its various online properties. As a result, when one types Facebook.com into a web browser, the browser has no idea where to find Facebook.com, and so returns an error page.
