Facebook and all of its apps go down simultaneously
By Mike Isaac and Sheera Frenkel
Facebook and its family of apps, including Instagram and WhatsApp, went down at the same time on Monday, taking out a vital communications platform used by more than 3 billion people around the world and adding heat to a company already under intense scrutiny.
Facebook’s apps — which include Facebook, Instagram, WhatsApp, Facebook Messenger and Oculus — began displaying error messages around 11:40 a.m. Eastern time, users reported. Within five minutes, Facebook had disappeared from the internet. Hours later, the sites were still not functioning, according to Downdetector, which monitors web traffic and site activity.
Technology outages are not uncommon, but to have so many apps go dark from the world’s largest social media company at the same time was highly unusual. Facebook’s last significant outage was in 2019, when a technical error affected its sites for 24 hours, in a reminder that even the most powerful internet companies can still be crippled by a snafu.
This time, the cause of the outage remained unclear. Several hours into the incident, Facebook’s security experts were still trying to identify the root issue, according to an internal memo and employees briefed on the matter. Two members of its security team, who spoke on the condition of anonymity because they were not authorized to speak publicly, said it was unlikely that a cyberattack had taken place because one hack was unlikely to affect so many apps at once.
Security experts said the problem most likely stemmed instead from a misconfiguration of Facebook’s server computers, which were not letting people connect to its sites like Instagram and WhatsApp. When such errors occur, companies frequently roll back to their previous configuration, but Facebook’s problems appeared to be more complex and to require some manual updating.
Andy Stone, a Facebook spokesperson, posted on Twitter: “We’re aware that some people are having trouble accessing our apps and products. We’re working to get things back to normal as quickly as possible, and we apologize for any inconvenience.”
The outage caused outrage and mirth online, as Facebook and Instagram users turned to Twitter to lament and poke fun at their inability to use the apps. The hashtag #facebookdown also quickly started trending.
But the outage was a blow to small businesses and others that rely on the platform to conduct outreach and advertising and to millions who use Facebook and its apps to communicate with friends and family across the world.
Gamers who livestream their play on Facebook Gaming and are paid by viewers and subscribers said Monday that they were trying to find alternatives.
“You definitely feel out of touch, and it’s scary, too,” said Douglas Veney, a gamer in Cleveland who goes by GoodGameBro. He said he had hoped to post videos and other content on Facebook for his followers before a planned livestream Monday night. “I have 300,000 followers there — you just cross your fingers that nothing’s gone when it comes back.”
Veney, 33, has a job outside of streaming as well, but he said he knew of other streamers living paycheck to paycheck who were making the jump to other sites to be able to keep making money.
“It’s hard when your primary platform for income for a lot of people goes down,” he said.
Inside Facebook, workers scrambled because their internal systems also stopped functioning. The company’s global security team “was notified of a system outage affecting all Facebook internal systems and tools,” according to an internal memo sent to employees. Those tools included security systems, an internal calendar and scheduling tools, the memo said.
Employees said they had trouble making calls from work-issued cellphones and receiving emails from people outside the company. Facebook’s internal communications platform, Workplace, was also taken out, leaving many unable to do their jobs. Some turned to other platforms to communicate, including LinkedIn and Zoom as well as Discord chat rooms.
Some Facebook employees who had returned to working in the office were also unable to enter buildings and conference rooms because their digital badges stopped working. Security engineers said they were hampered from assessing the outage because they could not get to server areas.
Facebook’s global security operations center determined the outage was “a HIGH risk to the People, MODERATE risk to Assets and a HIGH risk to the Reputation of Facebook,” the company memo said.
A small team of employees was soon dispatched to Facebook’s Santa Clara, California, data center to try a “manual reset” of the company’s servers, according to an internal memo.
Several Facebook workers called the outage the equivalent of a “snow day,” a sentiment that was publicly echoed by Adam Mosseri, the head of Instagram.
Facebook has already been dealing with plenty of scrutiny. The company has been under fire from a whistleblower, Frances Haugen, a former Facebook product manager who amassed thousands of pages of internal research and has since distributed them to the news media, lawmakers and regulators. The documents revealed that Facebook knew of many harms that its services were causing.
Haugen, who revealed her identity Sunday online and on “60 Minutes,” is scheduled to testify Tuesday in Congress about Facebook’s impact on young users.
In Facebook’s early days, the site experienced occasional outages as millions of new users flocked to the network. Over the years, it spent billions of dollars to build out its infrastructure and services, spinning up enormous data centers in cities including Prineville, Oregon, and Fort Worth, Texas.
The company has also been trying to integrate the underlying technical infrastructure of Facebook, WhatsApp and Instagram for several years.
John Graham-Cumming, chief technology officer of Cloudflare, a web infrastructure company, said in an interview that Monday’s problem was most likely a misconfiguration of Facebook’s servers.
Computers convert websites such as facebook.com to numeric internal protocol addresses, through a system that is likened to a phone’s address book. Facebook’s issue was the equivalent of removing people’s phone numbers from under their names in their address book, making it impossible to call them, he said. Cloudflare provides some of the systems that support Facebook’s internet infrastructure.
“It was as if Facebook just said, ‘Goodbye, we’re leaving now,’” Graham-Cumming said.