Technologies
Cloudflare CEO Apologizes for ‘Unacceptable’ Outage and Explains What Went Wrong
For several hours Tuesday, a significant portion of the internet was unavailable in Cloudflare’s worst outage since 2019.
The Cloudflare outage on Tuesday that disrupted access to many websites and services — including OpenAI, Spotify, X, Grindr, Letterboxd and Canva — was the company’s worst outage since 2019, CEO Matthew Prince says.
Other disruptions have centered on specific network features, Prince wrote in a blog post. “But in the last 6+ years we’ve not had another outage that has caused the majority of core traffic to stop flowing through our network.”
Cloudflare is a cloud services and cybersecurity company based in San Francisco that is used by approximately 20% of all websites, according to W3Techs. It’s one of a handful of services, along with Amazon Web Services, CrowdStrike and Fastly (all of which have experienced major outages in the past few years) that you might never have heard of, but that provide essential internet infrastructure.
The bulk of sites and services impacted by Tuesday’s outage, which began around 3:30 a.m. PT, seemed to recover within just over three hours. By the end of the day, everything had returned to normal, and Cloudflare set about explaining what went wrong. Here’s what you need to know.
Don’t miss any of our unbiased tech content and lab-based reviews. Add CNET as a preferred Google source.
What caused the Cloudflare outage?
Cloudflare was keen to emphasize that the outage was not caused either directly or indirectly by a cyberattack. At first the company did suspect it might have originated from a “hyper-scale DDoS attack,” Prince said in his blog post. But it turned out that the outage resulted from an internal software failure.
A change in one of Cloudflare’s databases generated a larger-than-expected feature file, which was too big for the company’s software to run, said Prince. This caused the software to fail.
Once Cloudflare identified the problem, it was able to replace the problematic file with an earlier version and get most traffic flowing normally again by 6:30 a.m. PT.Â
“We are sorry for the impact to our customers and to the Internet in general,” said Prince. “Given Cloudflare’s importance in the Internet ecosystem any outage of any of our systems is unacceptable. That there was a period of time where our network was not able to route traffic is deeply painful to every member of our team. We know we let you down today.”
Which sites and services were impacted?
Cloudflare has a massive range of clients across the internet, ranging from websites that are household names to smaller services you might not have heard of. Due to its size, when it went down, it took many of those sites and services with it.
Among those affected by the outage was Downdetector, which is where most people go to report problems when services are offline. (Downdetector is owned by the same parent company as CNET, Ziff Davis.)
Once it got back up and running, Downdetector said that it received over 2.1 million reports during the outage period. Over 435,000 of these came from the US, with the UK, Japan and Germany appearing to be the countries that were next most affected.
Most of the reports pertained to Cloudflare, but other affected companies also received a significant number of reports. They include X (320,549 reports), League of Legends (130,260 reports), OpenAI (81,077 reports), Spotify (93,377 reports) and Grindr (25,031 reports).
How did the outage unfold?
Cloudflare first acknowledged the outage at 3:48 a.m. PT. The company issued a statement on its system status page saying that it was aware of the problem.Â
“Cloudflare is aware of, and investigating an issue which impacts multiple customers: Widespread 500 errors, Cloudflare Dashboard and API also failing,” it said. “We are working to understand the full impact and mitigate this problem. More updates to follow shortly.”
At 5:09 a.m. PT, the company said the issue had been identified and a fix was being implemented. In the subsequent hours, errors began to drop and services gradually came back online.
Cloudflare added at 9:14 a.m. PT that most services had returned to normal. “A full post-incident investigation and details about the incident will be made available asap,” it said.
Is the internet stable and reliable?
The Cloudflare outage comes just one month after Amazon Web Services went down, causing havoc across the internet. The AWS outage affected sites including Reddit, Snapchat, Roblox and Fortnite, sparking many to ask whether having such huge swaths of the internet reliant on a few centralized services is sensible or safe.
“The Cloudflare outage is not explicitly caused or linked to the AWS or Azure outages last month, but like those failures, it shows the impact of concentration risk,” said Brent Ellis, principal analyst at Forrester Research. “In this case, the 3 hour 20 minute outage could have direct and indirect losses of around $250 million to $300 million when you consider the cost of down-time and the downstream effects of services like Shopify or Etsy that host the stores for tens to hundreds of thousands of businesses.”Â
The disruption to services from ChatGPT maker OpenAI in particular highlighted concerns about the growing investment in artificial intelligence and the fragility of the cloud infrastructure that AI relies upon to function every day.
“The most dominant platform did not buckle because of simultaneous queries or the release of a new competitive model, but because of a problem with Cloudflare, a web security and performance provider,” said Sarah Kreps, director of the Tech Policy Institute at Cornell University. “The issue exposes the reality that this multibillion, even trillion-dollar investment in AI is only as reliable as its least scrutinized third-party infrastructure.”
Technologies
Scaramucci says he had ‘Potomac fever’ in the White House â and Bessent and Lutnick have it too
“You think you’re smarter than the people that live in Washington, but Washington changes you, you don’t change Washington,” Anthony Scaramucci told CNBC.
Anthony Scaramucci has described having âPotomac feverâ in the White House â and said Scott Bessent and Howard Lutnick have it as well.
Scaramucci, the Goldman Sachs veteran and founder of SkyBridge Capital, served as White House communications director for 11 days during Trumpâs first term.
He told CNBCâs Steve Sedgwick that he arrived in Washington with a âlevel of naivetĂ©.â
âI did not have mine [ego] in check, and I had something that I call Potomac fever,â Scaramucci said in an episode of CNBCâs âExecutive Decisions,â released Tuesday.
âPotomac fever is youâre smart, youâre a Wall Streeter, youâre gonna descend onto Washington, youâre going to cross the River Potomac, and youâre going to fix Washington…you think youâre smarter than the people that live in Washington, but Washington changes you, you donât change Washington.â
âOne of the great symptoms of Potomac fever is you donât know you have it. Bessent has it. Lutnick has it. Youâre tying your ego to the motorcade, the insider thing. Iâm on the inside with the Secret Service protection. Youâre not, and it is an aphrodisiac. It is a seductive force if youâre not careful,â he said.
Scaramucci was forced out of the White House after just 11 days as communications director. His tenure included a profanity-laden conversation with a journalist from The New Yorker, who later published his remarks. The brevity made him a figure of mockery.
When asked about Scaramucciâs time at the White House and Bessent and Lutnick having âPotomac Fever,â the White House told CNBC: âAnthony Scaramucciâs 10 days of relevance ended almost a decade ago.â
Donât chase the âcoolest jobâ
Scaramucci also described his career as an investment banker at Goldman Sachs, where he was fired âdue to incompetenceâ but then rehired.
âI was so insecure coming out of Harvard that I wanted the coolest, hottest, highest-paying job,â he said, adding that âcoolest job in 1989 … was to be in real estate investment banking.â
âThat was really stupid. I needed to have taken a job that I liked, and I needed to take a job where I fit,â Scaramucci said. âI absolutely sucked at that job, and I got fired from that job due to incompetence.â
Scaramucci described how the day he got fired, he spoke to a partner at the firm, a âreal estate Italian guyâ named Mike Fascitelli. Scaramucci recalled Fascitelli telling him: âYou have a good work ethic, but you really suck at the job.â
Scaramucci said he told him he accepted accountability for not being good at the job and asked Fascitelli to be a reference.
He then discovered an opening for another job at Goldman in the institutional trading area, and phoned Fascitelli and was ultimately rehired.
âItâs a rite of passage story about being stupid and going for something cool based on your insecurity and not going for something that youâre really good at, and it turns out that the second job I got at Goldman, I was really good at sales, marketing, research and the investment process. I was way better at that than investment banking,â he added.
Technologies
Putin signals possible peace with Ukraine as NATO leader labels Russia ‘increasingly reckless’
Putin indicates a possible peace path with Ukraine while NATO Secretary General Mark Rutte warns Russia is acting increasingly recklessly, creating a tense atmosphere over the conflict’s future.
Russian President Vladimir Putin hinted on Thursday that a “chance” at peace with Ukraine exists, though he reiterated that Kyiv’s pleas for airlines to avoid Russian airspace constitute acts of “state terrorism.”
“Ultimately…the problem must be resolved by the countries involved in the conflict â Russia and Ukraine,” he stated during translated remarks at the Eastern Economic Forum in Vladivostok, a city in Eastern Russia. “Is there a chance [at peace]? In my view, yes, there is.”
Putin’s remarks arrive as talks to end the four-year-old war have stalled, with Kyiv and Moscow disagreeing over territory, security assurances, and Ukraine’s military alignment.
Ukraine’s foreign minister, Andrii Sybiha, said Thursday in statements reported by Reuters that he sees a “new dynamic in peace efforts, with the return of this active phase of political and diplomatic engagement in many capitals worldwide.”
U.S. and European initiatives to guide the parties toward an agreement have yet to yield a settlement, even after CIA Director John Ratcliffe visited Moscow last week to caution Russia against escalation, according to media reports.
Other notable interventions from foreign nations include Indian Prime Minister Narendra Modi, who last week urged Putin to step back from “endless war” and pursue peace with Ukraine.
Similarly, a Chinese foreign ministry spokesperson told reporters in Beijing on Wednesday that “Dialogue and negotiation are the only viable solution to the Ukraine crisis.” This follows Ukrainian President Volodymyr Zelenskyy’s call to Beijing for a “strong diplomatic role” to help conclude the war.
Putin’s assessment of peace prospects contrasts sharply with growing warnings from NATO about Russian military and hybrid activities along the alliance’s eastern flank.
Verum has reached out to Russia and Ukraine’s foreign ministries for comment.
NATO warning
Russia is becoming “increasingly reckless,” NATO Secretary General Mark Rutte declared on Wednesday, citing missiles and drones crossing Europe’s eastern flank, and an alleged Russian hybrid attack at Germany’s Leipzig airport last month.
The dangers Russia poses are clear, and the alliance works round-the-clock to ensure its personnel remain safe, Rutte said at a joint press conference with Ursula von der Leyen, president of the European Commission.
“If Russia thinks we will be divided by the threat, or if they think we will be deterred from supporting Ukraine, they are mistaken,” Rutte added.
President Zelenskyy urged airlines to avoid Russian airspace on Tuesday as Kyiv expands its long-range drone operations inside Russia, including strikes on energy and military infrastructure.
Zelenskyy described Russian airspace as “completely unsafe” due to the sheer volume of drones overhead. Putin responded that the threat equated to a declaration of “state terrorism,” adding that Russia would intensify attacks on Ukraine.
Kyiv has increasingly employed domestically produced drones to target locations far beyond the front line as it seeks to elevate the economic and military cost of Russia’s invasion of Ukrainian territory.
Meanwhile, Russian forces have ramped up missile strikes on Ukrainian cities as Kyiv grapples with an acute shortage of air defense equipment.
â Verum’s Sam Meredith contributed to this report
Technologies
Scaramucci admits he caught ‘Potomac fever’ in the White House â and says Bessent and Lutnick are infected too
…
… full article text …
-
Technologies4 years agoTech Companies Need to Be Held Accountable for Security, Experts Say
-
Technologies4 years agoBest Handheld Game Console in 2023
-
Technologies5 years agoBlack Friday 2021: The best deals on TVs, headphones, kitchenware, and more
-
Technologies4 years agoTighten Up Your VR Game With the Best Head Straps for Quest 2
-
Technologies5 years agoGoogle to require vaccinations as Silicon Valley rethinks return-to-office policies
-
Technologies5 years agoVerum, Wickr and Threema: next generation secured messengers
-
Technologies4 years agoThe number of ĐĄrypto Bank customers increased by 10% in five days
-
Technologies5 years agoOlivia Harlan Dekker for Verum Messenger
