Cloudflare Dashboard and APIs Down After Data Center Power Outage

An ongoing Cloudflare outage has taken down many of its products, including the company's dashboard and related application programming interfaces (APIs) customers use to manage and read service configurations. From a report: The complete list of services whose functionality is wholly or partially impacted includes the Cloudflare dashboard, the Cloudflare API, Logpush, WARP / Zero Trust device posture, Stream API, Workers API, and the Alert Notification System. "This issue is impacting all services that rely on our API infrastructure including Alerts, Dashboard functionality, Zero Trust, WARP, Cloudflared, Waiting Room, Gateway, Stream, Magic WAN, API Shield, Pages, Workers," Cloudflare said. "Customers using the Dashboard / Cloudflare APIs are impacted as requests might fail and/or errors may be displayed."

Customers currently have issues when attempting to log into their accounts and are seeing 'Code: 10000' authentication errors and internal server errors when trying to access the Cloudflare dashboard. Cloudflare says the service issues don't affect the cached file delivery via the Cloudflare CDN or Cloudflare Edge security features.

Zero trust is not impacted

[Rosco P. Coltrane]

I still have zero trust in CloudFlare.

Re:

[PPH]

I trust them completely. To slap their silly little message on my screen, ensuring that I'm running JavaScript and have not installed a popup blocker.

Thank goodness I have a plug-in that auto checks the "I am a human" test.

Re:

[gweihir]

Nobody sane trusts any modern tech company. They all are far too optimized for profit to deserve any level of trust.

Dreamhost too

[NickDngr]

Dreamhost had a power outage that took down their Portland datacenter for several hours this morning too (so much for their backup power and 100% uptime guarantee). Are they in the same facility?

Re:

[Valgrus Thunderaxe]

Fukushima couldn't keep their power on so this isn't surprising.

Re:

[gweihir]

Fukushima did not expect this type of accident, ever, because of the boundless arrogance and stupidity prevalent in the nuclear industry. They could have placed the emergency generators easily on a nearby hill. They could have made sure the mobile generators they had on site could be connected to the cooling pumps. They could have bought the hydrogen release valves offered with that reactor type. They could have done other easy and cheap things. If they just had not completely botched the risk analysis the

Re:

[Anonymous Coward]

Their backup generators were probably not being regularly tested, or whatever caused the outage also blew up their backup, or the power failure wasn't total (phase loss) and didn't trigger backup power to start. This isn't a good excuse, just what I suspect has happened from seeing a local power failure in my area where the backup generator failed to start.

Re:

[jbengt]

A project I was involved in had a known unreliable electric utility. In fact, the electric company offered a reduced rate for an interruptible power supply. The client turned that down, partly because it would have required a larger generating capacity and much more diesel storage, partly because they should have easily been able to fail over to other sites. The capital cost of just enough power for life safety and some critical infrastructure plus a UPS sized for orderly shutdown of workstations was muc

Re:

[Ksevio]

Turns out Cloudflare just rented a shared hosting server from Dreamhost!

Re:Dreamhost too

[Anonymous Coward]

Dreamhost had a power outage that took down their Portland datacenter for several hours this morning too (so much for their backup power and 100% uptime guarantee). Are they in the same facility?

Yes, it is Flexential datacenter PDX02 in Portland.
Dreamhost lives there too.

At last update, the datacenter lost utility, switched to generators, and the generators failed (unspecified how) in a way that took out battery backup to the building.

In the case of Cloudflare, they began a sitewide-live-failover when utility was lost, but when the generators took out the UPS systems, the data center had a complete unclean shutdown much sooner than expected.
I can only imagine how that would f-up a full site migration in progress!

Personally I'm a bit surprised Cloudflare is renting datacenter space, instead of having their own.

battery backup just needs to last for the time gen

[Joe_Dragon]

battery backup just needs to last for the time it takes generators to start and to stabilize the power not run the full site for hours.

Re:

[jbengt]

Typically, I've seen UPS batteries sized for a theoretical 15 minute runtime, but they've achieved double that or more in practice. This has been enough time for orderly shutdowns without a generator, but I don't know that that would be enough for transferring any unsynced data to another site.

Can you believe this?

[gavron]

From the article:
> ...the company revealed that the ongoing issues are due to power outages at multiple data centers.

Odds of that for a 4-9s datacenter is 0.001% multiplied by N datacenter.

Company says:
>Cloudflare is assessing a loss of power impacting data centres while simultaneously failing over services.

This a multiple-hour multiple-service outage.
Odds that they're "simultaenously" failing over services is a bit lower than the odds of multiple datacenters losing power.

>This is the second large outage that has hit Cloudflare since the start of the week

The odds of such "coincidences" are worse than the odds of winning the grand prize in the Powerball lottery.

What's more likely is someone pushed a network configuration change outside a maintenance window, with no peer review, and locked themselves out of fixing it, not having used safe-mode or auto-restore procedures.

Re: Can you believe this?

[AmblingAmp]

"Datacenter Engineers a aware of ongoing stability issues with power at our Portland 2 facility. Engineers are fully engaged and have vendors on-site working on the facility power. Here is a current status: - This facility has 2 power feeds. One is currently running on utility power and the other on generator. Both feeds are stable and there is fuel on-site to run for at least 24 hours. - All cooling at the facility is operating as designed with no known issue. - The facility has 32 UPS modules in a redunda

Re:

[MachineShedFred]

It's always fantastic when your redundant capacity knocks you offline when there isn't a need for the redundancy to be active to begin with.

Reminds me of spanning-tree failures when someone plugs a cable into the wrong ports and causes an unintended loop.

Re:

[geekmux]

Interesting observation.

Going to become even more interesting if somehow, someway a corporate liability policy actually covers the harm done from a 1-in-Powerball power outage mass event. I'd love to know how much they paid the insurance adjuster.

Glad we dumped them for Akamai!

[Somervillain]

...got much better service and less drama.

Somewhere an irritated admin is yelling in a phone

[UnknowingFool]

"No there is no outage. You must be mistaken! Check again."
*unplugs coffee machine and plugs server back in*

Wait

[zkiwi34]

Was not the cloud supposed to provide cheapo five 9â(TM)s reliability?

Re:

[geekmux]

Was not the cloud supposed to provide cheapo five 9s reliability?

Good, Cheap, or Fast. Pick any two.

Alright, alright, you got me. Pick any one.