Cloudflare Dashboard and APIs Down After Data Center Power Outage (bleepingcomputer.com) 22
An ongoing Cloudflare outage has taken down many of its products, including the company's dashboard and related application programming interfaces (APIs) customers use to manage and read service configurations. From a report: The complete list of services whose functionality is wholly or partially impacted includes the Cloudflare dashboard, the Cloudflare API, Logpush, WARP / Zero Trust device posture, Stream API, Workers API, and the Alert Notification System. "This issue is impacting all services that rely on our API infrastructure including Alerts, Dashboard functionality, Zero Trust, WARP, Cloudflared, Waiting Room, Gateway, Stream, Magic WAN, API Shield, Pages, Workers," Cloudflare said. "Customers using the Dashboard / Cloudflare APIs are impacted as requests might fail and/or errors may be displayed."
Customers currently have issues when attempting to log into their accounts and are seeing 'Code: 10000' authentication errors and internal server errors when trying to access the Cloudflare dashboard. Cloudflare says the service issues don't affect the cached file delivery via the Cloudflare CDN or Cloudflare Edge security features.
Customers currently have issues when attempting to log into their accounts and are seeing 'Code: 10000' authentication errors and internal server errors when trying to access the Cloudflare dashboard. Cloudflare says the service issues don't affect the cached file delivery via the Cloudflare CDN or Cloudflare Edge security features.
Zero trust is not impacted (Score:5, Insightful)
I still have zero trust in CloudFlare.
Re: (Score:2)
I trust them completely. To slap their silly little message on my screen, ensuring that I'm running JavaScript and have not installed a popup blocker.
Thank goodness I have a plug-in that auto checks the "I am a human" test.
Re: (Score:2)
Nobody sane trusts any modern tech company. They all are far too optimized for profit to deserve any level of trust.
Dreamhost too (Score:3)
Re: (Score:2)
Re: (Score:2)
Fukushima did not expect this type of accident, ever, because of the boundless arrogance and stupidity prevalent in the nuclear industry. They could have placed the emergency generators easily on a nearby hill. They could have made sure the mobile generators they had on site could be connected to the cooling pumps. They could have bought the hydrogen release valves offered with that reactor type. They could have done other easy and cheap things. If they just had not completely botched the risk analysis the
Re: (Score:1)
Their backup generators were probably not being regularly tested, or whatever caused the outage also blew up their backup, or the power failure wasn't total (phase loss) and didn't trigger backup power to start. This isn't a good excuse, just what I suspect has happened from seeing a local power failure in my area where the backup generator failed to start.
Re: (Score:3)
Re: (Score:2)
Turns out Cloudflare just rented a shared hosting server from Dreamhost!
Re:Dreamhost too (Score:4, Informative)
Dreamhost had a power outage that took down their Portland datacenter for several hours this morning too (so much for their backup power and 100% uptime guarantee). Are they in the same facility?
Yes, it is Flexential datacenter PDX02 in Portland.
Dreamhost lives there too.
At last update, the datacenter lost utility, switched to generators, and the generators failed (unspecified how) in a way that took out battery backup to the building.
In the case of Cloudflare, they began a sitewide-live-failover when utility was lost, but when the generators took out the UPS systems, the data center had a complete unclean shutdown much sooner than expected.
I can only imagine how that would f-up a full site migration in progress!
Personally I'm a bit surprised Cloudflare is renting datacenter space, instead of having their own.
battery backup just needs to last for the time gen (Score:1)
battery backup just needs to last for the time it takes generators to start and to stabilize the power not run the full site for hours.
Re: (Score:2)
Can you believe this? (Score:4, Interesting)
From the article: ...the company revealed that the ongoing issues are due to power outages at multiple data centers.
>
Odds of that for a 4-9s datacenter is 0.001% multiplied by N datacenter.
Company says:
>Cloudflare is assessing a loss of power impacting data centres while simultaneously failing over services.
This a multiple-hour multiple-service outage.
Odds that they're "simultaenously" failing over services is a bit lower than the odds of multiple datacenters losing power.
>This is the second large outage that has hit Cloudflare since the start of the week
The odds of such "coincidences" are worse than the odds of winning the grand prize in the Powerball lottery.
What's more likely is someone pushed a network configuration change outside a maintenance window, with no peer review, and locked themselves out of fixing it, not having used safe-mode or auto-restore procedures.
Re: Can you believe this? (Score:1)
Re: (Score:3)
It's always fantastic when your redundant capacity knocks you offline when there isn't a need for the redundancy to be active to begin with.
Reminds me of spanning-tree failures when someone plugs a cable into the wrong ports and causes an unintended loop.
Re: (Score:2)
Interesting observation.
Going to become even more interesting if somehow, someway a corporate liability policy actually covers the harm done from a 1-in-Powerball power outage mass event. I'd love to know how much they paid the insurance adjuster.
Glad we dumped them for Akamai! (Score:2)
Somewhere an irritated admin is yelling in a phone (Score:2)
*unplugs coffee machine and plugs server back in*
Wait (Score:2)
Re: (Score:3)
Was not the cloud supposed to provide cheapo five 9s reliability?
Good, Cheap, or Fast. Pick any two.
Alright, alright, you got me. Pick any one.