Southwest Delayed Hundreds of Departures Due To a Networking Glitch

Southwest Airlines has fixed a technical issue that delayed hundreds of flights across the country. In a statement, Southwest Airlines spokesperson Dan Landson says the company resumed operations after working through "data connection issues resulting from a firewall failure." From a report: The airline started having issues at around 10:30AM ET, with data from FlightAware suggesting that over 1,700 Southwest flights have been delayed so far. The Federal Aviation Administration paused departures at the request of Southwest Airlines around this time and later unpaused flights at 11:10AM ET. "Early this morning, a vendor-supplied firewall went down and connection to some operational data was unexpectedly lost," Landson says. "Southwest Teams worked quickly to minimize flight disruptions."

wat

[drinkypoo]

"data connection issues resulting from a firewall failure" never happened

"data connection issues resulting from a failure to install redundant equipment" is a thing

A more common and probable thing is "data connection issues resulting from a firewall configuration failure"

Re: wat

[reanjr]

Are you basing this assessment on anything? Firewalls can - and do - fail. Obviously, redundancy is critical, but it's not uncommon for failure modes to be poorly tested.

Re:

[drinkypoo]

Obviously, redundancy is critical, but it's not uncommon for failure modes to be poorly tested.

Which brings us back to misconfiguration error

Re: wat

[reanjr]

It can be both. A hardware firewall can fail, but due to misconfiguration, the secondary firewall doesn't immediately pick up the traffic.

Re:

[John Smith 2297]

Your brain is misconfigured. You are a lazy piece of shit and do not deserve to live. You contribute nothing.

Re:

[John Smith 2297]

You fail at life. Can't even support yourself. Good for nothing parasite. Go take a long walk off a short pier, you piece of shit.

Re:wat

[xforce]

Repeat after me: Redundancy is not redundancy if not tested, and not tested periodically.

Re: wat

[MobileTatsu-NJG]

Redundancy is not redundancy if not tested, and not tested periodically. ...and if ya liked that one, stick around for my skit about airline food!

Re:wat

[kdekorte]

Probably an SSL cert expired.. happens all the time.

Re:

[Virtucon]

Operations usually have things like dashboards that monitor critical data paths too.

airlines really need to start upgrading old IT

[Joe_Dragon]

airlines really need to start upgrading old IT systems.
It seems that lots of them are still running very old systems that can fail big time.

Re: airlines really need to start upgrading old IT

[S_Stout]

They have no incentive. Government just hands them money when they screw up.

That's not the problem

[rsilvergun]

the problem is gov't let the airlines merge, so there's only a few left. It's not incentive they need, it's competition.

We have so few airlines at this point they should just be nationalized. As long as we don't let the pro-privatization crowd sabotage things (like we always do... sigh...) it can only get better.

Re:

[antdude]

Government and many others have this same problem. :(

Re:

[ArchieBunker]

Age doesn't mean a whole lot. I have more faith in a 10 year old AS/400 than the latest cloud offerings or anything Microsoft based. This is a budget airline we're talking about here.

Windows rebooted for updates again?

[xack]

And then stayed stuck on the boot screen because it wanted a login to an ms account.

Iâ(TM)m going to go with stupid human error

[ajmcello78]

Should have a staging/test bed but this is the preferred âoediscountâ airline. December was also a bad month for SW if anyone remembers. âoe More than 90% of Wednesday's US flight cancellations were Southwest flights, according to flight tracking website FlightAware. Southwest canceled more than 2,500 flights. The next highest: SkyWest, with 77.â If I was a stock market shorter, Iâ(TM)d wager a short position rn.

Re:

[DarkOx]

yeah staging/test....

You know for route/switch/firewall is the one area where I have never really seen (even at the two F50 companies I worked for) have representative stage test environment.

Its just to expensive, to really duplicate the network fabric layer for a test environment. So what you end up with is a bunch of smaller (not identical same generation if you are lucky) equipment in a lab and you hope it works the same on the real hardware and there are no gotchas.

Then you try it out at the 'hot' site

Re: Iâ(TM)m going to go with stupid human err

[ajmcello78]

Completely agree. Looks like my encoding or something is borked ðY Worked a lot with redundant routers, load balancing and NIC teaming (tho most of the time they are plugged into the same switch, lol). Firewall redundancy except on routers, I have not seen this.

Re:

[DarkOx]

redundant firewalls are really pretty common. they even replicate tcp state to each other so that active connections can survive, but if foul up a rule change, the firewall is intentionally a choke point, unlike a router, so there probably is no other path for the traffic.

That is really my point on that one.

MBA?

[Tablizer]

I bet several years ago an executive fired the IT upgrade teams to "trim the fat", got big fat bonus for cost cutting, retired with a big stack of money, and is laughing now at the poor schmucks left holding his bag.

Re:

[Black Parrot]

Or still works there and browned his shorts.

Again?

[smithmc]

< crosses arms > What, again? < womp womp >

Seen this before...

[rickb928]

...and if you let your platform get stale, you can't recover fast enough to survive.

Sad, SWA was cool but it's old and cold. Dead on its feet.