One Failed NIC Strands 20,000 At LAX

The card in question experienced a partial failure that started about 12:50 p.m. Saturday, said Jennifer Connors, a chief in the office of field operations for the Customs and Border Protection agency. As data overloaded the system, a domino effect occurred with other computer network cards, eventually causing a total system failure. A spokeswoman for the airports agency said airport and customs officials are discussing how to handle a similar incident should it occur in the future.

That's all it takes

[Marxist Hacker 42]

Though I heard it was a switch. Same idea though- all it takes is one malfunctioning card flooding the LAN with bad packets to bring it all down.

Re:

[Jeremiah Cornelius]

Then that would lead me to think "hub", not switch. Or just a really shitty switch...

Re:

[COMON$]

apparently you are not familliar with what a bad nic does to even the best of switches.

Re:That's all it takes

[KillerCow]

I am not a networks guy... but it's my understanding that a switch acts like a hub when it sees a TO: MAC address that it doesn't know what port it's on. They learn the switching structure of a network by watching the FROM fields on the datagrams. When the switch powers up, it behaves exactly like a hub and just watches/learns what MAC addresses are on which ports and builds a switching table. If it starts getting garbage packets, it will look at the TO field and say "I don't know what port this should go out on, so I have to send it on all of them." So garbage packets would overwhelm a network even if it was switched.

It would take a router to stop this from happening. I don't think that there are many networks that use routers for internal partitioning. Even then, that entire network behind that router would be flooded.

It depends on the switch

[camperdave]

You're right to a point. An ethernet frame, along with the source and destination addresses, has a checksum. A switch that is using a store and forward procedure is supposed to drop the frame if the checksum is invalid. If the nic was throwing garbled frames onto the network, it would have to be garbled in such a way as to have a valid checksum (assuming they are using store and forward switches in the first place).

Re:That's all it takes

[Vengance Daemon]

Why are you assuming that this is an Ethernet network? As old as the equipment they are using is, it may be a Token Ring network - the symptoms that were described sound just like a "beaconing" token ring network.

Re:

[HowIsMyDriving?]

That is the first thing I thought, "I bet they are still using Token Ring." Man, when a Token Ring card went bad, it was hell on the network, nothing worked because the token would not get passed properly.

damn tokens...

[myowntrueself]

Man, when a Token Ring card went bad, it was hell on the network, nothing worked because the token would not get passed properly.

The worst thing is when a user decides to unplug the cable to move something or whatever. Then the token can fall out and you have to spend hours on your hands and knees with a magnifying glass trying to find the damn thing!

Its true! I saw it in a Dilbert cartoon!

Re:

[scottv67]

I don't think that there are many networks that use routers for internal partitioning.

We have *tons* of routers that separate the various subnets (which map 1-to-1 to VLANs) on our internal network. How do you get from one "broadcast domain" to another? Via the default gateway on your own subnet. That is passing through a router. (It may not be a physical box called a 'router' but the packets are still being routed).

References?

[TypoNAM]

Got any references or links to various tutorials and/or documents on how I could setup my network to notify me about a rogue NIC?

Re:That's all it takes

[Svet-Am]

Of course they're running old and outdated hardware. When thing work, particularly in a mission critical situation, you don't touch them! Even if the IT admins knew that computer was old and on the brink of dying, how are they supposed to convince the suits and beancounters of that? Non-technical people take the approach that since computers are inherently binary (work or no-work) that if the machine is up and running _right now_ then there is no problem and no sense on spending money to replace it.

If the IT folks were clueless about this machine's age or condition, then the blame lies solely with them for not knowing what the hell they were doing. However, if it was the other folks who shot the IT folks down about upgrading then "welcome to the current state of business", unfortunately.

Re:That's all it takes

[EmperorKagato]

Even if the IT admins knew that computer was old and on the brink of dying, how are they supposed to convince the suits and beancounters of that?

You show the suits and bean counters how much it costs the company if the system failed and time was spent recovering that system.

Re:That's all it takes

[quanticle]

You show the suits and bean counters how much it costs the company if the system failed and time was spent recovering that system.

That's very difficult to do, and your estimates of the costs will be called into question. Its often impossible to predict how long it'll take to diagnose and fix a problem unless you've already diagnosed and fixed a similar problem.

Making this kind of estimate also places you into a lose-lose position. If your estimate was high, then management sees you as "chicken little" and will be more likely to dismiss further concerns as more fearmongering. If your estimate was low, then the blame for the outage will cascade down onto you for not showing/convincing management that new equipment was needed.

Re:

[itwerx]

That's very difficult to do, and your estimates of the costs will be called into question.

Right, but that's why IT doesn't provide the numbers. It just provides the scenario and it's the bean-counters (BC) that provide the numbers.

IT: "We have some really old hardware that's going to fail any day now..."

BC: "So what?"

IT: "Well, that's a good question, we know it's going to cost $Bazillion to fix so we need to find out if it's worth it or not. Here's what will happen when it dies - LAX completely shuts dow

Re:

[dbIII]

Then they do not believe you until you can point at 20,000 people stranded at LAX. At this point you are fired since you knew about the problem, made some fuss, but did not make enough fuss to actually convice the suits and bean counters. It does help others that can then point at the problem of somebody else and get their suits and bean counters to pay attention. This is why infrastructure failure disasters go in cycles determined by the attention span and age of management - each new generation has to

Re:That's all it takes

[dbIII]

No - it implies a great deal of management has become a shallow oral tradition with all the problems that implies. They are not learning from anything before them and react with great surprise when a Rupert Murdoch or a Bill Gates that does know how to learn from the mistakes of others leaves them with effectively nothing but their underwear. It's like Cortez in South America - he used tactics of Roman Generals that he had read about against those that did not have a written history.

In contrast technical staff get to hear a lot about the Tacoma Narrows Bridge, Liberty Ships, Titanic or similar disasters from long ago as illustrations of how things can go wrong before they get let out of their first year of training. Some management would discard those lessons as things from the days of dinosaurs which is why we seem to have maintainance, infrastructure and contingincy plans reduced to nothing every decade and then be seen as important in the years immediately following a string of expensive or deadly disasters.

Re:That's all it takes

[ThinkingInBinary]

Of course they're running old and outdated hardware. When thing work, particularly in a mission critical situation, you don't touch them! Even if the IT admins knew that computer was old and on the brink of dying, how are they supposed to convince the suits and beancounters of that? Non-technical people take the approach that since computers are inherently binary (work or no-work) that if the machine is up and running _right now_ then there is no problem and no sense on spending money to replace it.

There's no reason you can't leave the almost-broken computer there and get a new one. You just build a backup system. Surely management understands that redundancy is good. Then, when the crappy one breaks, you can swap it out instantly. That way, you don't have to mess with things prematurely, but you're only down for hopefully a few minutes. (Of course, replacing it "intentionally", before it fails, is more reliable, but keeping a backup system is a viable alternative if nobody wants to touch the working system.)

Re:

[Greventls]

The new system is usually extremely expensive. Why spend all that money on a new system when the old one works? I know programmers who refuse to update their code from VB3.

Re:

[quanticle]

Surely management understands that redundancy is good.

No. In managements' eyes, redundancy is bad. You're paying twice as much, but you're not getting any extra functionality in return.

Re:

[leeward]

Of course they're running old and outdated hardware. When thing work, particularly in a mission critical situation, you don't touch them! Even if the IT admins knew that computer was old and on the brink of dying, how are they supposed to convince the suits and beancounters of that?

And why shouldn't the bean counters expect the old, outdated hardware to work. Quite a bit of the air traffic control system is hardware that is decades old, though it is slowly being replaced. You have to go to the bean count

The scope of the problem

[WheelDweller]

I agree, but the scope of the problem is much larger.

Americans are still designing systems (and I'm talking WHOLE systems, not just the computers) for the industrial revolution. Much the same way, we're educating our kids for the same purpose- to make them cogs for manufacturing.

The Japanese have a more 'cellular' structure, as opposed to the 'pyramid' designed back a couple of 'turns of the century' ago. One man on top drives five, who drive 200, who drive them all. But the Japanese model is more like object orientation: each unit has private parts. So long as the command it's given produces the proper results and stays within budget, who cares?

Assembly lines gather at their meetings and decide policy on their own. "Fred has been late 3 times this week; do we care?" and the only people to whom it matters, decide. There's no need for a strict, top-down policy, especially since only tiny organizations all do only one job.

Imagine the broken structures in a holding company; they own a newspaper, a carwash and a grocery store; the top man can't say "We'll only use glass containers", because that would be a disaster in a car wash. They can't say "we choose leaded inks" which might be fine for the car wash, but danger at the newspaper. Each unit has it's own purpose.

So how about giving the network admins the power to do *whatever* it takes to let them keep the equipment up to date? As long as it runs, under budget, and doesn't get'em on the newspapers, who cares about the specifics? Why not let the unused budget from every year sit in an account (not being taken back) and use THAT to improve infrastructure?

If these guys were able to have that kind of control, this discussion wouldn't be happening.

Re:That's all it takes

[Kadin2048]

Would you think that LAX is running anything that out-of-date or crappy?

I assume that they're running everything with spit, duct tape, wishful thinking, ancient custom software, near-fossilized hardware, and Excel spreadsheets ... just like pretty much everything else in the public sector.

I've seen what's running some government agencies, and it's frightening.

Re:

[spun]

I work in the public sector, and we don't use spit or duct tape much. We have custom software, it's not not ancient but it's written in COBOL anyway. The hardware is mostly new IBM blades and blade centers and we're phasing out the older stuff. We use Access databases, not Excel spreadsheets. But then, we're a state agency, not the Federal Government, so we may be doing it wrong.

Re:

[Kadin2048]

I pity you, your state and everyone else using Access.

Yeah, Access is a piece of shit. Unfortunately, it's a lot better than using Excel as a database, which is in many cases the alternative that I've witnessed.

There are also a lack of alternatives: you have FileMakerPro, which is neat (I like it) but not very appealing to some because it has a significant learning curve compared to Access and is also proprietary and expensive; aside from that you have OO.org's Base, which is still immature; and then you've

Re:

[Shagg]

Would you think that LAX is running anything that out-of-date or crappy?

I'm surprised they're even using IP. Many airline systems are still running on X.25 and mainframes.

So, no, running out-of-date hardware wouldn't surprise me at all.

Re:

[COMON$]

Having worked for the gov't I think you underestimate the quality of employees there...how does that saying go "Two things are infitite the universe and gov't stupidity?". Could be a hack but they wouldnt know unless they brought in someone from the private sector who is smart enough to charge a bagillion an hour to show them how to properly plug in the nic.

Yes, I am glad to be out of that velvet lined rut and in a world where there are actual professionals.

Re:

[COMON$]

Yes, but this is slashdot so I feel the compulsive need to render quotes in a way to fit my needs.

Re:

[SatanicPuppy]

Sure, if you're buying consumer grade switching hardware, and you have only one subnet, or all your subnets are weirdly bridged or whatever.

For my money, this should never have happened from a problem with one machine. That's wholly unacceptable. My home network is robust enough to handle one bad machine without going down completely...Hell, I could lose a whole subnet and no one on the other subnet would notice a thing.

If this system or switch or whatever is critical, there should have been a fail over. Th

Whiskey Tango Foxtrot

[SatanicPuppy]

According to the effing article, it wasn't even a server, but a goddamn desktop. How in the holy hell does a desktop take down the whole system? I can't even conceive of a situation where that could be the case on anything other than a network designed by chimps, especially through a hardware failure...A compromised system might be able to do it, but a system just going dark?

For that to have had any effect at all, that system must have been the lynchpin for a critical piece of the network...probably some Homeland security abortion tacked on to the network, or some such crap...This is like the time I traced a network meltdown to a 4 port hub (not a switch, and unmanaged hub) that was plugged into (not a joke) a T-3 concentrator on one port, and and three subnets of around 200 computers each on the other 3 ports. Every single one of the outbound cables from the $15.00 hub terminated in a piece of networking infrastructure costing not less than $10,000 dollars.

This is like that. Single point of failure in the worst possible way. Gross incompetence, shortsightedness, and general disregard for things like "uptime"; pretty much what we've come to expect from the airline industry these days. If I'm not flying myself, I'm going to be driving, sailing, or riding a goddamn bicycle before I fly commercial.

Re:Whiskey Tango Foxtrot

[Jeremiah Cornelius]

Well.

Token ring sure used to fail like this! 1 bad station sending 10,000 ring-purge messages a second? Still, it was a truck. Files under 1Mb could be transferred, and this was TR/4, not 16!

Re:

[sigipickl]

This totally sounds like a token ring problem.... Either network flooding or dropped packets (tokens). These issues used to be a bear to track down- going from machine to machine in serial from the MAU...

Ethernet and switching has made me fat- I never have to leave my desk to troubleshoot.

Re:

[hedley]

Back in the late 80's we had a network of Apollo DN300's. Days of lost time when the ring went down. Once and entire day lost. IT scrambling about in the ceiling tiles trying to TDR the cable, finally, a NIC out in a workstation in a closed office, the employee was on vacation. Whilst on vacation that NIC, like a bad xmas tree bulb took down the whole lan whilst it wouldn't play nice and pass on the token. Total junk and a total waste of time. I used to dream of taking that Apollo from CA to Chemlsford mas

Re:

[flappinbooger]

But Token Rings are, like, obsolete and stuff, surely there wouldn't be something that obsolete in a place like an airport, right?

Right?

[crickets chirping]

Right?

Re:Whiskey Tango Foxtrot

[Jaxoreth]

Still, it was a truck.

Which explains why it's not used in the Internet.

Token Ring upgrade

[camperdave]

Perhaps they upgraded their token ring to thinnet.

Re:

[mhall119]

A compromised system might be able to do it, but a system just going dark?

The article says it was a partial failure, so I'm guessing the NIC didn't "go dark", instead it started flooding the network with bad packets.

Re:Whiskey Tango Foxtrot

[MightyMartian]

If the NIC starts broadcasting like nuts, it will overwhelm everything on the segment. If you have a flat network topology, then kla-boom, everything goes down the shits. A semi-decent switch ought to deal with a broadcast storm. The best way to deal with it is to split your network up, thus rendering the scope of such an incident significantly smaller.

Re:

[SatanicPuppy]

Yup. I've never really seen a situation where you'd have more than a dozen or so computers on a crappy layer 1 switch. Higher quality hardware would throttle this stuff down to the very most local layer, unless you're specifically multicasting across the whole network, which is a security horror story.

Re:

[Spazmania]

And even a marginally competent network administrator ought to be able to recognize that they face a packet storm and isolate the problem in about 30 minutes through the simple expedient of, "Unplug this switch. Did the problem stop? No. Plug it back in and unplug the next switch."

I'll bet the sucker not only keeps his job but gets a commendation for finding the problem.

Re:

[Solandri]

Yeah, I had that happen at a small business I consulted for. Their flat LAN died. I eventually tracked the problem down to a cheap unmanaged switch which had a network cable plugged into it for people to plug their laptops into. Whoever used it last thought leaving the unplugged cable laying on the desk looked untidy, so they "helpfully" plugged it into an empty socket on the same switch.

Re:

[Billosaur]

And beyond that... how come there is no redundancy? After 9/11, every IT organization on the planet began making sure there was some form or fail-over to a backup system or disaster recovery site to ensure that critical systems could not go down as the result of something similar or some other large-scale disaster. Not only was this system cobbled together apparently, there was no regard for the possibility of it failing for any reason.

Re:

[dave562]

They concentrated all of the redundancy dollars into layer B of the OSI model... the bureaucracy. There wasn't anything left for the lower layers.

Re:

[ZachPruckowski]

Redundancy only helps when you have a system that stops working, not one that malfunctions:

For instance, imagine a RAID 1 in which the data is becoming corrupted. Having redundancy doesn't help: you just have two copies of a corrupted file.

In this instance, a network card started spewing out crap. Because it could fill it's pipe, and most of the packets were rebroadcast down most of the other cables, they also filled those cables.

Re:

[DA-MAN]

Um no? A number of large organizations do not have a disaster recovery site. Just the other day Cisco.com was down for a few hours.

Business continuity style disaster recovery doesn't really take public facing websites into account as high priority. Usually it's the payroll, accounts receivable and things needed to keep a business moving forward in case of a disaster.

Letting customers visit your public website is probably the lowest priority in recovering from an actual disaster.

Re:

[gEvil (beta)]

I told the boss we should get a proper network connection. But noooooo, he insisted that getting a consumer-level DSL connection and using Windows Internet Connection Sharing was the way to go...

Re:

[Joe The Dragon]

that is what the M$ tests talk about in there network setups.

Re:

[jafiwam]

According to the effing article, it wasn't even a server, but a goddamn desktop. How in the holy hell does a desktop take down the whole system? I can't even conceive of a situation where that could be the case on anything other than a network designed by chimps, especially through a hardware failure...A compromised system might be able to do it, but a system just going dark?

Los Angeles World Airports is a unique system of four airports owned and operated by the City of Los Angeles.

Any further questions?

Probably the lowest bidder union labor designing and setting it up. Shoulda called IBM.

Re:

[UbuntuDupe]

How in the holy hell does a desktop take down the whole system? I can't even conceive of a situation where that could be the case on anything other than a network designed by chimps

Heh, I think you're starting to get the sensation I had when one tiny error in GRUB locked me out of my computer entirely, to the point where even having the Ubuntu Install CD couldn't gain me any access to any OS whatsoever.

Geez, what kind of chimp would allow such a damaging failure to occur along such a vital path, right?

Re:

[SatanicPuppy]

No, not really. You've got to expect to lose machines; failure happens. Could have been a motherboard or a power supply. I'd still expect you to be able to boot from CD though. You try knoppix? You should be able to boot to knoppix, then mount the /boot partition and have your way with grub.

The thing is, a network topology is wildly different from a computer. It should be designed for parts of it to drop off, and parts to go berserk...These things happen all the time. It should be designed with a minimum of

Re:

[UbuntuDupe]

No, not really. You've got to expect to lose machines; failure happens. Could have been a motherboard or a power supply.

Yeah, then I could go to the local store, buy a part, replace it, and move on with my life, like I've done before (a power supply did in fact go out on me before).

I'd still expect you to be able to boot from CD though. You try knoppix? You should be able to boot to knoppix, then mount the /boot partition and have your way with grub.

Hey! Capital idea! I'd download it on that computer I c

Re:

[quizwedge]

[Quote]This is like the time I traced a network meltdown to a 4 port hub (not a switch, and unmanaged hub) that was plugged into (not a joke) a T-3 concentrator on one port, and and three subnets of around 200 computers each on the other 3 ports. Every single one of the outbound cables from the $15.00 hub terminated in a piece of networking infrastructure costing not less than $10,000 dollars.[/Quote]

LOL. Perhaps they ran out of funding after buying all of the rest of the hardware? :)

Re:

[MightyMartian]

Yessirree. Let's put one of the busiest airports on the planet on a paper-messenger boy backup. Yeah, that'll clear the backlog real well.

Re:

[SatanicPuppy]

Oh, no doubt. It's clearly what happened...This kind of thing is almost impossible with modern switching hardware, and not even the really expensive stuff, but the reasonable consumer stuff as well.

Fricking stupid. People think it'll never come back to bite them, and it always does.

Re:

[kylemonger]

Do these people hire idiots with no training or experience or what?

I think just hiring idiots would be enough. No need to train them.

Re:

[Kadin2048]

Do these people hire idiots with no training or experience or what?

Probably they do to some extent, but if it's like other places I've worked, they probably hire people who have a clue, but then tell them to do little bits and pieces, and never give them enough resources to actually do the job right.

It's a lot of "we'll pay you to come out and install this." They don't want to hear 'well, you should really re-think the architecture of your whole network' as a response. They just want the new piece grafted on, and if you don't do the job, they'll just find somebody who wil

Re:

[SatanicPuppy]

So? Does it give you confidence in the rest of their equipment when one misbehaving computer can bring down their entire network for nine hours?

Bunch of monkeys. The reason I don't fly commercial anymore has nothing to do with the planes. It has everything to do with the airports.

Re:

[badasscat]

So? Does it give you confidence in the rest of their equipment when one misbehaving computer can bring down their entire network for nine hours?

The point is ATC is not controlled by "their equipment". The airport authority and the FAA are two wholly separate agencies.

It would be like worrying about the strength of the US Army based on the fact that a prisoner escaped from a police car owned by the NYPD. Sure, both agencies involve guys carrying guns, but they otherwise have nothing to do with each other.

T

Re:

[Kadin2048]

Idiot intern nonwithstanding, should that really have been possible? I thought that any decent router would see that a loop had occurred and shut down the port connected to it, rather than forwarding all the broadcast storm packets.

After all, preventing layer 2 loops is what Spanning Tree is all about, and I thought Cisco had some similar system for figuring out if a link was unidirectional (if you're sending packets down to something and not getting anything back, it can shut it down, to keep it from just

Re:

[Lehk228]

i think the problem occurs when the switches "learn" a circular route and so packets get stuck in that circular route at whatever speed the network runs at.

In other news...

[djupedal]

"...said airport and customs officials are discussing how to handle a similar incident should it occur in the future."

What makes them think they'll get another shot? Rank and file voters are ready with their own plan...should a 'similar incident' by the same fools happen again.

The backup plan

[Animats]

DHS's idea of a "backup plan" will probably be to build a huge fenced area into which to dump arriving passengers when their systems are down.

Re:

[djupedal]

:)

I hear EMA has several new/used camp trailers I'm sure DHS could avail themselves of.

No, a multi-front plan

[EmbeddedJanitor]

Arrest all NIC designers, engineers, network stack developers, IT managers,... on suspicion of conspiring to cause the problem.

Change to Wifi because that can't have NIC faults.

C'mon folk... help me out here!

Re:

[toetagger1]

Arrest all NIC designers, engineers, network stack developers, IT managers,... on suspicion of conspiring to cause the problem. Change to Wifi because that can't have NIC faults. C'mon folk... help me out here!

Print each package to be sent over the network, use the USPS first class mail to send it to the right destination on time, and hire a bunch of undocumented immigrants to enter the data again.

I'm sure they already have a nice database to use to find prospects that could do the data entry!

Re:

[fm6]

What makes them think they'll get another shot?

You mean, besides the fact that DHS still has the same inept upper management they had during Katrina? And the fact that voters won't have any say in the matter until November 2008?

You figure it out

[COMON$]

Let me know, knowing how to prevent failure to to a flaky nic on a network is a very large issue.

First you see latency on a network, then you fire up a sniffer and hope to god you can get enough packets to deduce which is the flaky card without shutting down every NIC on your network.

Of course I did write a paper on this behavior years ago in my CS networking class. Taking a Snort box and a series of custom scripts to notify admins with spikes on the network outside of normal operating ranges for that device's history. However implementing this successfully in an elegant fashion has been beyond me and I just rely on Nagios to do a lot of my bidding.

Re:You figure it out

[GreggBz]

One not to unreasonable strategy is to set up SNMP traps on all your NICs. This is not unlike the cable modem watching software at most Cable ISPs.

At first, I can envision it being a PITA if you have a variety of NIC hardware especially finding all those MIBs. But they are all pretty standard these days, and your polling interval could be fairly long, like every 2 minutes. You could script the results, sorting all the naughties and periodic non-responders to the top of the list. That would narrow things down a heck of a lot in a circumstance like this.

No alarms, but at least a quick heartbeat of your (conceivably very large) network. A similar system can be used to watch 30,000+ cable modems, without to much load on the snmp trap server.

Re:You figure it out

[ctr2sprt]

One not to unreasonable strategy is to set up SNMP traps on all your NICs.

That doesn't make much sense. If the NIC goes down or starts misbehaving, the chances of your NIC's SNMP traps arriving at their destination is effectively zero. You probably mean setting up traps on your switches with threshold traps on all the interfaces, the switch's CPU, CAM table size, etc. Which would be more useful. You could also use a syslog server, which is going to be considerably easier if you don't have a dedicated monitoring solution.

But they are all pretty standard these days, and your polling interval could be fairly long, like every 2 minutes.

You're not thinking of traps if you're talking about polling. Traps are initiated by the switch (or other device) and sent to your log monster. You can use SNMP polling of the sort that e.g. MRTG and OpenNMS do which, with appropriate thresholds, can get you most of the same benefits. But don't use it on Cisco hardware, not if you want your network to function, anyway. Their CPUs can't handle SNMP polling, not at the level you're talking about.

No alarms, but at least a quick heartbeat of your (conceivably very large) network. A similar system can be used to watch 30,000+ cable modems, without to much load on the snmp trap server.

I think you are underestimating exactly how much SNMP trap spam network devices send. You'll get a trap for the ambient temperature being too high. You'll get a trap if you send more than X frames per second ("threshold fired"), and another trap two seconds later when it drops below Y fps ("threshold rearmed"). You'll get at least four link traps whenever a box reboots (down for the reboot, up/down during POST, up when the OS boots; probably another up/down as the OS negotiates link speed and duplex), plus an STP-related trap for each link state change ("port 2/21 is FORWARDING"). You'll get traps when CDP randomly finds, or loses, some device somewhere on the network. You'll get an army of traps whenever you create, delete, or change a vlan. If you've got a layer 7 switch that does health checks, you'll get about ten traps every time one of your HA webservers takes more than 100ms to serve its test page, which happens about once per server per minute even when nothing is wrong.

And the best part is that because SNMP traps are UDP, they are the first thing to get thrown away when the shit hits the fan. So when a failing NIC starts jabbering and the poor switch's CPU goes to 100%, you'll never see a trap. All you'll see are a bunch of boxes on the same vlan going up and down for no apparent reason. You might get a fps threshold trap from some gear on your distribution or core layers, assuming it's sufficiently beefy to handle a panicked switch screaming ARPs at a gig a second and have some brains left over, but that's about it. More likely you won't have a clue that anything is wrong until the switch kicks and 40 boxes go down for five minutes.

Monitoring a network with tens of thousands of switch ports sucks hardcore, there's no way around it.

Re:

[asphaltjesus]

It's called teaming on windows and we use it. In fact, we had a flaky NIC just the other day. I'm not sure how many cards/vendors support teaming outside of HPaq.

On linux, it's called bonding. This is a killer feature.

I had some very limited professional experience with LAWA in the last couple of years. (LAWA runs LAX) I have no doubt there is quite a bit of consultant the usual chicanery going on whereby they don't actually hire qualified IT people, just people an elected official or two or three may k

Re: Follow-up

[asphaltjesus]

It seems the controller in L.A. suggested the same thing years ago. (pdf)

http://www.lacity.org/ctr/press/ctrpress18616087_1 2152003.pdf [lacity.org]

Social not technical problem.

[twitter]

How about doing regular police work instead of pre crime, so that passengers don't have to stand around while your network flakes out?

Re:

[SatanicPuppy]

The AC is right. Your network topology should be spread out over a number of subnets, and they should only talk to each other where it's critical. The subnets should be separated by expensive managed switches, or by custom hardware configured to monitor packet traffic and isolate problems. Critical systems should be largely inaccessible to the vast majority of the network, and where they are accessible the access is monitored and throttled. If one machine takes too much traffic, you need a second machine se

Re:

[COMON$]

You are correct and you speak far truer than the AC. Basic VLANs should be a part of any admin's topology, I worked on a network once where all 100 servers were connected to the same VLAN as an additional 120PCs subnet was just about full and they were asking for the problem above. I agree with you wholeheartedly, but my question is more related to local subnets 100 PCs here 100PCs there and I would like to detect a faulty nic before it disturbs the rest of the PCs.

Re:

[t0rkm3]

You can see a similar behavior from Cisco IPS if you enable and tune the anomaly detector engine. This in turn feeds MARS... which is groovy except the alerting stinks within MARS. So you have to beat up Cisco and they'll hash out a xslt that will prettify the XML garbage into a nice little HTML email for the desktop support guys to chase down the offender. Couple that with some Perl to grab the fields and shove them in a DB for easy reference...

It works, and it works a lot more easily than anything else th

Sigh, ignorance is bliss

[COMON$]

no wonder you posted AC an answer like that is just begging to be flamed (modders here is your chance to click that button). Subnetting alone is not going to fix a flaky nic issue. Personally I love walking into a network where the admin doesnt understand the purpose of a subnet and just cuts everything up without segmenting the actual physical network. You need to go beyond /25 or /32, and create a substantial amount of subnets. A flakey nic will broadcast across and tear up that network as if there wer

Head of IT for LAX should be fired...

[Glasswire]

...for not firing the networking manager. The fact that they were NOT terrified that this news would get out and were too stupid to cover it up indicates he/she and their subordinates SIMPLY DON'T KNOW THEY DID ANYTHING WRONG by not putting in a sufficently montiored switch architecture which would rapidly alert IT staff and lock out the offending node.
Simply amazing. Will someone in the press publish the names of these losers so they can be blacklisted?

Re:Head of IT for LAX should be fired...

[Rob T Firefly]

They have to find someone who can not only design a vital high-traffic network and maintain it... but who didn't have fish for dinner.

Re:

[kschendel]

RTFA. This was a *Customs* system. Not LAX, not airlines. The only blame that the airlines can (and should) get for this is not shining the big light on Customs and Border Patrol from the very start. I think it's time that the airlines started putting public and private pressure on CBP and TSA to get the hell out of the way. It's not as if they are actually securing anything.

CBP deserves a punch in the nose for not having a proper network design with redundancy; and another punch in the nose for not h

Re:

[failedlogic]

That's the easy way out but probably not the best one. Often, is the case in government, the "officials" think they're always right on areas outside their expertise. The admin probably knows what he's doing but is having to convince people who don't know anything about anything to change their policies so that he can make the network work correctly. They don't see anything wrong with the way its setup - it was working well *before* the failure right? So, no need to change!

Re:

[bigstrat2003]

Yes, let's fire people because they don't cover up their mistakes. Then we'll have no one working in any position of importance but liars. Brilliant!

IF (and that's a big if) we accept your logic that the lack of a cover-up means the IT head/network admin (your post isn't terribly clear on this point) didn't realize there was anything wrong with the way things were being done, then yes, I suppose that person should be fired. However, I call that logic "bullshit". Maybe I'm too damn optimistic, but I'd pref

LACP

[dy2t]

Also known as IEEE 802.3ad supports aggregating NICs to both improve overall bandwidth as well as gracefully deal with failed links.
More info at http://en.wikipedia.org/wiki/Link_Aggregation_Cont rol_Protocol [wikipedia.org]

Systems seem to be more commonly shipping with multiple NICs (esp. servers) so maybe this will be used more and more. It is important to note that the network switch/router needs to be able to support LACP (dumb/cheap switches do not while expensive/managed ones do) so that might be a barrier. Cisco s

Let that be a lesson to you...

[urlgrey]

To all you novice net admins out there: network cards do *not* like chunky peanut butter! Smooth/creamy only, please.

Now you see what happens when some joker thinks [s]he can get away with using chunky for something as critical as proper care and feeding of network cards. Pfft.

Bah! Kids these days... I tell ya. Probably the same folks that think the interwebnet is the same as the World Wide Web.

Great, Scott! What's next?!

The whole system is pointless anyway

[Potent]

When the U.S. Government is letting millions of illegal aliens cross over from Mexico and live here with impunity, then what the fuck is the point with stopping a few thousand document carrying people getting off of planes from entering the country?

I guess the system exists to give the appearance that the feds actually give a shit.

And then the Pres and Congress wonder why their approval ratings are as small as their shoe sizes...

nic can take down a segment

[KDN]

Years ago we had a 10BT nic go defective so that whenever the nic was plugged into the switch it would obliterate traffic on that segment. The fun part: EVEN IF THE NIC WAS NOT PLUGGED INTO THE PC. Luckily that happened in one of the few areas that had switches at the time, everything else was one huge flat lan.

Re:

[zmollusc]

Maybe they were surprised that the fault didn't go away when that pc was powered down, so decided to test the card alone?

Re:

[KDN]

Excuse me, but why the hell did you test for that in the first place?

It was during the debugging phase. We got it to occur, and then turned off one machine at a time. When all the machines on the segment were off and the switch was still jabber isolated we all went "WTF?!" and then started unplugging cables.

"A similar incident"

[The One and Only]

A spokeswoman for the airports agency, said airport and customs officials are discussing how to handle a similar incident should it occur in the future.

Except in the future, the incident isn't going to be similar, aside from being similarly boneheaded. This attitude of "only defend yourself from things that have already happened to you before" is just plain dumb. Obviously their system was set up and administered by a boneheaded organization to begin with, and now that same boneheaded organization is rushing to convene a committee to discuss a committee to discuss how to prevent something that already happened from happening again. The root flaw is still in the organization.

Re:

[Unordained]

Isn't that basically what this whole thing is about, anyway? We sniff your shoes because we had a shoe-bomber. We check liquids because we heard someone might try that. We put cops on planes and tighten security on the cockpit doors because someone took advantage of that hole. We check a very few cargo containers coming into ports because we heard that was an idea thrown around. We suddenly learn to block physical access to the parking lots of FBI buildings because oddly enough, someone made use of that gap

Re:

[geekoid]

So you are saying that a NIC will never go down again in the LAX system? That's quite a bold statement.

Blaming the Wrong NIC

[Doc Ruby]

The NIC that failed isn't the part that's at fault. NICs fail, and can be counted on to do so inevitably, if relatively unpredictably (MTBF is statistical).

The real problem NIC is the one that wasn't there as backup. Either a redundant one already online, or a hotswap one for a brief downtime, or just a spare that could be replaced after a quick diagnostic according to the system's exception handling runbook of emergency procedures.

Of course, we can't blame a NIC that doesn't exist, even if we're blaming it

Managed switches are FTW

[Sehnsucht]

Where I work, if there's a packet storm someplace (server is getting attacked, server is attacker, or someone just has a really phat pipe on the other end and is moving a ton of data) we get a SNMP TRAP for packet threshold on the offending port. BAM! You know where the problem is, and since we have managed switches you just shut off the port if you can't resolve the problem.

Having said that, since the managed switches are gigE uplinked and each port is only 10/100, I don't think we've ever had a problem wh

not too suprised

[mytrip]

I used to work for a very large travel agency and have seen queues of travel resevations get pretty backed up and cause problems before although on a smaller scale.

Most reservations are checked for problems automatically but pushed through by a person and moved from one queue to another. If the program that checks them crashes, it can back things up.

I remember a program crashing and a queue getting 2000+ reservations in it before someone figured out what was going on and it had things screwed up for abo

IT is not that advanced

[Herkum01]

This brings out an obvious point, despite the advances we have made in computing and IT, it is still relatively young and not that robust.

This is the equivalent of your car stops working and the 'check engine' light does not even come on. At least now some of the technology for cars is getting to the point that it will find the problem for you. The same still cannot be said for large computer networks.

When people stop treating computers as flawless wonder machines, then we shall see some real progress

Re:

[MightyMartian]

Ethernet is hardly a new technology. Anybody with an ounce of knowledge should know about broadcast storms, and should know the standard techniques for isolating them (rebooting router/switch/hub is an awfully good start, and then from there taking individual NICs physically off the segment). There are tools that can help, but even if you're stuck with antiquated hubs and the like, there are still ways of dealing with this.

Where I come from is over a decade of hard-won experience dealing with network issu

sadly... this may be typical

[bwy]

Sadly, many real-world systems are often nothing like what people might envision as them as. We all sit back in our chairs reading slashdot and thinking everything is masterfully architected, fully HA, redundant, etc.

Then as you work more places you start seeing that this is pretty far from actual truth. Many "production" systems are held together by rubber bands, and duct tape if you're lucky (but not even the good kind.) In my experience it can be a combination of poor funding, poor priorities, technical management that doesn't understand technology, or just a lack of experience or skills among the workers.

Not every place is a Google or Yahoo!, that I can imagine look and smell like technology wherever you go on their fancy campuses. Most organizations are businesses first, and tech shops last. If software and hardware appears to "work", it is hard to convince anybody in a typical business that anything should change- even if what is "working" is a one-off prototype running on desktop hardware. It often requires strong technical management and a good CIO/CTO to make sure that things happen like they should.

I suspect that a lot of things that we consider "critical" in our society are a hell of a lot less robust under then hood than anything Google is running.

Are They Saying...?

[Nom du Keyboard]

Are they saying that one bad card destroyed other cards? That seems a bit unusual.

Re:

[tomstdenis]

This is what happens when laymen report tech news...

Tom

Fragile System

[monopole]

Like most of the TSA and the DHS poor design abetted with high levels of secrecy and politicization results in a fragile system. With a large number of single points of failure and a lack of fall-backs and fail-safes means that one problem hoses the system. One wonders what happens when things really go pear shaped in an unanticipated fashion.

Um, what about a paper backup??

[LeRandy]

Am I the only one laughing that back in old, antiquated Europe, our passport control have the ability to read the documents, with their own eyes? Oh I forget, how are you supposed to treat your visitors like criminals if you can't take their photograph, fingerprints, and 30-odd other bits of personal data to make sure we aren't terrier-ists (fans of small dogs). It doesn't help prevent terrorist attacks, but it does give you a nice big data mine (and how are you supposed to undermine people's rights effectively if you don't know everything about them).

It is laughable that there is no non-computerised backup for the system. (How about filling out the forms and scanning them in later?)

A Cisco Config to prevent this

[ScaredOfTheMan]

Yes NICs can go crazy and start blasting broadcasts or Unicasts over your network, if you have a Cisco switch (or any other that supports storm control like features) you may want to enable it, it costs you nothing but the time it takes you to update the config. on the access switch (the one connected to your PCs) get into config mode at type this on every interface that connects directly to a PC (use the interface range command to speed things up if you want). Switch(config-if)#storm-control unicast level X where X is the percent of total interface bandwidth you specify as the threshold for cutting access to that port. Its measure every second, so if you have 100 meg port and you set it to 30, if the PC pushes more than 30 meg a sec in unicasts the switch kills the port, till the pc calms down, if its a 10 meg port the 30 then equals 3 meg, etc etc. You can also add a second line to control broadcasts by changing the word unicast to broadcast. If that had this in place, when the NIC went nuts, the switch would have killed the port, and no outage (I assume a lot here, but you get the point).