Please create an account to participate in the Slashdot moderation system

 



Forgot your password?
typodupeerror
×
Security

Malformed Packet Causes Cisco Router DoS 124

MoreBeer writes "Patch 'em if you've got 'em... Cisco Security Advisory: Cisco IOS Malformed OSPF Packet Causes Reload states that a malformed OSPF packet can cause a router 'reload' (reboot). Vulnerable IOS versions include 12.0S, 12.2, and 12.3 ... If you're not screening OSPF at your perimeter and using OSPF Authentication, now would be a GREAT time to start."
This discussion has been archived. No new comments can be posted.

Malformed Packet Causes Cisco Router DoS

Comments Filter:
  • Setup OSPF (Score:5, Informative)

    by BoldAC ( 735721 ) on Wednesday August 18, 2004 @04:20PM (#10006476)
    I notice that Cisco isn't displaying this on their front page. It seems like they should be screaming for everybody to fix the problem.

    Quick walkthrough that I usually reference:
    Easy example how to setup OSPF Authentication [tech-recipes.com]

    AC
    • Re:Setup OSPF (Score:5, Informative)

      by w1r3sp33d ( 593084 ) on Wednesday August 18, 2004 @04:22PM (#10006494)
      That would be the same front page that they didn't address the IOS source theft on for several days???
    • by hot_Karls_bad_cavern ( 759797 ) on Wednesday August 18, 2004 @04:25PM (#10006531) Journal
      at the risk of stating the obvious: if you were a new customer and went to a company's site and it was splattered with all manner of warnings, update calls, and exploit workarounds....would you buy that product?

      If you have a cisco, you should already know where the errata, update, exploit-watch pages are and read them everyday. You should already know this. Why would cisco put that shit on the front page?

      • If you have a glaring security hole, you better tell everybody to patch it because you risk losing your rep.

        Reference:
        Microsoft's previous security plan.


        I love it when I got a company's webpage and they say... "We found out about the error yesterday and we are posting the fix today. Thanks for your support."

        AC
        • by hot_Karls_bad_cavern ( 759797 ) on Wednesday August 18, 2004 @04:31PM (#10006594) Journal
          "... If you have a glaring security hole, you better tell everybody to patch it because you risk losing your rep..."

          If you spent that much on the product, you should be the emails lists about that product. If you really spent a lot, there are plenty of companies that have closed lists for dissemenating exploit info extremely quickly...to the people that should know about it (ie, the people that HAVE the product. If you love reading that companies have fixed things so much, dig about on the site and find the pages with that info...not the front page. i understand what you are saying, but i'm trying to explain to the OP why this might not be on the front page of the site.
          • by Davak ( 526912 )
            We are just going to have to agree to disagree.

            I think security (especially patches for known exploits) should be as public as possible. The trouble-makers scan the security mailing lists more than the average device owner.

            btw, great pics on your site.

            • Understood :) ..and thank you :)
            • In Cisco's defense, though...

              Let's say you own products X, Y and Z. Do you regularly check www.x.com, www.y.com and www.z.com for important updates? Either you pay attention to the appropriate channels or you don't. It doesn't astonish me that they don't overturn their normal front page to provide alerts to people who might randomly stumble across them.

            • It's not at all clear that posting on the front page of the web site will help. How often do you browse the Cisco home page or the home pages of all of the vendors of technology that you use. Targeted email lists that alert you are the way to disseminate this information... Anyway, as you say, we all have different ideas of how to make this info available...
        • It depends (Score:3, Interesting)

          by Flower ( 31351 )
          I don't have to patch a single router. We don't use OSPF and it isn't turned on by default. This isn't like there is some hidden service that I'm not expecting the device to be running and now I must absolutely patch.

        • If you have a glaring security hole, you better tell everybody to patch it because you risk losing your rep.

          Reference:
          Microsoft's previous security plan.



          Using Microsoft as an example doesn't really help your argument. Microsoft (during their "previous security plan" days) made billions upon billions and is now the dominate software company in the world.

        • That's the real reason Open Source tends to be more secure than closed source.

          I would expect anything serious with Red Hat to show up on Red Hat's main page before Slashdot's main page.
          I can fully justify reading Slashdot at work just to keep on top of the latest in Microsoft wormage.

          I suspect that the Open Source coders are better, but that is an "extra". Because of the response patterns, Open Source coders would have to be much worse to break even with Closed Source coders.

          Put a 20' 2x12 on the ground
      • Would you buy a product from a company that doesn't inform their customers of a serious problem? Personally I'd rather have a company just admit they fucked up and fix it myself.
        • Would you sell a product to a customer who came to you and said: "Hey, I like your products. Especially that OSPF thing. I think I'll let my network respond to OSPF packets from anywhere instead of just internal, well-trusted sources"?

          As has been pointed out numerous times in the discussion here, if you haven't got OSPF filtered out right after the telco line, you've got bigger issues than a Cisco bug.

    • conf t

      access-list 150 deny ip 10.0.0.0 0.255.255.255 any
      access-list 150 deny ip 127.0.0.0 0.255.255.255 any
      access-list 150 deny ip 169.254.0.0 0.0.255.255 any
      access-list 150 deny ip 172.16.0.0 0.15.255.255 any
      access-list 150 deny ip 192.168.0.0 0.0.255.255 any
      access-list 150 deny ip 224.0.0.0 15.255.255.255 any
      access-list 150 deny ip 240.0.0.0 7.255.255.255 any
      access-list 150 deny ip 248.0.0.0 7.255.255.255 any
      access-list 150 deny ip host 255.255.255.255 any
      access-list 150 deny 89 any any
      • You know..I can't recall all the exploits ... but there have been several for cisco devices now and it seems that my access-list is growing rather large to avoid ciscosploits...
        • Dude, just download the fixed IOS/CatOS/PixOS and get rid of those performance eating access lists.

          If you're too worried that it won't come back up following a reload, then you really shouldn't be entering config mode either.
      • same thing a bit shorter for high bandwidth links.

        access-list 150 deny ip 10.0.0.0 0.255.255.255 any
        access-list 150 deny ip 127.0.0.0 0.255.255.255 any
        access-list 150 deny ip 169.254.0.0 0.0.255.255 any
        access-list 150 deny ip 172.16.0.0 0.15.255.255 any
        access-list 150 deny ip 192.168.0.0 0.0.255.255 any
        access-list 150 deny ip 224.0.0.0 31.255.255.255 any
        access-list 150 deny ip host 255.255.255.255 any
        access-list 150 deny 89 any any
        access-list 150 permit ip any any

        just some summarization. makes the list f
    • Re:Setup OSPF (Score:4, Informative)

      by Cramer ( 69040 ) on Wednesday August 18, 2004 @06:15PM (#10007520) Homepage
      The home page (www.cisco.com) is not where it belongs. Security notices are available at http://www.cisco.com/go/psirt [cisco.com] That's where security people will be looking. (and they'll be subscribed to any number of Cisco emailed alerts.)
    • Relax chicken little. Every new DoS or exploit isn't cause for thinking that the sky is falling.

      The world is now aware of the problem, Cisco will now fix it.

      Nothing to see here. Move along.

      LK
    • Because Cisco should be held accountable for the people that, for some unknown reason, accept packets for an interior routing protocol from an unknown exterior host?

      What should the front page say? "Download this patch" or "Buy one of our firewall products?" "Send your IT person to our courses," perhaps? If their customer is running a gateway that listens to such ports on its external connections, it's going to take far more than this patch to secure their network.

      This isn't Windows. Users aren't allow
  • by Anonymous Coward on Wednesday August 18, 2004 @04:21PM (#10006486)
    Before someone has a chance to reset my r
  • LessBeer (Score:2, Funny)

    by riptide_dot ( 759229 )
    Kinda old news actually - the article posted @ 15:00GMT, which is 8:00am my time. But I drank too much beer last night so I wasn't awake...:)
  • Bleh (Score:5, Funny)

    by Rosco P. Coltrane ( 209368 ) on Wednesday August 18, 2004 @04:27PM (#10006558)
    Patch 'em if you've got 'em...

    What a crock of shit. Everybody knows Cisco boxes are no route to host
    • Re:Bleh (Score:2, Funny)

      by Anonymous Coward
      Hahaha LO^^^^}}}}}}&..=3R"}'}'[NO CARRIER]
  • OpenBSD (Score:3, Informative)

    by Understudy ( 111386 ) on Wednesday August 18, 2004 @04:29PM (#10006570) Homepage
    May I recommend OpenBSD [openbsd.org] with carp [openbsd.org] as a alternative.
    • Re:OpenBSD (Score:2, Interesting)

      by w1r3sp33d ( 593084 )
      Perhaps you can help me. I would like a bsd box that can terminate 200 802.1Q vlans, two ATM modules for load balancing FROATM links, eight T1's, and some PRI's used as part of a VOATM/VOFR network between pbx's around the world. Which cards can I buy to do this in a PCI form factor?

      Not to call you out, but you are pretty screwed when it comes to routers, honestly Juniper is the only other choice for most of this scenario. Honestly Cisco is the best / only choice for many environments. Personally I like

      • Re:OpenBSD (Score:4, Informative)

        by Understudy ( 111386 ) on Wednesday August 18, 2004 @06:08PM (#10007464) Homepage
        T1 cards are readily avaliable in PCI form [google.com]

        OpenBSD at work [openbsd.org]

        Here is one example That uses 802.1Q VLANS.

        # Empire Net (now known as My180.net)
        An ISP in Bend, Oregon, uses OpenBSD on AMD, Intel, and Sun based hardware, for routing, firewalling, IPsec (VPN), bandwidth limiting, web hosting, database servers, network monitoring, intrusion detection, mail servers, backup servers, cache servers, and workstations. One of their OpenBSD routers handles traffic on between a T3 and eight fast ethernet ports, also with several 802.1Q VLANs to separate networks for co-location customers and business park tenants. An OpenBSD mail server handles e-mail storage/retrieval and RADIUS authentication for over 5,000 users. Several OpenBSD web servers each handle over 300 web sites.

        The Frame Relay over ATM (FROATM) is supported and this card [64.233.167.104] works with OpenBSD. From the website:
        Sangoma's T1/E1 WAN cards have PCI bus interfaces and incorporate an integrated combination T1 and E1 DSU/CSU for a direct connection between the client's server and the demarc. The cards support major protocols including ATM, Frame Relay, PPP, HDLC and X.25 under all popular operating systems including Linux, Windows, FreeBSD, OpenBSD, Unix and Sun Solaris.

        You can look at the OpenBSD hardware list [openbsd.org] for more information.

        Currently Asterik (a VOIP system)is being ported to FreeBSD and OpenBSD. I am not sure if those are complete yet or not but, that can work in coordination with your Voice over ATM (VOATM) and Voice over Frame Relay (VOFR). I realize that VOFR/VOATM is not VOIP but the system is being designed with that support in mind.

        I realize this may not answer all your points but it will help.
        • Re:OpenBSD (Score:3, Interesting)

          by w1r3sp33d ( 593084 )
          WOW! What great information on WANizing a BSD box! Thank you very much for the links.

          Asterik is a wonderful little system that I have been a fan of for quite a while. I cannot wait until it can support the large type of environment I generally build and support.

          VoATM and VoFR is really not a big deal when you get into it, just a few extra caveats to watch out for once you have a solid understanding of VOIP. I do feel the need to blast out one rant: I wish people could understand the difference between:

    • Re:OpenBSD (Score:5, Insightful)

      by xrayspx ( 13127 ) on Wednesday August 18, 2004 @05:45PM (#10007268) Homepage
      No, in this case, I don't believe you may :-)

      I can't get 1 hour support for an Intel/OBSD server from a service provider with a worldwide reputation. If I do get such support, it would have to be guaranteed that they would have every combination of T1 and FastEther card in stock, power supply, etc that would possibly break.

      Sometimes standardization on one vendor worldwide is a GOOD thing. It's no problem to find Cisco support in Europe, South America, North America, Asia, etc. If a company has a router in Singapore and that router fails, would they rather try to find support for an OpenBSD whitebox, or call 1800-Go-Cisco and have someone go replace it immediately? Many international offices don't have full-time IT staffing, so there may not be anyone within an 8 hour plane flight capable of fixing the issue within the company.

      Cisco and the other infrastructure providers make a lot of money for a very good reason, people trust them and can get support anywhere they happen to be.

      Certainly, for a home user or single office with 20 people, one of whom is a BSD junkie, a Unix based router might be a fine idea. However for global organizations with multiple high-bandwidth links between branches, for example, for whom downtime costs many thousands of dollars per hour, there aren't very many options. It's a good thing that what options there are are very solid.

      • by Alejo ( 69447 )
        I wouldn't recomend OpenBSD as replacement for everyone. Actually IMHExperience most network admins don't know the real protocols below their Cisco routers. They are more about the manuals and cisco howtos. Sure there are many great guys knowing a lot, but these are rare lately (in proportion, ppl don't dissapear or forget all they know).

        So I recomend ppl to go study the noncomercial docs (books specs rfcs papers whatever) FIRST, then do the manuals. Else you don't know for real how things work. You're al

        • True, and right you are. If I hadn't blown mod points by posting, I'd have bumped you. There are several companies you can get gear from that have large support bases, but none as large as Cisco. I remember wanting to get some Extreme switching for our core, and getting outvoted because it's "not the same" as the rest of our gear. Oh well, I guess if we are going to make everything match, at least we're making it match with a good vendor :-) But Cisco is in a huge position to do lots of evil or good.
      • I can't get 1 hour support for an Intel/OBSD server from a service provider with a worldwide reputation. If I do get such support, it would have to be guaranteed that they would have every combination of T1 and FastEther card in stock, power supply, etc that would possibly break.

        So what? It's still much cheaper than a real Cisco router. 8-)
        • So what? It's still much cheaper than a real Cisco router. 8-)

          Spoken like someone who has never had to calculate the cost of a large-scale service outage.
          • Spoken like someone who has never had to calculate the cost of a large-scale service outage.

            Cisco routers fail, too, and hardware support contracts are extremely expensive, even if you opt for the 4 hour response time variant ("response time", not "time to fix").

            If technically feasible, I'll always choose commodity PC hardware over special-purpose equipment. It's so much cheaper that you can actually afford redundancy. Unfortunately, some interface cards or protocol implementations are hard to find for
      • Re:OpenBSD (Score:3, Interesting)

        by Oddly_Drac ( 625066 )
        "However for global organizations with multiple high-bandwidth links between branches, for example, for whom downtime costs many thousands of dollars per hour, there aren't very many options. It's a good thing that what options there are are very solid."

        While I agree with almost all of your points...Dude, CISCO have a vulnerability to malformed packets and they appear to be staying quiet about it. How solid are you trying to tell people is solid? Have you ever tried to get Cisco out to swap out a router
        • Yeah, it is naive to rely on a company simply because of they claim to have good support. I've found a good route is to rely on reputable local Cisco vendors who do nothing all day except deal with the supply chain.

          For example, if I have a bad linecard, and I call Cisco directly, I'm going to sit on the phone for 20 minutes before I get a level one CC rep, who will give me a case number and take my name. If I call my local vendors 24 hour support number, THEY can deal directly with whatever Cisco suppl
          • "They often have a disk in the air within 20-30 minutes of a failure, worldwide."

            That is good, but I'm vicariously curious as to why you don't have spares...is it because of variable products used?

            "I like it because the infrastructure is there to get things handled extremely quickly."

            That's fair enough. I've tended to move more towards reliability and support than cheapness myself in most fields. I think experience teaches that after a while.

            "I don't agree with them keeping hush-hush about this
            • That is good, but I'm vicariously curious as to why you don't have spares...is it because of variable products used?

              For critical gear, we have spares on hand. However we have several locations which are only serviced by local contractors. In those cases, we require that the contractors have spares.

              Everyone's so afraid of messing with stock prices, to the point of failure to be forthcoming. Or in the case of Google removing OS Stats in that other story, to the point of even having meaningless stats,
      • Great post, and I agree that service is probably the single most important concern, but I'll add two more:

        1. Finding admins. Go to monster.com and search for CCNA, CCNP, CCIE. Now search for "OpenBSD, WAN, BGP" Compare number of hits.

        2. Performance. Hot swappable line cards running CEF on ASICs are going to vastly outperform even specialized PCI cards.
      • Comment removed based on user account deletion
  • I admit (Score:5, Informative)

    by tomee ( 792877 ) on Wednesday August 18, 2004 @04:32PM (#10006599)
    I had to look it up. OSPF [wikipedia.org]
  • by w1r3sp33d ( 593084 ) on Wednesday August 18, 2004 @04:36PM (#10006647)
    That rules out most routers, and most switches. If you have followed best practices in your deployment, no internet edge device should be running OSPF so that shouldn't be a consideration, basically it should boil down to who within the company is trying to crash your routers?

    What a great time to post a link to www.routergod.com! Here are the two parts of Seven of Nine's lecture on OSPF:

    http://www.routergod.com/sevenofnine/ [routergod.com]

    http://www.routergod.com/sevenofnine/ospf_part_2.h tml [routergod.com]

    • by Anonymous Coward
      who within the company is trying to crash your routers?
      As of tomorrow, someone who clicks on an email attachment....
      • damn...

        I don't worry too much about the routers, exploits have been in the wild for years without too many people scripting for them. I deploy some software for Cisco that runs on a 2000 server, been hit twice with viruses on production boxes. It will be sometime before the IOS devices are nearly as easy of a the target of those servers.

    • Thanks for the link! Oh the irony. I've spent all day reading up on OSPF but having difficulty finding good links. Then I slack off at /. and someone posts one for me. Beauty.
    • who within the company is trying to crash your routers?

      well, for one, no non-network devices should be allowed to form adjacencies, and the easiest way to guarantee that is to set "passive-interface default", then explicitly allow adjacencies on a per-interface basis. Second, use a password, third, use the MD5 hash function to keep it secret. Finally, force the router ID for each OSPF node, and explicitly allow them in a neighbor access list.

      Then the only people left on the parent's list of "who's hack
      • Passive-interfaces will still process incoming OSPF packets, it just won't send any out. From reading the bug description, it doesn't sound like an adjacency is required, so the first two suggestions, while sound practices, wouldn't protect you from this bug.

        You are correct that OSPF Authentication will protect you, and that's Cisco's currently suggested "workaround" for this problem.
  • by JakiChan ( 141719 ) on Wednesday August 18, 2004 @04:53PM (#10006787)
    To be honest, if this causes trouble for you then it's your own damn fault. If you accept OSPF packets from the Internet and/or you're not doing OSPF authentication then you deserve to be pwned.

    1. Don't use an IGP on an exterior interface.
    2. Don't send out routing updates on subnets/interfaces that don't need it. (For those of you with L3 switches that means using the passive-interface command on your vlans.)
    3. If your routing protocol offers an authentication option then use it.

    I used to think these things were obvious. Then I started interviewing other "senior" network engineers and realized they may not be...

    (BTW, kiddies, if you say you're a "senior network engineer" and you say that you know OSPF and I ask you if OSPF uses multicast or unicast and when does it use it/them then you had better be able to answer the question...)
    • I actually agree with your solution(s) as a whole but to be a bit nitpicky (just because of your condescending attitude concerning the "kiddies") #2 alone will not protect you from an exploit of this vulnerability. Passive = not sending ospf but will still receive. But since you have "senior" level knowledge and all, I'm sure you knew this already.
      • That is most true, but I was thinking about more than this vulnerability. It's all about making sure you know who you are adjacent to...
      • Passive = not sending ospf but will still receive.

        And passive interfaces cannot form adjacendies. The packet gets discarded. Therefore, no exploit.
        • A Cisco device receiving a malformed OSPF packet will reset and may take several minutes to become fully functional. This vulnerability may be exploited repeatedly resulting in an extended DOS attack. This issue is documented in bug ID CSCec16481.

          I don't see where the article details that an adjacency is necessary. Maybe you could point that out for me? I see 'a malformed packet' as in one packet, received by the router.

    • Or my favorite interview question: "please list all of OSPF's LSA types and what they do." You can pretty much work through a big pile of resume's really fast with that one alone.
      • So you get someone who can memorize a list? Just what I want in a senior engineer. This is about as retarded as having people recite the 7 layers of the OSI model.
      • 1. Router LSA, fundamental LSA. Used to describe each router-node, particularly their links, in SPF

        2. Network LSA, fundamental LSA. Generated by the designated router on each network, to describe that network for SPF.

        3. Summary LSA, used to aggregrate networks between areas

        4. ASBR summary, OSPF AS global LSAs to describe ASBRs, in particular, without a corresponding ASBR-summary for an originating router, an AS-External route is not valid.

        5. AS external, to describe routes external to the OSPF domain (O
    • by Zarhan ( 415465 ) on Wednesday August 18, 2004 @05:53PM (#10007341)
      (BTW, kiddies, if you say you're a "senior network engineer" and you say that you know OSPF and I ask you if OSPF uses multicast or unicast and when does it use it/them then you had better be able to answer the question...)

      I know most of these things, altough I'm not sure right now (2 am, and I've been on vacation for the last three weeks) what (if any) are the considerations on point-to-point or p-mp (Non-broadcast) links or other more special cases. However, I wouldn't in my right mind call myself a "senior network engineer".

      Oh well, I guess it comes with the fact that the more educated you are, the more modest you get.

      However, I don't really thing that the details are too important. I know OSPF is a link-state protocol where every node in a network knows states of all the other links in an area and calculates Shortest Path using Dijkstra's algorith. IS-IS is similar. RIP is not. If I need to suddenly remember what exact numbering scheme was there for the link-types 1-7 I can always look up a reference (L5 are external routes, L7 are NSSA routes, cannot really remember the rest nor do I care? Show ip route ospf tells me all I need to know on whether it is intra-area or inter-area).

      Just pointing out that you really cannot evaluate someone's knowledge by posing questions about minor details unless you are perhaps hiring somebody with a CCIE (and then you can probably start with more obscure ones about DECNet).

      (Anecdotal note: I was hired as a trainee by my current employer probably because I confessed in the interview setting up LANs with IPX/SPX back in -94 so that all us kids could play Doom. I guess they went for the enthusiasm and my genuine interest. Granted, later I was able to shine when the boss was around and I was just discovering an obscure bug in two different vendors' BGP stacks timer synchronization - Don't know if that had any effect by I got hired permanently).
      • Just pointing out that you really cannot evaluate someone's knowledge by posing questions about minor details unless you are perhaps hiring somebody with a CCIE

        Well, if the person you are interviewing says they are an expert on OSPF I think that it is a fair question. What's kinda curious is the number of CCIEs that can't answer the question. I guess when I'm looking for people I want someone who knows what the protocol is supposed to do not just how to configure it.

        As the man said, "You have to learn
      • Oh well, I guess it comes with the fact that the more educated you are, the more modest you get.

        thats not really modest now is it?

        • thats not really modest now is it?

          Probably not. It was meant as a general statement as in passive "you" meaning everybody in general, not myself. It's just something that I've noticed, especially in cases where you can quite easily measure someone's skills. One clear example that I see quite often is what people put in their CV's in the "language skills" (This is a Finnish POV, so everybody usually at least learns English).
          It's just that folks that have studied in university quite often just put "good"
    • 1. Don't use an IGP on an exterior interface.
      I don't think it is going to help. After reading the advisory, I have an impression that, as long as OSPF process is up,any interface is vulnerable- regardless of participation in OSPF routing.
  • Alcatel... (Score:2, Informative)

    by zxflash ( 773348 )
    Seems like the kind of flaw that Alcatel hopes to profit from...
    Alcatel hopes security will get users to switch [com.com]
    Although as we all know if Alcatel was the market leader more people would be finding flaws in Alcatel products instead of Cisco...
  • by mobby_6kl ( 668092 ) on Wednesday August 18, 2004 @05:06PM (#10006886)
    Malformed color scheme Causes Eyes to Bleed

    "Slashdot Security Advisory: Slashdot Color Scheme states that a malformed IT Color Scheme can cause a eyes to 'bleed' (fall out). Vulnerable /. sections include IT and Games. If you're not already using a /. deuglifyer, you should use the fix provided here [slashdot.org]."
    • I like the colors. I'm colorblind you insensitive clod!

      Seriously though, I always thought that Cisco recommended running IGRP or EIRGP unless you are doing variable-length subnet masking. Isn't it compatable with OSPF?
      • Re:In other news, (Score:2, Interesting)

        by Cramer ( 69040 )
        "compatable"? Are you kidding?! [E]IGRP is Cisco's proprietry [and patented] routing protocol. However, it is very well publicly documented. If EVERY network device needing to participate in routing is made by Cisco, then yes, [E]IGRP is a good choice. However, this is rarely true and OSPF is the only viable alternative.

        (BTW, Cisco supports EIGRP for more than just IP.)
  • by SuperBanana ( 662181 ) on Wednesday August 18, 2004 @05:09PM (#10006914)

    A few years ago I worked at a place where we had two Cisco PIX (the 1U widgets, dunno what model, sorry) in a failover configuration. For those that don't know, you can run two kinds- stateful and non-stateful failover.

    In stateful failover mode, the two units share their connection state info over a dedicated ethernet crossover cable- in theory, if one unit's hardware shits the bed, the other one immediately notices and takes over, and all users will notice is maybe a few seconds pause in everything, if that. It's all very clean and good, the slave even takes over the MAC address of the failed unit (something they've patented, and hence isn't useable in Linux HA; Linux has to force an ARP announcement, which is messier. Goooooo Cisco!)

    Anyway, that's great, except when you have a software defect. Oh, say...where the PIX OS (PIXes didn't run IOS or whatever, they ran a separate OS unique to the PIX family) gets into a certain situation based on state and locks up hard.

    Well, guess what happens to its twin, running the same PIX OS version, and sharing the same data? Yup, it crashes too.

    The pair actually did it once right in front of us- one stopped blinking its lights...the master/slave light blipped on the backup unit, and then a few seconds later, it too crashed- and everything ground to a halt.

    It was terribly amusing that Cisco was incompetent enough to not include a hardware watchdog in the PIX box so that if it hung it would reboot itself; my Sonicwall SOHO has this, why can't a PIX for chrissakes? The problem only happened every few days, and would have been manageable(ie ignorable ;-) if they had both simply rebooted themselves. Instead, someone had to trundle in and power cycle both of them, until we figured out that it was state-based, and disabled stateful failover. Then someone just had to check every day to make sure one of them hadn't kicked the bucket.

    • You might want to look into OpenBSD and CARP [countersiege.com].
    • Sounds like the 515 with a failover bundle. Every fault tolerance mechanism is designed to overcome some potential scenario. You wouldn't bitch if you had two system disks in a mirror and your motherboad getting a power spike killed your system would you? It is not a logical assumption that because you have some level of fault tolerance you will never experience downtime, it's not magic.

      I constantly have to deal with people who have been oversold on Cisco and what it can do for them. These perceptions ar

    • (First a correction... the "failover cable" is not ethernet, it's serial. Take the cover off and look where on the *ahem*PC MOTHERBOARD*cough* the cable goes.)
      • Cisco was incompetent enough to not include a hardware watchdog in the PIX

      If you knew your history, you'd know Cisco didn't design those machines. Cisco bought that company (I forget the name.) The only thing that makes the Pix a Pix is the flash memory card inside there -- in ealier models, it's an ISA card; they have 16M PCI ones now. With o

  • by twigles ( 756194 ) on Wednesday August 18, 2004 @05:36PM (#10007195)
    TAC is a little shell script that pretends to correspond with you a little bit, then tells you to upgrade your IOS. Seriously, I've opened a lot of tickets with TAC in the last few years and that has been their answer in every single one.

    At least they could have used perl or something so the correspondence part didn't take as long.
    • AGREED (Score:2, Interesting)

      I've been working with Cisco equipment for over 6 years. The TAC used to be OUTSTANDING. Now they are for shit. In my opinion it all started once they started moving most of the calls to Mexico etc.

      Step 1.) Tell customer to upgrade ios even though you cannot pin point a root cause or data that supports this as a reasonable solution.

      Step 2.) Tell customer they have a worm running rampant in the network. When asked by the customer why you think this is the case, do not repond for several days. When y

      • Re:AGREED (Score:2, Interesting)

        by Cramer ( 69040 )
        You have to get past the first 3 or 4 tiers of toadies to find a real engineer. We had problems with the 7400 for 6 months. We even had our very own IOS build for it (didn't help)... Then one day the arguements got high enough to find a real TAC Engineer (tm) -- his very first question fixed the whole problem: What's the PCB model number (the last two digits, -XX)?... Motherboard engineering defect that had been published for 4 months. (show tech includes the show diag output, so they knew it was bad had
      • sounds like everquest tech support.

        Problem:patcher won't start
        Solution: remove router and all security software

        Problem:connection lags:
        Solution: remove router and all security software

        Problem:server crashes
        solution: La La La i can't hear you the problem must be your equipment go away.
      • I've been working with Cisco equipment for over 6 years. The TAC used to be OUTSTANDING. Now they are for shit. In my opinion it all started once they started moving most of the calls to Mexico etc.

        I've been working with Cisco gear about the same amount of time. I would agree that US daytime support has gotten very bad. I generally can't get anyone that speaks intelligible English during my normal business hours. I'll let you in on a little secret I've discovered. If you need to talk to TAC and it's

  • This well reported several months ago.
  • I have a PIX 501-- is it affected by this vulnerability? It's hard to tell from the report..
  • So, does this count as a chernobylgram?

    TTFN
  • IIRC, the PGP client sent out some sort of malformed LDAP packets when it sent its key to the server. I managed to crash my university's router something like 19 times before I realized it was me. I cut off all Net access for days, people were fuming. Maybe this was it.
  • BGP outside, OSPF inside, and firewall in between. Authenticated OSPF isn't a fix, MD5 has ben broken too.

Solutions are obvious if one only has the optical power to observe them over the horizon. -- K.A. Arsdall

Working...