Stories
Slash Boxes
Comments
Search

News for nerds, stuff that matters

Slashdot Log In

Log In

Create Account  |  Retrieve Password

Marriott IT Exec Shares Network Horror Story

Posted by Zonk on Sat Jun 09, 2007 10:56 PM
from the waps-that-kill-ip-layers-run-amok dept.
Networking The Internet IT Technology
alphadogg writes "Neil Schubert is only partly kidding when he calls Marriott International's move toward a converged network a horror story. 'I'm here to tell you a terrifying tale of network design, support and administration,' he said at an IT conference in Boston, referring to a major bandwidth crunch caused by guests wielding Slingboxes and other network devices that overran the hotel chain's outdated network. 'One of the things we've learned about our guest networks is we have one of the most foreign, hostile environments known to man in the network administration world ... I can take 100,000 customers a night on that infrastructure and we actually have less incidents of harm than we do on our corporate back-office infrastructure.'"
+ -
story
This discussion has been archived. No new comments can be posted.
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.

Marriott IT Exec Shares Network Horror Story 50 Comments More /

 Full
 Abbreviated
 Hidden
More
Loading... please wait.

  • hurm? (Score:5, Funny)

    by spazmonkey (920425) on Saturday June 09 2007, @11:11PM (#19455355)
    So what was the point of that article again? I must have missed it. Perhaps the PR flak who subbed it could explain it to me. I want that two minutes of my life back now /.

    • Re:hurm? (Score:5, Funny)

      by CrazyJim1 (809850) on Saturday June 09 2007, @11:13PM (#19455365) Journal
      And here I was hoping the comments would be exciting to make up for my boredom reading that article.

    • Re:hurm? (Score:5, Insightful)

      by WarlockD (623872) on Saturday June 09 2007, @11:27PM (#19455431)
      Agreed. Whats the freaking point? The article is a fluff piece as it doesn't even describe what, if any, the problems they had to overcome. I rather have it explain WHY the hotel customer network was safer than there internal network than just it be said:P

      Marriott a sponsor of Slashdot?
    • Trying to parse it on lexical level is OK, moving to syntax is a bit challenging (have to make some assumptions!), but then on semantic level (taking above-mentioned assumptions into account) -- IT JUST DOES NOT PARSE!!! :) There is a feling of some text semi-randomly generated but not that smart an AI... (in other words, reminds me of spam!)

      On which network he could accomodate 100,000 customers, the one before the great unification, or the one after? Which network gives him his headaches?

      Now, I'm curious t
    • the thing I took away from this, is that even some well educated people don't know the difference between "less" and "fewer". The rest of the article didn't really match up to that exciting insight.
    • One of the points was that their current network implementation sucked. They have multiple networks, each handling a different type of medium, be it telephone, computer data and TV. This makes for huge complexity when administering the networks. Their current data network was not designed in an optimal manner and quickly got over run. Their plan is to provide one IP network for all data types (voice, fax, data and TV), instead of dedicated networks, and then have a box in each room which would provide an in

  • weilding slingboxes? (Score:5, Funny)

    by ChiChiCuervo (2445) on Saturday June 09 2007, @11:13PM (#19455367) Homepage
    I just get this mental image I'm not going to be able to shake....

    "Some call it a slingbox, I call it appleTV. nnnngggggghhhh"

  • One lesson from the article... (Score:4, Insightful)

    by BrowserCapsGuy (872795) * on Saturday June 09 2007, @11:19PM (#19455395) Homepage
    There needs to be better coordination between marketing and IT. IT had no idea marketing was running commercials showing customers using all this high-bandwidth stuff so there's no way IT could be prepared for it. Imagine 160 customers just trying to view websites on one DSL line! I admire this guy for his honesty if nothing else. He'll probably catch hell for it from his superiors!

    • Re:One lesson from the article... (Score:5, Insightful)

      by timmarhy (659436) on Saturday June 09 2007, @11:41PM (#19455495)
      that's what marketing do dumby! they make outrageous claims then handball it to technical, and when technical can't make the impossible happen marketing make it look like the techies failed. marketing will then tweak their bullshit slightly to cover their own ass's and make it look like they saved the day.
      • You just described the entire IT industry :)

      • Re: (Score:3, Insightful)

        by Kjella (173770)
        that's what marketing do dumby! they make outrageous claims then handball it to technical, and when technical can't make the impossible happen marketing make it look like the techies failed. marketing will then tweak their bullshit slightly to cover their own ass's and make it look like they saved the day

        Marketing aren't half bad, they haven't got a clue what they're talking about and answer "yes" to most questions when they have no idea or not. Most have a fairly relaxed attitude to that and know that thei

      • Whenever evaluating marketing, the first and foremost thing to keep in mind is that it is their JOB to successfully pass frosted dog turds off as wedding cake. No matter who they work for, the number one thing they will market is THEMSELVES. Once they successfully market themselves to their employer, they may (or may not) actually market the product. After all, they don't HAVE to make the product look good every time, just convince the employer that they did!

    • Biggest lesson learned...... (Score:5, Insightful)

      by Nick Driver (238034) on Sunday June 10 2007, @12:02AM (#19455615)
      ...is that the worst threats to your network do not necessarily come from outside.... they almost always come from your very own moronic employees.
  • The business units of most organizations typically make promises to their customers without comprehending or even considering the IT implications. Account Executive to customer: "Sure! We can provide you and your thousands of users seamless B2B connections from your network to ours wirelessly from any global location!" Account Executive to IT department: "Ok, you guys figure out how to do that."
    • Yep, seen that at plenty of places. At one place, the IT department clearly spelled out what was possible and in what timeframe. Marketing oversold anyway. However, this isn't just IT, and that's the scary part of it. Airlines routinely overbook aircraft. Package holiday companies sell hotel rooms for hotels that haven't yet been built. You too can place advance orders for books that haven't been written or buy computers that have not yet been built. (One company I know has made substantial money off a computer for which even the prototype does not yet work.)

      The problem with marketing is that it is not about selling what you have, it's about selling what the person wants to buy. If there's a discrepancy between the two, well, that's not your department. Complaints is three doors down, across the hall from Abuse.

      • That the reason they do it is because statistically a percentage of people don't make a given flight. Sometimes it ends up in conflicts, especially since they tend to err on the side of being full rather than no conflicts, but there is good reason to do it. If 100% of people who wanted on a flight showed up, they'd never overbook. However about 10-15% of people cancel their reservations or otherwise fail to show. Thus it makes sense to overbook the aircraft.

        • That the reason they do it is because statistically a percentage of people don't make a given flight. Sometimes it ends up in conflicts, especially since they tend to err on the side of being full rather than no conflicts, but there is good reason to do it. If 100% of people who wanted on a flight showed up, they'd never overbook. However about 10-15% of people cancel their reservations or otherwise fail to show. Thus it makes sense to overbook the aircraft.

          That might almost be an excuse except that they sell non-refundable "you die, you fly" tickets, supposedly for the exact same reason. Those empty seats are already paid for. They are trying to make additional money off of them at the cost of double booking. Like most businesses, they get you both ways and make you deal with the mistakes and inconvenience.

            • My most recent trip home from New Orleans on Continental was overbooked and they were offering initially $300 and then $400 to get off and take a flight 3 hours later.

              I personally didn't take it because I was feeling ill and had a doctor's appointment that evening that I'd have missed on the later flight... but the bump would have paid for two round trip tickets at the price I paid.

              (wouldn't you know, my train broke down on the way to the doctor)
    • Ok, you guys figure out how to do that.

      To be honest, I've never had a problem with that. For me (and most of the people I've worked with), it only becomes a problem when they add, "Oh, and we need it tomorrow", or "There's no extra money on the budget to spend on this", or both.

  • Anybody have a "buzzword bingo" card for this? (Score:4, Interesting)

    by MonkeyBoyo (630427) on Saturday June 09 2007, @11:35PM (#19455477)
    I may have understood some of the article but it seems to have been mainly an exercise in trotting out what somebody thinks are the most trendy buzzwords.
    • Whats there to understand? Basically he just said that Marriot is bumping up their network backbone to better hand VOIP and Slingbox [wikipedia.org] type devices (basically broadcasts and manipulates your tv via the net, and you can watch it via the Slingbox app on your laptop or supported cell-phone), along with adding docking stations and cables to hook up electronics and laptops and whatnot to your Marriot room's A/V system and network connection...

      Really there weren't any buzzwords at all in it - Slingbox isn't a bu
  • Placing all access points in a single telecom closet for what are generally rather spacious properties requires that 2.4 GHz signals be carried through coaxial cable that is very lossy at that frequency - it might be fair to expect up to 90% of the signal to be lost in the wire. There is an FCC limit on the transmitted power, and even if you manage to boost that at the antenna you will be boosting noise as well. And this attenuation and noise will of course hurt receiving too. This is in general going to result in lower wireless quality than desired, much lower than possible.

    Instead, get zero-management access points that do not do NAT, routing, etc, and treat them just like antennas once you set the SSID. Do the protocol processing in the telecom closet with a higher grade of hardware than consumer equipment. Cache DNS and web transfers there. Work with Slingbox to engineer channel aggregation with multicasting that bypasses the home units while transmitting the same programming, because so many of those folks are watching the same sports game. I can think of some interesting approaches to the possible legal issues with Slingbox aggregating channels, no doubt they can as well. Can an in-house video alternative be made as attractive as Slingbox? That's another solution.

    Bruce

    • Cisco has a solution that puts the PHY on the antenna out in the premises and just send back the partially processed signal to the MAC built into either an AP or linecard.

    • Can an in-house video alternative be made as attractive as Slingbox? That's another solution.

      Yes, it can. It just needs to be cheaper to build/purchase it then bandwidth.

      • Oh sure, Slingbox is going to switch from being P2P to a multicast provider of copyrighted content in order to let Marriott go cheap on bandwidth.

        I am well aware of the legal issues and did not mean to pose them as being simple. Hotel chains have lucrative relationships with sports enterprises.

        Cache the Internet on a local server and connected each room via a quad fiber ATM connection.

        This proposal not solve problem, which is limtied by the pipe to the building rather than the pipe within the building. Ca

      • Slingbox is not P2P at all. You stream content from your computer back home to wherever you want to watch it.

          • Re: (Score:3, Informative)

            by Franso6 (976942)
            In that definition everything that's unicast is Peer to Peer.
            The term is nowadays used for a form of content distribution that's based on using end-user-owned, non-specialised machines working collectively.
            What you're referring to is a client-server model.. Usually considered as more or less the opposite of a P2P model.
            I agree that the naming peer to peer is unfortunate, though.

      • Cache the Internet on a local server and connected each room via a quad fiber ATM connection.

        So if I understand correctly, you are suggesting that Marriott needs to build a Google installation for each one of its hotels. Do you suppose Mattiott would be able to stay in business if it does this? When are they going to hold the auction on used bits left over from the former Marriott Corporation? I'd like to get a rack full of computers and raid drives... - or - Maybe I am misreading what you are suggestin

      • Re: (Score:3, Informative)

        by Bruce Perens (3872) *
        I wasn't talking about leaky coax, but TV coax - RG-6, -57, -59, etc. Some of the "antenna distribution systems" not only use that, they might even use the same wire that is used to distribute television around the building. Obviously a solution that does not require wire pulling is attractive to these properties, that's why so many of the hotel-room wired Ethernet devices are really a sort of short-haul DSL piggybacked on the room phone line. I bet they feed the WAPs with that pseudo-DSL, too.

        When you sai

  • Astroturf? (Score:5, Insightful)

    by UESMark (678941) on Saturday June 09 2007, @11:54PM (#19455577)
    This seems like a thinly veiled ad for Marriot internet access.
    • "Plans for hotels include a distributed antenna system with a converged voice and data network, having all access points located in the telecom closet, up to four antennas used for coverage, and up to four access points used for bandwidth."

      Dunno, if it is an astroturf, then they probably don't want the author to write articles like this. Sounds like a clueless person on the technical side of things.

      Can anyone please explain what this means? I presume these antenna's are fixed to the access points, so of cou

      • In the same sentence, the article refers to both an "access point" and a "distributed antenna system," indicating that when they say "access point" they're probably not talking about the $2.99 device from CompUSA.

        It seems like when they say "access point" they mean "connection to another network outside the hotel". Currently, data might come in a variety of places -- maybe they have a directional antenna on the roof for each local HD tower, plus a satellite dish, plus an internet connection, plus a cable

  • While I was at the 2005 MysqlUC in Santa Clara, Microsoft put out one of their massive patch releases. Despite a large number of people running OS X or some Linux variant, there were still enough geeks in the hotel running Windows that they essentially DoS'ed the hotel's internet connection for about 2 days. I went down to reception at the time to find out if it was just the wireless, but the front desk people were similarly frustrated (they shared the same connection). Packet loss was at about 99%.

    That was probably just a T1 or something, but still, pretty funny. I wonder if Microsoft realises the damage potential..

    • Re: (Score:3, Insightful)

      by imac.usr (58845)
      What kind of fucking idiot updates their laptop during a conference? You wait to do that shit until you get back home in case it screws your machine.

      • Re:Microsoft update + Public Network = Instant DOS (Score:4, Interesting)

        by Nezer (92629) on Sunday June 10 2007, @10:31AM (#19458189) Homepage

        What kind of fucking idiot updates their laptop during a conference? You wait to do that shit until you get back home in case it screws your machine.


        Let's see... At a conference your computer is connected to hostile networks nearly all the time. Depending on the conference, there are potentially a LOT of people that know about 0-day exploits and might want to try something dumb.

        I dunno. I can see your argument but there may be very good reasons to patch your system ASAP. I used to work in an environment where NOTHING got patched because they were afraid of fucking-up production services. I argued until I was blue in the face that we needed to do something and have a plan for deploying patches. I even went so far as to make proposals explaining the benefits, the risks, and the costs. No one would listen to me because I was a UNIX admin on a Windows team. Eventually I was let go and no one else took-up my cause (perhaps the cause was a large reason I was let go). No one on the team, except me, felt that there was any risk because the networks were "isolated" behind three layers of firewalls. About three months after I left some nasty work managed to find it's way into this "isolated" network and wreak much more havoc than we ever could have patching the damned servers.

        I know that this isn't exactly the same thing as updating your laptop while on the road, but sometimes the updates are just worth the risk.

        Perhaps the hotels should consider a caching proxy for just these sorts of events. Let the first user wait for the the download to come down the pipe and everyone else can leach from the proxy.

  • Ironically... (Score:3, Interesting)

    by EReidJ (551124) on Sunday June 10 2007, @05:56AM (#19456953) Homepage
    With some irony, I am reading this story from a Marriott hotel room at the Marriott Boca Raton. I've had mostly no problem with their services, but here are a few things I would call interesting: * My laptop can often see multiple nodes, some very fast, some blazingly slow. If you stay in a Marriott, try out the different nodes you can see. * Some Marriott properties give free Internet access, some cost $10/day. I wish Marriott would be consistent across all their properties. * If you stay in a Marriott that does charge for access, as for a low floor. This is because often there's a single wireless connection in the business center that is free, but other access points cost money. So if you can get a room near the business center, you'll be able to hook up to that one for free. * Finally, I've never had a problem with BitTorrent uploading at any Marriott property. I don't know if they leave all their ports open or what, but I traditionally leave my uploading port for BitTorrent open on 34567, and I've never had a problem with a torrent at a Marriott.

  • I can take 100,000 customers a night on that infrastructure and we actually have less incidents of harm than we do on our corporate back-office infrastructure.'"

    That says less about the robustness of the hospitality net and more about the poor planning and administration of the enterprise intranet.

    • Re:Where's the horror? (Score:5, Insightful)

      by luvirini (753157) on Saturday June 09 2007, @11:05PM (#19455325)
      It is a "horror story" because the network manager was not prepared for the customers to actually USE the service...

      • Re: (Score:3, Interesting)

        by dwater (72834)
        Of course that's not an accurate description. It was that he was not prepared for them to use it to the extent they did - he wasn't prepared for the degree of success that occurred.

        On that note, I wonder what turns a healthy network into a broken one? Is there something different between a broken network and one that's just very slow?

        It reminds me of the problems we had in Asia (I'm in Beijing) earlier this year due to the earthquake in Taiwan. Network congestion was so bad that we figured using a 56K modem
        • if you did a 56k dial in to a modem bank at your destination you would have, if it was 56k to a local ISP you would have been just as fucked
          • Well der, of course I meant a modem on the other side of the blockage, else there wouldn't any point at all.
            I was thinking of something along the lines of an AOL or earthlink account, or just have someone hook up a modem at our US office...
        • With a 56k modem, you have guaranteed bandwidth between you and the ISP, not between you and the internet.
          • Yeah, so?

            If the ISP is on the other side of the blockage, it effectively gets me onto the other side of the blockage, no?

        • Is there something different between a broken network and one that's just very slow?

          One way for a network to be broken by design is if the total capacity decreases as the utilization increases; in other words, past a certain point, adding more load decreases the total throughput of the network.

          • Re: (Score:3, Funny)

            by dwater (72834)
            Well, my mum told me I shouldn't arrange to meet people off the net'...but, er, maybe...

    • Come on! (Score:2, Funny)

      by Anonymous Coward
      For running this slashvertisement, Zonk's getting one years free accommodation from the Marriott chain, so what's the problem?

      Can't an "Editor" graft in peace?

      • Re: (Score:2, Insightful)

        by Kyojin (672334)
        Whoever modded this "insightful" needs to get a clue. The article doesn't even have 3 pages, it has 2. Also the only mention of "Linux" is in the ads around the story, not in either of the two pages of the story.

      • Re:The Horror (Score:5, Insightful)

        by eln (21727) on Sunday June 10 2007, @03:41AM (#19456503) Homepage
        Sounds like a lack of capacity planning. If the marketroids were planning a huge marketing push, the IT manager should have been given the resources to increase network capacity prior to that push. If the IT manager was given proper warning and funding prior to the marketing campaign, it's his fault for not effectively utilizing it. Either way, the problem could have been avoided with proper management.

All trademarks and copyrights on this page are owned by their respective owners. Comments are owned by the Poster. The Rest © 1997-2009 Geeknet, Inc.