Forgot your password?
typodupeerror
Security Software Linux

Ubuntu Servers Hacked 330

Posted by CmdrTaco
from the zomg-alert-the-media dept.
An anonymous reader noted that "Ubuntu had to shutdown 5 of 8 production servers that are sponsored by Canonical, when they started attacking other systems. Canonical blames the community, saying they were community hosted, and were poorly maintained. However, kernel upgrades couldn't be done because of poor backwards compatibility with the very hardware that Canonical had sponsored! While people point fingers at each other it is pretty clear that both sides are equally to blame, the community administrators for practicing bad security practices, such as using unencrypted FTP transfers with accounts, not properly maintaining the system. However Canonical should have been well aware of what they are hosting. The question remains, if any of the files distributed to users have been compromised. A major blow for Canonical though who are attempting to enter the business market with Ubuntu Server."
This discussion has been archived. No new comments can be posted.

Ubuntu Servers Hacked

Comments Filter:
  • by Anonymous Coward on Wednesday August 15, 2007 @09:38AM (#20236491)
    Spambuntu
  • Hacked... (Score:5, Funny)

    by andrewd18 (989408) on Wednesday August 15, 2007 @09:40AM (#20236521)
    You keep using that word. I do not think it means what you think it means.
    • Hacked... You keep using that word. I do not think it means what you think it means.

      True, it has an entirely different meaning when applied to a FOSS organization rather than a commercial closed source company.
    • Re:Hacked... (Score:5, Insightful)

      by Lord Ender (156273) on Wednesday August 15, 2007 @11:27AM (#20238003) Homepage
      Language changes with time. This particular word has changed meanings (or at least got a new meaning) in the English language. You don't have to like that fact, but bitching on slashdot isn't going to change that fact.

      People in the industry are aware that "hack" used to mean "cleverly manipulate a device into doing something its designers did not intend." People also know that "wherefor" used to mean "why." In both cases, the original definitions no longer apply.

      Language changes. You'll get over it. There are more important battles to fight.
  • by ChazeFroy (51595) on Wednesday August 15, 2007 @09:40AM (#20236523) Homepage
    This isn't the only Linux distro security breach being disclosed recently. One of Gentoo's web applications was compromised and they are investigating it:

    http://bugs.gentoo.org/show_bug.cgi?id=187971 [gentoo.org]
    • by dattaway (3088)
      And instead of shooting the messenger and arresting him on terrorism charges, action was taken and he was given many words of thanks for helping to identify the problem.
      • by jcgf (688310)
        And instead of shooting the messenger and arresting him on terrorism charges

        and it's usually in that order too.

  • Don't worry (Score:4, Funny)

    by just_another_sean (919159) on Wednesday August 15, 2007 @09:42AM (#20236551) Homepage Journal
    This is just a transitional feature designed to make Windows users more comfortable using Ubuntu.
  • by QuantumRiff (120817) on Wednesday August 15, 2007 @09:42AM (#20236563)
    Since this is a community based, open source project, I would love in the near future (after the investigation and cleanup are done) to read about how they determined that the machines were compromised, what the attackers did, and more importantly, how Ubuntu cleaned them up...

    This could really help the community as a whole, and I know I would enjoy reading it..
    • I would love in the near future to read about how they determined that the machines were compromised

      Well. I mean, 5 of 8 machines were already totally owned by the time they worked it out. I don't think documenting the discovery process is going to do anyone any favors. Unless we're going to be composing a Linux Administration HOWTO: Best of Bloopers.

      • by Frosty Piss (770223) on Wednesday August 15, 2007 @10:24AM (#20237177)

        I don't think documenting the discovery process is going to do anyone any favors.

        Isn't that part of the Linux/Microsoft Double Standard? Now, if Microsoft this type of issue and had been less than totally open about the cause and methods, you know as well as I do that there would be a high-pitched wailing from the Slashdot World.

        • Re: (Score:3, Insightful)

          by _Sprocket_ (42527)

          Isn't that part of the Linux/Microsoft Double Standard? Now, if Microsoft this type of issue and had been less than totally open about the cause and methods, you know as well as I do that there would be a high-pitched wailing from the Slashdot World.

          I'm not so sure this is any kind of double standard. The last time Microsoft was compromised there wasn't a "high-pitched wailing from the Slashdot World" demanding details. Nobody really expected to hear any details. And we didn't get any. I'm sure there were some who would have been interested in them... and others who didn't care. And this is the situation we're in now.

          Some people care about these details and some don't. The parent apparently thinks there's nothing to learn. I disagree. There mi

          • Re: (Score:3, Interesting)

            by rtb61 (674572)
            Technically speaking, if there was high pitched wailing every time a windows server got hacked (these were not Canonical servers they just pay for them for use and care by others), then nearby star systems would start complaining about the noise.

            Could you imagine the data load if everybody wanted the information about how every windows server that ever got hacked (I assume M$ takes greater care of it's servers than general users, just as Canonical does).

        • Re: (Score:3, Insightful)

          by nuzak (959558)
          > you know as well as I do that there would be a high-pitched wailing from the Slashdot World.

          You mean the high-pitched wailing from the Slashdot World actually stops at some point?
      • by discord5 (798235) on Wednesday August 15, 2007 @10:40AM (#20237429)

        Unless we're going to be composing a Linux Administration HOWTO: Best of Bloopers.

        I could fill about a 100 pages on my own from stupid things I've done and stupid things I've seen coworkers/customers do.

        The funniest one is still one where one of my coworkers nuked /lib on a fairly important machine unintentionally because he just loves his spacebar:

        rm -f /home/user/project /lib/*

        Upon which of course by he proceeded to ask everyone "Hey, suppose I deleted something like /lib, is there a way to get it back?", followed by 10 people laughing, followed by a minute of silence as soon as we realized what machine he just did that on. He never got a root password for an important server after that incident. In hindsight, that was a funny incident, and a valuable lesson to us all (we all became paranoid of rereading what we just typed).

        Yes, we had backups... Yes, tape drives are still slow

        • Re: (Score:3, Informative)

          by Nimey (114278)
          Why the hell did he have root anyway? Only people with /need/ should have root, and then they should just use sudo anyway.

          Your server was poorly administered.
        • by mickwd (196449) on Wednesday August 15, 2007 @12:52PM (#20239069)
          "The funniest one is still one where one of my coworkers nuked /lib on a fairly important machine unintentionally"

          "He never got a root password for an important server after that incident. In hindsight, that was a funny incident, and a valuable lesson to us all (we all became paranoid of rereading what we just typed)."

          I hope the decision to deny him root access was based on more than that one unintentional incident. It could have happened to any of you. After all, why else would it be a "valuable lesson" to you ? Isn't the person who made that mistake the least likely to make it again ? And you did also say you "could fill about a 100 pages on my own from stupid things I've done".
        • by houghi (78078) on Wednesday August 15, 2007 @01:59PM (#20239991)
          That is why I use `rm directory -rf` instead of `rm -rf directory`. It saved me a few times already.
    • It sounds like a compromise based on using a flaw in an ftp daemon to exploit a kernel flaw to escalate privileges. The question I'd have is which ftp daemon were they running? FTP - even the old, unencrypted kind - IMHO can be run with tight security if you choose a daemon that can run in chroot with virtual-account privilege separation for each user. A few daemons do that, and do it well, most don't. So was this a known-problematic ftp daemon that Ubuntu's Loco servers were running, or a fresh exploit aga
    • by gmack (197796) <gmackNO@SPAMinnerfire.net> on Wednesday August 15, 2007 @10:31AM (#20237307) Homepage Journal

      It's important to note that the servers may not have been actually rooted. There is a large number of ssh dictionary breakin attempts on every machine I administrate on several completely different ip blocks. The worst hit is usually my personal server that tended to get hit with several thousand attempts per hour(enough that legitimate logins were a problem) before I installed countermeasures. Even now the countermeasures are locking out 5 to 8 hosts per day.

      They have managed to get user accounts on a few occasions and most of the time they never even attempt to gain root. They just start scanning for new hosts.

      I'm now running a python script called DenyHosts [howtoforge.com] to find and lockout dictionary attacks. "apt-get install denyhosts" for debian users. Even on much more liberal settings than the default it's lowered my cpu load considerably and locks out attacks in the first minute rather than the hour it would otherwise take me to notice.

      • by TheLink (130905)
        On my personal server I just run my ssh server on a different port. One that's not likely for trojans or other stuff to scan.

        Others can go say "bah security by obscurity" for all they like, I think they're mostly stupid/ignorant anyway ;).

        Actually what I do is run the ssh server on 127.x.y.z:someport and internal.ip:someport.

        Then I have the firewall redirect all accesses to external.ip:extport to 127.x.y.z:someport.

        That way even if the firewall rules aren't present (or messed up), it's likely that people ou
  • uh ho (Score:4, Funny)

    by FudRucker (866063) on Wednesday August 15, 2007 @09:45AM (#20236603)
    Ubuntu made a boobootu
  • The real test (Score:5, Interesting)

    by ZachPruckowski (918562) <zachary.pruckowski@gmail.com> on Wednesday August 15, 2007 @09:46AM (#20236611)
    The real test is how they react to this, and how they clean up their mess. Everyone screws up, but what separates good people from bad is how they react to problems and screw-ups.

    It sounds like that part at least is still underway, with a meeting (FTA) in "#ubuntu-locoteams on Tuesday, August 14, 2007 at 2:00PM UTC". Seeing as that's yesterday, we should probably reserve judgement a day or two to see how they respond.
  • sorry... (Score:2, Insightful)

    by cosmocain (1060326)
    administrators, but:

    who the hell places such exposed servers like these on the net without applying security patches and following simple rules? yeah, the freaking old hardware, compab problems, i sure understand that. but then make a fuss 'bout it. threat to stop maintaining the hardware if the networks cards aren't changed. if that REALLY is the only problem with the hardware which prevented updates, then i just don't understand how the hell this could happen. NICs, even though this would be no consumer
    • Re:sorry... (Score:5, Insightful)

      by ZachPruckowski (918562) <zachary.pruckowski@gmail.com> on Wednesday August 15, 2007 @09:56AM (#20236749)
      Oh, from the sounds of it, all that you say is well-warranted. They were running a version of Ubuntu from October of 2005, which was obsoleted in April of this year, and they weren't using encryption. This is security 101, and they didn't do it. This does sound a lot more like an administration problem than a software problem.

      Ultimately, I'd say that if this does wind up being an admin problem, then Ubuntu Server will not suffer. The bottom line is that a poorly administered server is a hacker target regardless of the OS.
    • Okay, maybe Canonical gave them hardware that was not ... or ... was ... okay, this is just difficult to conceptualize.

      The NIC's worked fine with version A.

      The NIC's did not work with version B. Where's the bug report?

      Breezy - this is where they stopped.
      + 6 months - Dapper - LTS, where is the bug report?
      + 12 months - Edgy - a bug report?
      + 18 months - Feisty - a bug report?

      If you just CANNOT apply a patch then you HAVE TO make sure that EVERYTHING else is locked down AND INCREASE YOUR MONITORING OF THAT SYS
  • sftp (Score:4, Insightful)

    by SolusSD (680489) on Wednesday August 15, 2007 @09:48AM (#20236639) Homepage
    it amazes me that people even use the plain old ftp protocol for anything important. sftp has been around forever.
  • Not like Debian (Score:5, Informative)

    by Bruce Perens (3872) * <bruce@perens.com> on Wednesday August 15, 2007 @09:51AM (#20236675) Homepage Journal
    This happpened to Debian once. I remember the very careful quality of the notifications, and the forensic analysis, and the fact that it was caught quickly and there thus wasn't much damage. It showed that a volunteer community can be right on top of this sort of problem with as much or more professionality than any paid staff. It's unfortunate that the configuration of Ubuntu and its loco teams has them pointing fingers at each other. And what about those systems that can't be upgraded? Are they, per chance, using proprietary network drivers? If so, well, folks should know better.

    Bruce

    • by simong (32944)
      The Debian servers were down for what seemed like ages though, which was frustrating for me as I was trying to build a few machines on it at the time. When providing a public service, there has to be a balance between fixing the problems and making sure that the service isn't down for too long.
      I would assume that the Ubuntu source is safely stored offline somewhere and can be recovered but one of the lessons that has to be learned is the value of a standardised production environment that's been designed in
    • by soupforare (542403) on Wednesday August 15, 2007 @10:19AM (#20237103)
      Maybe they should've been running deb stable. ;)
    • by un1xl0ser (575642)
      There is no word on what was compromised exactly, but network drivers shouldn't affect their ability to update the userland portion of Ubuntu whatsoever. That is assuming that there was a remote exploit in one of the services that they ran, and that someone didn't just sniff their unencrypted FTP authentication.
  • It's all the same. You can lock up a system tighter than a dolphins ass, but no security in the world can mitigate pebkac.
  • how ironic (Score:4, Insightful)

    by Anonymous Coward on Wednesday August 15, 2007 @09:52AM (#20236681)
    had these been windows servers we would have heard cries of a flaky operating system being the problem. in this case, since they're linux servers, we hear that the fault lays on the administrators of the boxen for not hardening the systems?
  • Seriously, better late than never.
    No software is perfect,no package is absolutely secure.
    Its good that these servers were compromised and detected too[i hope withing time].
    This means either admins are not doing their job properly or the culprit packages are buggy.
    Either way it is an eye opener to the community and especially Canonical.
    This calls for better auditing and more effort to be put into security on Ubuntu server systems as well as packages which make their way into Ubuntu.
    This may possibly mean mor
    • by plague3106 (71849)
      I wonder if the tone would be so even headed if this was a recent MS operating system.
      • by jedidiah (1196)
        Is there a similar sort of problem in Windows that was fixed 10 years ago and is now something you have to go out of your way to subject yourself to?

        Most Windows problems tend to be about what the system will do by default, not what sort of ways you can screw yourself up if you really try hard and insist on ignoring decades of other people's mistakes.
  • by twitter (104583) on Wednesday August 15, 2007 @09:56AM (#20236743) Homepage Journal

    It's like NT all over again [slashdot.org]. God only knows what bad things they can do with that.

  • With signatures in place, and verification by default when packages are installed, you'd need more than just breaking into a server to cause serious damage.

    Ubuntu seems to have something in place already, but from my look at it, doesn't seem nearly as insistent on security as it should be.
  • no upgrades past breezy due to problems with the network cards and later kernels

    So wait, this old hardware has no PCI slots? No USB ports? Nothing that could allow one to simply NOT USE THE UNSUPPORTED NIC CARD???

    What the HELL is going on here? This isn't just an 'oops', this is really, really friggen lazy! Last I checked, 3Com and Intel still have about a billion NICs out there in the great wide world. Hell, I could mail them a few myself... ;)

    No?

    • Re: (Score:2, Insightful)

      by greedyturtle (968401)
      It's a lot harder to remotely install a PCI card than it is to complain about it on an internet message board.
      • by BobMcD (601576)

        Admin: You see, boss, I wasn't there. I can't exactly reach through the pipes!

        Boss: I see. So should any hardware fail, it can never be replaced? No one has any kind of physical access to the hardware at all? I suppose the servers are encased in concrete??

        Admin: Well no. Not exactly...

        Sure, that'll fly. I'll use it on my boss. "I couldn't replace the drive from home, and didn't feel like driving in, sorry."

        Sheesh
      • by jedidiah (1196)
        No it isn't.

        Call the datacenter. Scream at the staff. Scream at the staff some more if the NIC isn't installed after the first round of screaming.

        It's not as if the datacenter isn't dying to help you for a fee.

        That's not even getting to the mind numbingly obvious option of schlepping over to the datacenter.
    • So wait, this old hardware has no PCI slots? No USB ports? Nothing that could allow one to simply NOT USE THE UNSUPPORTED NIC CARD???

      I wonder if they could use some of my NE2000 NICs. They should be compatible. I'll even toss in some 50 ohm terminations.
  • by HerculesMO (693085) on Wednesday August 15, 2007 @10:04AM (#20236869)
    Linux systems are only as secure as the admins who manage them.

    And for bonus "hate" points, even MS servers can be secure if they are admined probably. Don't worry though, I have my flame suit on. :)
  • Some clarification (Score:5, Informative)

    by joe_cot (1011355) on Wednesday August 15, 2007 @10:05AM (#20236881) Homepage
    As one of the people affected by this issue, I'd like to give some clarification on this. Firstly, the servers affected were Local Community (LoCo) Team servers, of which I maintain ubuntu-us.org While I'm personally annoyed that the site is down (given it was on the front page of Digg last week), these servers are far from "production" servers; they host LoCo team resources and websites. I'd like to know what "compromised" software would have been downloaded by users, given that these servers did not host user repositories, and for the most part hosted news pages, blogs, and localized documentation. The issues were twofold: the servers were not upgraded past breezy, leaving them open to vulnerabilities after Breezy's EOL; LoCo team users were running an array of web applications (Drupal, Wordpress, Mediawiki, etc), but not updating their systems with new security patches. Top that with ftp logins and no ssh keys, and you have yourself a problem. Canonical is moving the installs to their facilities, retrieving the data, and building the installs (including the aformentioned web applications) from scratch, assuming that everything has been compromised. Hopefully in the next few days this will all be over.
  • It happens (Score:5, Informative)

    by popeydotcom (114724) on Wednesday August 15, 2007 @10:19AM (#20237091) Homepage
    Firstly these servers were not "Canonical Hosted" as the anonymous readers suggests. They were hosted in a DC which Canonical paid for, but the community maintained them. So Canonical system admins had very little to do with them.

    My site - http://screencasts.ubuntu.com [ubuntu.com] was one of them that was affected, so I was of course concerned that there might be some data loss. I only use SCP to copy files up to the site, and logon with my ssh key, so don't think that all Ubuntu community members are using FTP, weak passwords and really old software, it only takes _one_ though to naff it up for everyone else.

    The Canonical system admins (on top of the work they already do) migrated the services from those servers to their own DC very quickly. My site went down on Tuesday and was back by Friday. For free hosting and oodles of bandwidth, I'm happy with that downtime - for a community site.
    • by a.d.trick (894813)
      Why is FTP even enabled? For anonymous transfers it makes a little bit of sense, but having it available for authenticated users is a exploit waiting to happen.
      • Good question. I don't know. I know it's a protocol that lots of web-newbies ask for. I guess someone made the duff decision to allow it.
  • Soviet? (Score:5, Funny)

    by Jugalator (259273) on Wednesday August 15, 2007 @10:23AM (#20237161) Journal
    "Ubuntu had to shutdown 5 of 8 production servers that are sponsored by Canonical, when they started attacking other systems."

    In Soviet Russia, server attack you?
  • by bealzabobs_youruncle (971430) on Wednesday August 15, 2007 @10:35AM (#20237365)
    to replace the horrid orange and brown default themes.

    I used to be an ardent Ubuntu supporter but since Dapper and the wider adoption there has been too much emphasis on making things more Windows-like and less on best practices throughout the Ubuntu community (note I said the community, not the developers). Stuff like Automatix and the general feeling that any script that or line of code that is posted on the Ubuntu forums is guaranteed safe has led to lax standards. I've brought this up a couple times and any valid discussion quickly descends into a flame-fest and the mods (rightly so) lock it down.

    The Ubuntu community has bent over backwards so far to prove they can include everyone they lost site of many of the things that make Linux a better choice for many people; time to get back to fundamentals and best practices, the sooner the better. Stop worrying about besting Windows at every silly thing (ahem, desktop transparency), stop trying to include aunt Tilly (who is never going to "switch" anyway) and remember that some things take more effort but are often worth it.

  • by Pecisk (688001) on Wednesday August 15, 2007 @10:41AM (#20237445)
    It is just became obvious recently that open source publishes their breaks as they are, because they can't actually hide anything. I bet breaks in coorporation servers are so frequent that is common practise to be silent about them.

    In mean time, there is a tradeoff between having one, LTS release which has rather old kernel with old drivers and new one, which has 18 month support but has everything up to date, including also unstable stuff of course. But in fact it doesn't even mather, because admin is who in charge.

    So Linux is more secure than Windows? You bet. Then why such break-ins happens? Because of lazy or hobbist admins who have no time or maybe not enough knowledge to lock down server to protect it from attacks. To lock down such Windows server/workstation is much harder because of "black box" mentality such software has. But it is also possible.

    So in resume - those are admins who are gulty persons here. Ubuntu Dapper and Feisty are secure enough releases to keep them locked down without causing trouble for services. And ohh, be careful to which persons you give access to and have good password management system.
  • by AndyCR (1091663) on Wednesday August 15, 2007 @11:09AM (#20237757) Homepage
    Thousands of Windows machines get exploited every day, and there's barely a word said about it. 3 Linux machines are exploited, and it's "OH MY GOSH!!111". I don't know whether this is a good thing, a bad thing, or, my best guess, both.

"The Mets were great in 'sixty eight, The Cards were fine in 'sixty nine, But the Cubs will be heavenly in nineteen and seventy." -- Ernie Banks

Working...