Ubuntu Servers Hacked

An anonymous reader noted that "Ubuntu had to shutdown 5 of 8 production servers that are sponsored by Canonical, when they started attacking other systems. Canonical blames the community, saying they were community hosted, and were poorly maintained. However, kernel upgrades couldn't be done because of poor backwards compatibility with the very hardware that Canonical had sponsored! While people point fingers at each other it is pretty clear that both sides are equally to blame, the community administrators for practicing bad security practices, such as using unencrypted FTP transfers with accounts, not properly maintaining the system. However Canonical should have been well aware of what they are hosting. The question remains, if any of the files distributed to users have been compromised. A major blow for Canonical though who are attempting to enter the business market with Ubuntu Server."

New distro name

[by Anonymous Coward]

Spambuntu

Hacked...

[andrewd18 (989408)]

You keep using that word. I do not think it means what you think it means.

Re:Hacked...

[Lord Ender (156273)]

Language changes with time. This particular word has changed meanings (or at least got a new meaning) in the English language. You don't have to like that fact, but bitching on slashdot isn't going to change that fact.

People in the industry are aware that "hack" used to mean "cleverly manipulate a device into doing something its designers did not intend." People also know that "wherefor" used to mean "why." In both cases, the original definitions no longer apply.

Language changes. You'll get over it. There are more important battles to fight.

Gentoo also recently disclosed security breach

[ChazeFroy (51595)]

This isn't the only Linux distro security breach being disclosed recently. One of Gentoo's web applications was compromised and they are investigating it:

http://bugs.gentoo.org/show_bug.cgi?id=187971 [gentoo.org]

I would like to read a report

[QuantumRiff (120817)]

Since this is a community based, open source project, I would love in the near future (after the investigation and cleanup are done) to read about how they determined that the machines were compromised, what the attackers did, and more importantly, how Ubuntu cleaned them up...

This could really help the community as a whole, and I know I would enjoy reading it..

Re:I would like to read a report

[gmack (197796)]

It's important to note that the servers may not have been actually rooted. There is a large number of ssh dictionary breakin attempts on every machine I administrate on several completely different ip blocks. The worst hit is usually my personal server that tended to get hit with several thousand attempts per hour(enough that legitimate logins were a problem) before I installed countermeasures. Even now the countermeasures are locking out 5 to 8 hosts per day.

They have managed to get user accounts on a few occasions and most of the time they never even attempt to gain root. They just start scanning for new hosts.

I'm now running a python script called DenyHosts [howtoforge.com] to find and lockout dictionary attacks. "apt-get install denyhosts" for debian users. Even on much more liberal settings than the default it's lowered my cpu load considerably and locks out attacks in the first minute rather than the hour it would otherwise take me to notice.

Re:I would like to read a report

[Frosty Piss (770223)]

I don't think documenting the discovery process is going to do anyone any favors.

Isn't that part of the Linux/Microsoft Double Standard? Now, if Microsoft this type of issue and had been less than totally open about the cause and methods, you know as well as I do that there would be a high-pitched wailing from the Slashdot World.

Re:I would like to read a report

[discord5 (798235)]

Unless we're going to be composing a Linux Administration HOWTO: Best of Bloopers.

I could fill about a 100 pages on my own from stupid things I've done and stupid things I've seen coworkers/customers do.

The funniest one is still one where one of my coworkers nuked /lib on a fairly important machine unintentionally because he just loves his spacebar:

rm -f /home/user/project /lib/*

Upon which of course by he proceeded to ask everyone "Hey, suppose I deleted something like /lib, is there a way to get it back?", followed by 10 people laughing, followed by a minute of silence as soon as we realized what machine he just did that on. He never got a root password for an important server after that incident. In hindsight, that was a funny incident, and a valuable lesson to us all (we all became paranoid of rereading what we just typed).

Yes, we had backups... Yes, tape drives are still slow

Re:I would like to read a report

[mickwd (196449)]

"The funniest one is still one where one of my coworkers nuked /lib on a fairly important machine unintentionally"

"He never got a root password for an important server after that incident. In hindsight, that was a funny incident, and a valuable lesson to us all (we all became paranoid of rereading what we just typed)."

I hope the decision to deny him root access was based on more than that one unintentional incident. It could have happened to any of you. After all, why else would it be a "valuable lesson" to you ? Isn't the person who made that mistake the least likely to make it again ? And you did also say you "could fill about a 100 pages on my own from stupid things I've done".

The real test

[ZachPruckowski (918562)]

The real test is how they react to this, and how they clean up their mess. Everyone screws up, but what separates good people from bad is how they react to problems and screw-ups.

It sounds like that part at least is still underway, with a meeting (FTA) in "#ubuntu-locoteams on Tuesday, August 14, 2007 at 2:00PM UTC". Seeing as that's yesterday, we should probably reserve judgement a day or two to see how they respond.

Not like Debian

[Bruce Perens (3872)]

This happpened to Debian once. I remember the very careful quality of the notifications, and the forensic analysis, and the fact that it was caught quickly and there thus wasn't much damage. It showed that a volunteer community can be right on top of this sort of problem with as much or more professionality than any paid staff. It's unfortunate that the configuration of Ubuntu and its loco teams has them pointing fingers at each other. And what about those systems that can't be upgraded? Are they, per chance, using proprietary network drivers? If so, well, folks should know better.

Bruce

Panic, They Might Have Gotten the Source Code!

[twitter (104583)]

It's like NT all over again [slashdot.org]. God only knows what bad things they can do with that.

Further proof..

[HerculesMO (693085)]

Linux systems are only as secure as the admins who manage them.

And for bonus "hate" points, even MS servers can be secure if they are admined probably. Don't worry though, I have my flame suit on. :)

Some clarification

[joe_cot (1011355)]

As one of the people affected by this issue, I'd like to give some clarification on this. Firstly, the servers affected were Local Community (LoCo) Team servers, of which I maintain ubuntu-us.org While I'm personally annoyed that the site is down (given it was on the front page of Digg last week), these servers are far from "production" servers; they host LoCo team resources and websites. I'd like to know what "compromised" software would have been downloaded by users, given that these servers did not host user repositories, and for the most part hosted news pages, blogs, and localized documentation. The issues were twofold: the servers were not upgraded past breezy, leaving them open to vulnerabilities after Breezy's EOL; LoCo team users were running an array of web applications (Drupal, Wordpress, Mediawiki, etc), but not updating their systems with new security patches. Top that with ftp logins and no ssh keys, and you have yourself a problem. Canonical is moving the installs to their facilities, retrieving the data, and building the installs (including the aformentioned web applications) from scratch, assuming that everything has been compromised. Hopefully in the next few days this will all be over.

It happens

[popeydotcom (114724)]

Firstly these servers were not "Canonical Hosted" as the anonymous readers suggests. They were hosted in a DC which Canonical paid for, but the community maintained them. So Canonical system admins had very little to do with them.

My site - http://screencasts.ubuntu.com [ubuntu.com] was one of them that was affected, so I was of course concerned that there might be some data loss. I only use SCP to copy files up to the site, and logon with my ssh key, so don't think that all Ubuntu community members are using FTP, weak passwords and really old software, it only takes _one_ though to naff it up for everyone else.

The Canonical system admins (on top of the work they already do) migrated the services from those servers to their own DC very quickly. My site went down on Tuesday and was back by Friday. For free hosting and oodles of bandwidth, I'm happy with that downtime - for a community site.

Soviet?

[Jugalator (259273)]

"Ubuntu had to shutdown 5 of 8 production servers that are sponsored by Canonical, when they started attacking other systems."

In Soviet Russia, server attack you?

Turns out the whole reason for the attack was...

[bealzabobs_youruncle (971430)]

to replace the horrid orange and brown default themes.

I used to be an ardent Ubuntu supporter but since Dapper and the wider adoption there has been too much emphasis on making things more Windows-like and less on best practices throughout the Ubuntu community (note I said the community, not the developers). Stuff like Automatix and the general feeling that any script that or line of code that is posted on the Ubuntu forums is guaranteed safe has led to lax standards. I've brought this up a couple times and any valid discussion quickly descends into a flame-fest and the mods (rightly so) lock it down.

The Ubuntu community has bent over backwards so far to prove they can include everyone they lost site of many of the things that make Linux a better choice for many people; time to get back to fundamentals and best practices, the sooner the better. Stop worrying about besting Windows at every silly thing (ahem, desktop transparency), stop trying to include aunt Tilly (who is never going to "switch" anyway) and remember that some things take more effort but are often worth it.

Breaks happens all the time

[Pecisk (688001)]

It is just became obvious recently that open source publishes their breaks as they are, because they can't actually hide anything. I bet breaks in coorporation servers are so frequent that is common practise to be silent about them.

In mean time, there is a tradeoff between having one, LTS release which has rather old kernel with old drivers and new one, which has 18 month support but has everything up to date, including also unstable stuff of course. But in fact it doesn't even mather, because admin is who in charge.

So Linux is more secure than Windows? You bet. Then why such break-ins happens? Because of lazy or hobbist admins who have no time or maybe not enough knowledge to lock down server to protect it from attacks. To lock down such Windows server/workstation is much harder because of "black box" mentality such software has. But it is also possible.

So in resume - those are admins who are gulty persons here. Ubuntu Dapper and Feisty are secure enough releases to keep them locked down without causing trouble for services. And ohh, be careful to which persons you give access to and have good password management system.

Re:sorry...

[ZachPruckowski (918562)]

Oh, from the sounds of it, all that you say is well-warranted. They were running a version of Ubuntu from October of 2005, which was obsoleted in April of this year, and they weren't using encryption. This is security 101, and they didn't do it. This does sound a lot more like an administration problem than a software problem.

Ultimately, I'd say that if this does wind up being an admin problem, then Ubuntu Server will not suffer. The bottom line is that a poorly administered server is a hacker target regardless of the OS.

Re:Following the M$ example. Re:BWAHAHAHA...

[Minwee (522556)]

Well, if they _did_ get broken into all the time, then that would be pretty embarrassing. The last thing they would want to do is publicize the fact, so it only makes sense that they would cover it up and say nothing about it.

Since nobody has _ever_ said anything about frequent break-ins, it's clear that they must be happening.

Why am I the only person who can see how obvious this is?

Re:Following the M$ example. Re:BWAHAHAHA...

[laederkeps (976361)]

No, but if M$ can't guard their precious source code, what can they guard?

Well, I heard that Ubuntu [ubuntu.com] isn't very good at that either...

Re:sftp

[by Anonymous Coward]

sftp and scp STILL do not allow anything like a REGET operations. Whenever anyone mentions this they got shot down in flames.

Re:sftp

[burner (8666)]

rsync works great for many use cases when transfers really need to be resumed.

Re:laziness and excuses

[Fred_A (10934)]

I've seen lots of dolphins but none of them had CAT5 coming out of their ass.

How right you are!

[spun (1352)]

I've never seen a paid individual make a stupid mistake like this. The captain of the Exxon Valdez was a volunteer with the Red Cross on a humanitarian mission. The Challenger and Columbia were piloted by kids from space camp. The original Tacoma Narrows bridge was designed by volunteers with Habitat for Humanity.

On the other hand, we all know that segregation & apartheid were both ended by paid professionals. If you want something big done right, only paid professionals can do it.