Virtualization Decreases Security

ParaFan writes "In a fascinating story on KernelTrap, Theo de Raadt asserts that while virtualization can increase hardware utilization, it does not in any way improve security. In fact, he contends the exact opposite is true: 'You are absolutely deluded, if not stupid, if you think that a worldwide collection of software engineers who can't write operating systems or applications without security holes, can then turn around and suddenly write virtualization layers without security holes.' de Raadt argues that the lack of support for process isolation on x86 hardware combined with numerous bugs in the architecture are a formula for virtualization decreasing overall security, not increasing it."

Uh oh

Theo de Raadt asserts...

CAUTION: flame war ahead.

Re:Uh oh

CAUTION: flame war ahead.

No there isn't! How dare you say that?? F-you! YOU GO TO HELL AND YOU DIE!!!

Re:Uh oh

I don't think anybody considers DJB a leader of the Free Software movement.

They consider him a brilliant man, and excellent programmer, and generous to let people download his code. They consider him a hero for taking on and beating the US government. They consider him a jerk. I've never heard anybody call him a leader of the Free Software Movement. I've never even heard his license-free software to be considered Free Software.

As an aside, many people call him a jerk for his style of writing information and documentation. I had to install a DNS server, and I found his you-must-be-a-moron-so-I-will-explain-everything-in-very-simple-terms documentation very informative, clear, and helpful. The security advantage is nice, but to me, tinydns' greatest advantage was the DJB's documentation.

Re:Uh oh

We should put Theo and Daniel J. Bernstein (DJB) [cr.yp.to] and see who survives. These so-called 'visionaries' and have a hard time forming an argument without degrading the argument with words like 'stupid'. It's a real shame that men like these are considered leaders the Free Software movement.

After reading vitriolic posts by these two fools, RMS doesn't seem all that bad.

I disagree. He seems to call it like it is. And I would agree that anyone deluded enough to think that adding another layer to the, already complex, PC model increases security is just stupid. Sure, it may be that they are not well versed in the inner workings of both the hardware and software, but does that make their assertion any more correct? And besides, he's on a mailing list where the majority of the readers should be close to his level of knowledge.. He may not be the most tactful guy in the world, but he's a hell of a lot smarter than most...

I've been on the fence about virtualization for a very long time now. Sure, it's quite convenient to install VMware, load up a guest OS, and tinker with new features. But to load up a server with multiple instances of the same operating system is ludicrous. It certainly doesn't scale well at all. And the marketing teams are incredibly good at making people believe that by installing their virtualization software, you'll suddenly have a bunch of "virtual" servers with the same capabilities as a single server. Sure, they all have the same capabilities from an OS standpoint, but performance isn't going to be anything close to a standalone server..

And as far as security goes, it's nonsense. Ok, so I install 5 copies of RHEL 5.0 on my virtual server. If the virtualization software itself is attacked and compromised, all 5 servers go down. If an OS level attack is successful, then all 5 virtual servers are likely vulnerable because it's an OS level attack. The only security "benefit" I can see is if a single virtual server is compromised through something like a web application. That application may not exist on the other virtual servers, so they're "safe".. However, once you get into that one server, DDoS attacks aren't far behind. At the very least, you'll take up resources and you can potentially impact the operation of the other virtual servers.

I'll stick with standalone servers for now.. At least until there's a better solution, of which I don't see one coming anytime soon...

Re:Uh oh

The fact is that very little hardening is typically done on the inside of the organization. A lot of organizations have the hard crunchy outside with a soft chewey center. (Don't remember who I heard make that analogy, but it's apt.) Most IT departments seem to have hardened servers at the border, but the inside is run-of-the-mill software and hardware. What this means is that maybe virtualization isn't great for the border proxies and firewalls, but it probably fits right into the controlled chaos on the inside, where nothing is especially secure anyway.

Re:Uh oh

I've been on the fence about virtualization for a very long time now. Sure, it's quite convenient to install VMware, load up a guest OS, and tinker with new features. But to load up a server with multiple instances of the same operating system is ludicrous. It certainly doesn't scale well at all. And the marketing teams are incredibly good at making people believe that by installing their virtualization software, you'll suddenly have a bunch of "virtual" servers with the same capabilities as a single server. Sure, they all have the same capabilities from an OS standpoint, but performance isn't going to be anything close to a standalone server..

Performance will take a hit from the overhead involved, but availability should increase. Most server applications don't fully utilize the CPU anyway, so sacrificing some cycles to run the apps in a virtualized environment is not really a big deal. Where virtualization shines is availability. If a server is malfunctioning or overburdened, the virtualized environment can migrate to another server without the server clients knowing this has taken place (other than some latency caused by the migration). This is actually the coolest part of this technology.

I never thought about using virtual servers to increase security. Except for running windows within Mac OS X, I really don't see virtualization making anything more secure.

I think this is much ado about nothing. It is only here because Theo is getting upset...

It's easy to defeat Theo's argument

> CAUTION: flame war ahead.

There doesn't need to be a flame war, because in this particular instance Theo's argument has a gaping hole in it. Consider the following two system architectures:

1) An ordinary multi-function Unix-type system which also runs a non-trivial component that is exposed to the world (all non-trivial components have bugs, as Theo is right to point out, and hence are attack vectors).

2) A machine running 2-guest virtualization, in which the non-trivial component runs in one guest, and the rest of the functions run in another.

Now consider what happens when the world-facing component gets compromised, and by one of many methods (because sysadmins are fallible) the attack gets promoted to root privilege. Security has failed in one guest, but has it failed in the other? Not necessarily, depending on whether the sysadmin has made repeated blunders and not just one. (Eg. a fool might be keeping ssh keys on the public-facing guest ...)

In this scenario, the isolation created by virtualization has given the syadmin an additional bulkhead against his own fallibility, and that is worthwhile for security, not only for better hardware utilization. The partitioning of the application and O/S space has reduced the cross-section of software open to attack.

Theo's argument also doesn't bear scrutiny at the hypervisor level, because while an O/S in dom0 is just as fragile as the one in domU that runs an exposed application, the instance in the hypervisor isn't exposed to attack. Theo seems to miss the distinction between endpoint fallibility and fallibility in the conveyance and resourcing that is done by hypervisors. They're different.

I like Theo's hard stance on security, but on this issue he's handwaving.

Re:It's easy to defeat Theo's argument

I don't think you've entirely grasped Theo's argument. He argues that your reasoning is invalid *because it assumes that the interface between the O/S in dom0 and the hypervisor has no security holes in it*. You don't get to just state that the hypervisor isn't exposed to attack. Now, you can argue that because of its limited nature, and because great pains are taken to avoid unwanted interaction between the hypervisor and the virutal O/S, it is more secure than ordinary software interactions. But I think Theo is not arguing that VMs are less secure than running all your stuff in one O/S. I think he's arguing that VNs are less secure than running your services *on actual separate machines*. And that stands a good chance of being true.

Chris Mattern

History teaches once again...

The Irish Potato Famine happened because Ireland was growing a small range of species of potato.
A virus hit the Potato and it spread so most of the potato's died thus causing the famine.
Other Areas had the same virus but it didn't cause a Famine because their stock was more diverse.
It wasn't because the other guys potatoes were immune to all virus or they were a heartier bread, but
because they had a wider diversity of product.

The same thing with Virtualization, each VM will not be completely secure and will have holes in it but
spreading will be reduced because only a smaller portion of application will use that OS to virtualize.
A Linux VM OS, a BSD VM OS, a Windows VM OS... Sure there will be security problems and patching and fixing
the problems in the Virtual OS will need to be resolved... But if there is an outbrake you will basicly loose your
VM application and perhaps some other ones that you may have running at the same time that uses the same OS. But now
If your OS Gets infected all your Apps are dead.

Re:History teaches once again...

It is not sustainable. It takes more calories to produce a child then you get from eating them.

This assumes that you eat (only) your own children.

Re:History teaches once again...

The Irish Potato Famine happened because Ireland was growing a small range of species of potato.

The same thing with Virtualization, each VM will not be completely secure and will have holes in it but spreading will be reduced because only a smaller portion of application will use that OS to virtualize.

I don't think that analogy applies here. I think TA's point is that the hypervisor itself may not be any more secure than the OSes it virtualizes. So now you're hypervisor OR the OS it's running may get hacked.

Re:History teaches once again...

Well right now most flaws are threw Applications anyways. I can run Open BSD and still make an application that runs as root and have such a major security flaw that it will allow people to have full access to my system. At least with it being virtualized it will at least protect it from quickly spreading to my other Apps.

Re:History teaches once again...

You're missing the point. Your virtualization product is an application, which weakens the security of the OS running under it. So now you can have attacks from both sides. As Theo says, now an OS crash (inside the VM) can become an attack on the host system, and application attacks on the VM can become an attack on the OS running in the VM.

His position has many facets. As I understand it:

* programmers make buggy code, and now programmers are programming virtual hardware
* the hardware they are emulating (PC architecture) is a nightmare, they have to do crazy, unsafe crap to implement it.
* application flaws in the VM can compromise the guest OS.
* OS flaws in the guest OS can potentially compromise the host OS.
* virtualizing hardware is inherently less secure than the physical segmentation of using actual, separate machines, so when you consolidate many machines onto a VM system you have a net loss in security.

Re:History teaches once again...

I think TA's point is that the hypervisor itself may not be any more secure than the OSes it virtualizes.

Actually Theo's argument was that software engineers can't write an OS without security holes, therefore they can't write a hypervisor without security holes.

The argument is, of course, full of shit. The hypervisor in question, Xen, is 50,000 lines of code. Compare this to the linux kernel which is about 6 million lines of code or Vista which is said to be 10s of millions. Theo also drags out his favorite attack about page protection. He is known for attacking a "vulnerability" in a C2D code segment limit/page accessed issue (AI90) as being "assuredly exploitable" in OSes other than OpenBSD, even though nobody has been able to propose a way to exploit it.

The problem with Theo attacking things is that he is so well respected in BSD-land that his word is taken for granted. Sometimes he gets it wrong, but unless someone equally high up wants to spend the time to rebut his ranting (a lot of work for no gain) everybody accepts what he says.

Re:History teaches once again...

The Irish Potato Famine happened because potato crops were the only thing they could grow on tiny plots of land in sufficient quantity to both pay their English Lords and feed themselves (the latter being the optional part), so when the potato crop failed they didn't have any other food. Other areas had a more diverse potato crop, but more importantly they had other crops entirely and thus even had all their potatoes died, it wouldn't have caused a famine (or at least not anywhere near the scale of the Great Famine) because they had other food to eat. The Irish had no other food source by design of the English, thus causing mass starvation.

There's still a lesson in diversity and computer security to be learned here. But it includes the harsher lesson that human leaders often don't care about the necessity for diversity and the cost to security (and thus the IT department), and can impose a homogeneity that is even worse than an IT department that just didn't consider diversity to be important.

Re:History teaches once again...

There's still a lesson in diversity and computer security to be learned here.

Indeed. Implementing proper security is no small potatoes.

Welcome to the rest of the IT world, Theo!

You're just NOW realizing this???

Re:Welcome to the rest of the IT world, Theo!

Not to put it as harshly as the AC, but you don't know what you're talking about. Are you a sysadmin of any kind? Security was never the direct reason for virtualization. Utilization was. Now, virtualization may not help with crackers, but it does help isolate configuration issues, runaway processes and things like that. Admins like to keep one app on one machine because in a production environment, we are afraid of borking things that are currently working. Virtualization lets us keep one app per virtual machine, while letting us more fully utilize our physical hardware. This cuts down on electrical and cooling bills, & frees up rackspace. Mainframes have been doing this for decades.

Counterargument

Virtualization layers can be much smaller than operating systems. Hypervisors don't have to do as much as a monolithic kernel does, so they're less prone to security holes.

Perhaps a Different Train of Thought

Well, here's his original post : [kerneltrap.org]

Virtualization seems to have a lot of security benefits.

You've been smoking something really mind altering, and I think you should share it.

x86 virtualization is about basically placing another nearly full kernel, full of new bugs, on top of a nasty x86 architecture which barely has correct page protection. Then running your operating system on the other side of this brand new pile of shit.

You are absolutely deluded, if not stupid, if you think that a worldwide collection of software engineers who can't write operating systems or applications without security holes, can then turn around and suddenly write virtualization layers without security holes.

You've seen something on the shelf, and it has all sorts of pretty colours, and you've bought it.

That's all x86 virtualization is.

It's highly probable that Theo is right. After reading the above post, it's highly probable he is a very abrasive and one sided individual. But this is a tech forum so I won't get into judging character.

However, his technical argument is ... somewhat unsound in my humble opinion. He seems to follow the train of thought that 1) people are, by nature, erroneous coders 2) virtualization means more code therefore 3) virtualization has more errors.

I'm going to point out some other things I know about coding. Although more lines of code usually means more bugs, this is not always the case. Correlation does not equal causation. It is correlated but only because the more lines of code, the more probability that more people contributed to the project which means it is highly probable one of them was a bad coder. Also, if you plan things out and follow a rigorous model, it is within your power to make very fully functional, very nice software.

My second point is a different way of looking at the problem. Let's take the naive approach of assuming a primary job of the operating system is to protect the user (and applications) from completely fouling things up in the hardware & memory realm. So it does an 'ok' job at this but, as Theo noted, some bugs still exist. Let's say it's something really bad like they don't stop programs from altering a very sensitive range of memory that is very vital to the correct execution of the operating system itself. Now, hypothetically, the virtualized layer on top of this would give coders a chance to catch this and correct it and protect the user from bringing down the operating system. In this way of looking at things you have two nets. Alone one lets many things pass through so you double it up and now you're catching more fish.

But my analogy is probably very flawed and I must confess I have coded neither of these pieces of software so I cannot confirm or deny this. I am quite shocked that Mr. de Raadt would react so abusively to a post where someone was merely saying that they 'appeared' to be receiving some amount of additional security from virtualization.

As for the very last comment Mr. de Raadt makes, I am confused. My employer uses virtualization on a mass scale to more effectively utilize hardware. I believe it has more uses than just bright shiny colors and wrapping--in fact I am interested in its potentials for hosting web OSs and other neat applications to users. It might not be the future like some people think it is but I think Mr. de Raadt was suffering a moment of frustration or dealing with irritable people when he authored this.

I do wish he were open to more ideas. The second you start to just outright dismiss all your options because they don't satisfy you on the surface you will find you are left with none and often miss the best.

Re:Perhaps a Different Train of Thought

This doesn't just go for Theo. Many geeks have a superiority complex that causes them to be acerbic, arrogant, and dismissive in technical discussions.

Actually caused by strong feelings of insecurity. The secure don't need to attack to try to constantly prove their superiority.

 

Theo is so full of himself he misses reality

Theo thinks so highly of himself, he is just wrong on this one. There is not one recorded/public example of someone breaking out of the isolation of a virtual environment! I dare someone to demonstrate otherwise, and I will eat my words.

VMware selloff

Thanks for the insider tip Theo, I just dumped all of my VMware stock.

Missing the point

Virtualization layers, and their cousins Separation Kernels are the darlings of the security crowd because they can be written in a relatively small number of SLOCs, which means there is a possibility to formally analyze them. Where Formal Analysis means proofs written in a well established mathematical notation, and machine checked. Green Hills Software has a separation kernel that should shortly be certified to a very high level (EAL 6 Augmented) CCEVS [niap-ccevs.org]

What are the big threats now?

A few years ago, it seemed email worms were constantly ravaging Outlook. That, I noticed. But that seems to have tapered off. Haven't noticed any panicked patching of zlib or ssh or sendmail lately. What is keeping people busy these days? Spyware-infested zombie boxes? Anything else?

Risk profiles

Let's consider the following:

1. Security is improved by minimizing the number of services your software layer exports.
   
2. Virtualization has a relatively small, well-defined number of services.
   
3. Operating systems do not.
   
4. ???
   

Virtualization is no doubt a complex problem to get right, but it's only one problem. There is a relatively fixed set of hardware any virtualization system claims to support. A reasonably complete virtualization system can be frozen at some level of functionality. An operating system can not; it must, by nature, constantly evolve to new requirements. Hardware, in contrast, is relatively more stable.

Operating systems running on virtualized systems also have the advantages of operating systems running any fixed configuration. While not quite as consistent as a completely emulated environment, virtualization gets most of the benefits, under reasonable assumptions.

So, in short, virtualization has the same sort of benefits microkernels were supposed to provide, albeit with a much more heavyweight solution: smaller core that's easier to secure. Virtualization has been used in the mainframe community for years. Virtualization is an even stronger form of process isolation than what operating systems provide.

Virtualization is much more costly to run than a standard operating system process. This should be a clue that it probably provides stronger isolation guarantees, even if you don't buy the rest of the argument.

I think it's a specious argument, as usual, to claim that securing the virtualization layer is no harder or easier than securing an operating system. I think securing the virtualization layer is going to be much easier, because while the problem itself is complex, it's still less complex than a complete operating system is.

A better argument would have been to point out that guest operating systems running under virtualization are no less vulnerable to being compromised than those running on real hardware. But then that would point the finger at operating system vendors, not virtualization ones.

Topic is x86 Virtualization

From TFA,

The topic is specifically about virtualization on the x86 platform.

Useless

Theo's side keeps asserting that "x86 virtualization isn't secure", but they seem to be perfectly comfortable at keeping the discussion at the level of a "I'm right, NO I'M RIGHT", without any corroborating statements (Hint: Theo's "I am familiar with x86 and its 'nastiness'" isn't one). What's not secure about SVM? What's not secure about VT-x? Why does Theo think that virtualizatio somehow has to imply legacy PC I/O emulation?

Ugh.

Re:Useless

What's not secure about SVM? What's not secure about VT-x?

VT-x and SVM provide paths for rootkits to integrate and hide. New rootkits like Blue Pill [bluepillproject.org] and Vitriol [theta44.org] utilize SVM and VT-x to virtualize the host platform and remain undetected and immune from removal. They're not widespread, but an attack vector exists, which implies the security concerns over them.

Makes sense to me.

I'm Not Sure I Buy His Analysis

The snippet presented seems to suggest that more security holes in virtualization = less secure operating system, or OS(X) + V(X), where OS(X) represents the operating system vulnerabilities and V(X) represents virtualization vulnerabilities.

However, I see this more as if the virtualization layer actually sits under the OS layer, then the actual security for remote intrusion would be, first, Y/OS(X), THEN Y/V(X), where Y is the number of people with the knowledge to exploit each vulnerability. Thus, someone who wanted to exploit the system would both have to be capable of exploiting an OS vulnerability, and THEN also exploiting a virtualization vulnerability.

(And we're talking about remote usage, because we all know it's virtually impossible to protect a system from anyone who has direct access to the hardware.)

I understand that reality may not be quite as tidy, but it still seems like a virtualized system would be much more secure that a non-virtualized system, if only because the increased level of knowledge involved means a smaller number of hackers capable of exploiting both layers. What am I missing?

But it's so fun

You mean my strategy of running Windows inside of Mac Parallels inside of Pear inside a VMWare instance in a Wine bottle isn't the most secure, stable environment ever conceived? Sheeze. Maybe I should just get a Mac. :)

--
http://www.metagovernment.org/ [metagovernment.org]
GOVERNMENT BY *ALL* THE PEOPLE

Theo rocks, as his usual!

And as there is no engineer that can develop hardware without security bugs, the only solution is to stay with insecurity!

Sounds to me like "those who can't, bitch"

This sounds suspiciously close to his comments about journaling filesystems when asked why OpenBSD didn't support them (which boiled down to "journaling sucks, use softdeps instead"). OpenBSD has native support for exactly zero virtualisation schemes, whereas NetBSD has native Xen support (something Opensolaris is working on -if they don't have it already), FreeBSD and Linux both have support for kqemu and Linux and Windows both have support for VMWare, Virtualbox and kqemu.

For fuck's sake, OpenBSD can't even offer a modern version of WINE in their ports (the one they offer is from 1999, and is broken to boot).

So instead of fixing OpenBSD so that it has native support for running some sort of native virtualisation scheme, Theo does what he usually does -bitches, whines and blames the technology for the flaws in his OS.

Re:When a Port is Lagging Behind the Mainstream

For fuck's sake, OpenBSD can't even offer a modern version of WINE in their ports (the one they offer is from 1999, and is broken to boot)

The ports tree is 3rd party stuff, not OpenBSD. Why don't YOU contribute instead of whining.

Because I have a wealth of already working solutions to choose from. Why in the fuck would I waste my time contributing to a project full of assholes (read the mailing lists for details) who don't even want to admit there's a problem in the first place -much less fix it?

Don't know about you, but I've got a life to live; if OpenBSD doesn't feel like offering virtualisation technology (or -in the case of WINE- compatibility functionality) then I'll simply move on and use operating systems which do.

And if Theo insists on making it sound as if the problem isn't OpenBSD's inability to support virtualisation but virtualisation itself; then I reserve the right to laugh my ass off at his extremely silly pompousness.

credibility?

Theo's childish, condescending and pointless choice of language seems to undermine his credibility. Although he may be an authority on the subject, I think he owes it to himself - as well as the rest of the community he helped to create - to communicate in a more professional, civilized and respectful manner.

He's in the same bucket as Dvorak - who wants to listen to the little twerp?

Its not just the developers...

I do a lot of prototyping and testing out of scenarios with virtual machines. (40+ iterations for servers and client) Not all are complete builds as I do a lot of cloning. If you fire up a virtual machine that hasn't been in use for a while, you may need to spend time with security updates. Also if you didn't place or adequately configure virus protection and a firewall in an original clone you may end up with a number of machines with poor security. On the other hand cleaning up viruses is easy with my scenario, I just delete a current clone and go back to one not infected. (Assuming the virus is readily identifiable.