Mark Russinovich on Windows Kernel Security 181
An anonymous reader writes to mention that in the final part of his three part series, Mark Russinovich wraps up his look at changes made in the Windows Vista Kernel by exploring advancements in reliability, recovery, and security. "Applications written for Windows Vista can, with very little effort, gain automatic error recovery capabilities by using the new transactional support in NTFS and the registry with the Kernel Transaction Manager. When an application wants to make a number of related changes, it can either create a Distributed Transaction Coordinator (DTC) transaction and a KTM transaction handle, or create a KTM handle directly and associate the modifications of the files and registry keys with the transaction. If all the changes succeed, the application commits the transaction and the changes are applied, but at any time up to that point the application can roll back the transaction and the changes are then discarded."
Not dupe, but almost (Score:1, Interesting)
On topic now, I don't like the way Russinovich is blowing Vista's horn now. I liked him more when he was more critical and analytical on what could be improved, instead of what has already been done.
Re:Not dupe, but almost (Score:4, Interesting)
Re: (Score:1)
Re:Not dupe, but almost (Score:5, Insightful)
He's working at Microsoft now, you do realize that "what could be improved" he's actually doing right now, so he can call it "already been done".
You don't really prefer pointless critique with no improvement, do you.
Re: (Score:2)
Re: (Score:2)
You don't really prefer pointless critique with no improvement, do you.
Point conceded.
Re: (Score:1)
Re:Not dupe, but almost (Score:4, Insightful)
Fresh blood always brings fresh perspectives. Mark has been on the outside looking in for long enough that he probably has a fairly asymetrical view of the NT kernel to that of the core MS developers, and in software development (especially at that level) that tends to be extremely useful.
Re:Not dupe, but almost (Score:4, Insightful)
Message to Microsoft (Score:2, Funny)
doesn't belong in the kernel (Score:4, Insightful)
Re: (Score:3, Insightful)
Re: (Score:2)
Re: (Score:1)
Not that you should be using ifdef to separate platform-specific code anyway... it should be put into another file.
Re: (Score:2)
It's like saying that PHP *is* HTML.
Re: (Score:2)
I have not seen a copy of the HTML specification that included any mention of PHP.
Re: (Score:2, Insightful)
Yes, I realize that.
And the preprocessor directives are part of that language. Or perhaps you missed sections 5.1.1 and 6.10 of the description [open-std.org] of that language? (Assuming the numbering stayed the same from draft to final.)
Re: (Score:2)
If you want to stomp your feet and pout, go ahead.
Still. Not. C.
Re:doesn't belong in the kernel (Score:5, Informative)
Userland apps don't have that kind of control over the registry. Hell they may not be sure to have that kind of control over the files they're manipulating.
Besides, I'd rather have this code once in a DLL than 10 times in 10 different apps. That's real bloat.
Re:doesn't belong in the kernel (Score:4, Insightful)
Of course it can. As long as the OS provides atomic I/O operations on individual files, the logic for implementing two-phase commit is the same in userland or in kernel space. For that matter, atomic I/O can be implemented entirely in userspace, as long as the OS provides a mechanism for userspace to find out when the data has actually been written. It's more convenient to put single-file atomic I/O in the kernel, though (or at least in the file system).
That's not an argument for putting it in the kernel.
Re: (Score:2)
Upon further thought, I stand corrected. There are lots of cases you can implement without kernel support, either with cooperating programs or with symbolic links, but a fully general implementation has to be in the file system.
Re:doesn't belong in the kernel (Score:5, Informative)
Of course it can be done in user space. If a user space app can't do it, neither can the kernel. And it isn't atomic I/O. There's no such thing as atomic I/O. I/O operations are reordered, split, combined, etc. by everything from the OS to the controller to the hard drive, and for network volumes, it is even worse. There's no practical way to guarantee atomicity, so you have two choices: have filesystems (including remote filesystems) with rollback capabilities (which still don't completely guarantee anything) or design a file structure that achieves the same thing (which still doesn't guarantee anything). The former is nice for a lot of reasons (e.g. so that every developer doesn't have to reinvent the wheel), but isn't essential by any means. It also would greatly increase the complexity of the VFS layer and filesystems written for it, so if that is the only purpose for doing transactions, it makes a LOT more sense to implement them in a user space library instead of in the kernel.
If your sole purpose is to be able to do multi-file rollbacks, user-space transactional support is as easy as designing your file format and/or layout around it. There are two easy ways to do this: files with built-in history and using swappable directories.
Files with built-in history:
For the initial modification pass, modify each file by appending what amounts to a diff footer. If an error occurs, you can undo all of the changes by truncating the files prior to the latest diff footer. Once these modifications are complete, you no longer need to worry about rolling anything back (except for cleaning up temp files if something fails in the second pass) because the data is safely on disk. (Note: this does require that the kernel and all devices and/or network disks reliably flush data to disk upon request. Don't get me started on buggy ATA drives.)
In the second (optional) pass, you coalesce the diff into a new copy of the file and swap the compressed version in place of the original file. This is generally an atomic operation in most operating systems. If anything occurs during the second stage, it is a recoverable failure, so there is no need to roll anything back. Heck, Microsoft's file formats pretty much do this anyway. (Notice the 500 megabyte single page MS Word document that occurs when you make lots of changes and always "save" rather than "save as".)
Swappable directories
The easiest way (tm) to handle system configuration files in an atomic fashion is to modify config files in the same way you would perform a firmware update: you have an active configuration directory and an inactive configuration directory. You read the active one, make changes, and write to the inactive one. Then you trip a magic switch (tm) that says that the previously inactive directory is now active, and vice versa. Assuming you don't have out-of-order writes going on (which the kernel can't really guarantee any better than user space, sadly), this is a very easy, effective way to perform an atomic commit. And if you have an "exchange in place" operation in which the data for two files or directories in the same directory are swapped in a single atomic operation, that's a really lightweight way to implement an atomic commit/rollback mechanism without most of the complexity.
Considering how easy this is to deal with in user space, the only legitimate reason I can think of to do it in the kernel is so that you can take it out of application control entirely (e.g. to make it easier to sandbox an untrusted application). Otherwise, it makes a lot more sense to do this in a library. Now if it had snapshotting where you could roll back the entire filesystem to arbitrary points in time, that might be interesting (for different reasons)... but basic transactional support in a filesystem is much less so, IMHO, unless your purpose is to be able to sandbox an application. If so, then all this other stuff basically comes for free. In that context, doing this in the filesystem layer makes sense. However, if that is not their purpose for doing this in Vista, then kernel bloat definitely strikes me as an accurate depiction.
Just my $0.02.
Re: (Score:1)
Re:doesn't belong in the kernel (Score:5, Interesting)
IO commands can be reordered as much as you want, but commands can be tagged for flush or immediate execution, and if a device doesn't comply with those, it is either badly writen or it has incorporated enough safeguards so it doesn't matter.
I'm laughing. No, really. A lot of devices lie in one way or another. It's not just a few badly written devices. You'd be shocked. My favorite are the drives that completely lie and say that the cache has been flushed when it really hasn't. Most of those drives, you can issue a second flush request and it will block, but there are some that always instantly return success for a flush request. It's not even all that uncommon, ESPECIALLY with external drives because of bridge firmware bugs.
I've heard some true horror stories about the number of problems companies find when qualifying off-the-shelf ATA drives for server use. If you heard them, you would cry. Truly, all you can assume is that the operating system has made a best attempt at a guarantee. Anything more than that is just kidding yourself... and you can get that same level of guarantee just as easily from user space (at least in UN*X) by executing two sync system calls in a row.
In addition, how do you expect a user mode component to be able to implement transactional services for kernel components?
WHAT!?!?!?!?!?! Why in you-know-where would anything in the OS kernel need filesystem transaction support? Stuff in the kernel shouldn't even be READING files, much less writing them! Yes, I'm serious. The kernel should provide only the most critical, basic system-wide OS services. Anything that requires such heavy lifting is NOT providing such a critical, low-level service, and thus, does NOT belong in the kernel. It's that sort of thinking that has led to such abominations as khttpd....
I'm going to go hide now. I'm very scared....
Re: (Score:2)
Re: (Score:2)
Of course, this doesn't completely solve consistency for the registry. Atomicity does not guarantee consistency with shared data structures. It just guarantees that everything (right or wrong) happens at once. Only exclusive locking can guarantee consistency.
For example, say that you have two values, A and B, and two processes, 1 and 2. Both of these are counters, both set to zero. Process 1 intends to increment both counters, so it reads the contents of both counters. Process 2 increments counter A
Re:doesn't belong in the kernel (Score:4, Interesting)
And yes, this is a really big deal. Other than maybe OS/400 I know of no system where file system operations can be done atomically with database operations. I can poll data, write a file and update that data all in one atomic operation. Either the data has been exported and marked as such, or not. No in-betweens. No incomplete files. No complete files despite a failure to mark the data in the database. No attempt to manually compensate for errors. It's all there, one operation, atomic and automatically so. This is absolutely massive for my world where financial and business transactions are moved constantly in such a fashion.
Yes, Microsoft does innovate sometimes. This is one of those occasions.
Re: (Score:2, Informative)
Well, DEC VMS had this capability decades ago, so is it really innovation?
http://h71000.www7.hp.com/commercial/decdtm/index
Re: (Score:1, Insightful)
The actual innovation is making a Kernel Transaction Manager, along with a resource manager for the filesystem. The KTM means that transactions can be inherited from parent process to child or joined by a cooperating process. Having a transactional filesystem means that all file operations can be all-or-nothing. Power goes out during a major update? No probl
Mod Parent (Farther) Up (Score:2)
This is cross-facility transaction management: registry and filesystem updates combined into a single transaction. The example in TFA that an entire install can be atomic: multiple filesystems, registry, everything appears complete and as requested, all at once, or it never happened.
It's extensible, if TFA is to be believed at all, and the facility works. It's actually there and in use, rather than an it'll be there someday and won't it be wizzo promise, so I'm in "trust-but-verify" mode. It'll be int
Re: (Score:2)
So "UNIX" has a non-general-purpose or non-kernel transaction manager, and has for decades. Seeing as how UNIX has been implemented at just about every conceivable point on the spectrum from microkernel to OneBigBlob, we can take that as a general assertion that you can find some kind of transaction manager out there somewhere for some variant of UNIX. Good. So what? Now Vista's going to have one too. Good for them.
I'm already on record here [slashdot.org] as saying that VM boundaries aren't the only way to mainta
Re: (Score:1, Informative)
Re:doesn't belong in the kernel (Score:5, Insightful)
For instance, how does a user-mode controlled file-system transactions rollback changes on reboot? Consider what happens when the change involves system files. NTFS handles this cleanly in Vista but any attempt to do this with a user-mode service would fail because the system files would already be in-use by the time the service started. Further, any transactions in-doubt would remain in-doubt until the service started (if you could get that far).
Similarly, what about settings in the registry? Consider if the transaction spanned installing a driver and updating its settings. What happens if the system powers down midway through that update? Without kernel-level transactions that are available at boot-time you cannot easily recover before inconsistent settings are used with the driver.
Beyond the boot-level support; another exciting aspect about the KTM is that this is a transaction-manager that provides transactions that can be shared across multiple processes. Try that with a user-mode transaction manager like XA or DTC and see your per transaction performance. By managing the transaction in the kernel, the 2PC performance is significantly improved.
In the end, NTFS and the registry live in the kernel -- so their tramsaction manager also has to live in the kernel. That is just the way it works.
Another exciting aspect covered is the ability to coordinate DTC transactions with KTM transactions. Meaning, you can coordinate your SQL Server (or Oracle) updates with your file-system updates. This is cool! No longer do you have to worry about finding orphan-files in workflow applications or other applications that manage both files and relational data.
Lastly, they are talking about full ACID properties with NTFS's transactions. Think about it -- you could update every file on your web server, in place, while its wild with activity. All your changes are isolated from the users until you say "commit" in your application.
Re: (Score:2)
I think he was talking about this:
Step 1.
Re: (Score:1, Troll)
Re: (Score:1)
Kernel enhancements? Are you sure? (Score:2, Insightful)
"Requires Ktmw32.dll."
Why would a kernel add-on require a
What is the registry in Vista? (Score:3, Interesting)
For years, the "Registry" was some weird mish-mash of binary files, many of which represented Jet databases.
Has Jet been completed abandoned in Vista?
If so, did they switch to the slimmed down SQLServer [that was supposed to be part of WinFS]?
Re: (Score:1)
Re:What is the registry in Vista? (Score:5, Informative)
There are two database engines that have been known as Microsoft "Jet", known as Jet Red and Jet Blue. Jet Red [wikipedia.org] is also known as the Access database engine. It is a fairly featureful SQL database. Jet Blue is now officially the Extensible Storage Engine [wikipedia.org] (ESE), and has been a system component since Windows 2000, backing WMI data, Active Directory, Exchange, and others. It is an ISAM database and is optimized for large sparse tables and also supports a transaction journal. Both are 100% user-mode and were not a part of the initial release of Windows NT. Microsoft has said that Jet Red is depreciated, and that future versions of the Access database engine will be integrated with Access and not have a public interface. Jet Blue's interface is well documented [microsoft.com] and will continue to see use for some time to come. Both being user-mode, dependent on Win32 and the wrong type of database (relational vs hierarchical), the Jet engines would not be suitable replacements for the registry.
SQL Server is a high-end SQL database engine. It was rumored that WinFS would use SQL Server Express and that Microsoft eventually plans to move some of the services that use Jet Blue to SQL Server (such as Active Directory). In any case, SQL Server is an even less possible replacement for the registry.
Microsoft has not gotten rid of the Registry in Vista. In fact, the new boot manager uses a registry hive to store boot configuration, replacing the old boot.ini.
So what is this thing? (Score:2)
The registry engine is implemented in kernel mode as an executive subsystem (inside ntoskrnl.exe), where it is known as the Configuration Manager. Registry hives use a transaction journal (like many filesystems do) to avoid corruption during a power failure or crash...
So you're saying that the engine which drives "the Configuration Manager" is neither Jet Red, nor Jet Blue, nor SQLServer Express?
So what is it? YAMIHDE [Yet Another Microsoft In-House Database Engine]?
Everything above is still the same
Re:Kernel enhancements? Are you sure? (Score:5, Informative)
Windows NT was initially designed to use single kernel for different subsytems (OS/2 subsystem, POSIX subsystems, etc.). Subsystems are implemented as dynamic modules talking with the kernel through LPC (Local Procedure Call, see http://en.wikipedia.org/wiki/Local_Procedure_Call [wikipedia.org] ). So in this case ktmw32.dll just wraps LPC calls in a nice API. That's actually a rather good design.
Re:Kernel enhancements? Are you sure? (Score:4, Informative)
Not just initially designed, it DOES use a single kernel for different subsystems. You can't get the OS/2 one any more, but the POSIX subsystem morphed into (part of) the Services for Unix which has become the Subsystem for Unix-based applications.
On 32-bit Windows, 16-bit Windows applications are handled by the "Windows on Windows" subsystem. On 64-bit Windows, 32-bit Windows applications are also handled by a "Windows on Windows" subsystem, though a different one than WOW16.
Re: (Score:1)
Re: (Score:1)
Hmmm... Upgrade to Ultimate Vista perhaps? (Score:4, Funny)
What, was my credit card declined for my upgrade to Vista Ultimate Edition?
Re: (Score:1)
Not exactly "error recovery" (Score:5, Insightful)
Not exactly most people's idea of robust and recoverable.
Re:Not exactly "error recovery" (Score:5, Insightful)
The format of the registry is largely irrelevant, but it is described to some extent in the "Inside Windows xxx" series of books (which Russinovich co-authored)
Which specific byte would you "scrozzle" in the registry to render a machine unbootable? How and why would you do it?
Extra credit:
Which files under
There are things to like or dislike about either a registry based approach (opaque data storage with a defined interface) vs a flat text based approach (clear data storage with an undefined interface). I don't think you make a compelling "anti-registry" argument with the points you list, however.
Re: (Score:3, Insightful)
What you've said is correct, the gp's gripe is really about using a binary configuration file.. a fairly stupid decision but that is only my opinion.
My argument for flat text (or xml or whatever) over binary, is the sam
Re: (Score:2)
What it comes down to is
Re: (Score:2)
Obviously what an app puts into its configuration files is undefined, but I'm pretty sure you can put Strings in the registry as well?
The registry is essentially a (bad) re-implementation of a filesystem, it's no more standardised (less so really) than just USING the filesystem.
Re: (Score:3, Insightful)
what is the comment convention?
What is the whitespace convention?
Does this version of this flavor use spaces or tabs in
What are the file locking semantics?
Which set(s) of files are logically related to the same peice of functionality?
What character encodings are supported?
What _line termination_ is supported?
nfs export your
Re: (Score:2)
Dunno about him, but... I'm crazy, not stupid.
Re:Not exactly "error recovery" (Score:4, Insightful)
Difference being you're not supposed to modify things in /boot and /sbin for all your settings, and /etc is text and therefore much harder to screw up. (you could put an EOF as the first byte in the file, but the system will still probably at least give you a "file x is empty" error message).
Users aren't supposed to modify the Registry, either.
What you've said is correct, the gp's gripe is really about using a binary configuration file.. a fairly stupid decision but that is only my opinion.
It was designed in the late '80s (well, arguably the late '70s) when a 20Mhz 386 with 4MB of RAM was "bleeding edge". Text-parsing is expensive.
I think what it really comes down to is this; If you decided to write a new OS today from scratch and wanted to have a central configuration database (a good idea, as shown by /etc), would YOU come up with the windows registry?
Windows NT wasn't designed today, it was designed in the late 80s.
Re: (Score:2)
"It was designed a long time ago" is not a valid argument for how something is today.
"Users" aren't supposed to modify the registry, but it's still where apps store settings, what app is storing settings in
Re: (Score:2)
So about when did /etc become the de-facto standard in Unix for configuation?
No idea, but since /etc and the Registry are worlds apart in function, purpose, usage and capabilities, it's hardly relevant.
"It was designed a long time ago" is not a valid argument for how something is today.
It most certainly is. You can't just remake something as big as a modern operating system from the ground up every time the latest cool fad comes around.
(And I'm sure you'd be happy to defend the train wreck that is /e
Re: (Score:2)
Not to dispute your conclusion, but one relevant aspect is that it seems to happen one hell of a lot more often in registry-based systems. For example, my development system at work is XP-based, and for no apparent reason my Recycle Bin disappeared. Off the desktop, My Computer, just gone. No idea where it went. I first tri
Re: (Score:2)
Re: (Score:2, Interesting)
But yes, I
Re: (Score:3, Informative)
However - a broken byte in an unbacked up (yeah a bad idea) registry [...]
The Registry is automatically backed up at the completion of a successful system boot. This has been true since at least Windows 2000, and probably longer.
Re: (Score:1)
Re: (Score:1)
Re: (Score:2)
Re: (Score:1)
either a registry based approach (opaque data storage with a defined interface) vs a flat text based approach (clear data storage with an undefined interface).
The GP's point was not entirely about automatic recovery from a broken structure; it was about human-led recovery. In a binary dump, even with defined interfaces, a single byte can render the entire structure non-human-readable. With a plain text file, a human can look at it and try to diagnose the problem -- simply by looking.
Re: (Score:2)
If you've booted the computer off of some other media and are attempting to repair the configuration info, you're using a special tool to do it (like a text editor for
Also, I've not seen evidence that a single byte defect renders the registry unreadable or the computer unbootable. I don't dispute that this is possible, but I'd be curious to know if this is a practical issue or a theoretical complaint. F
Re: (Score:2)
Re: (Score:2)
>Which files under /boot, /etc, /sbin, and so on would you be willing to stake your career on being "safe to corrupt by 1 byte" and still guarantee a bootable system?
Apples and Oranges. I can edit the text files and cp the binary files. Can't do either with the Registry.
>The fo
Re: (Score:1)
If you screwed up the registry you could always make them bootable again, you just had to screw around at the command line. Linux does not make leaps and bounds over this fix. X not booting? Time to go screw around at the command line again.
Re: (Score:3, Informative)
Re: (Score:2, Informative)
Re: (Score:1)
Even if you accept that, it's still easier to diagnose and recover from the extra newline.
Re: (Score:1)
The GP's point stands. The registry is no more vulnerable than any number of Unix config files. An extra newline in /etc/password will cause the same problem.
Firstly, there is no such file /etc/password, it is /etc/passwd since K&R didn't like typing and skimped on letters.
/etc/passwd in a linux-pam system.
/etc are much easier to diagnose and fix than an arbitrary (and slow) binary registry where the typical entry is something like "0x67386768787ab7aab87b78cd87687"
Secondly, I just tested this and it is completely untrue. extra newlines have absolutely no affect on the functioning of
Thirdly, transparent human readable and logically laid-out config files in
Re: (Score:1, Interesting)
Re: (Score:3, Interesting)
Re: (Score:2, Funny)
Re: (Score:2)
Re: (Score:1)
Re:Not exactly "error recovery" (Score:4, Insightful)
You can't pump V1@grA spam or DDoS packets out of a dead machine, and malware writers are definitely in it for the cash nowadays. (IIRC, a large percentage of malware specifically hides in the registry...)
'Protected Processes' and PC games (Score:5, Insightful)
With the huge amount of popularity this approach seems to have (I personally suspect it's a result of some very, very aggressive marketing on the part of the driver's developers), I wouldn't be suprised to see many games start demanding that users run them on Windows Vista, so that the 'protected process' mechanism can be used to fully 'protect' the games from users' interference. While you'd at least be able to end-task them, I can't say I see this as an improvement. It's saddening that many companies believe the solution to security is a series of hacks, workarounds, and black boxes - the only real solution is careful, methodical design and engineering. It seems very likely to me that within a few years, many PC games will refuse to run on anything except a Vista system with nothing but signed drivers loaded, and that's saddening. I dislike the notion that I am denied even basic rights to investigate what an application is doing on my machine simply for the sake of 'security', when it's trivial to set up a second machine to inspect and modify a game's network packets and cheat all I want.
Re:'Protected Processes' and PC games (Score:4, Insightful)
Outside of some very specific gov't spooks contract work, no one is willing to adopt the other because it's more expensive and requires more thought and risk than the average PHB can handle.
Case in point: The number of cashless casino gaming systems that are centrally managed. The average casino manager understands their casino systems are a single, massive, point of failure. Just wait till they go to cashless floors and someone engineers a jackpot for themselves.
Security programming is hard. Really hard. The developers at these companies may _want_ to do the right thing, but they don't because of the complexity and limited resources.
The big-digital-bank-robbery just hasn't happened yet.
Re: (Score:2)
That would be a neat trick to do with the end-user UI of a slot machine. Physical security is a pretty important first step.
> The big-digital-bank-robbery just hasn't happened yet.
Actually, when you look at identity theft and fraud, you can see it's happening every single day. Think breadth and not depth.
Re:'Protected Processes' and PC games (Score:4, Interesting)
Re: (Score:2)
The backend running those games is on a network.
What are the chances they've got a gateway that goes to the interweb?
What are the chances they are relying on Windows for their security?
The people running the casino are pretty busy with their day-to-day stuff, not infosec.
Re: (Score:2)
Re: (Score:1, Informative)
This is not true, and it's also not what a rootkit is. These games use rootkits to hide files and drivers from the Windows API, which you can do yourself just by creating a share that starts with '$' or a regis
Russinovich knows his stuff (Score:1, Flamebait)
roofles (Score:2)
Can't you get some new material, you poor, twisted fool? Come on. Keep me entertained over here!
Dear Mr. Kowalski, (Score:2)
I like picking fights with you because I know you cannot ever let anything go. I knew it would be utterly impossible for you to see something posted about you somewhere without your massive ego needing to reply to it with your usual string of self-important bombast. Do you actually presume that I have anything personally invested in calling yo
Figures you'd be a conservative bonehead, too (Score:2)
I've seen that before... (Score:1)
So, they reinvented the wheel once again? It seems to be: every database more complex than a flat file processed by a pair of simple perl scripts has support for transactions like this. So they invented nothing, just applied an old patch to new code.
Re: (Score:2, Insightful)
Yes, DBs do. But traditionally file systems don't. The only other system that provides this that I know of that isn't a full-fledged database is VMS.
i will explain what he is saying... (Score:1)
Sysinternals Utilities (Score:5, Informative)
Re: (Score:2)
Re: (Score:2)
slap it on mytempdir.com or rapidshare.de and then anyone can grab it.
Re: (Score:2)
This is... (Score:5, Funny)
Can we please drop file systems in favor or RDBMS? (Score:2)
A file system is still needed (Score:2)