Diskless Booting For the Modern Age 99
An anonymous reader writes "Ever wonder what happened to PXE? Intel's popular standard for diskless booting hasn't been updated since 1999, and has missed out on such revolutions as wireless Ethernet, cloud computing, and iSCSI. An open source project called Etherboot has been trying to drag PXE into the 21st century. One of their programmers explains how to set up diskless booting for your cloud, using copy-on-write to save space."
How is it slow? (Score:4, Interesting)
TFTP over UDP on a LAN, doesn't seem slow at all. It's stupid, but sufficient to bootstrap a small kernel to access the real meat of your OS. 1-10MB TFTP downloads over 100mbit is no big deal. You can't get good 1gbit performance (let alone 10gbit) out of the dumb drivers in a PXE boot ROM, but that's OK.
Re:How is it slow? (Score:4, Insightful)
Re:How is it slow? (Score:5, Funny)
Is it possible to use multicast? The multicast could run continuously or on demand. Can you route multicast?
It would be pretty impressive if one could attach 3 nodes to the network and they all boot up.
It would be a Borg network. Everything you add to it would be immediately taken over!
NODE (DHCP): Hi I'm a Dell workstat- ...
BORG NODE #1 (x.x.3.43): We are borg
NODE: ACK
BORG NODE #2 (x.x.3.44): We wil add your likeness to our own
NODE: RST RST RST
BORG NODE #3 (x.x.3.40): Resistance is futile
NODE: Downloading borg....
BORG NODE #2: We will add your likeness to our own (as soon as you have downloaded.
BORG NODE #4 (x.x.3.45): We are borg.
Hard Drive is the Enemy. Eliminate. (Score:2)
xCAT (Score:2)
xCAT at its core focuses on network boot (and for x86 portions, includes gPXE). It does Windows, Redhat, SuSE, VMware. It takes a lot of the tedium out at scale of network install, statless boot, and iscsi boot.
Re: (Score:1)
gPXE (the Etherboot project) is used in Fedora's BFO project [fedoraproject.org] (based on BKO [kernel.org]). If you have a local server, that is a cool project
I'm sorry that PXE and Etherboot get no love around here. I've been using Etherboot since ~2000, when it was the common boot method for LTSP (and K12LTSP). EB and gPXE are awesome projects that show how much you can do with so little space to work with.
When network booting several computers, I've found the boot to be much faster over the network than locally once the first computer
Re: (Score:2)
Re: (Score:2)
What it be possible for a multicast to work with unicast corrections?
The receiving node must be aware when a packet is not received, if it is not at a packet level, perhaps at a more logical level like a block of data. Rather than wait for the next multicast, it explicitly requests a retransmission of that block to that node via unicast.
I am of course assuming that only a few packets would get lost.
Or perhaps another corrective multicast that streams corrected packets when enough people request them.
MULTICA
Re:How is it slow? (Score:4, Informative)
Issues:
-tftp multicast is inherently limited to smaller than 98MB images with sane MTU. The same block number wrapping in unicast can't work in multicast. When you want speedup the most, tftp multicast can't even work
-multicast only buys you something if a large number of clients are acquiring the same payload at the same time. In a large scale 'cloud 'configuration, things are generally heterogenous enough to negate any such hypothetical benefit.
-Most ethernet fabrics are either incapable or not configured for IGMP/MLDv2 snooping required to properly scope multicast resulting in all multicast traffic degrading to broadcast. This has very adverse results unless every entity on the network only cares about the transfer.
Re: (Score:2)
Re: (Score:1)
Back in the 90s, there used to be a proprietary-but-free scriptable boot system that used multicast (the name started with a "B"). You would boot each computer you wanted to do an installation on, set them to receive multicast, then once the computers were all connected to the server you had put DHCP, TFTP, and FTP on, the server would multicast an install image and a post-installation script.
I seem to remember that this was before Ghost has multicast.
Re: (Score:1)
Re: (Score:1)
Indeed. That was it. Wow. Thanks for that. I couldn't remember the name. Interesting that it went on from being a fairly small project to an IBM product.
I guess that KA-Tools [ka-tools.imag.fr] became a popular replacement for BPBatch, but KA was still really immature back when I last used BPBatch. Now KA just seems abandoned, probably in favor of gPXE.
Re: (Score:3, Informative)
Yes, Apple has been able to do it for a couple of years now (since the PowerPC era).
Basically a DHCP server says: I have a boot image
Client1 says: thanks - starts downloading 0-10%
Client2 comes on: starts downloading 10%-100%
Client1 continues downloading 10%-100%
Client1 boots up
Client2 requests 0%-10%
Client2 boots up
I think your example involves P2P but unless your client also has the boot image and a mechanism to give it to others after it has booted (which could be potentially a security risk) it doesn't
Re: (Score:2)
The mechanism you describe is only used when imaging a computer, not durring the netboot phase that is often used to deliver the new image. It is really nice if you can run broadcast ASR, but like most multicast has some real limiations when crossing subnets. It is also a bit of a bear to set all the variables so that you get good speed without taking down a network. But once you get the combination for your network it really works out well.
Re: (Score:3, Informative)
Well, one issue is boot payload is getting bigger and bigger. One distro has about 20MB of download that would be tftped in the default case. Windows uses tftp for a *lot* more.
tftp has the following issues:
-16-bit block indexes. Most firmware won't go above 1400 or so blocksize (with good reason, if they go higher and the network is set for jumbo frames, transfers will fail in many scenarios). This means a cap of about 98MB before you overflow the counter. Most tftp servers nowadays can deal with it
Re: (Score:2)
As I mentioned above, most tftp clients go above and beyond the spec and for unicast transfers can outdo the spec (multicast a no-go).
half-duplex is not perfectly fine when there is a relatively accessible better technology.
http (or perhaps more appropriately ftp) are perfectly suitable protocols. The technology has progressed sufficiently far that a TCP stack is feasible in firmware, and there's no point in not doing it.
Re: (Score:2)
This is nonsense. Nobody in their right mind loads up the boot image in the first stage. The first stage is to download syslinux/grub/etc via TFTP. From there, you have all the features included in the boot loader of your choice, and can ignore PXE limitations. Want support for HTTPS? Good idea! Go include it in your boot loade
Re: (Score:2)
The thing is, syslinux/grub/etc all make PXE calls to retrieve files, which is TFTP. gPXE as a firmware (or as an undi driver delivered via PXE) gives syslinux http protocol support.
Since you mentioned IPv6, the DHCPv6 RFC is just about to come out, and specifies a url syntax for protocol flexibility, as well as recommending http as the protocol of choice (FTP kinda gets the shaft...).
Re: (Score:2)
Well that would be because PXE is pretty ancient stuff.
I have a gPXE deployment on my home network which screams like a banshee on roller blades.
Gigabit speeds, transfers over http and a good deal more configuration options.
With gPXE you can bootstrap a machine from Taiwan over the wan. The kicker is you still need a local agent to serve the tftpboot portion.
Beyond speeds the main benefit I find with gPXE is the ability to now load balance the kickstart servers. Even dhcpd has some features built in to assi
PXE/DHCP/AoE (Score:1, Interesting)
I did a similar thing a while ago with an Ubuntu desktop image and ATA over Ethernet. Worked nicely. Didn't get round to COW though ...
http://www.s-mart.net/aoe.txt
Re: (Score:1, Redundant)
There's always GOAT or even SOY.
Re: (Score:2)
And that gets to the point of answering the question "what ever happened to PXE?" - it got replaced by virtualised storage for booting virtual machines, running on a fat software hypervisor. It's still nifty though, and useful when you do it right.
Re: (Score:2)
Authentication (Score:3, Interesting)
The one thing missing from PXE is authentication: A PXE system will accept any DHCP address and with it any boot server configuration. Without cryptographic boot image authentication, network security is the Achilles' heel of PXE.
Re:Authentication (Score:5, Informative)
Re: (Score:2, Interesting)
It's still an omission. If you use PXE for remote administration (instead of using it for completely diskless operation), then there is local data which can be compromised by a hostile PXE payload. How hard would it have been to verify a cryptographic signature against a public key stored in the BIOS configuration?
Re: (Score:1)
It's still an omission. If you use PXE for remote administration (instead of using it for completely diskless operation), then there is local data which can be compromised by a hostile PXE payload. How hard would it have been to verify a cryptographic signature against a public key stored in the BIOS configuration?
Yes it was missed in PXE, however, based on the context of when PXE was developed I doubt we thought we needed it. The point of gPXE is that PXE hasn't been developed in line with changes in the computing arenas and that these omissions needed addressing.
I've checked further and even though there is a level of authentication built into the command line it doesn't yet have enough development to support 802.1X in gPXE, however, WEP, WPA and WPA2 are now supported so the remote boot from wireless can be un
Re: (Score:2)
So we're getting into cloak-and-dagger territory but wouldn't this be a simple way to have a malicious image installed on the client that'll relay logins back to you? If you're first redesigning it I'd make it so you can both verify cryptographic signatures and use SSL. It would be great if you can plug in your diskless machine into a presumed hostile network and know that it'll only connect to your designated boot server no matter what.
Re: (Score:1)
Yes it would and unfortunately its straightforward, however, there are lots of levels of mitigation in the infrastructure if network admins understand that it is there and use it - whether this is the case for the cloud is another matter.
What we are saying in a number of these posts is quite simply - "I don't trust core services on the network I'm connecting to". The response is, "Don't connect to that network or allow that network to connect to you".
Firstly - do you trust your router or DHCP server
Re: (Score:2)
If you permit rogue DHCP servers on your network, you have bigger problems than PXE.
The correct solution is to block DHCP response packets from unauthorized ports at the switch level, and make it a policy to smash the knees of anyone who connects a 'home router' to your corporate network.
Re: (Score:2)
If untrusted machines are being attached to the same network segment you are, you've got a hell of a lot of problems, netbooting or not...
i dont wonder...... (Score:3, Interesting)
i am using pxe often.
i have setup a few linux install "CDs" for network install, a few live CDs for an emergency OS. LTSP is using it too and a small intel atom box gets its kernel over tftp/pxe... the pxe provides the parameters for the nfsroot mount. :) :)
old win2k netinstall for ppl without a RIS uses that system too
the tftpd box that provides all that stuff is a small amd geode that is normally my router
i often thought about making a sourceforge project out of it.... :)
Re: (Score:2)
I also use PXE. My home server (an old P3 laptop) runs from an USB drive, but since the laptop doesn't support USB booting, I use PXE to boot it.
The annoying part is that I can't reboot it remotely, as it won't have a TFTP server to feed the Linux image,
Re: (Score:2)
i often thought about making a sourceforge project out of it.... :)
I have often searched for how to set that up.
Re: (Score:2)
http://www.netboot.me/ [netboot.me] + PXE = ultimate network config.
After setting up my own PXE setup, I put the settings in my DD-WRT. After completely hosing the only computer in my house (I left my laptop at work). I would have been SOL (no floppies, no CD's) Not so with Netboot.Me. Fired up a Debian recovery disk. The have recovery disks, minimal install disks, partition disks, FreeBSD, Linux.
I haven't been wondering... (Score:5, Interesting)
Re: (Score:2)
The trick is to use a wired card with a PXE ROM attached directly to a wireless router with firmware that supports a bridged mode. In my case, I used a WRT54G running DD-WRT.
You configure the router to bridge the wireless and wired networks, and enter the appropriate information for your wireless network. Then, you configure your netboot system normally (DHCP, TFTP, etc.). When the client boots up, the PXE ROM thinks it is directly attached to a wired network so everything works normally.
The downside is you
Re: (Score:1)
I'm guessing that the "machine *effectively* netbooted wirelessly" means that the machine was connected via ethernet cable to the wireless router (which was in bridge mode....)
Not really sure if my explanation serves to clarify....
cheers,
Re: (Score:2)
Obviously you've never used a Mac...
Re: (Score:2)
Macs since about 2007 can netboot wirelessly. Before that, you had to connect via Ethernet. But MacBook Air has no Ethernet, or optical drive. The way you install an OS is to boot from the DVD in another machine on the Wi-Fi network. Optionally, you can use a USB-to-Ethernet adapter to speed things up, or a USB DVD drive. But a Wi-Fi netboot is the default way.
The Mac firmware is savvy about all of the system's hardware. It can also connect to Bluetooth devices, so your keyboard and mouse continue to functi
Re: (Score:1)
Cloud? (Score:5, Funny)
Warning / Rant: The last 5 years of computing have been pretty lame. Concurrency and solutions to it using functional high-level languages are the future. That's where we should have been five years ago when it was so obvious that chips with large numbers of cores were the future. These cloud solutions are just a stupid name for the same old monolithic crap. It doesn't scale and isn't modular in a Unixy way. Modern applications just suck because they're so inflexible. Why can I do so many things from a little text terminal, but I can't easily script the behavior of my web browser without special add-ons? Why aren't modern applications flexible like this, with simple interfaces for communicating with other programs? Where is the equivalent of a shell pipe, in modern applications? It's like somebody threw away all the lessons of the past, and said "But this is the new way, we don't need the old way, because this is new." Fuck that, computing should be better than this. It should be better than these stupid clouds and old piece-of-shit reinvent-the-wheel C and C++ programs with buffer overflows and other ancient problems. Or the HTML / Javascript / whatever jerry-rigged "web applications" that run on some opaque "cloud" that a random company has. Why is it that languages like Smalltalk and Lisp have been around for so long, and nobody learns from them or uses them? It's like the chips keep getting faster and faster, and people keep getting dumber and dumber.
Re:Cloud? (Score:4, Informative)
It doesn't scale and isn't modular in a Unixy way. Modern applications just suck because they're so inflexible. Why can I do so many things from a little text terminal, but I can't easily script the behavior of my web browser without special add-ons?
http://en.wikipedia.org/wiki/Uzbl [wikipedia.org]
Re: (Score:2, Informative)
http://www.uzbl.org/faq.php
Cool, so I can install uzbl and have a "Unixy" browser that does next to nothing without a headache and a weekend of tinkering ... or I can install Firefox and actually get shit done. Sounds about right.
Re: (Score:2)
http://www.uzbl.org/faq.php
Cool, so I can install uzbl and have a "Unixy" browser that does next to nothing without a headache and a weekend of tinkering ... or I can install Firefox and actually get shit done. Sounds about right.
Cool, so I could Firefox and actually get shit done, or I could install uzbl and have a "Unixy" browser that does next to nothing without a headache and a weekend of tinkering.
FTFY (this is /., after all...) *goes away to play with uzbl*
Re: (Score:2)
It's like the chips keep getting faster and faster, and people keep getting dumber and dumber.
I believe somebody said once "the amount of intelligence on Earth remains constant"...
Re: (Score:2)
I believe somebody said once "the amount of intelligence on Earth remains constant"...
I wager it goes down during wars, dictatorships, televised games, and elections. Orders are to destroy anyone who is nonconforming.
Re: (Score:1)
It's like the chips keep getting faster and faster, and people keep getting dumber and dumber.
I believe somebody said once "the amount of intelligence on Earth remains constant"...
while the amount of people it is spread between increases exponentially?
Re: (Score:2, Interesting)
My grandfather was telling me about how he used to walk inside of an old cray with 5 processors like 50 years ago. What happened to the people who used to know how to program those things efficiently? Apparently multi-core programming isn't so new.
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
When you're paying a few million for a Cray, paying top salaries for programmers who can take advantage of it is the only sensible way of getting a return on that investment. When you're paying $500 for a multicore server, paying top salaries costs a lot more than just upgrading to a $5000 server.
$1200 for a multi core server with three Chinese indentured slave programmers, service for ten years. China software-factory sweat shop escape-and-rebellion insurance and translation services not included.
Re: (Score:2)
What the hell do you mean "Apparently"?
I was doing commercial multi-threaded programming a decade ago, and it wasn't new then. Multi-process across multiple cores has been with us for decades.
Parallel programming is not massively difficult, but here on Slashdot people seem to think it's some crazy new thing that's really hard and nobody's ever done it before.
No it isn't and yes they have.
Re: (Score:2, Offtopic)
There are two types of people who use the word "cloud". People who are in charge who have absolutely no idea how technology works, and those who make fun of those who use the word "cloud". Back in the day, we use to call the "cloud" the "network". Of course, the clueless now confuse the cloud with components of the cloud, such as clusters, SAN's, and mainframes
Re:Cloud? (Score:4, Insightful)
Re: (Score:2)
Unless that person happens to be your boss.
Re: (Score:2)
Re: (Score:2, Insightful)
I guess you've never heard of Applescript ;P
Re: (Score:1, Insightful)
Warning / Rant: The last 5 years of computing have been pretty lame. Concurrency and solutions to it using functional high-level languages are the future. That's where we should have been five years ago
But five years later, that's not where we are; the world moved into a different direction. What are you going to do? You can either cling to your ideas and refuse to give them up, or accept reality and try to improve what we've got now, instead of what should have been but isn't.
Example: Plan 9 and Inferno. Much better than Unix, but didn't go anywhere. Pity? Yes, but it's futile to still hope for a takeover that'll never come.
Re: (Score:2)
Tried uzbl [uzbl.org]? Also, you don't need any addon to make a Firefox addon. Greasemonkey simplifies the process, but it's not required.
Re: (Score:2)
Good grief, everything is a "cloud" now. Have some servers on a rack? Those aren't servers, that's a cloud! It's like some retards took a Cisco networking diagram, and went crazy when they realized that everything could be simplified into one of the "clouds"..
Question: I tried to put my boot into the cloud but it just went through. What do I do?
Answer: You need our new solid cloud state technology at an additional cost of $3M a year.
the perfect attack vector (Score:2)
Cloud related ranting apart, can you imagine a better attack vector than wireless booting? Just place a rouge access point in a crowded cafe or station, broadcast a boot image and infect all of the poor smucks that start-up their laptop in the area. You just need to patch their windows system and then start it, it will take 30 sec longer than the ususal 5min boot, they won't ever notice...
Works pretty well and scales well too (Score:4, Interesting)
Re: (Score:2)
iSCSI boot from a Linux SAN
If you call that a SAN, my calculator is a computer.
Re: (Score:2)
I took a look at the Dell Equallogic that we have at work, which had a nice pricetag, also runs Linux. So Linux as a SAN doesn't sound all that stupid. Now I wouldn't expect Dell Equallogic to be the top of the line, but they do have all the 'enterprise features'.
Re: (Score:2)
A few interesting things... (Score:5, Interesting)
The DHCPv6 netboot standard about to come out recommends http as the protocol of choice where tftp would have been used, but uses URLs so the protocol is selectable.
The iSCSI portion of this is a wider standard, implemented by many firmware configurations out of the box.
Finally, I'm going to plug xCAT as a tool to wrap dhcp, dns, ntp, active directory, gPXE, iSCSI, PXE, bootp/tftp, ipmi, blades, vmware, kvm, xen, LPARs, and more to deploy vmware, windows, linux, and aix systems and do hardware management. It mostly pays off at larger scale, but it is a project that aims to understand how to best utilize those various technologies.
EEPROMs (Score:2)
Re: (Score:2)
My workstation boots off a 128GB chunk of solid-state storage - I don't know what you're talking about.
But I still use PXE... (Score:1)
It works out quite well, when it works, and it's neat to do. I generally use this tutorial [hugi.to] to set up the TFTP server on my Windows machine and just change whatever release they talk about to whatever release is current.
It's fun, frankly.
Fedora has OpenSharedRoot (Score:2)
I've been using OpenSharedRoot over NFS at my workplace on three different clusters. It works pretty well. Ultimately, I would love to be able to install gigabit switches everywhere and give everyone disk-less workstations that PXE boot. Unlike a thin-client solution, the engineers would still be able to use their desktop's CPU resources and local RAM, plus the local HD could be used for swap and data files, but not the OS itself.
Boot over the Interwebs (Score:1)
http://boot.kernel.org/
http://boot.fedoraproject.org/
Although they are pretty slow at times, and I have had them freeze a few times (this may have been due to bad hardware though). It a pretty cool way to install a Linux distro.
Boot syncing your distro. (Score:1)
Not limited to small files (Score:2)
I am not sure where the idea that PXE boot files are limited to 32KB comes from, but we are booting FreeBSD 8.0 with a 240KB boot file with PXE and tftp and have not had to do anything special. We also boot Linux (Fedora 11) with a 4MB initrd over tftp and that has not posed any difficulties either. Our FreeBSD experience is documented at http://www.nber.org/sys-admin/FreeBSD-diskless.html [nber.org] - it works quite well for us. I looked at gPXE and it doesn't really solve any problems we have had. Actually, we have
Re: (Score:2)
I've PXE booted 9MB images for X terms. PXE grabs a tftp/UDP bootstrap program (only a few kB), and the rest is downloaded over TFTP. but it could have just as easily been grabbed over HTTP or NFS with a different bootstrap. (u-boot has some nice mini NFS client code capable of quickly grabbing files in a very small "kernel", easy to drop into other projects)
And we less computer-oriented nerds (Score:4, Funny)
USB Flash booting works fine (Score:2)
PXE is nice if you have an entire cluster to build. We use it and it can be great to get a disk image installed on a new cluster. But honestly, I've only seen two cases were PXE really works...small "toy" setups where somebody boots this system on the left side of the desk from an PXE/TFTP/NFS server on the right side of the desk. Or using it to clone/install large blocks of systems. (That is how ROCKS works.) But I don't know of anyone using PXE to boot a significant number of systems...workstations o