Forgot your password?
typodupeerror
Data Storage Networking IT Linux

Diskless Booting For the Modern Age 99

Posted by timothy
from the who-needs-moving-parts dept.
An anonymous reader writes "Ever wonder what happened to PXE? Intel's popular standard for diskless booting hasn't been updated since 1999, and has missed out on such revolutions as wireless Ethernet, cloud computing, and iSCSI. An open source project called Etherboot has been trying to drag PXE into the 21st century. One of their programmers explains how to set up diskless booting for your cloud, using copy-on-write to save space."
This discussion has been archived. No new comments can be posted.

Diskless Booting For the Modern Age

Comments Filter:
  • How is it slow? (Score:4, Interesting)

    by OrangeTide (124937) on Saturday May 08, 2010 @05:21AM (#32137272) Homepage Journal

    TFTP over UDP on a LAN, doesn't seem slow at all. It's stupid, but sufficient to bootstrap a small kernel to access the real meat of your OS. 1-10MB TFTP downloads over 100mbit is no big deal. You can't get good 1gbit performance (let alone 10gbit) out of the dumb drivers in a PXE boot ROM, but that's OK.

    • Re:How is it slow? (Score:4, Insightful)

      by emj (15659) on Saturday May 08, 2010 @05:29AM (#32137294) Homepage Journal
      Apparently PXE only allows a bootloader payload of 32KB, which makes it slow, and these days it wouldn't be a problem to fit a much more competent bootloader in flash on the mother board. So the question is why should I have to boot/download a 4MB Linux kernel to be able to use the 1Gbps drivers?
    • by improfane (855034) on Saturday May 08, 2010 @06:30AM (#32137428) Journal

      Is it possible to use multicast? The multicast could run continuously or on demand. Can you route multicast?

      It would be pretty impressive if one could attach 3 nodes to the network and they all boot up.

      It would be a Borg network. Everything you add to it would be immediately taken over!

      NODE (DHCP): Hi I'm a Dell workstat-
      BORG NODE #1 (x.x.3.43): We are borg
      NODE: ACK
      BORG NODE #2 (x.x.3.44): We wil add your likeness to our own
      NODE: RST RST RST
      BORG NODE #3 (x.x.3.40): Resistance is futile
      NODE: Downloading borg....
      BORG NODE #2: We will add your likeness to our own (as soon as you have downloaded. ...
      BORG NODE #4 (x.x.3.45): We are borg.

      • Network multi-booting is great to allow people to change OS's, but it's pretty complex to set up, and there is little knowledge of it.
        • by Junta (36770)

          xCAT at its core focuses on network boot (and for x86 portions, includes gPXE). It does Windows, Redhat, SuSE, VMware. It takes a lot of the tedium out at scale of network install, statless boot, and iscsi boot.

          • by Daengbo (523424)

            gPXE (the Etherboot project) is used in Fedora's BFO project [fedoraproject.org] (based on BKO [kernel.org]). If you have a local server, that is a cool project

            I'm sorry that PXE and Etherboot get no love around here. I've been using Etherboot since ~2000, when it was the common boot method for LTSP (and K12LTSP). EB and gPXE are awesome projects that show how much you can do with so little space to work with.

            When network booting several computers, I've found the boot to be much faster over the network than locally once the first computer

      • Multicast would not work well for something like this. The problem with multicast is it doesn't care if you dont get all the packets. So if your network looses one packet, your whole boot will fail and will have to wait for the next multicast start.
        • by improfane (855034)

          What it be possible for a multicast to work with unicast corrections?

          The receiving node must be aware when a packet is not received, if it is not at a packet level, perhaps at a more logical level like a block of data. Rather than wait for the next multicast, it explicitly requests a retransmission of that block to that node via unicast.

          I am of course assuming that only a few packets would get lost.

          Or perhaps another corrective multicast that streams corrected packets when enough people request them.

          MULTICA

          • Re:How is it slow? (Score:4, Informative)

            by Junta (36770) on Saturday May 08, 2010 @08:51AM (#32137874)

            Issues:
            -tftp multicast is inherently limited to smaller than 98MB images with sane MTU. The same block number wrapping in unicast can't work in multicast. When you want speedup the most, tftp multicast can't even work
            -multicast only buys you something if a large number of clients are acquiring the same payload at the same time. In a large scale 'cloud 'configuration, things are generally heterogenous enough to negate any such hypothetical benefit.
            -Most ethernet fabrics are either incapable or not configured for IGMP/MLDv2 snooping required to properly scope multicast resulting in all multicast traffic degrading to broadcast. This has very adverse results unless every entity on the network only cares about the transfer.

        • by Daengbo (523424)

          Back in the 90s, there used to be a proprietary-but-free scriptable boot system that used multicast (the name started with a "B"). You would boot each computer you wanted to do an installation on, set them to receive multicast, then once the computers were all connected to the server you had put DHCP, TFTP, and FTP on, the server would multicast an install image and a post-installation script.

          I seem to remember that this was before Ghost has multicast.

          • by mawhin (635345)
            Are you referring to BPBATCH (Free as in beer)? That became Rembo (Not free, but we got a forever unlimited site licence for some few thousand pounds...)? That is now IBM Tivoli Deployment Manager (Massively not free)? We've used it for ten years, and it's excellent.
            • by Daengbo (523424)

              Indeed. That was it. Wow. Thanks for that. I couldn't remember the name. Interesting that it went on from being a fairly small project to an IBM product.

              I guess that KA-Tools [ka-tools.imag.fr] became a popular replacement for BPBatch, but KA was still really immature back when I last used BPBatch. Now KA just seems abandoned, probably in favor of gPXE.

      • Re: (Score:3, Informative)

        by guruevi (827432)

        Yes, Apple has been able to do it for a couple of years now (since the PowerPC era).

        Basically a DHCP server says: I have a boot image
        Client1 says: thanks - starts downloading 0-10%
        Client2 comes on: starts downloading 10%-100%
        Client1 continues downloading 10%-100%
        Client1 boots up
        Client2 requests 0%-10%
        Client2 boots up

        I think your example involves P2P but unless your client also has the boot image and a mechanism to give it to others after it has booted (which could be potentially a security risk) it doesn't

        • by larkost (79011)

          The mechanism you describe is only used when imaging a computer, not durring the netboot phase that is often used to deliver the new image. It is really nice if you can run broadcast ASR, but like most multicast has some real limiations when crossing subnets. It is also a bit of a bear to set all the variables so that you get good speed without taking down a network. But once you get the combination for your network it really works out well.

    • Re: (Score:3, Informative)

      by Junta (36770)

      Well, one issue is boot payload is getting bigger and bigger. One distro has about 20MB of download that would be tftped in the default case. Windows uses tftp for a *lot* more.

      tftp has the following issues:
      -16-bit block indexes. Most firmware won't go above 1400 or so blocksize (with good reason, if they go higher and the network is set for jumbo frames, transfers will fail in many scenarios). This means a cap of about 98MB before you overflow the counter. Most tftp servers nowadays can deal with it

      • by evilviper (135110)

        Well, one issue is boot payload is getting bigger and bigger. One distro has about 20MB of download that would be tftped in the default case. Windows uses tftp for a *lot* more.

        This is nonsense. Nobody in their right mind loads up the boot image in the first stage. The first stage is to download syslinux/grub/etc via TFTP. From there, you have all the features included in the boot loader of your choice, and can ignore PXE limitations. Want support for HTTPS? Good idea! Go include it in your boot loade

        • by Junta (36770)

          The thing is, syslinux/grub/etc all make PXE calls to retrieve files, which is TFTP. gPXE as a firmware (or as an undi driver delivered via PXE) gives syslinux http protocol support.

          Since you mentioned IPv6, the DHCPv6 RFC is just about to come out, and specifies a url syntax for protocol flexibility, as well as recommending http as the protocol of choice (FTP kinda gets the shaft...).

    • by Cylix (55374)

      Well that would be because PXE is pretty ancient stuff.

      I have a gPXE deployment on my home network which screams like a banshee on roller blades.

      Gigabit speeds, transfers over http and a good deal more configuration options.

      With gPXE you can bootstrap a machine from Taiwan over the wan. The kicker is you still need a local agent to serve the tftpboot portion.

      Beyond speeds the main benefit I find with gPXE is the ability to now load balance the kickstart servers. Even dhcpd has some features built in to assi

  • PXE/DHCP/AoE (Score:1, Interesting)

    by Anonymous Coward

    I did a similar thing a while ago with an Ubuntu desktop image and ATA over Ethernet. Worked nicely. Didn't get round to COW though ...

        http://www.s-mart.net/aoe.txt

    • Re: (Score:1, Redundant)

      by MrEricSir (398214)

      There's always GOAT or even SOY.

    • by grantek (979387)

      And that gets to the point of answering the question "what ever happened to PXE?" - it got replaced by virtualised storage for booting virtual machines, running on a fat software hypervisor. It's still nifty though, and useful when you do it right.

  • Authentication (Score:3, Interesting)

    by Anonymous Coward on Saturday May 08, 2010 @06:15AM (#32137398)

    The one thing missing from PXE is authentication: A PXE system will accept any DHCP address and with it any boot server configuration. Without cryptographic boot image authentication, network security is the Achilles' heel of PXE.

    • Re:Authentication (Score:5, Informative)

      by rathaven (1253420) on Saturday May 08, 2010 @06:48AM (#32137500)
      It wasn't designed for it - PXE boots without authentication on the client so that the hardware gets the image thinly and then auth takes place when the OS is installed. It assumes control of the local LAN is in place and it is trusted. If you are looking for auth at this level you'd need to look at authentication to the switch or wireless on the network - pre-authentication using something like 802.1X. I'm not 100% clear but I believe gPXE has something that probably covers that in the docs as it has scripting capability pre-receiving DHCP addresses (at the level for wireless authentication and possibly 802.1X)...
      • Re: (Score:2, Interesting)

        by Anonymous Coward

        It's still an omission. If you use PXE for remote administration (instead of using it for completely diskless operation), then there is local data which can be compromised by a hostile PXE payload. How hard would it have been to verify a cryptographic signature against a public key stored in the BIOS configuration?

        • by rathaven (1253420)

          It's still an omission. If you use PXE for remote administration (instead of using it for completely diskless operation), then there is local data which can be compromised by a hostile PXE payload. How hard would it have been to verify a cryptographic signature against a public key stored in the BIOS configuration?

          Yes it was missed in PXE, however, based on the context of when PXE was developed I doubt we thought we needed it. The point of gPXE is that PXE hasn't been developed in line with changes in the computing arenas and that these omissions needed addressing.

          I've checked further and even though there is a level of authentication built into the command line it doesn't yet have enough development to support 802.1X in gPXE, however, WEP, WPA and WPA2 are now supported so the remote boot from wireless can be un

      • by Kjella (173770)

        So we're getting into cloak-and-dagger territory but wouldn't this be a simple way to have a malicious image installed on the client that'll relay logins back to you? If you're first redesigning it I'd make it so you can both verify cryptographic signatures and use SSL. It would be great if you can plug in your diskless machine into a presumed hostile network and know that it'll only connect to your designated boot server no matter what.

        • by rathaven (1253420)

          Yes it would and unfortunately its straightforward, however, there are lots of levels of mitigation in the infrastructure if network admins understand that it is there and use it - whether this is the case for the cloud is another matter.

          What we are saying in a number of these posts is quite simply - "I don't trust core services on the network I'm connecting to". The response is, "Don't connect to that network or allow that network to connect to you".

          Firstly - do you trust your router or DHCP server
    • by Burning1 (204959)

      If you permit rogue DHCP servers on your network, you have bigger problems than PXE.

      The correct solution is to block DHCP response packets from unauthorized ports at the switch level, and make it a policy to smash the knees of anyone who connects a 'home router' to your corporate network.

    • by evilviper (135110)

      The one thing missing from PXE is authentication: A PXE system will accept any DHCP address and with it any boot server configuration. Without cryptographic boot image authentication, network security is the Achilles' heel of PXE.

      If untrusted machines are being attached to the same network segment you are, you've got a hell of a lot of problems, netbooting or not...

  • i dont wonder...... (Score:3, Interesting)

    by Ruede (824831) on Saturday May 08, 2010 @06:24AM (#32137416)

    i am using pxe often.

    i have setup a few linux install "CDs" for network install, a few live CDs for an emergency OS. LTSP is using it too and a small intel atom box gets its kernel over tftp/pxe... the pxe provides the parameters for the nfsroot mount.
    old win2k netinstall for ppl without a RIS uses that system too :)
    the tftpd box that provides all that stuff is a small amd geode that is normally my router :)

    i often thought about making a sourceforge project out of it.... :)

    • I also use PXE. My home server (an old P3 laptop) runs from an USB drive, but since the laptop doesn't support USB booting, I use PXE to boot it.

      The annoying part is that I can't reboot it remotely, as it won't have a TFTP server to feed the Linux image,

    • by h00manist (800926)

      i often thought about making a sourceforge project out of it.... :)

      I have often searched for how to set that up.

    • http://www.netboot.me/ [netboot.me] + PXE = ultimate network config.

      After setting up my own PXE setup, I put the settings in my DD-WRT. After completely hosing the only computer in my house (I left my laptop at work). I would have been SOL (no floppies, no CD's) Not so with Netboot.Me. Fired up a Debian recovery disk. The have recovery disks, minimal install disks, partition disks, FreeBSD, Linux.

  • by langelgjm (860756) on Saturday May 08, 2010 @06:27AM (#32137418) Journal
    I still use PXE to boot a diskless MythTV client. For a while I had the machine connected to a wireless router set up in bridge mode, so the machine effectively netbooted wirelessly.
  • Cloud? (Score:5, Funny)

    by plasticsquirrel (637166) on Saturday May 08, 2010 @06:38AM (#32137466)
    Good grief, everything is a "cloud" now. Have some servers on a rack? Those aren't servers, that's a cloud! It's like some retards took a Cisco networking diagram, and went crazy when they realized that everything could be simplified into one of the "clouds".

    Warning / Rant: The last 5 years of computing have been pretty lame. Concurrency and solutions to it using functional high-level languages are the future. That's where we should have been five years ago when it was so obvious that chips with large numbers of cores were the future. These cloud solutions are just a stupid name for the same old monolithic crap. It doesn't scale and isn't modular in a Unixy way. Modern applications just suck because they're so inflexible. Why can I do so many things from a little text terminal, but I can't easily script the behavior of my web browser without special add-ons? Why aren't modern applications flexible like this, with simple interfaces for communicating with other programs? Where is the equivalent of a shell pipe, in modern applications? It's like somebody threw away all the lessons of the past, and said "But this is the new way, we don't need the old way, because this is new." Fuck that, computing should be better than this. It should be better than these stupid clouds and old piece-of-shit reinvent-the-wheel C and C++ programs with buffer overflows and other ancient problems. Or the HTML / Javascript / whatever jerry-rigged "web applications" that run on some opaque "cloud" that a random company has. Why is it that languages like Smalltalk and Lisp have been around for so long, and nobody learns from them or uses them? It's like the chips keep getting faster and faster, and people keep getting dumber and dumber.
    • Re:Cloud? (Score:4, Informative)

      by fauxhemian (1281852) on Saturday May 08, 2010 @07:01AM (#32137546)

      It doesn't scale and isn't modular in a Unixy way. Modern applications just suck because they're so inflexible. Why can I do so many things from a little text terminal, but I can't easily script the behavior of my web browser without special add-ons?

      http://en.wikipedia.org/wiki/Uzbl [wikipedia.org]

      • Re: (Score:2, Informative)

        by Anonymous Coward

        http://www.uzbl.org/faq.php

        Cool, so I can install uzbl and have a "Unixy" browser that does next to nothing without a headache and a weekend of tinkering ... or I can install Firefox and actually get shit done. Sounds about right.

        • http://www.uzbl.org/faq.php

          Cool, so I can install uzbl and have a "Unixy" browser that does next to nothing without a headache and a weekend of tinkering ... or I can install Firefox and actually get shit done. Sounds about right.

          Cool, so I could Firefox and actually get shit done, or I could install uzbl and have a "Unixy" browser that does next to nothing without a headache and a weekend of tinkering.

          FTFY (this is /., after all...) *goes away to play with uzbl*

    • by sznupi (719324)

      It's like the chips keep getting faster and faster, and people keep getting dumber and dumber.

      I believe somebody said once "the amount of intelligence on Earth remains constant"...

      • by h00manist (800926)

        I believe somebody said once "the amount of intelligence on Earth remains constant"...

        I wager it goes down during wars, dictatorships, televised games, and elections. Orders are to destroy anyone who is nonconforming.

      • by sea4ever (1628181)

        It's like the chips keep getting faster and faster, and people keep getting dumber and dumber.

        I believe somebody said once "the amount of intelligence on Earth remains constant"...

        while the amount of people it is spread between increases exponentially?

    • Re: (Score:2, Interesting)

      by warrior389 (314070)

      My grandfather was telling me about how he used to walk inside of an old cray with 5 processors like 50 years ago. What happened to the people who used to know how to program those things efficiently? Apparently multi-core programming isn't so new.

      • by vbraga (228124)

        For sure it's not new. It's wide availability that's new. And algorithms and solutions using it for desktopish requirements were almost non existent.

        Cray then also used vectored processing now available on desktops as GPUs.

        Ways to develop for these technologies for sure exists but had another set of requirements in mind when it were made. A large number of programmers grew up without being able to play with it. It will take some time while people learn it.

      • When you're paying a few million for a Cray, paying top salaries for programmers who can take advantage of it is the only sensible way of getting a return on that investment. When you're paying $500 for a multicore server, paying top salaries costs a lot more than just upgrading to a $5000 server.
        • by h00manist (800926)

          When you're paying a few million for a Cray, paying top salaries for programmers who can take advantage of it is the only sensible way of getting a return on that investment. When you're paying $500 for a multicore server, paying top salaries costs a lot more than just upgrading to a $5000 server.

          $1200 for a multi core server with three Chinese indentured slave programmers, service for ten years. China software-factory sweat shop escape-and-rebellion insurance and translation services not included.

      • by Nursie (632944)

        What the hell do you mean "Apparently"?

        I was doing commercial multi-threaded programming a decade ago, and it wasn't new then. Multi-process across multiple cores has been with us for decades.

        Parallel programming is not massively difficult, but here on Slashdot people seem to think it's some crazy new thing that's really hard and nobody's ever done it before.

        No it isn't and yes they have.

    • Re: (Score:2, Offtopic)

      by antirelic (1030688)

      There are two types of people who use the word "cloud". People who are in charge who have absolutely no idea how technology works, and those who make fun of those who use the word "cloud". Back in the day, we use to call the "cloud" the "network". Of course, the clueless now confuse the cloud with components of the cloud, such as clusters, SAN's, and mainframes

    • Re: (Score:2, Insightful)

      by fredrickleo (711335)

      I guess you've never heard of Applescript ;P

    • Re: (Score:1, Insightful)

      by Anonymous Coward

      Warning / Rant: The last 5 years of computing have been pretty lame. Concurrency and solutions to it using functional high-level languages are the future. That's where we should have been five years ago

      But five years later, that's not where we are; the world moved into a different direction. What are you going to do? You can either cling to your ideas and refuse to give them up, or accept reality and try to improve what we've got now, instead of what should have been but isn't.

      Example: Plan 9 and Inferno. Much better than Unix, but didn't go anywhere. Pity? Yes, but it's futile to still hope for a takeover that'll never come.

    • Why can I do so many things from a little text terminal, but I can't easily script the behavior of my web browser without special add-ons?

      Tried uzbl [uzbl.org]? Also, you don't need any addon to make a Firefox addon. Greasemonkey simplifies the process, but it's not required.

    • by syousef (465911)

      Good grief, everything is a "cloud" now. Have some servers on a rack? Those aren't servers, that's a cloud! It's like some retards took a Cisco networking diagram, and went crazy when they realized that everything could be simplified into one of the "clouds"..

      Question: I tried to put my boot into the cloud but it just went through. What do I do?
      Answer: You need our new solid cloud state technology at an additional cost of $3M a year.

    • Cloud related ranting apart, can you imagine a better attack vector than wireless booting? Just place a rouge access point in a crowded cafe or station, broadcast a boot image and infect all of the poor smucks that start-up their laptop in the area. You just need to patch their windows system and then start it, it will take 30 sec longer than the ususal 5min boot, they won't ever notice...

  • by pdbaby (609052) on Saturday May 08, 2010 @07:56AM (#32137686)
    We do this at work - we chain-load gPXE using PXE and then use that to iSCSI boot from a Linux SAN which uses LVM COW snapshots. It's pretty good - the etherboot project rocks! We've been doing it for a while but it always gives me a kick when I type something at the commandline which wakes up a machine using IPMI & then boots it off some SAN volume
    • iSCSI boot from a Linux SAN

      If you call that a SAN, my calculator is a computer.

      • by Lennie (16154)

        I took a look at the Dell Equallogic that we have at work, which had a nice pricetag, also runs Linux. So Linux as a SAN doesn't sound all that stupid. Now I wouldn't expect Dell Equallogic to be the top of the line, but they do have all the 'enterprise features'.

      • by pdbaby (609052)
        We get a much better bang for our buck by using commodity hardware - we can build & maintain a few linux SANs for the price of an 'enterprise' one. We just build software which can deal with the underlying hardware failing or the network splitting.
  • by Junta (36770) on Saturday May 08, 2010 @09:04AM (#32137938)

    The DHCPv6 netboot standard about to come out recommends http as the protocol of choice where tftp would have been used, but uses URLs so the protocol is selectable.

    The iSCSI portion of this is a wider standard, implemented by many firmware configurations out of the box.

    Finally, I'm going to plug xCAT as a tool to wrap dhcp, dns, ntp, active directory, gPXE, iSCSI, PXE, bootp/tftp, ipmi, blades, vmware, kvm, xen, LPARs, and more to deploy vmware, windows, linux, and aix systems and do hardware management. It mostly pays off at larger scale, but it is a project that aims to understand how to best utilize those various technologies.

  • The only diskless OS booting I'm interested in and waiting for is booting from EEPROM. Near instant boot times. Why has this never really taken off?
    • My workstation boots off a 128GB chunk of solid-state storage - I don't know what you're talking about.

  • I have a laptop which cannot boot from USB and whose CD-ROM doesn't work. It's good enough for a Linux install, so I've taken to installing Ubuntu via PXE.

    It works out quite well, when it works, and it's neat to do. I generally use this tutorial [hugi.to] to set up the TFTP server on my Windows machine and just change whatever release they talk about to whatever release is current.

    It's fun, frankly.
  • I've been using OpenSharedRoot over NFS at my workplace on three different clusters. It works pretty well. Ultimately, I would love to be able to install gigabit switches everywhere and give everyone disk-less workstations that PXE boot. Unlike a thin-client solution, the engineers would still be able to use their desktop's CPU resources and local RAM, plus the local HD could be used for swap and data files, but not the OS itself.

  • I thought these were pretty interesting ideas:

    http://boot.kernel.org/

    http://boot.fedoraproject.org/

    Although they are pretty slow at times, and I have had them freeze a few times (this may have been due to bad hardware though). It a pretty cool way to install a Linux distro.
  • Of topic, but instead of "updates available, download and update". Just reboot your system it syncs with the latest distro of choice. Should be nice, is there a solution already?
  • I am not sure where the idea that PXE boot files are limited to 32KB comes from, but we are booting FreeBSD 8.0 with a 240KB boot file with PXE and tftp and have not had to do anything special. We also boot Linux (Fedora 11) with a 4MB initrd over tftp and that has not posed any difficulties either. Our FreeBSD experience is documented at http://www.nber.org/sys-admin/FreeBSD-diskless.html [nber.org] - it works quite well for us. I looked at gPXE and it doesn't really solve any problems we have had. Actually, we have

    • I've PXE booted 9MB images for X terms. PXE grabs a tftp/UDP bootstrap program (only a few kB), and the rest is downloaded over TFTP. but it could have just as easily been grabbed over HTTP or NFS with a different bootstrap. (u-boot has some nice mini NFS client code capable of quickly grabbing files in a very small "kernel", easy to drop into other projects)

  • by Wilson_6500 (896824) on Saturday May 08, 2010 @04:12PM (#32141098)
    naturally laugh at the mere suggestion of dickless booting.
  • PXE is nice if you have an entire cluster to build. We use it and it can be great to get a disk image installed on a new cluster. But honestly, I've only seen two cases were PXE really works...small "toy" setups where somebody boots this system on the left side of the desk from an PXE/TFTP/NFS server on the right side of the desk. Or using it to clone/install large blocks of systems. (That is how ROCKS works.) But I don't know of anyone using PXE to boot a significant number of systems...workstations o

"I got everybody to pay up front...then I blew up their planet." "Now why didn't I think of that?" -- Post Bros. Comics

Working...