Follow-up on EVE's Boot.ini Issue

Krinsath writes "CCP, publishers of Eve Online, have posted a Dev Blog detailing the circumstances leading up to the deletion of XP's boot.ini file, which was earlier discussed on Slashdot. The blog post has intimate details about how the mistake occurred (a new installer from their normal one), how they responded and what CCP has learned from it. While fairly dry, it is to the company's credit that they're being open about one of the more serious bugs to crop up in gaming's recent history."

That's actually a really straightforward response.

[Silverlancer (786390)]

Now if only more businesses acted this way.

Re:That's actually a really straightforward respon

[Harmonious Botch (921977)]

But they should delete greater percentages of XP...

Straightforward, sure.. but... | also, the bug

[Animaether (411575)]

what of the users who did lose valuable computer time due to this problem? The proverbial kid handing in their homework (or dissertation paper or whatever), for example. Apologizing and willing to pay for a third party tech support service (e.g. Geek Squad) is nice and all, but does that cover damages incurred? doubtful. Perhaps that EULA will finally get a test.

As for the bug itself... the installer code is NSIS script; quite powerful, but you do need to know what you're doing. Especially with a command such as "Delete", I can't help but wonder who failed to RTFM (TFM reads, as they point out, that "Delete" requires a full path to be safe or else it expects the path to be root) and instead made an -assumption- on how it would work.

Now, to their defense, NSIS is also a little inconsistent (RMDir needs /r to be recursive, but DeleteRegKey needs /ifempty to NOT be recursive; whatthe.) and I've wiped my entire root myself while developing an installer with it, although via a more complex bug.. NSIS simply doesn't have any built-in "you dumbass"-protection like most commercial installers.

Although I think it's nice of them to say that they're not blaming Windows for their own mistake, I do honestly think that Windows should protect such vital files at all cost - including against Administrator level process (e.g. a prompt "you dumbass - are you sure?" will do).

Re:Straightforward, sure.. but... | also, the bug

[TooMuchToDo (882796)]

what of the users who did lose valuable computer time due to this problem? The proverbial kid handing in their homework (or dissertation paper or whatever), for example. Apologizing and willing to pay for a third party tech support service (e.g. Geek Squad) is nice and all, but does that cover damages incurred? doubtful. Perhaps that EULA will finally get a test.

Almost never will damages be covered. Come to think of it, I think in this case I can say "Damages will never be covered." You have to show value and proof of destruction of that value. Your homework being destroyed? Your dissertation being destroyed? While it may have a large amount of value to you, monetarily it has very little value.

Re:

[Thomas M Hughes (463951)]

Almost never will damages be covered. Come to think of it, I think in this case I can say "Damages will never be covered." You have to show value and proof of destruction of that value. Your homework being destroyed? Your dissertation being destroyed? While it may have a large amount of value to you, monetarily it has very little value.

Lost homework is usually only about 1-3 weeks of lost work. Often less. Dissertations are a whole different beast.

A lost dissertation has a lot more value than sentimental value. You've spent X years of your life working on it, with the clear expectation that you have a high probability of getting a PhD. Having a PhD means getting a job that pays better than the pay of a graduate student. If graduate student pay is $Y, and reasonable post-doctoral pay is $Z, and you lost X years of work due to the bug

Re:

[Ost99 (101831)]

Even though the bug deleted a file, it's not possible to lose data due to this bug.
The only thing you stand to lose is a few minutes (or hours) of production time due to a non booting system.

Re:

[deathy_epl+ccs (896747)]

what if in those few hours (if you're not familiar with Windows at all and don't have a boot or rescue disk, etc. etc.) you needed access to those files? Do you file away every new e-mail to an external drive ready to be taken to the nearest computer cafe in case your machine goes wonky? I doubt it.

Do you expect to get paid by somebody every time your computer gets corrupted when you need to finish some vitally important work?

Also, the window was incredibly thin in which it took them to catch it... they have a couple hundred thousand players, and approximately 200 players got hit by it (I'm rounding, I know the number is larger) which indicates that the response time had to have been pretty quick, since the players were undoubtedly waiting with baited breath to download the patch as soon as the ser

Re:Straightforward, sure.. but... | also, the bug

[zippthorne (748122)]

what of the users who did lose valuable computer time due to this problem?

That's a good point. And generates some good advice for future student/gamers: Do not install any new software of any kind a week or two before a paper is due*

*at least, not without having some kind of back-up which can be read and worked on on another computer and which you regular test.

TFM reads, as they point out, that "Delete" requires a full path to be safe or else it expects the path to be root

That sounds like the the opposite of a good way for delete to fail.

Re:

[LarsG (31008)]

Do not install any new software of any kind a week or two before a paper is due

ITYM "Don't install an addictive game before a paper is due."

Re:

[Atlantis-Rising (857278)]

Although I think it's nice of them to say that they're not blaming Windows for their own mistake, I do honestly think that Windows should protect such vital files at all cost - including against Administrator level process (e.g. a prompt "you dumbass - are you sure?" will do).

Isn't the first thing most people do on Vista is turn off the Administrator "Are you sure?" prompts?

(I know that personally I do not- I don't get them more than a half-dozen times a day, if that, so it's really not that big a deal.)

Cancel or allow?

[Rix (54095)]

If you really want that sort of behaviour, why haven't you switched to Vista? That sort of behaviour is a big part of why hardly anyone is?

Re:

[Animaether (411575)]

I have Vista running on one of my machines, as I need to be able to test on it.

And no, I'm not referring to Vista's behavior of demanding Admin rights for a ton of things.. though I don't have a direct problem with that.. if somebody does, have them run as administrator.
I'm only referring to popping up a big fat warning when you are (or something else is) about to do something to a critical system file where the change could very well leave the machine unbootable without a boot or rescue disc. As long as t

Re:

[aussie_a (778472)]

Why weren't these people backing up their work (let alone definitely needing it when using an operating system well known for its bugs)?

Re:

[houghi (78078)]

Well, the people who sufferd will have learned some valuable lessons:
1) Do not install anything anymore once a system is running and you have critical data on it.
2) Backup
3) Be able to restore that backup

To me it is comparable to changing a tire. If you are not able to do that, you should not even be given a driving licence.

Re:

[cheater512 (783349)]

Its interesting to note that its impossible to do stuff like that accidentally under Linux. /boot/ is usually unmounted once its not needed and *nothing* should be touching anything in there anyway.

Oh well. Thats what Windows users get for using a really bad file layout.

oh yes it is...

[cheekyboy (598084)]

When cloning my linux from old hd to new hd, I accidentally forgot to change the label name back for the root= entry in grub. and couldnt boot unless I fixed it with the repair CD console login.

Its as fragile as boot.ini, though I personally have a few copies of it and /boot.copy /local/boot.old

What linux needs is like the old VMS days of auto versioning, grub.conf;1 etc...

If its a system core component config, then its smart to do it for that, HD space is plenty for text files.

Re:

[cheater512 (783349)]

a) You were moving partitions around, not installing a game patch. :P
b) Why didnt you simply press 'e' at the grub screen? You can edit how it boots from its self.

Re:

[Stevecrox (962208)]

You realise most people on Slashdot attack Vista for doing what you suggested.

Re:

[DeadChobi (740395)]

Actually I've found that EVE provides such a nuanced experience that it would be totally lost on anyone not older than 16. The game isn't the GAME, it's the little political games that we play within the rules of the game. For example, having Band of Brothers, the largest alliance in the game, put in an appearance when we're fighting a smaller alliance indicates that that smaller alliance is allied with BOB.

There are more games to be played against other players than just fleet battles. You can fight allian

Re:

[Ecuador (740021)]

Yeah, if only more businesses did not test products enough before deployment, or read TFM when using delete commands...

AND THEN they send GEEK SQUAD to "fix" your computer. Talk about adding insult to injury!

How is that even possible

[AndrewBuck (1120597)]

From the article...

"Why doesn't Windows protect its system startup files? That's a good question, one that I have asked myself in these last few days and wish I knew the answer. But of course I'm not going to blame Microsoft for our mistake. Windows doesn't protect those files and therefore software developers must take care not to touch them. We should have been more careful."

That is a good question. I am not an EVE player myself so I don't know if this update had to be run with admin privileges but it doesn't appear to be that way from the question and reply. If you are not running as admin then how is it even possible to remove a system file that is necessary to boot the system. Unlike the EVE representative making this statement I am going to blame Microsoft, it should not be the developers responsibility to make sure they don't break the OS, it is the OS developers responsibility to make sure that it cannot be broken without admin/system/root access.

-Buck

Re:How is that even possible

[Osty (16825)]

That is a good question. I am not an EVE player myself so I don't know if this update had to be run with admin privileges but it doesn't appear to be that way from the question and reply. If you are not running as admin then how is it even possible to remove a system file that is necessary to boot the system. Unlike the EVE representative making this statement I am going to blame Microsoft, it should not be the developers responsibility to make sure they don't break the OS, it is the OS developers responsibility to make sure that it cannot be broken without admin/system/root access.

Two things to note:

1. This was an XP problem. Technically it could've happened on Vista, but I haven't seen anything that said it did. As such, this falls into the same category of problems that Microsoft attempted to fix in Vista with UAC -- nearly everybody ran XP as admin, and many apps expected you to be running as admin.
2. This was a problem with an installer/uninstaller. Since nearly everything on Windows installs into %programfiles% and that's a shared location, installers need admin access (installers that ask if you want to install for "Just this user" or "Everyone" are not going to install in %userprofile% if you choose "Just this user". They're just looking to see if the Start Menu shortcuts should go into "%appdata%\Microsoft\Windows\Start Menu" or "%allusersprofile%\Microsoft\Windows\Start Menu"). Vista will elevate your privleges when you try to run an installer (you'll get a UAC prompt), after which a misbehaving installer could screw up boot.ini. Regardless of operating systems, you almost always install applications as administrator. Yes, you can install apps in $HOME on *nix systems, but 9 times out of 10 you'll use sudo on the installer (sudo apt-get install foo). Therefore this is technically a bug that could happen on any OS. It's not difficult to imagine an application install that deletes your kernel image, for example.

The real WTF here is that they have an important game file named "boot.ini". That's an exceedingly poor choice of filename. Think of it like having a game file called "autoexec.bat" or "vmlinuz" that actually has nothing to do with the DOS boot process or the Linux kernel. The only defense they give for that is "legacy".

Re:

[Doogie5526 (737968)]

This could indeed happen on any system. I saw a Perl script in Linux just today that said ${directory}/file. The directory variable was empty and it tried to write to /file. Fortunately, it didn't have permissions to do anything damaging. Hopefully it would have been better written if it did run with those permissions.

Didn't Quake have an autoexec.bat file as a startup script?

Re:How is that even possible

[RulerOf (975607)]

Didn't Quake have an autoexec.bat file as a startup script?

Quake 3, and I assume for 2 and 1, contained a file called "autoexec.cfg." I always thought it was aptly named, being a DOS veteran myself, because it contains game configs like default keybindings (e.g. bind w +move) and such that actually allow you to control the game in the first place, and it's always called during game startup. Very similar in function to the file that it is named after.

Re:

[mav[LAG] (31387)]

Technically it could've happened on Vista, but I haven't seen anything that said it did.

Well, that would require a group of people who have Vista installed.

Re:

[m4g02 (541882)]

Technically it could've happened on Vista

Wrong, Vista no longer use a boot.ini file, changes to the boot process can only be made by running bootsect.exe in a CMD window.

Re:

[LordLucless (582312)]

Really? You think your average user would pay attention to that? They know they're installing software, and they trust the source - unless they know what boot.ini and that it really is a bad idea to delete, chances are they'll just hit "Allow". That's one of the fundamental flaws of UAC as security. If the user trusts the installer, they're going to hit allow. If the user doesn't trust the installer, they wouldn't have run it anyway. UAC is good for things that try to stealth-install, or do things behind th

Re:

[m4g02 (541882)]

I don't understand your reply, he said UAC wouldn't stop it from happening but that Vista has a new boot system that no longer use a boot.ini file (changes are made with bootsect.exe). He is right and... I don't know what are you discussing. What's your point? You seem to be off-topic.

Re:

[LordLucless (582312)]

Sorry, you're right, I misread the wouldn't as would.

Re:

[Bellum Aeternus (891584)]

I did patch EVE as a non-privileged user on my XP Pro system and the problem didn't happen to me. It does seem to be that since most people basically have to run as an administrative account to make XP "work", CCP was able to damage the OS as they did.

This is a multi-part failure. One part Microsoft for making an OS that almost requires standard users to run a privileged account all the time to make basic applications work. One part CCP for developing software that damaged the underlying OS.

My only hope i

Re:

[Deviate_X (578495)]

"how is it even possible to remove a system file that is necessary to boot the system. Unlike the EVE representative making this statement I am going to blame Microsoft,"

The boot.ini file is actually protected. It is specifically marked as a System File, Read-Only and Hidden. This means that to modify the file you need to remove these attributes in a specific order first before you can modify or delete it, even if you are logged in as Administrator.

The only way to prevent software from doing bad things

Re:

[dangitman (862676)]

Why the hell does a game need a producer?

Uh, to produce it?

Re:

[jythie (914043)]

Exactly.

Having worked on games with and without a producer.... yeah, they earn their paycheck (and this is coming from one of the software engineers)

Re:

[Perseid (660451)]

That's one of the flaws in XP - chances are you can't get away with not being administrator. Some things won't install. A few things won't even run. Not being Admin will lead to an unpleasant experience.

It's nice when companies are honest

[_Shad0w_ (127912)]

If only more companies were so honest and straight forward when they cockup. It almost makes me feel like playing EVE again. CCP can consider themselves as being given a virtual karma bonus.

Although I can't help but wonder if the "honesty is the best policy" choice was because of their handling of the last PR cockup.

Re:

[soupforare (542403)]

They're trying to make up for lost karma in the seemingly unending employee-player scandals.
I'd probably give it another go, it's not really that bad for an online spreadshzzzzZZZZZZ

Weak!

[toadlife (301863)]

The Operation Flashpoint dedicated server bug that deleted every file of the drive that it was installed on makes this one look weak.

Wish I could find a link right now but I can't. It will quite a scene on the BI forums when that one came out.

Only half the answer we need

[Chuu (307073)]

While it's nice to know the technical reasons for problem, it still does not answer why they failed to take note of this problem when it was originally reported and discussed on the beta server's message board two days before the expansion hit the live servers.

It's in the eight post of this [eve-online.com] thread.

Alright!

[Cheezymadman (1083175)]

Finally, a benefit to Vista! Vista users like myself were 100% unaffected by this. It was awesome.

I cant help but wonder...

[Nezer (92629)]

I have never been into MMOs. I just didn't get them. However, in the last week things have changed and it's due, in part, to this bug.

You see, until this bug happened EVE was totally off my RADAR screen. When I read about the bug on /. last week I went to the companies website and found myself intrigued. Further discovery that they didn't charge $50 for the box on top of the monthly fee was also appealing. Further, I see client software for Macs and Linux. Intrigued I download the Mac client and create the trial account. Two days later I'm hooked and sending them my CC #.

If it hadn't been for this bug, I probably would have never bought their product! They say that any publicity is good publicity and I think this is true. Sure the SNAFU was pretty bad yet the product was still compelling enough to buy it despite a pretty bad QA miss. This latest response from the company will only help further get their name out there and is truly an opportunity to make lemonade from lemons.

boot.ini ? heh

[l3v1 (787564)]

Now if I could count how many hours of my life have I wasted because of disappeared boot.ini's, geez. Nice move, still.

That was a very, very good analysis of the problem

[Antique Geekmeister (740220)]

Having violated /. policies and actually Read The Fine Article, it was a good analysis. I wish more people would write their bug reports this well, and explain how they're going to address the problem.

I also wonder if they wouldn't benefit from a nice virtual environment system to do QA testing of new releases with? Capturing the full graphical behavior of an OS is difficult in virtual systems, due to the overhead of the virtualization itself, but it might be a lot cheaper than keeping a dozen different hardware configurations around.

Package Management

[Jessta (666101)]

When is MS Windows going to get proper package management?
These sort of problems are not something that should be occuring in a modern operating system.

Can you imagine how this happened?

[Daimanta (1140543)]

Programming guy 1: So we're finally done with coding everything.
Programming guy 2: Yeah finally.
*Programming guy 2 tries to make a joke*
Pr. guy 2: Hey pr. guy 1, look at this
Pr. guy 1: lol, you appended a del boot.ini
Pr. guy 2: Well, I'm going to take a coffee break
Pr. guy 1: Yeah, me too
Pr. guy 2: Wait, lets put a sticky-note on the board that we're done
Pr. guy 1: Sure
*Pr. guy 2 puts sticky on the notice board*
*both walk off*
*manager walks in*
*manager looks at the board*
Clueless manager 1: Nice, the work is finally done.
Cl. manager 1: Ahhh, I'm on a tight schedule. Lets send this file to the head programmer so he can compile everything.
*Tries to click close*
Cl. manager 1: What, changes have been made? Whatever, save.
Cl. manager 1: Ok, open outlook. Send. Done. Wow, I know this will be a spectacular release.
*Cl. manager walks of*

Installer "technology"

[kabdib (81955)]

I'm not aware of a single installer package on Windows that isn't a useless, complicated, badly-documented, bug-ridden piece of crap. The times when I've had to use (say) InstallShield, I've seriously thought about finding a new job. This stuff sucks *and* blows. (Don't get me started on USB support and BlueTooth. Oh my God. Don't even think about reading about that stuff: Once you crack open the docs and see the wavy tentacles, the squamous mouths, the eyes, the eyes, the eyes that . . . well, you'll never be quite the same again. T-tr-trust me on th-that).

Remember the happy days of "just copy" installs, which worked great on MacOs in the 90s? Upgrade to a new system? Just copy your "apps" folder over.

The question, "What kind of installer should our OS have?" is like asking, "Should we drink the red poison or the green one?" Just asking the question seals your doom.

Re:

[_Shad0w_ (127912)]

CCP write it as EVE on the website.

Re:

[B3ryllium (571199)]

I would, but I had a power outage that terminated my 4-week uptime. :)

Worst part of it was, I was baking pecan butter tarts at the time. In an electric oven. They don't make UPSes for that.

Re:

[m4g02 (541882)]

I run Vista, so this would never happen to me as it no longer use a boot.ini file... But I do have to reboot quite often because the #$% thing keeps crashing for no reason :P

How long is too long?

[DigitalReverend (901909)]

He was one of 215 users that was affected, it's possible they didn't even know the cause. All you have if someone whining that they waited "too long". How long is too long? 15 minutes, 1 hour, 8 hours a day, multiple days? The user's post on that forum is very subjective and some people are just plain impatient.

Re:

[coolGuyZak (844482)]

Further, who would admit they were wrong when they do get help? Online?