Seagate Firmware Update Bricks 500GB Barracudas 559
Voidsinger writes "The latest firmware updates to correct Seagate woes have created a new debacle. It seems from Seagate forums that there has yet to be a successful update of the 3500320AS models from SD15 to the new SD1A firmware. Add to that the updater updates the firmware of all drives of the same type at once, and you get a meltdown of RAID arrays, and people's backups if they were on the same type of drive. Drives are still flashable though, and Seagate has pulled the update for validation. While it would have been nice of them to validate the firmware beforehand, there is still a little hope that not everyone will lose all of their data."
Huh.... (Score:1, Interesting)
I didn't even know that people updated the firmware on their drives.
Pwnt. (Score:1, Interesting)
Mirrors are not a backup, and now a raid array isn't either. I'm gonna stick to printing my porn and storing it in a cargo container.
If Seagate keeps this up (Score:2, Interesting)
Re:If Seagate keeps this up (Score:0, Interesting)
Everything Maxtor (both good and bad) was thrown away in 2006. This is all Seagate.
As the owner of 4 of the 1 TB drives... (Score:4, Interesting)
Not even an acknowledgment that they have looked at my tickets. I got a "your ticket was created" email, and that is it.
Seagate is getting very close to losing a lot of customers.
Re:I have a solution for long term data storage. (Score:1, Interesting)
Although that was probably(?) meant as a joke, I'm wondering if that might not end up being the way things go. Think for a moment. CDs work by blistering aluminium foil with a laser. It's not perfect, but it works.
If you had a more stable structure and a little more oomph on the writing laser, it is quite possible that there are ceramics or metals you could etch with an information density every bit as good as a hard drive.
As you'd be altering the structure of something, rather than playing with very weak magnetic fields on a medium that doesn't hold them very well, the longevity should be every bit as good as that of the baked clay tablets found Mesopotamia.
Not that there's anything wrong with magnetic fields. Core memory is considered good for 100+ years under normal conditions and is still used today for extreme radiation environments. The magnetic field recorded in lava flows has lasted hundreds of millions of years.
Re:If You Can Reflash It, It's Not Bricked (Score:3, Interesting)
Because a brick can't be fixed to do anything better. The term as originally used is wonderfully descriptive - I think I first heard it used about the PSPBrick trojan, which really turns the PSP into a BRICK. Like, you can't do anything with it anymore. There isn't a thing in the world that you can do with your PSP, ever again, except keep a table from wobbling. I like having a term for that sort of hardware-disabling software problem, and I can't imaging there's anything as evocative as "brick" for the purpose.
Re:bad Seagate, bad! (Score:3, Interesting)
Meta-suggestion for eds (Score:5, Interesting)
Can we, for God's sake, just permanently ban the use of the word "brick" or "bricked" in the summaries. I have yet to see it used correctly.
Brett
THE FACTS (Score:5, Interesting)
I work for Seagate. I was there when the fit hit the shan, and I saw everything going in internally, as well as externally.
I really love my job, so please excuse the sock-puppet nature that creating a brand new account and claiming to be an authority on the subject I must seem to be. But I am a geek, and I really think you all need to know the true story behind the scenes.
This whole thing started with the 1.5 Terabyte drives. It had a stuttering issue, which at first we all thought was a simple bad implementation of SATA on common chipsets. Seagate engineers promptly jumped in and worked to try to duplicate the issue and prove where the problem was. This wasn't a massive rush as 1.5tb drives are what? 5% of the drives on the market. When it became obvious that the issue was more widespread, they buckled down and put out a couple of firmware revisions to fix it.
Now, in the 1.5tb drives, there are 2 main revisions. the the product line that gets the CC* firmware, and the line that gets the SD* firmware. They came out with firmware CC1H and SD1A to fix these issues and started issuing them.
But, seagate has always been restrictive of handing out their firmware, so such updates required calling in with your serial so that the people who had access to hand out the firmware could check a) model, b) part number, and c) current firmware just to make absolutely sure that they were giving the right firmware out. This has been a procedre that has worked for YEARS up until now.
Then the bricking issue came to their attention. It took so long because it's an issue that's hard to track down - pretty much the journal or log space in the firmware is written to if certain events occur. IF the drive is powered down when there are 320 entries in this journal or log, then when it is powered back up, the drive errors out on init and won't boot properly - to the point that it won't even report it's information to the BIOS.
This is a rare, but still obviously bad issue. Up until now, we all figured it was just some standard type of failure, as it was such a rare event, so we'd RMA the drives.
So, for whatever reason, mid management started freaking out (as it could be a liability for seagate, I suspect - ontop of the already potentially liable issue of the stuttering problem causing drives to fail in RAIDs). So, they pushed the release of the SD1A firmware to the general public. They took a few days to 'test', though it was mostly just including some code in the batch file that kicks off the firmware updater, to check that it is a BRINKS drive, and the proper model number. Then it was kicked out to the public.
Please understand, this firmware had to go through five different checks to make sure it applies to the specific conditions to qualify sending to a customer, before now. 5 chances for us to go your drive needs the other (or none) firmware update. Suddenly, it's down to ONE check, and even that was more designed for a contingency just incase the wrong firmware was sent out.
Of course, it starts bricking drives.
Right now, the engineers are crapping themselves, the firmware's been pulled, the support agents are told to say "The firmware will be released soon" and no real procedure to fix this issue is in place. Our phones are flooded so bad that it locks the system up when there are too many calls in queue, and emails are coming in at hundreds an hour.
We simply cannot keep up.
The good news is, the chance of your drive simply not spinning up one day is very low. And for those of you who flashed the wrong firmware - be patient. It's not bricked, just unable to write data to the platters properly. When they have a *GOOD* firmware out, a new flash should un-brick the drives. If not, flashing it back to SD15 should make it work again.
Seagate really pushes the idea of being open and honest as much as we can without being sued to hell. They let agents make choices and use their skills instead of scripting us to death. They worked hard to bring their support back t
Re:THE FACTS (Score:2, Interesting)
This isn't anything to do with SATA per se (Score:3, Interesting)
Re:THE FACTS (Score:3, Interesting)
Re:THE FACTS (Score:5, Interesting)
A thank-you! (and some questions) (Score:3, Interesting)
Maxtorman, I'd mod you up if I had the points. Your comments are the first ones to alleviate a very significant knot that formed in my stomach after reading this. I'm still a little concerned though, and have some questions at the bottom I hope you could answer.
I'm a little late to the party because I only use these only for non-critical stuff like home office and family PC's, but the prospect of having all my drives inevitably die really scares me. I've bought 18 drives (ST31000340AS and ST3500320AS all w/ FW SD15) in the last half of 2008 that sound like they match those reported to fail on the forum.
Funny enough I was complaining to my vendor about 4 drives that had to be replaced because they died within a month of use. Thought it was a bad batch they were pawning-off on customers, but I still trusted the Seagate brand.
So, my questions:
1) Is this definitely fixable in firmware? Should I be buying new drives right now?
2) What are the honest chances of a drive dying before a working firmware patch? My critical stuff is in RAID5 so I can always rebuild, but the gaming rig is RAID0, and off-site stuff like mom's media PC is only a single drive.
I appreciate your comments. Good to know there's a guru in the Slashdot community :)
-Matt
Re:At least no censoring (Score:3, Interesting)
Oh they are trying. Trying to bugger up systems! Surely if they validated the firmware update before releasing it the problems would have been caught in the QA process? I'd love to have been a fly on the wall in the QA meeting after the latest fix was released.
Re:A thank-you! (and some questions) (Score:2, Interesting)
I sincerely smile in relief every time I hear someone other then myself say that phrase. It's a sign of someone who truely 'gets it' as far as hardware is concerned.
Thing is, I know Seagate really does try to push for high manufacturing standards (for example, did you know that every last Refurb drive *must* go through the full new-drive qualification before it's sent out? - something only a percentage of actual new drives have to go through because it's time consuming).
But WD and Hitachi and most other HDD companies have only slightly worse failure rates, negligible difference in failures really in the grand scheme of things. No matter if you go with WD, Seagate, Hitachi, IBM.... whatever. If there isn't an outstanding issue, then you're really looking at about the same chance of failure no matter which you buy.
The real decision is more 'how easy will it be to replace this if it fails?'
Funny though, I have in my system a 250Gb WD drive, that used to be in an external enclosure, until the USB interface controller died and I ripped the drive out - and now, after about a year of use, the internal drive's starting to fail... I rolled the crapshoot on this drive it seems.
Re:Seagate's forum is on fire from this mess. (Score:3, Interesting)
Deleted already :(
Slashdotted ?
Re:THE FACTS (Score:5, Interesting)
As I've noted below, it was an emergency release that shouldn't have been, and was never designed for release to the general public.
They should have redesigned the delivery system, but there was too much public pressure on them to get a fox out *now*...
But then again, it was somewhat their own damn fault - if they had just came out an explained the details of the issue to everyone instead of keeping it in-house, people would have realized quickly it wasn't as dangerous a situation as it seems at first glance. Just inconvenient to the few who run into it more then anything. But the ambulance chasing lawyers smelled blood during the 1.5Tb issue and forced management into a hole.
Re:THE FACTS (Score:2, Interesting)
I've worked tech support for a while now, and that's something that's struck me working for seagate.
Just about everyone there actually.. well, -cares- about what they do. The atmosphere inside is completely different then say, dell or cox or AT&T (from what I hear - I've never worked at any of those places, but many of my coworkers have).
The management lets support agents have license to use their experience and skills in fixing issues, instead of reading from a script. They don't heavy-hand them, and they are more concerned in fixing an issue in as few of calls as possible, then caring about the number of calls taken a day or keeping the calls as short as possible. It means that tech support can actually *fix* the problem instead of trying to get people off the phones as fast as possible.
And thanks, always glad to hear the positive feedback.
Re:At least no censoring (Score:1, Interesting)
So far, there is no indication that they even have a QA process...
Indeed.
What the summary fails to mention is that the original package Seagate posted to their web site had a flash update utility that would SEGV before doing anything.
Good thing, too. Because otherwise I would have bricked five drives.
Yes, Seagate released firmware that bricks drives with a flash update utility that SEGVs.
That's epic fail. Truly epic.
Re:THE FACTS (Score:4, Interesting)
First, let me apologize, I'm gong to withhold employment details such as tenure and experience mostly due to the fact that many of us at Seagate (including some in management) are Slashdot regulars.
That said, I really do enjoy my time at Seagate, and it has been an absolutely wonderful company to work for.
As far as "BRINKS" "MOOSE" "GALAXY" etc.. are concerned, they are pretty much the internal development names of the drive family. There can be overlap, but most "BRINKS" drives are 7200.11, I believe, while "MOOSE" drives are almost all 7200.10, and "GALAXY" drives are 7200.9. Generally, those names don't make it out into public, but if you were to tear into the SD1A firmware, you'll notice that it looks for the "BRINKS" drive before it flashes the firmware to the drive. There can be different internal names for different revisions of the drive itself, but generaly they stick to one revision per family - a new internal name would only be used for a MAJOR revision on the drive.
I don't have my documentation handy, but I'll look that up later in the week and try to give you a better answer.
Finally, thank you for your kind comments.
Re:Fail (Score:3, Interesting)
Seagate never played Whack-a-mole growing up.
The game you played with a server full of seagate drives "growing up" is that if it was off long enough to cool down it was a virtual certainty that at least one of those drives wouldn't spin up. The odds of another disk developing stiction while you were taking the first one out of the case and whacking it with a screwdriver approached infinity as the number of disks became large enough to make the machine or enclosure heavy. Is that enough like whack-a-mole for you?
On the plus side, most of those drives had really huge filters in them. I had a 40MB RLL disk that I opened, de-stuck, and closed with nary a data error. I just did it in my bedroom (huh huh) with no dust or static control in the environment whatsoever. That drive eventually burned a trace right off of its control board, and then burned off the jumper wire I replaced it with, but that was over a year later and I'm quite sure one had nothing to do with the other.
I don't know how Seagate actually got its good name. I've used a lot of their disks, since way back when, and I just don't see it. I used to like Maxtor in spite of the noise, but WD has pretty much been my best friend all along. Today I will hardly buy anything else.
Comment removed (Score:3, Interesting)
Re:THE FACTS (Score:3, Interesting)
Hahaha. That's so true, but the whole point of middle management is to make bad engineering decisions for political (read: "stupid") reasons, because the people who know enough wouldn't, and the people above them think it'll save money to have a political layer (that is, a stupidity) like that inbetween. The people at the top can't make those decisions directly, because when they screw up, someone has to take the blame. So these people are pushed to make "the hard decisions", then get blamed for it when they turned out to be "the stupid decisions". So, execs keep their noses clean, and engineers get bypassed. Of course, if middle management has their way, they push the responsibility on down as far as they can, but that's irrelevant from the perspective of the execs who put the stupidity layer in place.
Sigh.
Re:If You Can Reflash It, It's Not Bricked (Score:2, Interesting)
Can we please stop saying this?
When a router is bricked (bricked is a layperson's term mind you, not a technical term!) it can nearly always be recovered through tftp in the first few ms of boot up (if it automatically listens in an automated failsafe recovery step like some Buffalo routers) or through a JTAG port. By your logic, a router is never bricked unless the NVRAM is fried, right? Wrong. It's bricked - just like iPhone users who jailbreak and then end up with dead iPhones. Sure, they can be recovered, but for the average joe, the thing is a "brick" good for little else than a paperweight or door stop (depending on mass) even though it can be fixed.
It's not a technical term by any means. It's slang for "ZOMG! IT DONE BROKE I DUNNO IF IT CAN BE FIXED!" then you find out it can be undone through a firmware update, even if by a JTAG port, then it's "I HAS SHINY FIXED! KTHXBAI!"
Now, let's get over the word "brick" and agree that its meaning is not necessarily "permanently broken" but its meaning is "non-working shiny which may or may not be reparable."
KTHXBAI!
Re:THE FACTS (Score:3, Interesting)
Re:Pwnt. (Score:3, Interesting)
I work in computer repair, and you would be surprised how many people lose all their data, despite having a backup. No single backup is a good idea. Multiple backups are ALWAYS a good idea. So here's what I do on a small budget with a ton of data:
NOTE: Backups need to be painless. If they're not, I'll never do them. Be honest with yourself. Making a note to sit down and burn DVDs every sunday will NEVER happen. It might happen a few times, but not *just* before you need it. Plan around convenience, for your data's sake.
1. Backup number 1 is basic. I have another desktop on my network, with a raid mirror and a network share. I run a backup regularly. Incremental backups to seperate files works nicely to conserve space. CHECK YOUR BACKUPS REGULARLY. Nothing's worse than realizing your backup has been bunk for 3 weeks, and then have a system go down.
2. My second backup is in case of major disasters that burn down my house or something similar. I use MOZY to do online backups. Now, with 350gb, it's not a very fast thing. But because it's incremental, eventually you get past the hump, and it only has bits and peices to do to keep up. A major point for me on online backups was security. My life is in this data, including SSNs, Pictures, Embarrassingly bad poetry, and so on, so I was a bit nervous about starting an online backup.
Mozy offers local encryption with a private key BEFORE uploading, although it's not neccessary. Mozy states in their terms, if you decide to go with the private key, and you lose that key- they can't help you. I like the sound of that.
If you don't use your own private encryption, they say they don't look at your data, but you know how companies fold to government wishes. I'm not hiding anything illegal, but I do want my privacy.
Mozy is $5/month. With my connection (768kbps up) it's taking months to finish the first upload. But after that's done, I'll have a bit more comfort, and for $5, it's not worth not doing. They allow you to throttle the bandwidth during different times of the day, so I have it upload during the night and while I'm at work, then have it stop while I'm home, so I have the internet connection all to myself!
*Note: I do not work for, nor am I affiliated with mozy.
Re:If You Can Reflash It, It's Not Bricked (Score:4, Interesting)
You're close, but bricked really just means "you can't fix it, nor can the average layperson". There is such a think as "unbricking".
For example, you might brick a motherboard by flashing it with some hacked BIOS you found on a tweak forum. If you're as dumb as the average forum troll, you're probably not clever, resourceful or brave enough to hotflash your socketed chip on a different board, but an experienced techie could do it.
There's also a pretty large market of "unbricking services", usually just some half-breed with a special cable he bought off of some other wannabe-crook on eBay. He'll reflash your PSP, cell phone or hacked FTA receiver for ten bucks, right from his ornate Honda Civic office.
There are very few cases where a "bricked" device is truly beyond repair by a skilled and equipped technician. If a gadget sells for $100, and your staff tech costs $50/hour, then as long as he can fix more than one unit every two hours (minus S&H and markdown), you fix the gadget. In practice, you end up seeing the same problems over and over, most of them very simple, so your tech might be able to fix 5+ per hour, and I'm being conservative here.
Throwing it in the trash is not a good idea, because if you don't try to fix the broken ones, someone else will buy your trash and do it behind your back. Then you have a bunch of poorly-repaired devices bearing your brand name, floating around generating forum posts and hate mail all over the web. The cost of junking returns can be greater than the cost of repairing them.
Re:I have a solution for long term data storage. (Score:4, Interesting)
Core memory is making a comeback (sort of) - http://en.wikipedia.org/wiki/Magnetoresistive_Random_Access_Memory [wikipedia.org]
Re:A victims point of view (Score:1, Interesting)
Which is all well and good.
Having said that, I first discovered that this was possible due last weekend's /. thread on this topic, and had I had a lot of fun tinkering with with an expendable drive.
If it's not under NDA, are there also serial commands to issue arbitrary ATA commands? I didn't see anything immediately obvious in the online help from the drive. (Also, again prefacing the question with "if it's not under NDA", is firmware-flashing-via-serial done with things like "DownloadGenericFile", or with variations on EnterBatchFile or RunBatchFile?)
Seriously, with your posts today you've cut a lot of support costs. Knowing that the root problem is actually understood is very cool; it means it's easier to wait a few days for a fix. Knowing that any bricked drives can be unbricked with a relatively simple (even though officially unsupportable) serial port hack was a huge breakthrough, as semi-shady overseas commercial data recovery "services" were aware of this workaround for weeks, but kept it under wraps in order to gouge customers $bignum for it. I'm glad the open community has finally figured out the secret.
Now that I know the problem's with the 320th entry in the logfile, I can use DisplayLogFile to see how long the thing is. If that's the logfile referred to with the Level L commands, maybe I can use CreateLogFile/DeleteLogFile/InitLogFile to zero it out if it's too long, or lengthen it to 320 entries to consistently reproduce the original misbehavior and then try to find a fix that doesn't involve unplugging the PCB from the drive.
Any hints you can provide (within the limits of NDA, and again, I'm experimenting on a drive whose data is expendable :) would be appreciated. Even if you can't provide any further hints beyond the output of the drive's "Q"uery (DisplayAsciiCmdInfo) online help, thanks again for kicking serious ass today. You've saved the support queues several unnecessary calls, and you've also helped restore trust in the brand. "It was a bug, the bug's known (here's the bug), the patch was rushed through, no drive affected by the bad patch needs to lose data (here's the workaround), no drive affected by the firmware bug has lost any data (here's the other workaround!)" isn't something that marketing can put a positive spin on, but you've given enough details about where each of those steps went wrong that IT people can appreciate.
Re:THE FACTS (Score:1, Interesting)
Actually, Moose and Brinks are both 7200.11 (the model numbers are different though), while Galaxy is 7200.10. 7200.12 is Pharaoh IIRC.
For example, all 7200.11 1TB drives with the model number ST31000333AS are Brinks drives, with 3 disks, while all ST31000340AS drives are Moose, with 4 disks. That having been said, as far as I know, all other times that the internal name has changed, the external name has as well - I.E. all 7200.10s are Galaxy, all 7200.12 are Pharaoh. I don't know any older than Galaxy though, so I can't be certain.