Tracking Down a Single-Bit RAM Error 277
Hanji writes "We have discussed here before the potential effects of and protections against cosmic ray radiation, but for the average computer user, it's an obscure threat that doesn't affect them in any real way. Well, here's a blog post that describes a strange segfault and, after extensive debugging, traces it down to a single bit flip, probably caused by a stray cosmic ray. Lots of helpful descriptions of Linux debugging techniques in this one, and a pretty clear demonstration that this can be a real problem. I know I'm never buying a desktop without ECC RAM ever again!" The author acknowledges that it might not have been a cosmic ray-based error, but the troubleshooting steps are interesting no matter what the cause.
Ugh, single bit errors (Score:3, Interesting)
One of my computers had an intermittent failure in a RAM chip/line/something somewhere that mostly manifested as SHA/MD5 failures when I was checksumming large files that I'd downloaded. Never showed up in Memtest86, but eventually I eliminated every other possibility. IIRC, I solved it by underclocking the machine and then replacing it when I was able.
Takes me back (Score:5, Interesting)
Easter Earthquake (Score:5, Interesting)
I don't know about cosmic rays, but immediately following the Easter day Earthquake in Guadalupe Victoria (about three hundred miles from where I was located) I tried to fire up my laptop and then my desktop, both of which had been suspended to RAM. Neither one would wake up, though the lappie displayed a garbled screen. No errors in the log files (Ubuntu 9.10 on the sys76 lappie, Deb Lenny on desktop).
RAM error? (Score:5, Interesting)
Forget a RAM error, I have seen a bit on a file on the disk flip.
After years of successful operation a Perl script quite working. On investigation a G was transformed to a W a difference of one bit. The file mod date was years old.
faulty RAM (Score:5, Interesting)
I've been working with some large microarray datasets recently, and so had to double my computer's memory to 8GB.
As I've done for years, I went to Fry's to get some Corsair chips... installed F13 64bit to replace my older 32bit distro... and crash-o-matic began. Mostly from Chrome and Mercurial.
I ran memtester86+ and sure enough, verified my first purchase of faulty memory.
So, I went back to Fry's and exchanged for another pair of Corsair 2GB chips. This time, I ran memtester86+ first thing... ANOTHER bad set, so back it sent to Fry's.
*Third* set of memory was Kingston, and a trip through memtester86+ verified no errors. Yay!
Computer has been stable, too.
With more and more RAM in computers, my next box will have ECC.
radioactive isotope in the chip (Score:4, Interesting)
Actually, I recall reading that in the early solid state memory days, they had problems with this. I don't remember what the solution was, but I thought it was to make the circuit somewhat resilient to it, as it was impossible to get 100% neutral epoxy, there's always going to be traces of something radioactive.
I think they tested the cosmic ray theory by running the same chip with and without lead shielding, and did not find a significant difference in errors, they then assumed it was impurities in the chips themselves decaying.
Re:It's not cosmic. It's from the die/package (Score:3, Interesting)
Re:RAM error? (Score:5, Interesting)
I experienced almost exactly that issue with a RAM error. My system was apparently stable, and then one day I got a syntax error in a system Perl script: one character had changed. The script was owned by root and otherwise untouched. After puzzling over it for quite a while I realized it could be a RAM error and ran memtest86. It reported a single permanently stuck bit in my 512MB of RAM. I found a kernel patch to manually mark problem RAM areas as reserved and kept on running with that RAM for a few years.
Are you sure that perl script issue was caused by a drive error? A RAM error can cause the same apparent problem, if the corruption happens in the kernel's cache. However, it shouldn't be permanent as it will not be written back to disk (the cache won't be dirty) unless someone actually modifies the file.
Old, old story (Score:5, Interesting)
Back in the early 80's, HP published a paper on random bit errors in RAM. They looked at chips from a variety of vendors and determined that the RAM coming out of Japan was the most reliable. That paper caused a lot of US RAM vendors to shutter their doors as there was a sea change in purchasing habits.
A few years later, I ran into John Scully while we were waiting for a flight. I mentioned the paper to him and asked him how Apple could seriously expect to sell a Macintosh specifically aimed at the Scientific community if it didn't have ECC. He blithely said "it's not a problem..." 20+ years hence and most of us still don't have ECC so it seems he was right.
Re:It's not cosmic. It's from the die/package (Score:3, Interesting)
Very old processed lead, such as that used for the roofs of some European cathedrals, has been used to build supercomputers since more of the radioactivity has decayed.
That sounds a bit fishy.
I _think_ I might be willing to believe the radioactivity of lead, presumably from contamination through some other source radioactive mineral in the ore that decays into radioactive lead. What I have a hard time believing though is that supercomputer makers wouldn't just use non-lead solder, which has been around for years and has actually been mandated for use in recent years in electronics.
Re:It's not cosmic. It's from the die/package (Score:1, Interesting)
And in lead-free solders, frequently the indium. Sure, it decays really, really, really slowly, but when you're looking at literally single particles potentially causing an issue, it's just another possible cause.
Had a MySQL problem once. (Once... ha.) (Score:2, Interesting)
I had a mysql replication server which was reading SQL commands from a binary log on a master server. One day after years of operation I noticed an update failed. I didn't see anything at first by looking at the query, but when I looked closely I noticed the query had a single character changed, and of that character only one bit had changed. It was something like a P becoming a Q and thus giving a syntax error.
True story.
Radioactive packaging (Score:3, Interesting)
The worst problem was with ceramic DIP packages -- the really good ones for when you needed reliability (partly because the plastic ones tended to allow moisture to get in, and then condensation on thermal cycling.) The standard ceramic packaging material contained trace amounts of thorium, which is an alpha emitter. The alpha bombardment was enough to flip bits.
There have been several fixes since then. Using materials that don't contain radioactive species was one. The one you're probably remembering is that the manufacturers apply a polymer coating to the surface of the die, which is enough to stop a lot of alpha particles and a fair number of electrons. Getting rid of lead in packaging is also good, because lead tends to contain some radioactive traces.
On the other hand, there's flat nothing to be done about cosmic rays and damn little to be done about X-rays and thermal noise (you do keep your memory cold, don't you? Thermal noise is proportional to KT/qe after all.) So at some point we get to where there are too many bits which need minimal energy to flip them -- and then you have errors.
Pity that so few mobos actually support ECC, though.
Re:It's not cosmic. It's from the die/package (Score:3, Interesting)
Maybe. It just sounds like an urban legend to me. I was also able to find a 25 year old patent claiming that gold-tin solder assured both high reliability in chip making.
http://www.google.com/patents/about?id=MZY1AAAAEBAJ&dq=4512950 [google.com]
Re:Ugh, single bit errors (Score:4, Interesting)
I think the original article showed why you'd want ECC in a desktop machine -- random bit errors do happen in real life. I don't see how a warranty makes this less of an issue -- if my machine silently corrupts data due to a bit error, getting a $50 replacement DIMM isn't really going to satisfy me. Does ECC really cost 5X over non-ECC?
If he was processing data or editing a spreadsheet, then that bit error could have corrupted his data. If he was compiling a program for distribution (perhaps to thousands of machines), that bit error could have corrupted his executable, causing errors on all of the machines it was deployed to.
After reading this article, the question that comes to mind is why am I *not* running ECC on my desktop?
Re:This would be important (Score:5, Interesting)
Billions of years in the ground, and only a few centuries on the roof and all of the radioactivity is gone! Wow!
The author needs to provide a reference, but there's a few ways I can think of that a processing stage, and a few centuries would produce something less radioactive than something produced more recently. I think all of them stem from the ore containing a source material that gets separated through the refining process, but the daughter products from the source don't. Here's one scenario:
Ore = Lead + radio-isotope a + radio-isotope B.
radio-isotope A decays to radio-isotope B
radio-isotope A: 4 billion year half-life.
radio-isotope B: 20 year half life, decays to stable isotope C.
during refining, radio isotope A gets nearly completely refined out to parts per trillion. radio isotope B is similar to lead chemically, and remains at 1 parts per million (at time of refining).
200 years go by. (10 half lives of radio isotope B)
radio isotope B is now at 1/2^10 concentration, or about 1 part per trillion. Significantly less than when it was first refined. The added radioactivity from radio-isotope A decaying into B is negligible due to the long half-life of A.
These numbers and process are obviously made up to show how it MIGHT work. It still remains to be seen if it's actually true or not.
Ksplice ... go figure (Score:5, Interesting)
The guy that posted this is a Ksplice developer. In case you didn't knew, KSplice allows you to patch your running kernel without rebooting. Nice.
Anyway, this guys sees a random memory error. He conveniently goes on a debugging rampage, while we all know the most logical first step would be rebooting that damn machine. Random memory errors do happen.
He says he "hasn't gotten around" to memtesting his RAM yet. So, let me get this straight ... he implies that random cosmic rays caused the error, but he hasn't yet tested his ram for what is the most possible cause of the issue?
Then he goes on to explain that you don't even need to reboot your machine due to damn cosmic radiation. Or kernel updates. Because you have Ksplice.
Come on.
Re:Cosmic rays, my ass. Occam's Razor time. (Score:5, Interesting)
On the subject of the imperfect nature of machines, I found this post by Richard D. James (aka Aphex Twin, a noted electronic music composer) quite interesting. He describes how the physical machinery of analog electronic music machines means it is near impossible to duplicate them in digital programs.
link [archive.org]
Author: analord
Date: 02-07-05 03:14
some people bought the analogue equipment when it was unfashionable and very cheap though.
some of us are over 30 you know!
anyone remember when 303`s were £50? and coke was 16p a tin? crisps 5p
also you have overlooked A LOT of other points because its not all about the overall frequency response of the recording system its how the sound gets there in the first place.
here are some things which you can`t get from a plugin,they are often emulated but due to their hugely complex nature are always pretty crass aproximations..
the sound of analogue equpiment including EQ, changes very noticably over even a few hours due to temperature changes within a circuit.
Anyone who has tried to make tracs on a few analogue synths and make them stay in tune can tell you this,you leave a trac running for a few hours come back and think Im sure I didnt fucking write that,I must be going mental!
this affects all the components in a synth/EQ in an almost infinte amount of tiny ways.
and the amount differs from circuit to circuit depending on the design.
the interaction of different channels and their respective signals with an analogue mixer are very complex,EQ,dynamics....
any fx, analogue or digital that are plugged into it all have their own special complex characteristics and all interact with each other differently and change depending on their routing.
Nobody that ive heard of has even begun to start emulating analogue mixer circuitry in software,just the aesthetics,it will come but im sure it will be a crap half hearted effort like most pretend synth plugins are.
they should be called PST synths, P for pretend not virtual.
Every piece of outboard gear has its own sound ,reverbs,modulation effects etc
real room reverb, this in itself companies have spent decades trying to emulate and not even got close in my opinion, even the best attempts like Quantec and EMT only scratch the surface.
analogue EQ is currently impossible in theory to be emulated digitally,quite intense maths shit involed in this if youre really that interested,you could look it up...good luck.
your soundcard will always make things sound like its come from THAT soundcard..they ALL impose their different sound characteristics onto whatever comes out of them they are far from being totally neutral devices.
all the components of a circuit like resistors and capacitors subtley differ from each other depending on their quality but even the most high quality milatary spec ones are never EXACTLY the same.
no two analogue synths can ever be built exactly the same,there are tiny human/automated errors in building the circuits,tweaking the trimpots for example which is usually done manually in a lot of analogue shit.
just compare the sound of 2 808 drum machines next to each other and you will see what I mean,you always thought an 808 was an 808 right?
same goes for 303`s they all sound subltey different,different voltage scaling of the oscillator is usually quite noticable.
VST plugins are restricted by a finite number of calculations per second these factors are WAY beyond their CURRENT capability.
Then there is the question of the physicallity of the instrument this affects the way a human will emotionally interact with it and therfore affect what they will actually do with it! often overlooked from the maths heads,this is probably the biggest factor I think.
for example the smell of analogue stuff as well as the look of it puts y
Re:Radioactive packaging (Score:1, Interesting)
It's also worth pointing out that it's not galactic cosmic rays (i.e. heavy ions) that are hitting the memory device. While traversing the miles of atmosphere, GCRs collide with atomic oxygen or hydrogen causing a particle / photon cascade. It's the neutral neutrons by-products that make it to the surface and in turn collide with atoms within the device. That final displaced atom is what will ionize a track through a memory cell transistor and cause the upset.
Re:Easter Earthquake (Score:3, Interesting)
Re:+1 Informative (Score:3, Interesting)
Been common with all kinds of things for some time. CD-ROMs, for example, use EFM, eight-to-fourteen modulation at their most base level. Eight logical bits are actually written as fourteen pits on the disc. Again the reason is error correction. They have more error correction at higher levels, and even more for data CDs. That's why data and audio CDs don't add up. An 80 minute CD holds 700MB of data. But do the math on 44.1kHz, 16-bit, 2-track audio and it takes 800MB of data to hold. Well in data mode the CD has additional ECC and that takes up the space.
As our media densities have increased, there is just no way to have 100% bit accurate recovery of data all the time. So, we just don't try. Instead, write it in a fashion that the errors can be dealt with.
For sure the PRML that HDDs use is the coolest and the most voodoo. It really seems like the kind of shit that should work. I mean taking samples (PCM style) of an analogue wave and comparing it to what it is likely to be? GTFO...
But it works, and works great.
There's all sorts of nifty shit like that at the hardware level that just fascinates me. Another one is 8b/10b encoding, which most popular buses (PCIe, 1394, etc) use. You take each byte and encode it as 10 bits, instead of 8. Why? Because it gives multiple different ways of encoding it. In particular, you have have different 1s and 0s patterns or on the wire, different high and low voltage. You then have the system balance those out, giving you no DC component to the signal. If you didn't, and you got a lot of 1s (high voltage) you'd have DC on the wire and that could cause all kinds of trouble. However 8b/10b solves that problem quite nicely. You get 0 DC and pass all your data. Does incur overhead though. There are newer schemes of a similar fashion coming online with more complexity to get the same effect with less overhead.
Re:Ugh, single bit errors (Score:3, Interesting)
One off single bit flips DO happen in otherwise perfectly good hardware.
I saw one years ago. A '386 running a single batch process (under DOS). It was supposed to be a massive sorting operation (500MB was a lot back then) and the results came out terribly scrambled. Each entry was fine except that they were not in order as they should have been. Since it was a batch I had the luxury of running it again. The error NEVER repeated. The same machine ran flawlessly for the rest of it's natural life after passing every test I could throw at it.
It could have been a cosmic ray, alpha decay or even a really unfortunately timed power spike.
ECC wouldn't have helped though, it had parity and that didn't detect anything, so the flipped bit was likely in the CPU itself.
Single bit errors (Score:3, Interesting)
This turned out to be a manufacturing problem with a particular run of RAM. I starting finding more of these bad parts and could reproduce the failures. I guess what I'm saying is that this could well be a manufacturing defect in the RAM.