Secret Service Runs At "Six Sixes" Availability 248
PCM2 writes "ABC News is reporting that the US Secret Service is in dire need of server upgrades. 'Currently, 42 mission-oriented applications run on a 1980s IBM mainframe with a 68 percent performance reliability rating,' says one leaked memo. That finding was the result of an NSA study commissioned by the Secret Service to evaluate the severity of their computer problems. Curiously, upgrades to the Service's computers are being championed by Senator Joe Lieberman of Connecticut, who says he's had 'concern for a while' about the issue."
Wow. (Score:5, Funny)
Re: (Score:2)
and get rid of the decimal.
What, don't you realize they could have 999% reliability?? This could be revolutionary!! Think what you are saying, you nearly threw that away.
Re: (Score:2)
Obligatory (Score:2)
That is silly, it would quickly discover the only way to win is not to play.
Re: (Score:2)
Upgrade... (Score:5, Funny)
To windows, and get 73% uptime!
Or.. that other OS that you don't have to license per seat, and get in the solid 90+% uptime.
Re: (Score:2)
If you can't manage two nines on a basic windows server, you're doing it wrong. If your service depends on a single server, you're still doing it wrong.
Lastly, is a performance reliability rating the same thing as uptime? I doubt it. If their server is down eight hours a day, they'd swap it immediately.
Re: (Score:3, Insightful)
As soon as they fill out all of the paperwork, and find a way to blame the downtime on someone with we don't like.
Re: (Score:2)
I'll pre-whoosh myself here:
*whoosh*
There, with that out of the way. Actually, it's probably going to take until they can resurrect their last COBOL programmer or find someone who they can train on a thirty year old system in less than a year.
Re:Upgrade... (Score:5, Funny)
666 666 -> Devilishly clever redundancy.
Re:Upgrade... (Score:4, Informative)
if you can't manage 2 nines on a an IBM mainframe your doing it wrong to begin with what makes you think they can do with something vastly more complicated as a massive windows deployment.
Re: (Score:2)
I made that point in my second sentence. I can't imagine a mainframe with an active support contract has less than 99% uptime. I'm pretty sure that "performance reliability rating" is not a euphemism for "service level availability."
Re: (Score:2)
Re: (Score:3, Informative)
Re: (Score:2)
An AS400 is a mid-range, not a mainframe...
Re: (Score:2)
You always expect the AS400 to be a bit more impressive the way the AS400 guys would talk about it. Then one day I walked into the room where it was kept and had to stifle a laugh.
Re: (Score:2)
Two nines? What, 9.9% ?
Even a monkey could manage that.
Re: (Score:3, Insightful)
Re: (Score:2)
90%? (Score:3, Insightful)
Re: (Score:2)
Sampling bias (Score:5, Informative)
Doesn't this constitute a sampling bias? (from netcraft)
Why do you not report uptimes for Linux 2.6 or FreeBSD 6 ?
We only report uptimes for systems where the operating system's timer runs at 100Hz or less. Because the TCP code only uses the low 32 bits of the timer, if the timer runs at say 1000Hz, the value wraps around every 49.7 days (whereas at 100Hz it wraps after 497 days). As there are large numbers of systems which have a higher uptime than this, it is not possible to report accurate uptimes for these systems.
The Linux kernel switched to a higher internal timer rate at kernel version 2.5.26. Linux 2.4 used a rate of 100Hz. Linux 2.6 used a timer at 1000Hz (some architectures were using 1000Hz before this), until the default was changed back to 250Hz in May 2006. (An explanation of the HZ setting in Linux.)
FreeBSD versions 4 and 5 used a 100Hz timer, but FreeBSD 6 has moved to a customisable timer with a default setting of 1000Hz.
So unfortunately this means that we cannot give reliable uptime figures for many Linux and FreeBSD servers.
Re: (Score:2)
Modded -1 Lier
Re: (Score:2)
Re: (Score:2, Informative)
Half of those show as IIS 5.0/Windows 2000. There is no way that a Windows 2000 box has stayed online for 1700 days (over 5 years!!!) without being pwned and crashed. For large sites that do load balancing and such, Netcraft is a better indicator of SITE uptime instead of uptime for a single particular box.
Here's An Idea ... (Score:5, Funny)
Re: (Score:3, Insightful)
I could point out that the cost of replacing this mainframe would mostly involve rewriting its applications to run on modern hardware. But then you'd be deprived of your joke, even if it is a pretty lame one.
Re:Here's An Idea ... (Score:5, Informative)
Re: (Score:2)
AC isn't lying, I'll back it up. Even if you wrote your stuff on an old System32, you can run it on most any IBM machine today.
Re: (Score:3, Interesting)
Should have RTFA. If we're talking an IBM mainframe, then you're certainly right. I cut my programming teeth on those 40 years ago (and haven't been near one in 35), and the basic architecure is still around, though many details have changed. In fact, I've long been convinced that this kind of need for backward compatibility is the only reason people still buy mainframes. Though there are those who are convinced they're fundamentally kewler.
The problem is probably as much political as bureaucratic, if not m
Re: (Score:2)
But then you'd be deprived of your joke
Are you sure? I assumed that the joke was that he was posting the ad on Slashdot because he didn't like Craigslist because anyone can post anything there (unlike Slashdot, where anyone can also post anything there too).
But then I guess it's not funny if you have to explain it.
Re: (Score:2)
Re: (Score:2)
Good point. I think even the oldest IBM mainframes are still around in emulator form.
Re:Here's An Idea ... (Score:5, Informative)
Actually, as AC already pointed out, the idea that you'd need to rewrite anything is incorrect. One could for added speed, but the IBM mainframe line runs the code for every IBM mainframe for the last few decades without changes. There are reasons people buy them, you know.
Re: (Score:2)
Re: (Score:2)
Of course I got modded down, but it doesn't matter. Virtue is its own reward!
Re: (Score:3, Informative)
Re: (Score:3, Interesting)
You can get a 486 (DX2 66 MHz?) with 40 MB ram and eventually four harddrives, 3com 509b NIC and I think two disk drives if you come pick it up here.
I live in Sweden.
Re: (Score:2)
Re: (Score:2)
Why do people bother using 'old' with P4 and especially a 286, 386, C64? Is is that there are 'new' ones that you can just buy on the market? Seems redundant.
I'm pretty sure the older you are the more often you use 'old' to describe things (I am, in fact, considered a 'dinosaur' in programming parlance). That being said, yes [newegg.com] you can buy them new if you were so inclined.
Re: (Score:2)
P4s? Pah, new kid on the block. Here, have a 6502. [jameco.com] Only $6!
Re: (Score:3, Funny)
We're on slashdot, right? Emulators? Think about it. We should take a poll to see how many members keep an emulated woman in their bottom dresser drawer, under the socks, where they don't think Mommy will find it. I hear that some people actually prefer fake over real.
Re: (Score:2)
Slowpoke.
My most reliable server is 3x Intel P3 Xeon 700MHz (and one empty slot) and 3GB (16GB max) PC100 RAM. It has triple redundant power supplies and 4 hard drives. It runs Win2k3 AD (backup DC), uTorrent and VmWare Server with two VMs running - Win2k3 and Win7 (so I can see the UI if I need to walk someone trough some settings, the UI is a bit different from XP version; also to try to make Win7 look like Win2k). It has enough speed for now, if it gets too slow, I can add one more CPU.
Re: (Score:2)
440BX was a brick shithouse of a chipset.
fully operational doesn't mean what it sounds like (Score:5, Interesting)
Mainframes of yore had a hell of a lot of moving parts: a large system might have dozens of tape drives and disk drives. Tape drives in particular broke down all the time and were taken offline until the maintenance guy came for his weekly or monthly visit and tightened the belts or whatever the hell they did. Knuth remarked on that situation in his magnum opus TAOCP vol 3 on sorting and searching. In the part about sorting with tape drives, he remarked that he'd never seen a large computer installation where all the tape drives were working. You'd have a computer with ten tape drives, two of them would be down pending repairs, and you'd use the other eight. In other words your computer was operational but not FULLY operational.
There is a similar situation in today's data centers. Even at the wimpy little shop I worked in last year (about 2000 computers) some were always down. We were doing pretty good if the number down at any moment was less than a few dozen. I don't think we ever had a single day of being fully operational (every single computer up at the same time). That was fine, it wasn't a requirement, it was a distributed system and the data and functions were all sufficiently replicated that we kept running, by design, even with parts of the system unavailable.
They are just going after some impressive numbers (Score:2)
I mean why settle for five nines when you can have... NINE FIVES! :D
Color me skeptical (Score:5, Insightful)
There's something about this whole thing that simply doesn't ring true. I believe parts, I believe they have a 1980's main frame, I believe it's not terribly reliable but something about the whole: leaked memo according to Joe Leiberman, we need more money, they won't give us more money' spiel sounds off. I suspect they have huge chunks of computing that's much newer and reliable, I'd be shocked if that IBM serves any significant purpose.
If nothing else I predict a large percentage of the umpteen million dollar final cost somehow going to Connecticut, but I'm probably just incredibly jaded.
Connecticut already gets billions (Score:3, Insightful)
If nothing else I predict a large percentage of the umpteen million dollar final cost somehow going to Connecticut, but I'm probably just incredibly jaded.
What's a few million? Connecticut is one of the top haulers, thanks to Electric Boat, where many nuclear subs (and a number of other ships) are made.
Every time the Pentagon tries to cut its budget, congrescritters get all up in arms about "jobs", so the Pentagon has all these useless projects (congress forces the programs it wants.) It's the primar
Hokey Illustration (Score:2)
1980's mainframes did not use reel-to-reel tape. They used tape cartridges, often managed by automatic tape libraries.
Re: (Score:2)
The picture looks an awful lot like late 60's drives for an IBM 360 model 44.
I guess they needed a picture that screamed 'main frame'.
Re: (Score:2)
TG Daily claims [tgdaily.com] that the Secret Service uses a IBM 704 [wikipedia.org]
Re: (Score:3, Informative)
Re: (Score:2)
Plenty of nine-track tape was still in use on mainframes in the 1980s.
Yes, I did backups at work on 9 track well into the 1990s. Admittedly we were late for an upgrade then. The coating used to come off on the heads on the old tapes. Cleaning was a chore.
At my current job I rescued an old 9 track tape which was going to be left behind for the cleaners when we moved offices. Its at my desk right now. Maybe somebody will come looking for it one day.
Re: (Score:2)
Nine track drives like that were still in use the 1980s. Into the 1990s, even.
Misleading photo (Score:5, Informative)
The story uses a stock photo [gettyimages.com] captioned "Obsolete mainframe super computers in [Computer History] museum". I don't think the Secret Service uses IBM 2401 magnetic tape units [ibm.com]
That's normal. (Score:3, Insightful)
The media uses stock photos whenever they don't have real photos of something. This is normal. I've even seen stock photos of Bumble Bee tuna used in contamination stories for another brand. (I forget which one.) Talk about misleading...
Re: (Score:2)
dis-information. (Score:2)
what more is there to be said aboutit?
Security by Venerability (Score:5, Funny)
Not curious at all (Score:2)
Its not curious. Don't confuse his desire to censor, restrict, or otherwise hinder the people's access to free information(the internet).
Doesn't mean he won't allow every resource into that same tech if security/administration needs it...especially if it achieves the former.
Pish, Overachievers.... (Score:2)
That's better than our goal of Nine 5's...up a little over half the time!
$187 million? (Score:5, Insightful)
They're claiming it will cost $187 million to replace. Bullshit. If the hardware is more than 15 years old, which it sounds like it is, it's impossible to conceive how they could spend more than $100k on hardware to replace it and still give 100x the performance and capacity. OK, let's splurge - spend 5 million on hardware.
These jackoffs would have us believe it's going to cost $180 million to replace some bullshit law enforcement database software that's 20 years old? Complete bullshit. Instead of the mythical $500 government hammer, now we've got the $180 million dollar software package that should cost
Re: (Score:2)
These jackoffs would have us believe it's going to cost $180 million to replace some bullshit law enforcement database software that's 20 years old?
We don't know what that software does. Thats why its called the Secret Service. My guess is that nothing will be delivered for less than 300 million USD. And yeah I do work on large civil/military projects, though on the European side.
Re: (Score:2)
Re: (Score:2)
Don't forget labor costs.
Labor with sufficient security clearance, proper technical certification, and adequate competence. It might be a "pick any two" situation...
Re: (Score:2)
Because they would spend 37k on hardware and hire 100 contractors at 1m/year to port everything over.
Re: (Score:3, Interesting)
These jackoffs would have us believe it's going to cost $180 million to replace some bullshit law enforcement database software that's 20 years old?
The rated you funny, but it's true. FBI CASE system also vintage 1980's mainframe system. They have tried and failed twice to rewrite 20 year old law enforcement database software at over half a billion spent so far. First time they said they didn't even have anything salvageable to show for it and threw out the entire project which aos hap
Re: (Score:2)
>> People blame the government bureaucrats, but the failed work is done by the huge consulting companies.
Billions of dollars sunk into failed projects, and yet the government bureaucrats keep hiring these consulting companies for software work. Therefore they share a large part of the blame: at the least it shows incompetence and negligence, at worse it implies collusion and fraud.
-dZ.
Re: (Score:2)
"Curiously"? (Score:5, Insightful)
"Curiously, upgrades to the Service's computers are being championed by Senator Joe Lieberman of Connecticut "
What's curious about that? It's not like the guy is a Luddite or something. The Secret Service, at the forefront of protecting POTUS, is a national security issue, and Lieberman is very involved in those issues. If the author threw that in because he doesn't like Lieberman's politics, then that's kind of lame. One would think that issues like keeping government IT systems up to date would transcend party politics.
Re: (Score:2)
They could mean it's curious that Joe Lieberman is doing something that isn't a waste of time.
Re: (Score:2)
then that's kind of lame
No, that's the quality of reporters these days when everyone has a blag, the true folks have forgotten what separated them.
Law enforcement systems are hard (Score:4, Interesting)
The article is bogus, but the problem is real. Computer support systems for investigators are hard to build. The FBI has struggled with this, taking about a decade to deploy their "Field Office Automation" system. They're hard for many of the same reasons medical systems are hard - much of the incoming data is unstructured, and many people enter data relevant to the same case. It's even harder than in the medical world, because links between various individuals and events are important, but unreliable. The "customers" aren't cooperative, they usually don't have unique identifiers, and a sizable fraction of the information is bogus. The security problems are tough to even define - exactly who's allowed to see what is a big issue.
The older law enforcement systems didn't offer much searchability. Unless you had a hard search key, like a driver's license number or a full name, you couldn't retrieve much. Now, everybody expects Google-like searchability, and the older systems just didn't have the machinery for that.
Re: (Score:2)
The telco wire tap system is out sourced to telco billing systems outside the USA and needs paper work.
If you in good with the billing system, you have real time insight into most legal wiretaps in the USA.
The NSA looks at all packets, but could not talk about its findings.
What would the Secret Service have? A card system like hyper card.
One person, one big virtual card, a pic, list of past issues ect.
Lieberman also got interested in a p2p hunting system to tag p2p shared media with
That's unacceptable (Score:2)
They should make an effort to get at least NINE sixes. Or more.
Why not outsource the whole ball of whacks? (Score:2)
Security problem? Loss of jobs?!? In your mind senator.
Re:don't trust it, it's about pork (Score:4, Funny)
Re: (Score:2)
Oynk, oynk. Joe Lieberman - the corporatist. Is his wife getting any money from insurance industry? Is he getting money from insurance industry? Yes they do, but obviously that became not enough. Now he also wants to force the secret service to build some ridiculous and most likely not needed IT systems in his state, I am sure, that is the only real reason for any of US politicians to come out with statements like that: we want pork.
Certainly he will pillage the system some more and then again before h
Re: (Score:2)
Underbid someone at 0 dollars for a contract and deliver what with that 0?
Lucky for US I am not it's citizen? As far as I am concerned - lucky for me.
what if the voting system itself is just as failed as the institution that implemented it?
- voting system will remain a failure as long as the people who vote do not care about understanding what they vote on. A system that implements voting in the first place is better than a system that has no voting at all, so there must have been something there in the beginning, but obviously over time money became the deciding factor for winning almost any
Re: (Score:2)
it has nothing to do with voting machines, it has everything to do with the corporations giving money to candidates from either political party, who are then given to you to vote for. Doesn't matter who you vote in, almost any politician from any side (R/D) is already a tool of the corporations.
The only right way to go is to vote for the non-establishment people, those who are not yet real politicians, those who have not being bought yet. You have some people like that on both sides: Ron Paul and Alan Gra
Re: (Score:2)
Re:Two Satans (Score:4, Interesting)
Can you convert that into a more familiar unit, like Library of Congresses?
You know the Library of Congresses is a pretty reliable machine. Does anybody know what its downtime is?
Re:Two Satans (Score:5, Funny)
Can you convert that into a more familiar unit, like Library of Congresses?
You know the Library of Congresses is a pretty reliable machine. Does anybody know what its downtime is?
The downtime for the Library of Congress [loc.gov] is 4:30 pm - 8:30 am, Monday - Friday, and all day Sunday. That translates into an uptime of about 28.6%. If you take the Secret Service 68% as uptime, then it would be 2.4 Library of Congresses.
Re: (Score:2)
Re: (Score:2)
Oh, for Christ sake.... will you people stop bashing DC for the snow debacle. The amount of snow that fell shattered every snowfall-related record in the books.
Criticizing DC for being unprepared for 40"+ inches of snow is akin to criticizing Chicago for being unprepared for an earthquake. I've spent most of my life up north in snowier climates, and can say with a high degree of certainty that both storms would have still been major events up there.
Re: (Score:2, Informative)
That's not downtime - that's no public access time. During the night is when the Library of Congress gets the most work done by magically book elves and their brethern the dust dwarves.
Re: (Score:2)
Yes but the LOC is a lot older than those Secret Service computers.
Re: (Score:2)
Ah, but what's the un scheduled downtime?
The scheduled downtime is there to take advantage of certain resources. Namely people who want to work when the sun is out, but don't actually care if they see the sun or not.
Re: (Score:2)
Re: (Score:2)
Wow. Sounds like they could really use a RAIL array....
Re: (Score:2)
> Isn't this an easy fix?
And you'll port their software for them at no charge?
Re:1980's mainframe? (Score:4, Insightful)
Ah. So you will just port all their data from their old proprietary database system to a new proprietary database. Piece of cake.
Re:1980's mainframe? (Score:5, Interesting)
Ah. So you will just port all their data from their old proprietary database system to a new proprietary database. Piece of cake.
You would need a security clearance for starters. Then the software would have to be developed to US Federal/Military standards. Maybe that requires CMMI-5 these days. So there's certification of the development processes, auditing and QA.
I think we are talking 100E6 USD before any code is actually written.
Re: (Score:3, Insightful)
If the only thing keeping them from upgrading was a "small consumer grade server" I'm pretty sure the NSA would have made one fall off the back of a truck and this would no longer be a problem.
The problem is more likely that the software running on the server is proprietary and closed-source, making upgrades incredibly expensive. Far more expensive than the incremental upgrades that the system should have seen in the 20+ years that it's been in production.
Re:1980's mainframe? (Score:5, Interesting)
Don't ever underestimate the difficulty of porting specialized applications
One Government agency I know of was informed with 5 years advance notice that their long time mainframe computer manufacturer would no longer be in the hardware business nor support the operating system. The Govt let a huge contract to port the applications. After several years, and millions spent in progress payments, that conversion attempt failed. So did several more. So after 10 years and about 4 attempts at conversions using some of the biggest software contract houses in the country they were still running on the original hardware and software and buying used equipment for backup. One of the few in the world.
It got done eventually I suppose.
Why, you ask, was it such a task to convert? Because they were attempting to replace something that had been custom built on top of and inside an operating system over perhaps 20 years. Distributed database and multiple geographic locations processing bits of the data using computers from multiple manufacturers communicating together long before the Internet (not that you could have put that kind of data on the net). So in order to convert, it took an understanding of how the whole thing worked and those that had that level of understanding had long since retired. It wasn't Cobol that was the problem but human limitations.
Re:1980's mainframe? (Score:5, Interesting)
The traffic signal system called SCATS was like that. It was hand assembled in PDP 11 machine code. There was business logic built into device drivers to get around executable image size issues. The people who wrote (more like built) it knew it inside out. They were just lucky to get it ported before those guys retired.
Re:1980's mainframe? (Score:4, Insightful)
It wasn't Cobol that was the problem but human limitations.
More likely it was the project itself, that is, replacing a pointlessly complicated system with an updated version of the same. If they sat down and looked at the real core requirements, instead of recoding a monster, they could have designed a simpler and better system for a fraction of the cost. I bet there were huge teams of designers and project managers who got rich off of each of those attempts.
Re: (Score:2)
Sure, get the custom-programmed code that requires the input and output of a mainframe rewritten to a $1000 server. The cost should only be a few million. That'd only buy a few new IBM mainframes that would run the existing code faster and more reliably than the old mainframe without any code changes.
I truly doubt it. (Score:2)
Never ascribe to malice or incompetence what can be explained by incompetent malice. I'm proud of myself, that sounded witty, but honestly I'd guess the original system "just worked" and slowly the needs outgrew it so that fewer people could use it at any given time than would like to use it. So slowly that the people who would have to explain the purchase could say "but its not that much worse than last year" instead of filling out enough forms to account for the mass of a sequoia, in order to do the req
Re: (Score:2)
Understandable. IBM dealt with them, too. Oh, and they had contacts in the Bush family as well. I believe W's grand daddy knew a few.