Email Offline At the Home of Sendmail 179
BobJacobsen writes "The UC Berkeley email system has been either offline, or only providing limited access, for more than a week. How can the place where sendmail originated fall so far? The campus CIO gave an internal seminar (video, slides) where he discussed the incident, the response, and some of the history. Briefly, the growth of email clients was going to overwhelm the system eventually, but the crisis was advanced when a disk failure required a restart after some time offline. Not discussed is the long series of failures to identify and implement the replacement system (1, 2, 3, 4). Like the New York City Dept. of Education problem discussed yesterday, this is a failure of planning and management being discussed as a problem with (inflexible) technology. How can IT people solve things like this?"
Nothing to do with Sendmail (Score:4, Insightful)
It's the backend. When you have too many connections on too few servers, with not enough storage
you usually see this kinda issue.
Re: (Score:3)
It's the backend. When you have too many connections on too few servers, with not enough storage you usually see this kinda issue.
I see it as yet another failure for the client/single server model.
It surprises me that people are still investing so much time and effort on centralisation of services when obviously the most practical technical[*] answer is the opposite. Simple, common protocols and decentralised infrastructure are the most robust model for overall survival of a communications system. DARPA proved that some time ago, but we seem intent on forgetting as much of that lesson as possible.
----------------
[*] Okay, I don't wan
Re:Nothing to do with Sendmail (Score:4, Informative)
Re:Nothing to do with Sendmail (Score:5, Funny)
It's the backend. When you have too many connections on too few servers, with not enough storage
you usually see this kinda issue.
Knowing the speed and flexibility of university upgrade policies, and knowing sendmail was born around 4.1BSD, and knowing the -BSDs were VAX only until 4.2 or 4.3 or so in the 80s, I'm guessing they're still using the original VAX it was developed on?
Re: (Score:3)
Many educational institutions lag behind because they're an ever-evolving door. Even when they've got dedicated and experienced IT staff, most of it's just in a managerial role for the student work studies (it saves money, of course).
It isn't an I.T. problem (Score:3, Insightful)
It's an economic one. It needs an economic solution.
e.g.
Have people buy a $10 ticket to get an account on the email server.
Re:It isn't an I.T. problem (Score:5, Insightful)
Pretty sure that's what tuition is.
Re:It isn't an I.T. problem (Score:4, Funny)
no I'm pretty sure tuition is more than $10
No, tuition is for education (Score:2)
Clearly email is an afterthought thrown in for free.
If you want a service to work, you have to fund it. You can try to fight for budgets against the football team or you can simply charge and the money automatically goes where it's needed.
Think of money as little packets of information. You buy something there is a need for it, you don't buy it, there is no need. Resource allocation without dozens of layers of management.
Maybe nobody cares about email and they can just shut it down. Charge for it and find o
Re: (Score:2)
Maybe they should offer redirection for free and paid storage. I find it useful to have an email address within my college domain, but I redirect everything to my main account.
Re: (Score:2)
At my university we split it into tuition and student fees. Fees go to all manner of things like clubs and activities and the rec center and other stuff.
Re: (Score:2)
Pretty sure that's what tuition is.
Tehnically, that's actually covered by a student fee, usually a "technology fee" in most universities. So yes, this cost should already be built into the cost of attending university. Whether that fee is enough to cover everything, including email, I'll leave to Berkeley.
IT has to deal with budgets, too (Score:3)
I hate it when people try to act as if IT isn't subject to budget constraints and having to prioritize spending like any other department of a large organization. Sure the money comes out of the "client" departments, but it's an issue that IT does have to plan for and deal with.
The summary asks "How can IT people solve things like this?"
Forward the emails and responses to the demands for planned capacity growth to the public.
Oh, you didn't keep the email from your manager refusing to pay for a neede
So the ultimate solution will be outsourcing (Score:2)
Re: (Score:2)
Re: (Score:2, Funny)
Wow, Squirrelmail. So at least they managed to migrate from pine at some point.
Carrier pigeons coming soon! (Score:2, Funny)
Wow, Squirrelmail. So at least they managed to migrate from pine at some point.
Yeah, they're planning the upgrade from squirrel to carrier pigeon as we speak!
Re: (Score:2)
Re:So the ultimate solution will be outsourcing (Score:5, Insightful)
Re: (Score:3)
Re: (Score:3)
Outsourcing would work, because when there is another failure they will have another party to blame instead of pointing fingers to a decision made in Spring 2011 (even as a total stranger I could feel the bitterness under that bullet point in the slides).
Telnet (Score:2)
Re:Telnet (Score:5, Insightful)
Students need school email addresses because that way all students have an email address.
At my school, students are expected to check their university email at least once every 24 hours. Many people forward it to a personal account, and obviously most people check it more frequently than that, but if the university issues an account to everyone, then there can be no debate about how they didn't get the email. The school takes responsibility for the email system (and any failures), and then professors can be assured that if they send an email out to the class, it will be (or should have been) read, leaving the onus on the student to actually do it. It's similar to why we provide computer labs - that way, each student unequivocally has a way to do electronic assignments, even if nearly everyone has their own machine.
Re: (Score:2)
They also need school email addresses because Hotmail et al are liable to mark a university as a spammer if enough luser students decide to click the "mark as spam" button for university-sent emails. Happened to my university a few times before It Was Decreed that all students are required to have a Google Apps for Domains email account resolving to our domain.
Re: (Score:3)
Re: (Score:2)
lusers reporting as spam because they don't want to bother unsubscribing is a problem. but can one unsubscribe from those emails without missing out on the important/relevant emails? that's what I wondered about as I ran into the same problem
Re: (Score:2)
I've had this problem at a big ten school. They have a mailing list where the disclaimer at the bottom of every email is that if you unsubscribe (even though they won't let you anyway), you may miss some important stuff (I'm paraphrasing). I've archived all of those email from almost a decade, and one afternoon went through them all. All of six messages out of almost 3000 were relevant to me, and most of them would be irrelevant to 90%+ of students they were reaching. Yet there was no way of unsubscribing:
Re: (Score:2)
Heh, yeah, that's legit. We send a bunch of crap that most of the *staff* don't care about, let alone the students, bulk emails about Spanish Club, bake sales, and so on. From two different email systems, yet.
Re:Telnet (Score:4, Interesting)
When I was admining at a small college, we DID NOT provide email for students, only for staff. We ran a listServe (sympa) and if the students gave us their personal email address, and checked a box, they would be added to a mail list for every class automatically..
Any student that didn't have an email would be sent to the library, where they would be shown how to sign up for a hotmail, yahoo, or gmail account.
We had students thank us, since they have gone to other schools, and though it was silly to have to check yet another account, when they already had 3 or 4.
The ONLY reason colleges give out emails is because they have been doing it since before email was a common thing. There is no actual reason for it.. (but I have heard some neighboring colleges give very, very very good sounding arguments on why they needed to drop a few hundred grant on a SAN and exchange)
Re: (Score:2)
The school takes responsibility for the email system (and any failures), and then professors can be assured that if they send an email out to the class, it will be (or should have been) read, leaving the onus on the student to actually do it.
That is peculiar coming from a scientific institution. Email offers no guarantee of reception. I grant that an extremely high amount of mails do arrive well. But if I were to avoid distracting discussions, I'd communicate with more than just email as a medium and I wouldn't rely on it being high available.
Re: (Score:2, Interesting)
facebook also gives me an email address
When did this start happening? Does it actually interoperate with other email services?
Re: (Score:2)
From news articles, it started a year ago, and it does interoperate with other email services (you get a [username]@facebook.com) but it doesn't let the sender choose the Subject line, add CCs, forward, etc.
Re: (Score:2)
No. (Score:5, Insightful)
Now I have an email addresses through hotmail, gmail and yahoo that I use for different things and facebook also gives me an email address. So, I doubt students really need email addresses provided by the university anymore.
You are quite wrong. Email addresses - especially .edu addresses - are still quite valuable. At lot of academic resources that take registration via email won't allow registration to go to a throwaway account (a la hotmail, gmail, yahoo, etc). Many organizations that are interested in real information on users insist that users use an actual unique account and not a freebie. And when you're in college and making very little money a lot of those things can be important.
I think it just shows that trying to build IT competence into a government agency basically a waste of money because the institutional culture of government
You're not very accurate on that, either. Government organizations need to be able to keep track of their email - especially internal communications - which they would not be able to do if they outsourced email and other telecom.
In short, all of these kinds of organizations could just offer email through gmail/google business or any number of other providers that will scale up almost infinitely.
With the various privacy breeches that have occurred, that would be a terrible idea. And on top of that, IT is a lot more than just email. Do you want the government to turn to comcast for networking support while their at it? What if the IRS web servers go down on tax day? Do you want them to have to lean on an outside company to get it back up?
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
In short, all of these kinds of organizations could just offer email through gmail/google business or any number of other providers that will scale up almost infinitely.
With the various privacy breeches that have occurred, that would be a terrible idea. And on top of that, IT is a lot more than just email. Do you want the government to turn to comcast for networking support while their at it? What if the IRS web servers go down on tax day? Do you want them to have to lean on an outside company to get it back up?
If you watched TFV...that is exactly what they're going to do. They're either going to go with Microsoft's 365, or Google's Gmail. They're just working out the contracts.
Re: (Score:2)
It's safer if the school has no access to your email so they should just drop that service except for those students who provably don't have the money for it.
I'd never use their servers the same as I won't use my ISP's. I use ones I have some insulation from and if I need more privacy than that I use encryption and if it's life or liberty threatening a drop off to a hollow tree in some random park. ;)
Re: (Score:2)
Re: (Score:2)
So it would be for official use only like work.
Re: (Score:2)
Yes, but for most students, I'll bet that their official school email address is still their primary email address for all the important stuff.
Re:Telnet (Score:4, Informative)
Also, email is used for a lot of very important stuff like sending reports, design files, etc. Having someone on campus that can fix problems is quite valuable. Your campus email will never be "accidentally" seized, locked out, etc. like people have experienced with google and yahoo. Because the campus maintains backups (or at least, they should), you data will never be suddenly gone with no chance for recovery like people have experienced with google and yahoo.
Re: (Score:3)
Actually, most schools require an Official school email address. This guarantees the uptime from the faculty's point of view; you can't claim you never got the assignment or that you turned it in on time and nothing was there. It's also important for them from a liability standpoint; my Registrar will not send me any bills unless it's to my .edu account, and professors are instructed to ignore any student emails from any other domain. They're also organized by real name, so the school has a working internal
Re: (Score:2)
Um, because the "internal directory" can't be implement using LDAP?! What's wrong with LDAP anyway? Even OpenLDAP seems to scale decently and I'd hardly consider it a "bother".
Re: (Score:2)
If the university can't competently provide its own IT infrastructure, why should they be expected to provide anything else competently? Perhaps it's just time to privatize the whole thing. I can't wait for GoogleU and Inteliversity.
Funding (Score:2)
Re:Funding (Score:4, Insightful)
Maybe it has something to with the fact that the state of california has cannibalized the funding for my beloved alma mater.
They wouldn't have to if they didn't have too many colleges (they do), and try to send too many kids to college (they do), many of whom may have no business being in college (they don't). Tax revenue is not an infinite resource. But California seems to have a community college on every two dirt roads, and several 4 year (or higher) colleges in a similar area.
Re: (Score:3)
Re: (Score:2)
Improper capacity planning (Score:4, Informative)
Briefly, the growth of email clients was going to overwhelm the system eventually, but the crisis was advanced when a disk failure required a restart after some time offline.
Capacity planning is supposed to account for reduced capacity due to component failures, system outages, and temporary demand spikes due to restart events.
Re: (Score:3)
In my experience this type of "planning failure" is caused when IT repeatedly tells management they need money to maintain and upgrade systems, and management consistently says no because they don't have the money for it. Not enough money or people to configure, install, support, and maintain any new systems because the budget won't allow any more. Yet somehow there always seems to be money for shiny new iPads and iPhones for the executives.
Re: (Score:2)
miniscule message storage limits like that are ridiculous anyway
In a perfect world, yes. But... (Score:2)
unions (Score:2)
http://slashdot.org/comments.pl?sid=2556922&cid=38249652 [slashdot.org]
IT should have unions so they are not the fail guy for management mess up's / lack of funds and or planing.
How IT people can solve this problem... (Score:2)
Of course, this requires IT people who are willing to put their foot down. We don't seem to have many of those...
Re: (Score:2)
Of course, the smart IT people are often not allowed into management -- they are too useful at their current level (keeping the systems running, turning the screws, etc), and would be a pain to be replaced. So they promote people who are easy to replace into management. For a smart IT person, usually the only way to get ahead is to move sideways, not up. Go somewhere else, or do something else... If you are good at what you do, there is little incentive for people to move you up.
Re: (Score:2)
excellent example of the Peter Principle
Re: (Score:2)
Unfortunately, those smart IT people who have spent years in the trenches and understand, in detail, how to build a robust and resistant infrastructure are often overruled by the a CFO who's only qualification is they have an MBA. In many tech companies the group that handles the infrastructure (DNS, email, backups, etc.) reports to the CFO not the CTO. Why is this? After 25 years in the computer field I still have not heard a rational explanation for this idiocy.
Re: (Score:3)
History.
Computers originally came into companies to do accounting and related work.
IT is not the Problem (Score:5, Insightful)
Re:IT is not the Problem (Score:4, Informative)
no way, I work at a Value Added Reseller of hardware and the good sales guy would definitely use your fears to sell you some expandable solution
outsourcing? (Score:2)
At the school where I teach, whenever there's a discussion of how much it costs us to run our own email, someone suggests outsourcing (e.g., to gmail), and then someone else says, "No, we can't do that because of privacy laws." Am I right in guessing that privacy laws don't in fact prevent outsourcing to google? I suspect the argument is basically a way for IT folks to have job security. There are certainly laws that say, e.g., that we can't give students' grades to third parties. But it's hard to believe t
Re: (Score:2)
That depends on where you are. Privacy laws in Canada most certainly do restrict Government agencies (and probably educational ones) from using Gmail because it's American and the Patriot Act is a rather severe problem that can't be mitigated.
Re:outsourcing? (Score:4, Insightful)
Typically universities have acceptable computer policies and at those institutions that run their own mail servers, such policies usually govern email. Students and faculty can demand changes to university policy if the policy does not properly align with the academic mission of the institution. Students and faculty have essentially no power over the terms of use that Google or Microsoft or any other third party email service imposes on them. It is easy to say, "Well, it is not like Google is going to demand something outrageous!" but there is really nothing preventing Google from doing so (if you do not think they have done so already). Google does not have the best interests of academia in mind when it sets its policies, nor is there any reason for Google to care about academic needs.
Re: (Score:2)
nor is there any reason for Google to care about academic needs.
Sure there is: If they don't meet the need, they'll lose the customers to someone/something else who does.
Re: (Score:2)
Can you provide an example of a policy that would get the students and educators up in arms?
Can you provide an example where the terms of access to outsourced services are not set by the organization who set the terms in their contract rather than blindly accepting whatever the outsourcing company tabled as a "standard" contract?
I thought not. More FUD. "Please panic, people, because they might possibly maybe perhaps breach their contract and do something we don't like."
Pfffftttttt.
Re: (Score:2)
Re: (Score:2)
I'm just going with past experience on outsourcing details. In every case where I've seen outsourcing done, the terms of uptime guarantees, access guarantees, priveleges, security -- all were negotiable items with give and take by both the outsourcing provider and the service purchaser.
If Google and Microsoft aren't allowing customers to set terms on critical things like system access and the validity of content, then companies are being very foolish to contract with them at all. Your provider should n
A lot simpler than that (Score:2)
There's a more mundane problem. Unless you are an incredibly huge customer the large service providers are just not going to care if there is an outage. One example I ran into last year is a University of 45,000+ students that lost their student email hosting (hotmail) for a week due to a DNS typo for a machine in a hotmail MS Exchange server farm. To get a
Re: (Score:3)
Google now manages e-mail for more than 2,000 colleges and universities, enabling students to transform accounts capped at 100 mb into Google-managed inboxes that allow for 70 times as much mail. Microsoft also provides free Web-based mail for thousands of schools, including colleges in 86 countries.
Here's the article: http://www.time.com/time/business/article/0,8599,1915112,00.html [time.com]. Now, a specific school?
Re: (Score:2)
The answer depends on:
1) Where you are and thus what laws are applicable to you.
2) Who you are. Healthcare, university, private company? If you are a university, are you a public university? If so, there may be additional laws and regulations.
3) What's being emailed. Patient records, classified documents?
What's acceptable for people in similar situations may not be acceptable for you. I go to
Solution: Join the Google Collective (Score:2)
Seriously, is Berkeley like the only college campus that hasn't outsourced their e-mail to Google yet?
Re: (Score:2)
Re: (Score:2)
The University of Washington has outsourced its student email as of a couple years ago; but for faculty and staff the old "deskmail" servers are still available with no announced EOL (yet) - although many do use Gmail too, since it offers an order of magnitude more storage.
Technically UW offers both Gmail and the cloud Windows mail. It used to be just Google Mail and Calendaring, but before he left UW President Emmert dictated the university needed to have Exchange available to everyone - probably coinciden
Only 70000 accounts? (Score:2)
Only 70000 accounts? That's not a big system at all. I was running systems with over million email accounts ten years ago, and by today's standards even those would be considered small.
Re: (Score:2)
Two decades ago we were supporting 78,000 users on a machine that was almost as fast as 4 Nintendo 64.
Re: (Score:2)
Not to mention not having lusers send 100MB attachments in their email to multiple recipients.
Re: (Score:2)
Nah, HTML mail was well and truely endemic by then (the ISP I worked for was even spamming our users with it every week).
That said, how much of an impact does that have anyway? Worst case, there's an HTML copy and a plain text copy in the same email, so ... ~2.5 times the original size? That's not all that much of an imposition.
Attachments might be more of a concern, but even today, the default maximum message size of Postfix is still only 10Mb. I daresay most servers on the internet are still running with
What does this have to do with Sendmail? (Score:4, Insightful)
Also, they mention that the cost of the system is something like $1.30 per account per month. I don't know much about IT budgeting, but that seems like a really low number for something as critical as messaging and calendaring. I have to imagine that they spend more money per user just cutting the grass around the campus.
Re: (Score:2)
They no longer use Sendmail; they use Exim.
Re: (Score:3)
Also, they mention that the cost of the system is something like $1.30 per account per month. I don't know much about IT budgeting, but that seems like a really low number for something as critical as messaging and calendaring. I have to imagine that they spend more money per user just cutting the grass around the campus.
Totally agree. One of my client did a major cost-cutting initiative for its email platform, and there was just no way to make it reliable under 9$ a month (per account). And this is when there is no Crackberry (which brings the numbers way up).
Re:What does this have to do with Sendmail? (Score:4, Informative)
Re: (Score:2)
That's a lot of money actually. $1.30/user is $13,000/month for what a simple server cluster (Postfix etc.) should be able to handle (~10,000 accounts). That's over $150,000/year.
I used to work for a hosting company and our server cluster (4 servers with a 10TB backend) handled well over 50,000 accounts about 5 years ago. Sure the load would be at 10 or so but that's not a big deal for a mail server.
Re: (Score:2)
the point was "how the mighty have fallen", not something about sendmail in particular
IT cannot solve this (Score:2, Interesting)
it's like saying IT can do heart surgery or IT can provide pscyhological counseling to a trauma survivor. IT is IT, it is not management and it is not leadership. IT is IT.
of course, shit rolls downhill, and leaders nowdays are incompetent buffoons who gain their positions largely through bribery, kickbacks, extortion, and other 'features' endemic to societies where the rule-of-law breaks down thanks to a greedy, corrupt elite.
again, IT cannot fix that.
Re: (Score:2)
oh man, good IT people are leaders, whether in management or not. They try to identify resource problems before they become issue, and have solutions.
The failure is leadership, planning, budgeting... (Score:4, Interesting)
I've only heard from people on one side of this but the story that I hear is that in the past, many departments had their own IT, mail servers, web, etc. When the campus built its centralized computing services facility, there was great pressure on departments to move to the central system. There was some griping about the costs for central services often exceeding the internal costs the departments formerly had but there was, I'm told, much need to justify the expense of and to pay for the new center. I've heard that some departments have been able to resurrect their internal systems to get through the outage.
Perhaps someone with more inside knowledge than I have can fill in and/or correct information from both sides of the story.
That slideshow is pure management-spin right from the opening "look how complicated and difficult this is..." I love how the "solution" to a system that is soon to outstrip its capacity is to stop expanding (and, it appears, properly maintaining) said system and hope it doesn't implode before you can toss the potato to an external party (who can then take the blame). Guess I was never learned at that school of capacity "planning".
Re: (Score:2)
They left out the slide where management get great big bonuses for being such swell thinkers.
Did the CIO just give up in the presentation? (Score:4, Insightful)
The press pretty much reads like this to me:
1) We didn't size the system large enough to handle the possible outages.
2) The outage we didn't size for happened, basically taking everything down.
3) My team is now working on a band-aid solution, which basically involves hobbling the application.
4) Since we're incompetent, we're going to outsource this next year.
I mean, if I was the CIO's boss I would have fired him on the spot. Maybe outsourcing is a better answer than putting in place a proper system and looking at that analysis could be interesting. I see no indication any of that was done here, basically the CIO gave the Barbie response, "Mail is hard, let's go shopping." If he doesn't understand how to do it in house, he won't understand how to arrive at a good outsourcing agreement.
Which means this pretty much sums up everything that is wrong with large org IT today.
Re: (Score:2)
"Mail is hard, let's go shopping."
IT isn't a university's core business. When IT was in its infancy there was a case for letting the CS faculty run IT with students volunteering. As IT has advanced in the mean time, the CS faculty can now concentrate on CS and reduce its hands-on stuff to leading edge technologies (e.g. research in super computing, semiconductors, etc...). Nowadays for the vast amount of tasks -even for most CS tasks- ample computing resources, technologies and software can be easily made available.
The email experts are not ... (Score:2)
Look up Microsoft live@edu and Google aps for education.
Re: (Score:2)
Microsoft exchange server is a reliable, secure and scalable solution? Bwhahahahaha, my employer uses that shit, it is none of the above. If it were, the internet's backbone would use it
Poor management (Score:2)
Sigh...
Look at the first bullet point of the timeline. Productivity suite approved, upgrade to Calmail cancelled. Then a week ago, they decided on an interim upgrade because not upgrading in the first place caused problems. So, rather than a planned upgrade, the IT folks were thrown into panic mode because their (probable) proposed timeline for safely doing an upgrade, including burning in and testing of new hardware, was cut to a fraction of what it should've been.
You can argue about the budgets, or the IT
Re: (Score:2)
That's only applicable if you're using FAT, sysvfs, ancient versions of NTFS or ext2 (before dir_index) and others of that age. Any modern filesystem can handle millions or more entries in a directory without going into the O(n^2) hole you're speaking of.
Re: (Score:2)
The MAIL environment variable has been around since before the early 1980s and IDA sendmail had patches to look up the mailbox per user using dbm files in the late 1980s. I can't see this every being a problem with a mail server but I have seen it on usenet servers that don't expire some groups.
Re: (Score:2)
Re: (Score:2)
Re:Hate Being First .... (Score:5, Insightful)
Believe it or not, maintaining a mail host for a larger, geographically diverse
If it were easy, there'd be no push to outsource it to "the Cloud" (or anywhere else), and countless organizations wouldn't be moving from the "burden" of administering something like Exchange (ie, a trivial amount of knowledge is required compared to any other MTA) to Office 365 or Google.
It's not just as simple as setting the mx to point to a 'working host', especially not in academia (though many try). Do you have to deal with this kind of thing?
As someone who has to deal with this stuff on a daily basis - I had dealings regarding CalMail last week on a similar mail related problem of their's - and with academic mail systems in general, let me clue you in:
* This is not your business mail system, where everyone has a uniformly specified mailbox.
* It is not dictated from the top down how mail is run. In a corporation, there is standardization. CalMail is the exception in academia, as far as I can tell, in that it's run somewhat like the business model. However, there is still somewhat of the "Greek" (vs. "Roman") model of management involved, and this does tend to lead to problems. (This is much more true with other academic mail systems, from what I can tell.)
* Unlike in the work place, there is very little systems experience where it is needed (ie in the actual administration). Even with dedicated IT, very few people are actually good with the mail system due to how broad and complicated mail management can be.
* Running a mail server effectively is now quite difficult. Not only do you have to "just make it work" - ie, dealing with all the misbehaving mail systems out there from other academic institutions and verifying the VIP email makes it through (regardless of how much spam that means letting through - but never let any spam through!) - but it's got to run like a top.
* Often, you're dealing with decades of systemic dependencies. Mail was the first connected application, after all, and nobody's had it as long as Berkeley. Based on my own experience with networks which grew around their mail system, small changes can compound any sort of change or update. Suddenly, there's something everywhere that needs a specific mail system functionality which can't simply be copied over during a move to replicate it.
* An organizational system like this is big, it's not garden variety email. Hell, i guarantee you they don't have as many IT people maintaining accounts as they have admissions people, probably not even a 10th. Yet the IT people have to actually make sure those records get to the right places all while assuring the admissions people that the information transits securely.
* There is undoubtedly a faculty member with his pet requirements for email. He probably has things which will not migrate properly.
* There will undoubtedly be the people using their mail account for file storage.
* Believe it or not, it's actually fairly difficult to migrate mail from, say, Cyrus IMAP to anything else. It takes time (and anything at all with Cyrus, which I'd not be surprised if they were using, takes a lot of time). Sieve scripts, procmail, IMAP states, et al. It's a pain in the ass, and takes a loooong time to do seamlessly. Doing it under duress of hardware failure is something else entirely.
From my reading of the events (and seeing some other things not mentioned in OP or linked article) there were a number of things which caused this prolonged outage. First and foremost, the system was not designed to be resilient so much as it was designed to scale up (or proper failure condition testing was not performed beforehand). Second, they either don't have the necessary (knowledgeable) human resources, or enough time allocated to those resources, to effectively manage this system. (You would not believe how difficult it is to find a "mail administrator". Everyone's done it, but nobody seems to like it or is all that good at it. If they are, they want a LOT in compensation.) Third, they may
Re:Hate Being First .... (Score:4, Interesting)
As a mail administrator for a big system, I completely agree with you.
The biggest problem was that they had everything on a single SAN, so when they ran out of IOPs, there was no spare capacity anywhere, and nowhere to mitigate it to. I've had people try to sell me on putting all our systems on a SAN too "it's so simple to administrate. It has plenty of IOPs, see, look at these shiny numbers". Fine when it's empty and you're only hitting the battery backed cache.
Which is why we have hundreds of separate little disk sets managed with templated configurations rather than any single points of failure. I'm really glad to be there!
Re: (Score:2)