Follow Slashdot stories on Twitter

 


Forgot your password?
Close
typodupeerror
×
Unix Operating Systems Software Programming IT Technology

Being Enron's SysAdmin 94

Posted by Zonk from the big-time dept.
CowboyRobot writes "FreeBSD's Kirk McKusick has a long interview with Enron's former SysAdmin, Jarod Jenson, where he describes the nuts and bolts of working in and managing such a large-scale operation." From the article: "EnronOnline was a Web-based trading application. We had several hundred, even thousands of commodities that we would price in realtime, the same way that equities are priced. We were trying to push realtime pricing information out to clients who could do instantaneous transactions on them. People who are familiar with financial markets--the commodity markets--would recognize EnronOnline as sort of the same thing. We had a lot of the same issues that the markets had trying to push out realtime data--not only within our local network but also to the customers--as quickly as we could globally, and trying to make sure that what every trader saw on the screen matched what every company in the world had on theirs."
This discussion has been archived. No new comments can be posted.

Being Enron's SysAdmin More

Being Enron's SysAdmin

Comments Filter:
  • Must have been a big headache to manage something that big.

  • Realtime performance (Score:5, Funny)

    by NightWulf ( 672561 ) on Saturday February 11, 2006 @02:49PM (#14695498)
    I wonder if EnronOnline had realtime showing of the stockholders assets flying out the window.
  • I think the Sysadmin had allot on his plate setting a goal like that. But it is a possible and probable goal to attempt and well worth it.
  • ... whos "printer freindly" webpage ends as -

    KM

    In terms of performance mistakes, the operations guy says it's the developer's fault, the developer says it's the operations guy's fault. Whose fault is it?

    JJ

    whereas the non-"printer freindly" version continues with the interview, this is why I don't join the ACM, they can't even take 9 seperate pages and format them to be printed, I now re-christen the ACM - ABM, Association of Broken Minds.

    • No kidding. It looks like the printer friendly version goes to the first question of page 7 of the regular version. If you change the 'page=1' part of the URL to 2 or some other number, you get that page of the article, but in the non-adridden semi-printer friendly style. I can't find a way to get pages 7-9 to show up in a single printer friendly page, though.
  • Since I know a lot of you aren't going to RTFA
    After the collapse of Enron, Jenson worked briefly for UBS Warburg Energy before setting up his own consulting company.
    It's good to see that his life didn't get fscked up by the Enron blowout.

    Enron the company was doing lots of good and innovative things in the marketplace as TFA shows. It was certain groups of people that drove the company into the ground.
    • "Enron the company was doing lots of good and innovative things in the marketplace as TFA shows."

      I don't know about that. You should watch an excellent documentary by the title 'Enron: The Smartest Guys in the Room'. The impression I get is that almost everyone at Enron knew that something shady was going on. For example, a lot of the power outages in California were due to traders calling up power plants and asking them to shut down in order to drive the price of energy up.

      The entire operation was extr

      • Enron the company was doing lots of evil things in the marketplace

        I'm a classic bleeding heart liberal, and old enough that I should have known better, but I always thought that folks like Ken Lay were sincere in their beliefs about the virtues of free markets. Then I read The Smartest Guys in the Room, and Den of Theives and finally twigged to the fact that folks like Lay, Skilling, Milken, and Boesky, speak publicly about the miracle of free markets, then go back to their boardrooms and do their damndes

        • that folks like Lay, Skilling, Milken, and Boesky, speak publicly about the miracle of free markets, then go back to their boardrooms and do their damndest to put the markets in their pocket

          I also just finished The Smartest Guys in the Room, and I was left with impression that Lay is an idiot, definitely not smart enough or driven enough to control the market. It seemed to me that Andy Fastow (former Enron CFO) was the dirtiest of them all, and Skilling set up the environment for Fastow to flourish. Lay



    • Problem with your statement is that a company is only as good as its people. A company isnt a free-willed creature.

    • Re:Enron didn't hurt him- Insightful??? (Score:5, Informative)

      by ralf1 ( 718128 ) on Saturday February 11, 2006 @05:25PM (#14696263)
      This was modded insightful? Bah. You statement shows a complete lack of understanding of the impact of the Enron collapse on many of its employees. The fact that he got another job is a simplistic, and quite frankly juvenile view of a major financial event in many peoples lives. Almost everyone who worked for Enron got another job. The Houston job market is pretty good, and particularly good for people with solid IT credentials. There was very little stigma associated with having Enron on your resume unless you were in the inner circle of Lay/Skilling/Fastauw.

      The real impact was that many people at Enron had the vast majority of their personal savings in Enron stock or other Enron securities. The company management strongly pushed employees to do that, and there was significant corporate cultural pressure to invest ALL of your 401K in Enron stock. When the company tanked, people who had worked for years for Enron or one of the companies Enron had acquired suddenly went from having accumulated enough wealth to be close to retirement to having to start over. Plenty of stories of folks who at age 50 suddenly found themselves going from being worth millions on paper to having no life savings and no fiscal security. This happened to folks in a matter of weeks, and while it was happening compnay management was encouraging those employees to stay the course - hold the stock - as it would come back. It changed the lives of tens of thousands of people. I have one acquaintance who went from having 85,000 in a 401K, who got a settlement check for 43.00 from the bankruptcy court. Fourtunately he had other monies and the Enron investments really didn't change much for him long term, but I mention it to give you a sense of the scale of the collapse for folks.

      The 'he got a job' comment may resonate with you folks who are young, have no long term obligations like a family, and are living paycheck to paycheck with no view of your future beyond how fast can I save up for a new Athlon dual core, but for those of us whose lives are a bit more established, it stinks.

      Disclaimer - don't work for Enron, never did, no one in my family or close circle of friends did/does. But I do live in Houston and have seen what its done to good, honest people who did nothing wrong but believe the propaganda delivered by the people for which they worked.
      • That was nobodys fault but theirs.

        Fuck the greedy bastards. They can always flip burgers.

      • If people worth millions (paper money or not) can't be arsed to buy "Investment 101" or "Savings for Dummies" they are setting themselves for failure.

        And don't bring the "they were forced to buy Enron" bullshit. I am pretty sure that nobody put a gun to the head of anybody to force them to risk all their saving eggs in one basket only.

        The bad luck of a fool is always regrettable and certainly one can feel a degree of pity for the fool, but the fool has to take responsibility for his own actions if he or she

    • Re:Enron didn't hurt him (Score:5, Interesting)

      by GodBlessTexas ( 737029 ) on Saturday February 11, 2006 @07:36PM (#14696951) Journal
      I also worked at Enron, and we were trying to get money and hardware resources pumped into several open source projects, namely the OpenBSD project, because we were one of the companies trying to use it in our production environment. Unfortunately my layoff came before we could get anything signed, at which point there was nothing left to give.

      Enron and its subsidiaries had a lot of great people working for them, and it was one of the few places where bright minds/tech people could get promoted for being great at what they did, and it didn't matter if you were some fresh-faced college graduate recruited for our analyst program or a guy who dropped out of high school but was brilliant at programming. It also left a lot of those bright people in a financial and professional lurch. What was worse than the sudden loss of employment was the loss of professional stature. I had prospective employers, even those in the energy trading business, actually deny me employment because I worked at Enron, as if I had something to do with their downfall as an upper level tech person. A lot of people thought we were part of some vast conspiracy, when many of us were the ones who got screwed in the process of Enron's downfall just like the other stock shareholders. The only difference between us and them is it was significantly more perseonal.

      • A lot of people thought we were part of some vast conspiracy, when many of us were the ones who got screwed in the process of Enron's downfall just like the other stock shareholders.

        I'm sure you, and most of the employees at Enron, were upstanding and decent folk, but didn't you have any inkling that things might be amiss? I mean the guys on the energy trading desk are a long way from the boardroom, but from their taped converstations they knew they were engaging in un-ethical, and perhaps illegal, behav

        • Enron never required publicly, privately, or explicitly that employees were required to purchase and keep Enron stock in their retirement portfolios. It was certainly part of the culture, but it was part of the culture that came from middle management as best I could tell. The only thing that involved money that made you visible to the upper level management types was time spent on community/charity work and donations to charities, especially United Way, as Enron matched dollar for dollar and actively enc
      • > I had prospective employers, even those in the energy trading business,
        > actually deny me employment because I worked at Enron, as if I had something
        > to do with their downfall as an upper level tech person.

        It occurs to me that the best response to such suspicions would be to point out that if you were one of the swindlers, you wouldn't need to look for another job.
      • Remember the Alamo, and God Bless Texas...

        Don't forget Goliad, either....
  • >and trying to make sure that what every trader saw on the screen matched what every company in the world had on theirs

    Keeping it honest is indeed very important!

  • Email (Score:3, Informative)

    by Helios1182 ( 629010 ) on Saturday February 11, 2006 @03:15PM (#14695628)
    There is a large collection of email from Enron vailable online. It has been usefull for research in natural language processing, text classification, and data mining. Check it out here: http://www-2.cs.cmu.edu/~enron/.http://www-2.cs.cm u.edu/~enron/ [cmu.edu]

  • Just two words... (Score:3, Funny)

    by ValentineMSmith ( 670074 ) on Saturday February 11, 2006 @03:40PM (#14695753)
    man shred(1)

  • I bet he had an easy job.. (Score:5, Funny)

    by alfrin ( 858861 ) on Saturday February 11, 2006 @03:42PM (#14695766)
    "rm -rf /" on everyone's machines.

  • Developers vs Admins... (Score:5, Interesting)

    by otis wildflower ( 4889 ) on Saturday February 11, 2006 @03:47PM (#14695784) Homepage
    Very interesting discussion about the relationship between developers and admins in TFA. My own take is that it's basically the only useful thing that team leads or managers can do anything about by setting the incentive structure for both to be somewhat similar.

    Essentially, if a developer's job relies on the same thing that an admin's job relies on (that is, stable, secure and reliable operations) then you have the foundation for harmony. If a developer's job relies on features and new functionality at the expense of stability, security and reliability, you have a recipe for hostility.

    You can tell the priorities at a company by how cranky its admins are.

    On the other hand, admins need to be open and available to developers, offering advice on OS, hardware, infrastructure, etc. and be able to clearly define the requirements for SSR so that any new designs or requirements can be supported from day 1.

    Oh, and a great way to get documentation from developers: give all their cell#s to the admins, so when something breaks at 3am on a sunday and there's no documentation, the admin has a little company. A few calls like that and developers can write some pretty handy documentation!
    • >>> You can tell the priorities at a company by how cranky its admins are.

      Is cranky good or is cranky bad? My guess is cranky is bad.

      The best developers I've worked with know how to create both features and stability. What makes sysadmins cranky is, to take an example, getting into an argument with a developer who has just noticed that all the soft links have permissions 777 and thinks it's a security problem.

      Actually the bigger the organization the less variation in crankiness. It may seem overly
    • Stupid ass admin's who do shit like mount application log files on the root filesystem... and stupid ass admins who don't have scripts to alert them when a file system is 99% full... or stupid ass admins who wait until a disaster strikes at 3am in the morning to do anything about it!

      • gdb into any live production code to unwedge something lately?

        And an admin is more likely to get shitcanned for that kind of incompetence than a developer is for writing shoddy/incomplete/nonexistent documentation or slamming an untested bit of code into production without telling anyone.

        I've seen admins fired for that kind of nonsense. I rarely see developers fired for shoddy workmanship, lax code management or other egregious crap.
      • 99% full: we have scripts thank you. We have tried many different policies to clean up once we are alerted: asking nicely (they ignore us), asking harshly (they complain), being draconian (removing older, bigger files) people get sarcastic at you (here, I give you 100 bucks, buy a new disk), imposing limits (everybody has a good reason to need more than their colleagues), doing careful audits (discovering all the porn, movies, pictures, unauthorized software, etc., people, kid you not, complain about their

  • Enron Wasn't Innovative In IT (Score:3, Interesting)

    by Anonymous Coward on Saturday February 11, 2006 @03:49PM (#14695788)
    I had two encounters with Enron during it's peak years:

    IT Job Offer From Enron:In the first I was offered a trading systems developer position. They used PowerBuilder on Windows for their trading systems - nothing wrong with that. But the IT Director kept talking about how quickly the laws changed that concerned their energy trading. I enquired as to how they maintained their business rules. He replied roughly that "it's all in code; when the laws change we rewrite the code." I asked what they would do if there were an audit say, 6 months later after they'd gone through 4 versions of code. He replied that they would set up a standalone network, load the old software and restore snapshots of the database and rerun the old data.

    Naturally I asked if he had heard of rule-based systems and hinted that, if they had a dynamic rulebase architecture, they could avoid the recoding and versioning. Instead they could have a much simpler system and better controls. He said he had never heard of such a thing and couldn't see how it could be done in their environment. But he was interested.

    Seeing that their viewpoint was extremely short-term and unenlightened, I turned down Enron's offer. Many of my cohorts at the time said I was a fool, but the IT situation at Enron seemed to be impoverished in thought although rich in resources. In time I realized my hunches were correct.

    Encounter With Enron Broadband:Flash forward to 2 years later. I've gone completely over to WWW development and to FOSS. I'm attending a Microsoft event on IIS mostly out of curiosity. Next to me is an outspoken individual who works for Enron Broadband. He speaks endlessly of the benefits of IE and ActiveX. I ask if Enron uses ActiveX much; he replies that they are tied to IE and that ActiveX is a necessary part of their architecture. I ask about Apache, Netscape, and FOSS; he replies that they have no capability in those areas to his knowledge. I once again decide that Enron has somehow missed the boat.

    Enron never carried significant loads on their web servers. Their limit was the number of energy traders in the U.S., which is a relatively small number. Today, small sites do much more than Enron did with fewere resources. There are few useful lessons to be learned from Enron's IT group or their BroadBand division fiascoes.

    • Considering I work in a highly regulated industry (lending), our developers learned from the previously design system in terms of rule-based systems.

      In our case we call it a "product". Each product has an associated set of rules such as interest rates, loan term and more. When we face regulatory changes, we modify the "product" but the original loan terms are stored at the time the were done so we can always go back and show you the state of a loan at any point and time.

      It also helps that we actually have a

  • Ad (Score:1, Insightful)

    by killerdark ( 922011 )
    The article sounds like a consultant who is in dire need of work and does the ultimate desperate action, getting free ad time on /. His story sounds like a fairy tale. Not like the life of a system administrator.
    • i've met the guy, i've sat through a class on Solaris that he was teaching, and i've been here as we've discussed getting him to take a look at our systems. i know that the last time we had him in, we had to schedule him some time in advance. i'm fairly certain that he's not hurting for work, based on my interactions...
      • I might have been a bit to pessimistic to be honest. I dont know the guy after all. Thanks for pointing out that he seemed to have plenty to do.

  • Enron - lessons in what NOT to do (Score:3, Insightful)

    by queenb**ch ( 446380 ) on Saturday February 11, 2006 @05:16PM (#14696223) Homepage Journal
    Frankly, all of my contact with Enron has shaped my opinion as such that if I had worked at Enron, I'd be reluctant to put it on my resume. I'd rather have a sabattical in Tibet, stay in rehab, or time in the peace core on there instead. Much of the criminal activity at Enron was IT-based. When your accouting system is an application, your IT had to know that the books were being cooked. The fact of the matter is that they chose not to report it. Instead,a lowly bean counter blew the whistle on all of them. Now that I know that this man has come out and proudly announced that he was part of such an organization, I would seriously recommend that he not be hired for any kind of a position of trust. I don't see a lot here to be proud of, ethically or morally.

    Anyone who's been an SA for any length of time knows that being an SA carries an ethical and moral burden. Just because you accidentally read an email while trying to fix something on the mail server doesn't mean you can go gossip about it. If you happen to see a file that has private contents on someone's desktop, you don't go gossip about it. If you happen to find kiddie porn, you inform the FBI. There are rules to this business. Annoucing that you've violated the most important ones doesn't exactly make you a desirable candidate.

    How can anyone, who claims to be providing web services for "the public", tie themselves to an IE/Windows/Active X architecture? I work for a university and, while our web traffic is quite atypical, IE accounts for only 1/2 of our traffic and has for some time. Windows only accounts for about 60% of our operating systems. Since we seem to "lead the curve" on rest of the net, I would suspect that in the next year or two most sites, at least ones that take international traffic, will start seeing a similar shift in their traffic.

    The whole paradigm of web services is that it's supposed to work on any OS, any browser, etc. without the need for any specific client software. If you're web application isn't browser/OS agnostic, you've totally missed the boat. Frankly that's not anything to be bragging about technically either.

    So, to sum it all up. He's bragging about being part of an ethically corrput and technically deficient company and wondering why he's not got a job.

    2 cents,

    Queen B
    • This is probably my biggest concern when interviewing people for System Admin positions. Ethics that is.

      We end up having power over everything. We have the ability to read everyone's email, look at everyone's personal files and in some cases screw the company out of millions of dollars. And we know the ways to cover our tracks because we know the environment better than anyone.

      I know that I would never abuse that position but it's tough trying to decide based on interviews if someone else would. It's the th
      • >I know that I would never abuse that position but it's tough trying to decide based on interviews if someone else >would. It's the thing that keeps me up at night when we bring a new person on board.

        This is a good reason for auditing. All actions taken by sysop's should be accounted for, both for security and maintainablility. Systems should be logged to a log box that only the senior sysop/CIO/HOD/whatever has access to, and the logs should be audited and any logins that aren't documented should be

    • Re:Enron - lessons in what NOT to do (Score:5, Insightful)

      by XorNand ( 517466 ) * on Saturday February 11, 2006 @06:16PM (#14696541)
      Much of the criminal activity at Enron was IT-based. When your accouting system is an application, your IT had to know that the books were being cooked.
      That's a pretty ignorant assumption. I dunno about everyone else, but as a systems dude, I don't even know how to use some of the applications I support. The network support guys are even further removed. Is a drunk's car mechanic liable if after a bender, he runs down a few kids?
      • Agreed. I was thinking the same thing.

        I work IT in a hospital. I'll admit i have no idea what's *actually* going on in the pathology or cardio systems.

        I'm not a cardiologist. Enron IT guys weren't accountants.
      • If you're doing what you should be, as a sys admin, which means checking logs, checking time stamps, etc. then you KNOW that the data is being edited. If audit data is being tinkered with that should be sending up all kinds of red flags, flares, bells, whistles, and maybe even a couple of roman candles.

        If you're running the right kind of an IT shop, this gets reported up the food chain that "Hey, this audit data is being messed with." until it gets to someone who can do something about it. Since he's in c
    • So, to sum it all up. He's bragging about being part of an ethically corrput and technically deficient company and wondering why he's not got a job.

      The impression I got from the article was was that he was pointing out the benefits you get from team work, and the usefulness of dtrace [theregister.co.uk].

    • Re:Enron - lessons in what NOT to do (Score:1, Informative)

      by Anonymous Coward
      Very ignorant comment...

      As an SA for a very large telecom -- we support such a wide variety of systems that I couldn't even begin to tell you what the data on the systems actually holds.

      The guy was an SA for a VERY large company, and their IT systems were NOT their downfall...
  • For taking such good care of your servers! We picked up a bunch of them when the company went bust, and they're still running fine today. Hell... We even left the Enron asset tags on them for kicks.
  • I can't believe how little effort some people put into writing grammatically correct prose.

  • EOL (Score:2, Funny)

    by superflytnt ( 105865 )
    Is anybody else seeing the humor in an application called "EOL"? (end-of-life) ?
  • Some of his comments about benchmarks and measuring performance bottlenecks go back decades to some comments I've seen time and time again. You have to know what you are measuring in order to know how to measure your performance. If you don't know what you are measuring, you're not going to get useful information.

    His points about 'taking ownership' of the problem appear to be spot-on right, when people take ownership of a problem it provides a better chance to solve the problem. This comes straight out of the book In Search of Excellence as a method of building a better company. Difficulties and loss of effort happen when the general environment is one of "it's not my problem." When one takes ownership of problems in order to fix them no matter where the actual problem is you can produce excellent results. But again, if you're not looking in the right place to find the solution, chances are you won't find it.

  • I read the article, and it has some useful ideas, not necessarily new. Jared is a skilled performance tuner and its a good discussion to read. What is disappointing is the flambait on this slashdot discussion board.

Slashdot Top Deals

An Ada exception is when a routine gets in trouble and says 'Beam me up, Scotty'.

Close