Forgot your password?
typodupeerror
Bug The Almighty Buck

Royal Bank of Canada Software Upgrade Goes Awry 602

Posted by michael
from the cosmic-justice-for-SCO-investment dept.
Reader mks113 writes "Many Canadians living payday to payday have been in for a shock this week. Canada.com along with many other sources is reporting how thousands of customers have been inconvenienced following an unsuccessful software upgrade at the Royal Bank of Canada on Monday. All government employees (including me) in several provinces had their direct deposits delayed by a day or more." RBC has a comment on the mess.
This discussion has been archived. No new comments can be posted.

Royal Bank of Canada Software Upgrade Goes Awry

Comments Filter:
  • by Anonymous Coward on Friday June 04, 2004 @12:41PM (#9335998)
    They actually pay Canadians?... With money?
    • by AchilleTalon (540925) on Friday June 04, 2004 @12:42PM (#9336021) Homepage
      Of course, but it's in canadian dollars!

      • by Zcipher (756241) on Friday June 04, 2004 @01:03PM (#9336254)

        So that'd be a no, then ^_~

  • Sticky karma.. (Score:5, Insightful)

    by the_rajah (749499) * on Friday June 04, 2004 @12:41PM (#9335999) Homepage
    I guess that bad karma is pretty sticky. Even selling their preferred A-1 shares to Baystar didn't save them.

    My Canadian friends are screaming bloody murder. I don't blame them.

    "Do the Right Thing. It will gratify some people and astound the rest." - Mark Twain
    • Royal Karma (Score:5, Funny)

      by conner_bw (120497) on Friday June 04, 2004 @12:49PM (#9336099) Homepage Journal

      These are the same assholes that were SCO's top investor. Karma to say the least, what goes around comes around.
      • Re:Royal Karma (Score:5, Informative)

        by grozzie2 (698656) on Friday June 04, 2004 @05:28PM (#9339750)
        These are the same assholes that were SCO's top investor.

        This has been widely mis-reported. RBC doesn't make such investments themselves, they act as the broker on behalf of clients, exactly the same as any other brokerage. In the case in question, the client chose to remain anonymous, and the shares in question were purchased by RBC, and then held 'in street name' for assignment to an internal client account. The actual details of who the account holder is, are protected by confidentiality laws, and would only become public information if the client requested certificates of shares issued in thier own name, rather than held by the brokerage in street name on thier behalf. the courts can also order such disclosures, but will only do so if there is a real requirement for said disclosure. In this case, there is no requirement for disclosure.

        RBC has recieved a lot of negative exposure in the linux community simply because they have respected privacy laws. They acted as the broker in the transaction, and held the shares on behalf of a client. I'd commend the bank, in the face of a lot of pressure, never once have they released or leaked to the public the name of the client they are acting on behalf of. This is as it should be.

        The real question in my mind, what individual/corporation outsourced this transaction to Canada, to take advantage of privacy laws that allowed them to do the entire deal anonymously, with the bank acting as the publicly visible broker of record?

    • I don't know if it is really the karma that is to blame. Maybe they were just using SCO software to begin with :).

      We all know it's so good that linux just had to steal from it to become enterprise-ready. RBC surely must've used it!
    • Yup (Score:3, Insightful)

      by Greyfox (87712)
      They have already angered the IS/IT Gods and will not be able to successfully use computers for at least 3 lifetimes. No amount of bailing out of SCO now will save them.

      It makes me wonder; if you piss off enough of the clueful folks in this industry, would they simply not apply at your organization, insuring that the only people your HR department sees are the dregs of the vocational schools? Since HR people can't tell the difference between good IT people and bad, no one would get wind of the situation u

  • by Anonymous Coward on Friday June 04, 2004 @12:42PM (#9336008)
    As an employee of Interplay, I can say that a 1 or 2 day delay in pay is nothing at all to worry about.
  • Coincidence? (Score:2, Interesting)

    by qorkfiend (550713)
    Coincidence, maybe, that England's air traffic control goes down during a software upgrade, and then the same happens to the Royal Bank of Canada?

    Paranoia keeps you healthy!
  • This affects a lot of people - even if you don't bank with Royal Bank but your employer does then you will be affected. The HR manager where I work sent out a bulletin today that should apply to anyone affected by this situation:

    ----
    All financial institution are on line with this issue, mortgage or automatic debit payments, will be honored, should anyone be charged interest , advise your bank,the Royal Bank will refund the charges.

    All financial institution will advance cash based on an employee presenting a pay stub, they will not advance the full amount of the pay stub , they will however provide cash for the weekend.
    ----
    • by eln (21727) on Friday June 04, 2004 @12:59PM (#9336201) Homepage
      What's their limit on "cash for the weekend?" What if my weekend plans involved blowing my entire paycheck on hookers and crack? Or spending the whole time in a casino playing high-stakes poker?

      Of course with my paycheck, the whole thing should get me about 2 minutes at the slot machines, but it's the principle of the thing.
    • I'm pissed as hell! Goddammned incompetent RBC does it again!

      I deposit my cheque May 31st. I make two transfers from my chequing account to my Visa and my Savings. Come today, there's no money in my Chequing account! I look at the transaction history, and there's no trace of the deposit I made, or the transfers. Strangely enough, the money I transfered out of the savings account is still in the chequing account. They were supposed to fix this when? two days ago? I went into the main branch here in Vancouve
  • They lie.. (Score:5, Informative)

    by hookedup (630460) on Friday June 04, 2004 @12:45PM (#9336046)
    I deposited a money order at an RBC branch on the 2nd of june, they told me it would take 12 hours. it still has not been put in my account, same with my pay from today.

    At least I have real reason why my rent is late this month..
    • Darn... (Score:5, Funny)

      by presarioD (771260) on Friday June 04, 2004 @01:28PM (#9336555)
      We finally reached an agreement with this Nigerean wife of an ex-general that she found $10,000,000 in a hidden vault behind the ex'es private toilet bin and agreed to give me 10% if I provide them with my checking account number.

      Darn Canadian Bank, now the whole deal might not go through...
  • Oh no! (Score:5, Interesting)

    by devphaeton (695736) on Friday June 04, 2004 @12:45PM (#9336059)
    had their direct deposits delayed by a day or more."

    Wait till your bank holds onto your payroll checks for 2 weeks.

    Once a bank of mine made an addition mistake, i wrote a pile of checks that all bounced. The bank acknowledged their mistake, and restored funds in my account, but refused to help out with all the check-bouncing fees.

    $25 X 17 Hurray.
    • Re:Oh no! (Score:3, Informative)

      by Anonymous Coward
      Holy crap, $25 an incident, who are you banking with??? You need to escalate your complaint. If your branch manager refuses to budge, contact the district manager, and so on. Eventually, somebody will will be willing to do something and you'll get your jack back.
    • ...would make me wonder how many other customers were victims of this kind of "mistake". They made money off the deal.
    • Did you try suing?
    • Re:Oh no! (Score:2, Insightful)

      by shawn(at)fsu (447153)
      Something like this happened at my company with the direct deposit so I understand you situation but shouldn't you have made sure that you had the money in the account before you wrote the check?

      I know it's a common practice to float a check, you take in to account that the check will take X days via us mail, it will then take x days to post, my paycheck will go in to my account in X - y days so I'll be fine.

      Just because it's common practice doesn't make it right. At some point you have to take responsibi
      • Re:Oh no! (Score:4, Informative)

        by Pedersen (46721) on Friday June 04, 2004 @01:12PM (#9336360) Homepage

        I've got to defend him in this case. He said an addition mistake occurred and was made by the bank. Now, if he does his part, reconciles his account monthly, etc, then he has no reason to believe that things failed to occur as he expected them to. For instance, consider this:


        June 1, he has a balance of $1,200 ready to pay his bills. In fact, he even confirms with the bank that he has this much money, and no outstanding checks/debits to his account. He does so ($700 rent, $150 phone, $50 cable, $75 electric, and $200 groceries to get the whole month's staples). That means he's spent $1,175. Now, the bank does an addition error. Let's say the first check to get to the bank is the cable bill, and their error is to add a 0 at the end of the check, meaning he just paid for $500 worth of cable. Second, in comes the rent, at $700. There's $1,200 out of his account. He has made zero errors here, but he will now be hit with insufficient funds charges for the groceries, and for the electric. And the elctric company and grocery store will also add their own charges on top of it. Some companies will run checks through repeatedly until they clear, incurring further insufficient funds charges from the bank and from the company. Furthermore, the grocery store is likely to stop taking his checks because of this.


        No errors on his part, the bank screwed up, but he gets hit with all the penalties. He did what he was supposed to do, what more could he do?

    • Re:Oh no! (Score:5, Interesting)

      by gmack (197796) <gmack@innerfir e . net> on Friday June 04, 2004 @01:31PM (#9336603) Homepage Journal
      That happened to me once because a teller made a mistake upgrading my account.. money that was supposed to be cleared wasn't.

      After spending 15 minutes tracking down the error.. the bank refunded the NSF fee it charged me and asked me to present reciept for the fee charged by my landlord. The refunded my landlord's fee and provided a letter of appology stating that it was all their fault.

      If your bank refused to do that then I suggest you find another bank.
  • I just hope their programmers aren't unionized. Heads should roll for this one. In cases like this, you should be lucky if you aren't held 100% liable.
  • Ah, Nostalgia... (Score:5, Insightful)

    by ackthpt (218170) on Friday June 04, 2004 @12:48PM (#9336088) Homepage Journal
    Remember back when companies had Q/A departments and procedures? Wrote test plans and tried various scenarios to make sure the software was idiot and bullet proof? When routine software updates didn't suddenly pull your pants down and slap a creme pie into your face? When companies didn't just write any old thing and throw it out there for their customers to actually perform the test?

    Geez, I'm showing my age again...

    • I dont see why this isn't the case nowadays really..

      Sure, some big company gets burnt trying to cut some corners; I'd be willing to bet that the vast majority of companies out there that demand high availability (also the gov't) still maintain policies and procedures for their upgrades.

      Heck, I do. And my employers are rather happy about that.
      • by ackthpt (218170) on Friday June 04, 2004 @01:12PM (#9336357) Homepage Journal
        I dont see why this isn't the case nowadays really..

        It's called "Risk Management"

        Sure, some big company gets burnt trying to cut some corners; I'd be willing to bet that the vast majority of companies out there that demand high availability (also the gov't) still maintain policies and procedures for their upgrades.

        The deal is you have companies which now asses the costs of proper testing verses the cost of defending themsevles against their product blowing up and opt for whichever is cheapest.

        There are companies which must maintain a higher standard, by law or existing contract. Unfortunately the trend I've been watching over the years is an acceptable level of incompetence or defects. Manufacturers of PC parts, f'rinstance, are fine with a 15% failure rate off the line. I couldn't imagine such being acceptable with pacemakers.

        Ironically, most of the PC's in the world run on some version of Windows and even XP still loses its marbles on a regular basis. Thanks to the complexity of some products, some companies simply weigh the rist and make a financial decision and some CYA plan for Image Damage Control -- Gee, sounds just like the war on Iraq, come to think of it, it's a pervasive attitude.

  • by SLot (82781) on Friday June 04, 2004 @12:48PM (#9336089) Homepage Journal
    why they wanted their money back from SCO. ;)
    • by Rupert (28001)
      Someone over at Groklaw [groklaw.net] this morning was claiming that RBoC is the only major Canadian bank not still running OS/2. RBoC uses an OS from a company that's fairly close to Vancouver that is unbeloved of /.ers.
      • by AlexCV (261412)
        Like all the major Canadian banks, the transactional system is on an IBM Mainframe. Running either OS/370 or OS/390. The transactional integrity is handled either with CICS or with custom home-grown software.

        As for OS/2, that was only on ATMs. And ATMs are running anything from VxWorks to System 7 unix to OS/2 to embedded windows.

        Finally, if you're going to cite a comment on a web site, as reliable as that is, please give a reference to it that is more then the URL to the whole site...
  • by iso (87585) <slash@warpzero. i n fo> on Friday June 04, 2004 @12:49PM (#9336094) Homepage
    I've heard a lot about this here in Canada over the past few days. Does anybody know what systems RBC was using, and what upgrade they were doing? It definitely seems suspicious that they were doing an upgrade at the *end* of the month (May), which is the busiest time for a bank (I know, from having worked at one). Was this really an upgrade gone wrong? Are there any more technical details?
  • Internet on every PC (Score:3, Interesting)

    by Anonymous Coward on Friday June 04, 2004 @12:49PM (#9336097)
    I visited RBC earlier this year to make changes to my retirement plan and I was shocked to see that the account manager used a single PC to manage the accounts and access the internet. When I pointed that out, he said "don't worry, we run the best anti-virus software there is" (McAfee by the look of the icon in the tray). Because, as we all know, it's those viruses that eventually steal passwords and break into the databases. *rollseyes*
  • I Need My Pay (Score:3, Interesting)

    by Tr0mBoNe- (708581) on Friday June 04, 2004 @12:50PM (#9336107) Homepage Journal
    I am a Royal Bank customer too... fortunatly my company uses CIBC, so I went down to the bank on my way to work this morning with my paystub and left with my pay, and all my funds from my account. I closed it and gave my financial buddy at work a new account with CIBC.

    Honestly though. Being a software developer and knowing the development cycle like the back of my computer leads me to wonder how in the world they didnt test it fully. I mean... comeon guys. And that kind of institution using SCO's brand of UNIX? face + palm

    Oh well... i dont care anymore... i close the accounts and visa card and when they asked me why, I just said: "I can't trust a bank that can't deal with this kind of glitch."
  • Just like the suits (Score:3, Informative)

    by Marxist Hacker 42 (638312) <seebert42@gmail.com> on Friday June 04, 2004 @12:50PM (#9336112) Homepage Journal
    And people wonder why I've become so incredibly disgusted with standard capitalism: "I owe them a Visa payment, but this glitch hasn't affected their ability to collect the money that's owed to them," Maria Janchenko, 26, said in Toronto, adding that she had been contacted three times this week by a collection agency.

    It's their fault that these people aren't recieving paychecks and they're still hassling them aobut paying bills?
    • You don't get contacted three times by a collection agency for being one day late with your Visa payment.
    • by Dark Nexus (172808) on Friday June 04, 2004 @12:59PM (#9336205)
      You know, if she's been contacted THREE TIMES this week about payment, then she's at LEAST 2 months overdue. Going to blame the other 2+ months on the computer glitch that started on Monday? Besides, their Visa bills aren't actually DUE until about 7 working days into the month, so if she was up to date, then her payment wouldn't even be due yet.

      No, that's just somebody who thinks the world owes them everything taking the opportunity to complain because it might get them something they don't really deserve.
    • What is your point? They don't send these things out to a collection agency until you've been past due for a while. Which means she was already late in paying the bill before the computer glitch happened. Which means that if this computer glitch didn't happen, they would STILL be calling her. Just that now, she has an excuse to make them wait a few more days.
  • This is why (Score:5, Funny)

    by Prince Vegeta SSJ4 (718736) on Friday June 04, 2004 @12:50PM (#9336115)
    I don't keep all of my money in a bank, I've never had the hole in my back yard, under the oak, next to the stream, 5 steps from the bush refuse to give me a withdrawal. Well, except for that one time the stream flooded.

    Sure, I don't earn interest on it, but at least I have some in an emergency

  • by Anonymous Coward on Friday June 04, 2004 @12:51PM (#9336122)
    No, Lou, you're in the unstable branch! Gaah!
  • Double Withdrawl (Score:3, Interesting)

    by CHaN_316 (696929) on Friday June 04, 2004 @12:52PM (#9336130)
    I made my credit card payment via online banking on June 1st, and the transaction went through. However, on June 2nd, the system decided to pay my credit card again a second time. Now I'm down a few hundred bucks.... should be fun getting this sorted out :|
  • I don't typically work on these all-or-nothing type of systems. I usually work on embedded controllers, so we have the ability to put, say 10 units into the field for trials. Given the English air troubles and now this, isn't there a way to deploy a system such as this where it can be tested with real-world loads but not be the only system in use? Our controllers are not placed directly on the assembly line on the first day out, just for this purpose.
  • Banking Hazards (Score:4, Interesting)

    by TaddS (538049) * on Friday June 04, 2004 @12:54PM (#9336146) Homepage
    I recently got an overdraft notice on my bank account, four days after depositing my paycheck, in the branch with a teller. After several days of wrangling with their phone customer service and various managers at the bank I finally found out what had gone wrong: the teller had entered the wrong account number into the computer and someone recieved my money in their account. After several more forms and a couple hours of waiting around in the bank they finally got me my money back, but this was after being without cash or check-card for a week. All this because someone, whos job is to be exact, typoed.

    I'm sure if this had been their money they would've gotten it back in less than 7 days, and levied some hefty time and inconvenience charges.
  • by j0eshm0e (720044) on Friday June 04, 2004 @12:54PM (#9336150)

    junioradmin@rbchost:/> rm -rf core *

    waiting

    waiting

    thinking...this is taking longer than it should

    phone rings.

    ctrl-| ctrl-| ctrl-| ctrl-|

  • Credit damage (Score:5, Interesting)

    by elrick_the_brave (160509) on Friday June 04, 2004 @12:57PM (#9336182)
    I love that.. any reasonable costs. How about the cost of damage to someone's credit when a payment can't go through... are they going to write a million credit apology notices? Are they going to write paper letters so you can keep a copy when someone calls into question your credit? The credit system is very damaging in these cases and has no easy fix.... I recommend all people go their RBC branch and get a letter explaining why payments were missed. Have them give you as many registered copies as you need for all your creditors affected.
  • 16bit! (Score:3, Funny)

    by dfltr (627954) on Friday June 04, 2004 @01:00PM (#9336209) Homepage
    i bank at RBC, and i have no money in my account right now. and it's my girlfriend's birthday. and she hates the present i got her. and my dog got run over by the bank manager... if i had a dog he would have anyways. it turns out that _some_ of the transactions from the weekend actually did go through, like the ones from my account, so when they reapplied everything yesterday i got double-debited for everything and it emptied my account. whee. fun side note: if you walk into a Royal Bank branch, you'll notice that the terminals behind the counter are running 16bit windows apps. check it out frank, we got this great new version, it's For Workgroups!
  • by eltoyoboyo (750015) on Friday June 04, 2004 @01:00PM (#9336212) Journal
    Nice planning: an end-of-month upgrade at a financial institution when, by their own admission, transactions are at their peak.

    Maybe they thought they would broaden their QA testing base to, say 20,000,000.
  • I wonder... (Score:2, Interesting)

    by TedTschopp (244839)
    I wonder if the software was written off shore?
  • by jwsd (718491) on Friday June 04, 2004 @01:02PM (#9336250)
    Why post a bug story on /. if we can't blame it on M$?
  • by aardwolf204 (630780) on Friday June 04, 2004 @01:04PM (#9336269)
    For the longest time I was living from paycheck to paycheck. Compared to my friends I made some pretty good money for being 20 (30K/Yr) but it didnt seem like much when you were living on your own with rent, insurance, car payments, electric, cable, phone, water, and a girlfriend. Its amazing how things add up. It just so happens that I receive the first paycheck of the month on the same day that rent is due. After paying rent I'm left with about $100. It is also convenient that the second and last paycheck of the month is received on the same day that all of the bills are due. After paying them I'm left with $300. Note I havent mentioned the G/F tax yet but that one is expensive.

    Anyway, my tip is, next time you get a bonus, tax returns, some lump sum of money, spend it on next months rent before you can do anything else with it. Trust me on this. If you put it in your savings you can too easily transfer it to checking when you see Wizz-Bang4000 on pricewatch for only $499! I do this every chance I get and it really helps out a lot.

    Now if I could only figure out what to do with the SO.
    • This is insightful? I hope aardwolf was actually trying to be funny. Otherwise his "don't save money because it is too easy to spend savings" plan will have him working until the day he dies. Personally, my SO and I live off of one paycheck and put the other into some form of savings or another. Using this plan, we're aiming at having a million smackers in the bank before we're fifty.

      Better still, this plan doesn't have us up Shit Creek when a paycheck (or twenty) is missed.

      If you have such a serious
  • I'd rather have the couple days delay be on the bank end. Working for a tanking company, where paychecks get delayed further and further till you aren't even sure if you'll get payed next week for the work you're doing now is a much bigger problem. Of course, having it happen right around rent time is a major bummer - but in this case the "It's the bank's fault" excuse is validated.
  • by mwillems (266506) on Friday June 04, 2004 @01:05PM (#9336282) Homepage
    HPUX and AIX, at least on their web servers, and no doubt also on their critical systems.

    See nextcraft: http://uptime.netcraft.com/up/graph?site=www.rbc.c om

    MW
  • by presmike (754040) on Friday June 04, 2004 @01:08PM (#9336311)
    a beer drinking beaver was found at the mainframe keyboard typing I AM CANADIAN over and over. More details to follow...
  • by Various Assortments (781521) on Friday June 04, 2004 @01:08PM (#9336313)
    A few short years ago, the Toronto Dominion's entire network went down for a whole day. I happened to be visiting a friend a train ride away, and could not get money out to get home out of any machine. It was frustrating, and it made me wonder why my service charges hadn't gone to redundant connections and machines, but I was able to borrow some cash and get home.

    But when it happened a second time, in less than a year, I got a little frustrated and switched to president's choice bank.

    My wife uses Royal Bank, and her pay has not gone through yet and it is now more than several days late. I certainly hope they work later than 4pm, monday to friday, to fix this. Some people who were supposed to be paid on the 31st have bounced their rent cheque!
  • by yoha (249396) on Friday June 04, 2004 @01:09PM (#9336316)
    It's makes you wonder what "money" really is, when a software error can make it disappear.
  • by pbailey (225135) on Friday June 04, 2004 @01:11PM (#9336344)
    Can you imagine working in that IT department right now. My first reaction when I saw this story was that I felt incredibly sorry for those IT guys and gals. Must be hell over there right now!

    I know I always sweat when releasing new software, at least I don't have to worry about effecting the bank accounts of millions of people. That would truly be scary!
  • Big Questions (Score:4, Interesting)

    by blueZhift (652272) on Friday June 04, 2004 @01:22PM (#9336464) Homepage Journal
    I just have a few big questions, not that the bank is likely to answer any of them. I suppose the inevitable lawsuits may flush out some of them though.

    1. What OS(s) were they running before this happened?

    2. Were they really doing an upgrade or a crossgrade, that is, switching to a new system altogether?

    3. Was this being handled by in house IT or was it being outsourced half way around the world?

    • Re:Big Questions (Score:5, Interesting)

      by Anonymous Coward on Friday June 04, 2004 @02:01PM (#9337007)
      I work for a financial firm. Not a bank, but we talk with a lot of banks.

      RBC is probably an IBM mainframe environment. OS/390 or similar OS. They probably have AS/400 and AIX hosts as well for other functions (printing, data transfer, etc.) IBM sells you a whole shop and then you're stuck with it.

      The OS is irrelevant, this is most likely an application (transactional database) error. Read the Globe And Mail article.

      You call in the vendor (IBM) to do system upgrades and the like. They are VERY careful. If it went wrong, RBC would be laying the blame on them right now, and publicly.

      Switching between (physical) systems for a mainframe environment is commonplace enough and would be fully supported by IBM. Parallel Sysplex and all that. One of my vendors did it over a weekend without incident - except they didn't wait until month-end to do it! Doh! That was dumb.

      I don't do anything around month-end or quarter-end if I can help it. It's asking for trouble.

      I'm pretty sure RBC does their IT in-house. These tend to be large, customized or internally developed systems, so outsourcing would mean almost certain death to the company. Even moving/consolidating operations between different groups in the same company is very painful.

      Unless they were trying to
      • Re:Big Questions (Score:5, Informative)

        by Ubergrendle (531719) on Friday June 04, 2004 @04:15PM (#9338758) Journal
        You're close to the money re: OS390, and it being a bank problem (not vendor). I work for another canadian bank, but have former associates at RBC. Here's what I've heard is happening:

        - May 31st = month end. new month end code gets run first time in prod (JCL), there's problems - not sure of name of system, but the general bank ledger for customers is fubared.
        - RBC rolls back all changes, so RBC is 1 day behind on Tuesday.
        - re-run batch with old code on Wednesday am... unfortunately recovery procedures are flawed/human error, batch is screwed up again.
        - now bank is 2-3 days behind...can't process transactions effectively, can't catch up with sequential batches in evenings because there's too much to run.
        - RBC departments start running independently based on May 31st data...can't afford to be down more than 2 days. now the roadmap is a mess for recovery, general ledger is still at May 31st state (might be June 1st after a successful batch run last night???)

        Apparently about 80-100 IT staff are living at the Skydome hotel in Toronto working 16 hour shifts (16 on, 8 off) to try and get caught up. Everything i've heard suggests that they know they can't get done during the business week...they're relying on 2 days of 24hr downtime on the weekend to reload the batches and get systems back in sync.

        Based on my experience in Canadian banking (7 years) plus stories of old timers, this is the worst outage in close to 10 years, maybe the worst in 20 if my rumours prove to be true. I have no direct mainframe experience, though, so take my descripton of the problems with a grain of salt...
        • Re:Big Questions (Score:3, Insightful)

          by rborek (563153)
          I guess this begs the question - why was the new code not tested, using live data, on the test/devel machine? In an environment such as this, you can't do half-assed testing - you have to run real, live data into it and test it out (ie one month before, duplicate the data - if it isn't already duplicating it real-time into the test machine, which I would expect - and run the code. Any problems, wait one month and try again).

          This comes down to poor testing. The manager(s) involved should (and probably will

    • Re:Big Questions (Score:3, Informative)

      by Alomex (148003)
      1. What OS(s) were they running before this happened?

      This question has "newbie" written all over. Bank applications run on true and tested OSes written decades ago. They run on large mainframes with uptime that often goes back to the day the computer was turned on sometime in the 70s or 80s.

      This problem is, in all likelihood, an application problem.

  • Expert guesses? (Score:3, Interesting)

    by phizman (742537) on Friday June 04, 2004 @01:33PM (#9336633)
    Who are these clowns that the media talk to for their "expert" opinion on computers?

    "George Geczy, a software developer and computer consultant based in Ancaster, Ont., guessed that the problem involves identification numbers assigned to transactions"

    Thousands of different reasons why their system cratered and some guy running a consulting firm from his basement nailed it for us! Guess his experience in installing MySQL a couple times helped him diagnose their massively huge database issue.

    Just because you have a IT job and a bank card, doesn't make you an expert.
  • by Graemee (524726) on Friday June 04, 2004 @01:39PM (#9336725)
    As someone effected by the mess, I have good news, My pay is in the bank. No news whether they "fixed" the problem or used a work around, which is more likely IMHO.
  • my take: (Score:3, Insightful)

    by jafac (1449) on Friday June 04, 2004 @03:11PM (#9337995) Homepage
    Couldn't happen to a nicer, more deserving bunch of jerkwads.

    (/former field-support rep for a vendor, who got *burned* by the incompetence and mendacity of RBC IT personnel who lied to their manager, and my manager, when THEY screwed up their evaluation of our product - AFTER they had dragged the evaluation out past the 12-month mark. . . how the hell do you justify evaluating a product for 12 months?).
  • by Tripster (23407) on Friday June 04, 2004 @04:24PM (#9338852) Homepage
    One thing I've never understood about bank computers, why is it they don't do transactions outside banking hours when it involves putting money INTO your account?

    Sure I can make a deposit at an ATM and have it instantly accessable, but what I am talking about is between bank transfers and such, for some reason these seem to take place only on weekdays.

    Is there really some peon sitting in front of a terminal approving every transaction?

    What peeves me is when I have something being transferred from say my merchant account, it can take 5 business days to get to my account, I mean these are computers we're talking about here and that type of delay really makes no sense when we live in a world where instant transactions are available.

    I've seen stuff start transfer on a Wednesday and take until the following Tuesday to show in the account, that is just sad.

"If I do not want others to quote me, I do not speak." -- Phil Wayne

Working...