Catch up on stories from the past week (and beyond) at the Slashdot story archive

 



Forgot your password?
typodupeerror
×
Bug The Almighty Buck

Royal Bank of Canada Software Upgrade Goes Awry 602

Reader mks113 writes "Many Canadians living payday to payday have been in for a shock this week. Canada.com along with many other sources is reporting how thousands of customers have been inconvenienced following an unsuccessful software upgrade at the Royal Bank of Canada on Monday. All government employees (including me) in several provinces had their direct deposits delayed by a day or more." RBC has a comment on the mess.
This discussion has been archived. No new comments can be posted.

Royal Bank of Canada Software Upgrade Goes Awry

Comments Filter:
  • Sticky karma.. (Score:5, Insightful)

    by the_rajah ( 749499 ) * on Friday June 04, 2004 @12:41PM (#9335999) Homepage
    I guess that bad karma is pretty sticky. Even selling their preferred A-1 shares to Baystar didn't save them.

    My Canadian friends are screaming bloody murder. I don't blame them.

    "Do the Right Thing. It will gratify some people and astound the rest." - Mark Twain
  • Re:Coincidence? (Score:5, Insightful)

    by Sique ( 173459 ) on Friday June 04, 2004 @12:44PM (#9336044) Homepage
    Now you know where the old wisdom "Never touch a running system" comes from.
  • Ah, Nostalgia... (Score:5, Insightful)

    by ackthpt ( 218170 ) on Friday June 04, 2004 @12:48PM (#9336088) Homepage Journal
    Remember back when companies had Q/A departments and procedures? Wrote test plans and tried various scenarios to make sure the software was idiot and bullet proof? When routine software updates didn't suddenly pull your pants down and slap a creme pie into your face? When companies didn't just write any old thing and throw it out there for their customers to actually perform the test?

    Geez, I'm showing my age again...

  • by iso ( 87585 ) <slash@warpze[ ]info ['ro.' in gap]> on Friday June 04, 2004 @12:49PM (#9336094) Homepage
    I've heard a lot about this here in Canada over the past few days. Does anybody know what systems RBC was using, and what upgrade they were doing? It definitely seems suspicious that they were doing an upgrade at the *end* of the month (May), which is the busiest time for a bank (I know, from having worked at one). Was this really an upgrade gone wrong? Are there any more technical details?
  • Comment removed (Score:5, Insightful)

    by account_deleted ( 4530225 ) on Friday June 04, 2004 @12:49PM (#9336098)
    Comment removed based on user account deletion
  • by BigGar' ( 411008 ) on Friday June 04, 2004 @12:50PM (#9336104) Homepage
    to cut back your lifestyle and save a bit. Even a dollar or two a day can be helpfull over the long haul. Also, if you think that you can't possibly save anything, then you're not trying, you just need to determine what your priorities are.
  • by gmletzkojr ( 768460 ) <gmletzkojr@NOSpaM.gmail.com> on Friday June 04, 2004 @12:53PM (#9336142) Homepage Journal
    I don't typically work on these all-or-nothing type of systems. I usually work on embedded controllers, so we have the ability to put, say 10 units into the field for trials. Given the English air troubles and now this, isn't there a way to deploy a system such as this where it can be tested with real-world loads but not be the only system in use? Our controllers are not placed directly on the assembly line on the first day out, just for this purpose.
  • by aardvarkjoe ( 156801 ) on Friday June 04, 2004 @12:54PM (#9336147)
    You don't get contacted three times by a collection agency for being one day late with your Visa payment.
  • by Dark Nexus ( 172808 ) on Friday June 04, 2004 @12:59PM (#9336205)
    You know, if she's been contacted THREE TIMES this week about payment, then she's at LEAST 2 months overdue. Going to blame the other 2+ months on the computer glitch that started on Monday? Besides, their Visa bills aren't actually DUE until about 7 working days into the month, so if she was up to date, then her payment wouldn't even be due yet.

    No, that's just somebody who thinks the world owes them everything taking the opportunity to complain because it might get them something they don't really deserve.
  • by eltoyoboyo ( 750015 ) on Friday June 04, 2004 @01:00PM (#9336212) Journal
    Nice planning: an end-of-month upgrade at a financial institution when, by their own admission, transactions are at their peak.

    Maybe they thought they would broaden their QA testing base to, say 20,000,000.
  • The scary thing (Score:1, Insightful)

    by Anonymous Coward on Friday June 04, 2004 @01:01PM (#9336225)
    Is that after several days they still don't know what the problem is!

    "The bank says millions of transactions, from direct pay deposits to bill payments, have been affected by an unknown computer error. "

    This could be anything from a simple bug to a disgruntled programmer.. industrial espionage.. etc.
  • Re:Oh no! (Score:2, Insightful)

    by shawn(at)fsu ( 447153 ) on Friday June 04, 2004 @01:01PM (#9336229) Homepage
    Something like this happened at my company with the direct deposit so I understand you situation but shouldn't you have made sure that you had the money in the account before you wrote the check?

    I know it's a common practice to float a check, you take in to account that the check will take X days via us mail, it will then take x days to post, my paycheck will go in to my account in X - y days so I'll be fine.

    Just because it's common practice doesn't make it right. At some point you have to take responsibility, you singed your name to that check and said that at from the date on this check forward I have the funds in my account to cover it.

    I'm not trying to be an troll or anything, I just think that you share a part of the blame for your checks bouncing.
  • by Pxtl ( 151020 ) on Friday June 04, 2004 @01:02PM (#9336237) Homepage
    No, this is management's fault. The only time a programmer fails is if something wasn't delivered on time, or they just don't produce, or their stuff doesn't make it past QC. Then fire them.

    If bad code makes it into the wild, then somebody signed off on it. Somebody cut corners on testing. Somebody decided deadline is more important than quality. Somebody insisted it had to run the newest Microsoft code.

    That somebody is the programmer's boss.
  • by Rude-Boy ( 25678 ) on Friday June 04, 2004 @01:02PM (#9336246)
    I'd bet on managers pushing through the update before it was ready.
  • by GoofyBoy ( 44399 ) on Friday June 04, 2004 @01:02PM (#9336248) Journal
    >Heads should roll for this one.

    But are the "correct" heads going to roll?

    >In cases like this, you should be lucky if you aren't held 100% liable.

    If you were suppose to be held liable, do you think anything would change? Were any Professional Engineers held liable for the big blackout last year?
  • by lordgarth ( 581960 ) on Friday June 04, 2004 @01:03PM (#9336257)
    These kind of things do happen quite often if the banking industry is examined as a whole. If it tends to happen to an individual bank on a recurring basis then the depositors should move their money. I myself have been in the position of living paycheck to paycheck in the past and I've nobody to blame for it but myself. The bank is at fault but if someone cannot survive a couple of days without their deposit going in then they should seriously look at how they manage money. If an unforseen emergency arises then they are going to find themselves in dire straits.
  • by Bull999999 ( 652264 ) on Friday June 04, 2004 @01:03PM (#9336262) Journal
    I totally agree. Most experts recommand that you have three to six months worth of funds saved up, which means that delay of payroll check deposit of a day, or even couple of weeks should be non-issue.
  • by Suit_N_Tie ( 128024 ) on Friday June 04, 2004 @01:04PM (#9336266)
    It's only Canadian money.
  • by aardwolf204 ( 630780 ) on Friday June 04, 2004 @01:04PM (#9336269)
    For the longest time I was living from paycheck to paycheck. Compared to my friends I made some pretty good money for being 20 (30K/Yr) but it didnt seem like much when you were living on your own with rent, insurance, car payments, electric, cable, phone, water, and a girlfriend. Its amazing how things add up. It just so happens that I receive the first paycheck of the month on the same day that rent is due. After paying rent I'm left with about $100. It is also convenient that the second and last paycheck of the month is received on the same day that all of the bills are due. After paying them I'm left with $300. Note I havent mentioned the G/F tax yet but that one is expensive.

    Anyway, my tip is, next time you get a bonus, tax returns, some lump sum of money, spend it on next months rent before you can do anything else with it. Trust me on this. If you put it in your savings you can too easily transfer it to checking when you see Wizz-Bang4000 on pricewatch for only $499! I do this every chance I get and it really helps out a lot.

    Now if I could only figure out what to do with the SO.
  • by pbailey ( 225135 ) on Friday June 04, 2004 @01:11PM (#9336344)
    Can you imagine working in that IT department right now. My first reaction when I saw this story was that I felt incredibly sorry for those IT guys and gals. Must be hell over there right now!

    I know I always sweat when releasing new software, at least I don't have to worry about effecting the bank accounts of millions of people. That would truly be scary!
  • by ackthpt ( 218170 ) on Friday June 04, 2004 @01:12PM (#9336357) Homepage Journal
    I dont see why this isn't the case nowadays really..

    It's called "Risk Management"

    Sure, some big company gets burnt trying to cut some corners; I'd be willing to bet that the vast majority of companies out there that demand high availability (also the gov't) still maintain policies and procedures for their upgrades.

    The deal is you have companies which now asses the costs of proper testing verses the cost of defending themsevles against their product blowing up and opt for whichever is cheapest.

    There are companies which must maintain a higher standard, by law or existing contract. Unfortunately the trend I've been watching over the years is an acceptable level of incompetence or defects. Manufacturers of PC parts, f'rinstance, are fine with a 15% failure rate off the line. I couldn't imagine such being acceptable with pacemakers.

    Ironically, most of the PC's in the world run on some version of Windows and even XP still loses its marbles on a regular basis. Thanks to the complexity of some products, some companies simply weigh the rist and make a financial decision and some CYA plan for Image Damage Control -- Gee, sounds just like the war on Iraq, come to think of it, it's a pervasive attitude.

  • by tokaok ( 623635 ) on Friday June 04, 2004 @01:13PM (#9336370)
    money is work that is all, money is just a means to represent work. instead of trading my cow for your virgin daughter. all we are trading is for the value of me raising my cow and yourself your daughter.
  • Re:Banking Hazards (Score:5, Insightful)

    by Corporate Gadfly ( 227676 ) on Friday June 04, 2004 @01:13PM (#9336371)
    I finally found out what had gone wrong: the teller had entered the wrong account number into the computer and someone recieved my money in their account.
    Hate to be a troll, but if you go to the bank teller, always ask them for a receipt and double-check your account numbers on that receipt. Yes, the teller shoulders some of the responsibility, but so do you.
  • Re:I wonder... (Score:5, Insightful)

    by tool462 ( 677306 ) on Friday June 04, 2004 @01:33PM (#9336625)
    Yeah, it was probably written in the US :)
  • Re:Banking Hazards (Score:4, Insightful)

    by CarrionBird ( 589738 ) on Friday June 04, 2004 @01:38PM (#9336691) Journal
    I would disagree. Yes you should double check, but doing thier job is 100% thier responsibility.
  • This is insightful? I hope aardwolf was actually trying to be funny. Otherwise his "don't save money because it is too easy to spend savings" plan will have him working until the day he dies. Personally, my SO and I live off of one paycheck and put the other into some form of savings or another. Using this plan, we're aiming at having a million smackers in the bank before we're fifty.

    Better still, this plan doesn't have us up Shit Creek when a paycheck (or twenty) is missed.

    If you have such a serious problem with raiding your savings account direct deposit can be a great tool for you. Have a small chunk of each check sent into this savings account and never touch it. Never ever. Hell, you'd probably be better off if you didn't even open your bank statements for that account but once a year. Whatever you do, living month to month is not the answer.

    Peter
  • Yup (Score:3, Insightful)

    by Greyfox ( 87712 ) on Friday June 04, 2004 @01:42PM (#9336764) Homepage Journal
    They have already angered the IS/IT Gods and will not be able to successfully use computers for at least 3 lifetimes. No amount of bailing out of SCO now will save them.

    It makes me wonder; if you piss off enough of the clueful folks in this industry, would they simply not apply at your organization, insuring that the only people your HR department sees are the dregs of the vocational schools? Since HR people can't tell the difference between good IT people and bad, no one would get wind of the situation until such time as there were a major failure in your systems brought on by your shitty IT department. Makes you stop and think, doesn't it?

  • by clart ( 685785 ) on Friday June 04, 2004 @01:44PM (#9336793)
    Try being fresh out of college. How the hell are you supposed to have 6 months of funds saved up when you've only been working full-time for 2 or 3? I'm sure that will teach those pesky kids trying to pay their rent and student loans for not saving anything up. Bottom line not everyone has that luxury.
  • by Anonymous Coward on Friday June 04, 2004 @01:47PM (#9336831)
    I'm one of the Canadians affected - the paycheque I deposited on June 1st disappeared from the system after I had made a withdrawal and a credit card payment, leaving my account in the negative.

    Fortunately, I've made it a habit to keep cash on myself at all times (hence the withdrawal), so I'll weather this glitch okay.

    A lot of Canadians rely completely on electronic cash (debit cards) for all their purchases. I used to be the same way, until I realized that the only ways the debit card had advanced our society was by:
    1. Making store checkout lines twice as slow, as every Dick and Harry tries four debit cards and two credit cards looking for an account with money.
    2. Making it easy to overspend because you lose track of your account balances.
    3. Leaving you cashless and stranded every time there's a network glitch, a power outage, a bank error, or what-have-you. This happens frequently, and never at a good time.

    So now, thanks to the remarkable invention of cash (physical money), I am no longer dependant on the workings of computer systems I have no control over!
  • Re:Instability? (Score:2, Insightful)

    by rainman_bc ( 735332 ) on Friday June 04, 2004 @01:48PM (#9336842)
    I've been saying that to IT people for years. While security is important, having a sound disaster recover plan is even more important. Your system cannot be 100% secure; it's practically impossible. It can be 99.99999999% secure, but that one time can still happen, and you had better be prepared for it.

    I've seen too many IT people focus entirely on security, and forget about having a sound backup strategy. Those IT people need to get fired. Sure, playing with Arcserve or Backup Exec or whatever sux, compared to finding some new hash algorithm to further encrypt your ssh session, and let's face it, we all like learning new things, but IT isn't about that kinda stuff.

    I've had yelling matches at my last employer with IT people; they insisted on not having ANY FTP server, were farting around with NDS, yet when we asked them for a restore from tape, every time there was an issue. They had a 100% failure rate on their backup tapes.

    There's too many retarded IT people out there. and too many retarded IT managers out there too.
  • by Anonymous Coward on Friday June 04, 2004 @02:01PM (#9337005)
    Oh, excellent economic analysis.

    Really, the only factor is whether the US is at war or not? Really!? Really, really!??

    Let's see ... seems to me Canada was in WWI and WWII as well, rendering your analysis on even the most facile level moot and not a little idiotic.
  • by Jtheletter ( 686279 ) on Friday June 04, 2004 @02:03PM (#9337032)
    Most experts recommand that you have three to six months worth of funds saved up

    So what do these "experts" recommend you spend during those 3 to 6 months?

    Honestly, if everyone had the ability to just save up 6 months worth of funds then no one would be bouncing checks or investing in short term disability insurance. I had bills before I had a job, no one gave me a grace period to get together some emergency savings.

    I save a pretty good amount, 20% of each check goes straight to savings, but it'll still take me most of a year to get 3 months of backup funds. Someone living on a more hand-to-mouth salary may never get that much saved.

  • by Webapprentice ( 608832 ) on Friday June 04, 2004 @02:09PM (#9337109)
    Since when does having a girlfriend mean having a money pit? It sounds like the girlfriend is more interested in material goods than you or you are out to make a big, but expensive impression on her.
  • by thelexx ( 237096 ) on Friday June 04, 2004 @02:41PM (#9337562)
    See my sig...
  • by GordoTheGeek ( 608960 ) <gordon.cruachan@ca> on Friday June 04, 2004 @02:59PM (#9337822)
    Stop! My Sides! You're killing me! Both you and the other moron are letting your ignorance show. You have quite succinctly [dict.org] demonstrated exactly what it is that I and a very large portion of the world do not like about what, unfortunately, passes for representative Americans. Bravo

    It's quite sad, really: any disagreement with you automatically means that I don't like Americans and any and all arguments can and will result in violence, in this case coupled with a strange assortment of insults, what I can only guess is a swipe at my sexuality, and a comment that shows your profound misunderstanding of geography. I must, however, point out that your message, such as it is, is getting garbled by your obvious mastery of grammar and spelling.

  • by Clod9 ( 665325 ) on Friday June 04, 2004 @03:04PM (#9337890) Journal
    This advice isn't bad as far as it goes, but it only gets you a one-month buffer.
    That's a highly desirable breathing space but still leaves you shackled to your paycheck. If you can learn discipline -- where "savings" means "that stack of money that keeps growing and that I will never touch unless my child is dying", you will be FREE.

    FREE, to take a six-month leave of absence to do something that's important to you.
    FREE, to quit the job that is making you ill with stress, even though you have no prospects at this time.
    FREE, where your boss and your company's CFO and any of the financial institutions you keep your money in -- all these have NO SAY in your life, except as far as you wish them to.

    You can't protect against everything, but 30K ought to be enough to get ahead. I started out at 15K a year in 1987, and gave some of that away to charities. I know what humble beginnings are like. ESCAPE THEM!

  • by pmsr ( 560617 ) on Friday June 04, 2004 @03:04PM (#9337901)
    Mate, according to the CIA World Factbook, Canada is "somewhat larger than the US". So, you have a neighbour country, you don't know it's size, say it is "probably" smaller than some random state in your country, and have the nerve to insult others in such a rash tone? I have to tell you this: american or not, you are a sorry excuse for a human being, and the world doesn't really need your kind. Well, maybe your mother does, but then again, what would be of some people without their moms?

    /Pedro

  • my take: (Score:3, Insightful)

    by jafac ( 1449 ) on Friday June 04, 2004 @03:11PM (#9337995) Homepage
    Couldn't happen to a nicer, more deserving bunch of jerkwads.

    (/former field-support rep for a vendor, who got *burned* by the incompetence and mendacity of RBC IT personnel who lied to their manager, and my manager, when THEY screwed up their evaluation of our product - AFTER they had dragged the evaluation out past the 12-month mark. . . how the hell do you justify evaluating a product for 12 months?).
  • by RetroGeek ( 206522 ) on Friday June 04, 2004 @03:32PM (#9338234) Homepage
    It's makes you wonder what "money" really is,

    Money is a concept built on mutual trust. I trust that the money you give me will be honored at its face value in another place.

    Be it dollars, euros, gold, or matchsticks.

    Money, as such, is meaningless without that trust.
  • by FreeUser ( 11483 ) on Friday June 04, 2004 @03:54PM (#9338519)
    Apologies for this offtopic post (soon to be modded -99 by those Republicans with moderator priveleges no doubt), but this comment, and the toxic meme it propogates, needs to be addressed.

    The USA would be a lot better place if you Democratic nutjobs would just let go

    No, it wouldn't.

    And I say this as one who did not vote for Gore, and who votes the issues, not the party (which means I vote as often Republican as I do Democratic ... although after this past four years that could very well change).

    Most people did what you advocated, and the results for America have been disasterous. We have overextended our military, spent ourselves into a deficit we may never recover from, lost virtually all of our world prestige, the vast majority of our close allies (alliances that had lasted more than half a century burned up in just four years of GW Bush's rule), and all of our credibility in the rest of the world. The list goes on, but I think you probably get the point.

    Indeed, the USA would be a better place if people had taken to the streets or risen up in outright revolt after the Republicans stole the election and discarded the expressed will of the American people, as it was cast at the ballot office (including Florida, which a full statewide recount sponsored by the media demonstrated conclusively that Gore had won. As an aside, it is interesting how the domestic media then unreported and spun their own study to favor Bush, while their overseas collegues reported it more accurately. Go figure.)

    Undermining the democratic process, as the Republicans and their supreme court appointees did in 2000, is terribly destructive irrespective of the qualifications of the usurpur who governs thereafter. The damage to our institutions is quite severe and will be quite long lasting, not least of which because the last branch of our government that had, to all appearances, remained somewhat unsullied by politics, namely the Judicial, has shown itself at its highest levels to be more interested in paying back political favors than ruling sensibly on constitutional law. (Yes, I've actually read their decision. Have you? It is the most convulated series of transparent justifications for violating the intent and will of the constitution and the American voters I've ever read ... and that is saying a lot.)

    Of course, in this case we've ended up with an incompetent usurpur to boot, who used his artificially inflated legitimacy post 9/11 to prosecute an unrelated war against his family's enemy in Iraq, thereby overextending the US military and quite possibly costing us a victory in the real war we should be fighting, namely the War on Terror (hint: Afghanistan, NOT Iraq, although Bush's activities have certainly made Iraq from an inhospitable secular environment for Al Q'aide into a veritable breeding ground and ideal staging area ... thanks Mr. President).

    - lose the rhetoric and actually address issues people are concerned about. If that had actually happened in 2000 Al Gore would be president right now.

    "People" are concerned about the economy and how much gasoline for the SUVs costs. We have very systematically and very effectively dumbed down our population, to the point where an election can be stolen out from under their noses and their main concern will be "but will this preempt my Saturday afternoon sports, Friends, or Everybody Loves Raymond?"

    Gore wasn't my choice, and had I been selecting the party nominee Kerry probably wouldn't have been my choice either. Both are vastly more competent than the current usurpur, however, but more importantly than that, we need a return to constitutional law in this country.

    That means an end to midnight raids on people's homes, an end to detention without due process, an end to dismissing and violating the Geneva conventions, an end to operating concentration camps of any kind, including Guantanamo, even if it is aimed at scum like
  • by AviLazar ( 741826 ) on Friday June 04, 2004 @04:15PM (#9338756) Journal
    On behalf of the United States, I would like to appologize to anyone who was offended by the moronic poster who decided to insult our Northern neighbor, as well as assume that Canada is smaller then Texas. To the insulting poster - buy a globe, buy a hammer. Study the globe, and smash your keyboard so you can't type anymore.
  • by Anonymous Coward on Friday June 04, 2004 @04:45PM (#9339193)
    No apology on his behalf is necessary. We appreciate the sentiment, but we do realize that most Americans aren't that stupid.
  • by fishbowl ( 7759 ) on Friday June 04, 2004 @05:14PM (#9339573)
    >If you can learn discipline -- where "savings"
    >means "that stack of money that keeps growing and
    >that I will never touch unless my child is dying"

    That's a good strategy and everything, but I learned that it would be better to go ahead and let that savings account go to about zero, if I can use it to get completely out of debt.

    If I had a "child is dying" incident, I'm sure it will mean going into debt anyway, and it can't hurt to have zero debt in a situation like that.
  • Re:Big Questions (Score:3, Insightful)

    by rborek ( 563153 ) on Friday June 04, 2004 @06:41PM (#9340448)
    I guess this begs the question - why was the new code not tested, using live data, on the test/devel machine? In an environment such as this, you can't do half-assed testing - you have to run real, live data into it and test it out (ie one month before, duplicate the data - if it isn't already duplicating it real-time into the test machine, which I would expect - and run the code. Any problems, wait one month and try again).

    This comes down to poor testing. The manager(s) involved should (and probably will) be fired - they would have had to have signed off on the code change, and odds are that they rushed the programmers/QA people through testing.

And it should be the law: If you use the word `paradigm' without knowing what the dictionary says it means, you go to jail. No exceptions. -- David Jones

Working...