Royal Bank of Canada Software Upgrade Goes Awry 602
Reader mks113 writes "Many Canadians living payday to payday have been in for a shock this week. Canada.com along with many other sources is reporting how thousands of customers have been inconvenienced following an unsuccessful software upgrade at the Royal Bank of Canada on Monday. All government employees (including me) in several provinces had their direct deposits delayed by a day or more." RBC has a comment on the mess.
Wait a minute... (Score:3, Funny)
Re:Wait a minute... (Score:4, Funny)
Re:Wait a minute... (Score:4, Funny)
So that'd be a no, then ^_~
Re:Wait a minute... (Score:5, Informative)
Here's a little history for y'all. In 1862, the loonie and the buck were worth about the same thing. Two years later, in 1864, the loonie was worth 265% of a buck. That's right, people could buy $2.65 of american money for a single canadian dollar. After the civil war, the buck slowly went back up to match the loonie.
The loonie began to lose value compared to the buck at the beginning of WWI, then slowly restabilized after, then lost value again during WWII. In 1945, right after the war, the loonie and the buck regained their equality.
In 1961, both were worth about the same, and in 1972 the loonie was worth about 5% more than the buck.
At the beginning of 2003, the loonie was worth slightly more than 60% of the buck, since then, it got a big boost and almost hit 80% to be at 75% today...
Notice how all those major fluctuations coincides with wars... Civil war, WWI, WWII, Viet-Nam, Irak... Re-elect Bush and see what happens to your buck...
Re:Wait a minute... (Score:5, Informative)
Dream on.
The canadian looney is artificially kept "low" by toying with the interest rates index to favor US exports, wich is the buyer of 90% of our exports.
Only once has the canadian dollar was worth more then the US dollar, and it proved disastrous for the economy.
Re:Wait a minute... (Score:3, Funny)
Re:Wait a minute... (Score:5, Insightful)
It's quite sad, really: any disagreement with you automatically means that I don't like Americans and any and all arguments can and will result in violence, in this case coupled with a strange assortment of insults, what I can only guess is a swipe at my sexuality, and a comment that shows your profound misunderstanding of geography. I must, however, point out that your message, such as it is, is getting garbled by your obvious mastery of grammar and spelling.
Re:Wait a minute... (Score:5, Insightful)
Re:Wait a minute... (Score:5, Insightful)
Sticky karma.. (Score:5, Insightful)
My Canadian friends are screaming bloody murder. I don't blame them.
"Do the Right Thing. It will gratify some people and astound the rest." - Mark Twain
Comment removed (Score:5, Funny)
Re:Royal Karma (Score:5, Informative)
This has been widely mis-reported. RBC doesn't make such investments themselves, they act as the broker on behalf of clients, exactly the same as any other brokerage. In the case in question, the client chose to remain anonymous, and the shares in question were purchased by RBC, and then held 'in street name' for assignment to an internal client account. The actual details of who the account holder is, are protected by confidentiality laws, and would only become public information if the client requested certificates of shares issued in thier own name, rather than held by the brokerage in street name on thier behalf. the courts can also order such disclosures, but will only do so if there is a real requirement for said disclosure. In this case, there is no requirement for disclosure.
RBC has recieved a lot of negative exposure in the linux community simply because they have respected privacy laws. They acted as the broker in the transaction, and held the shares on behalf of a client. I'd commend the bank, in the face of a lot of pressure, never once have they released or leaked to the public the name of the client they are acting on behalf of. This is as it should be.
The real question in my mind, what individual/corporation outsourced this transaction to Canada, to take advantage of privacy laws that allowed them to do the entire deal anonymously, with the bank acting as the publicly visible broker of record?
Re:Sticky karma.. (Score:2)
We all know it's so good that linux just had to steal from it to become enterprise-ready. RBC surely must've used it!
Yup (Score:3, Insightful)
It makes me wonder; if you piss off enough of the clueful folks in this industry, would they simply not apply at your organization, insuring that the only people your HR department sees are the dregs of the vocational schools? Since HR people can't tell the difference between good IT people and bad, no one would get wind of the situation u
That's nothing. (Score:5, Funny)
Coincidence? (Score:2, Interesting)
Paranoia keeps you healthy!
Re:Coincidence? (Score:5, Insightful)
Re:Coincidence? (Score:4, Funny)
Re:Coincidence? (Score:3, Funny)
Affects not just RB customers (Score:5, Informative)
----
All financial institution are on line with this issue, mortgage or automatic debit payments, will be honored, should anyone be charged interest , advise your bank,the Royal Bank will refund the charges.
All financial institution will advance cash based on an employee presenting a pay stub, they will not advance the full amount of the pay stub , they will however provide cash for the weekend.
----
Re:Affects not just RB customers (Score:5, Funny)
Of course with my paycheck, the whole thing should get me about 2 minutes at the slot machines, but it's the principle of the thing.
Re:Affects not just RB customers (Score:2, Interesting)
I deposit my cheque May 31st. I make two transfers from my chequing account to my Visa and my Savings. Come today, there's no money in my Chequing account! I look at the transaction history, and there's no trace of the deposit I made, or the transfers. Strangely enough, the money I transfered out of the savings account is still in the chequing account. They were supposed to fix this when? two days ago? I went into the main branch here in Vancouve
They lie.. (Score:5, Informative)
At least I have real reason why my rent is late this month..
Darn... (Score:5, Funny)
Darn Canadian Bank, now the whole deal might not go through...
Oh no! (Score:5, Interesting)
Wait till your bank holds onto your payroll checks for 2 weeks.
Once a bank of mine made an addition mistake, i wrote a pile of checks that all bounced. The bank acknowledged their mistake, and restored funds in my account, but refused to help out with all the check-bouncing fees.
$25 X 17 Hurray.
Re:Oh no! (Score:3, Informative)
Re:Oh no! (Score:3, Informative)
TD Canada Trust is another big one. The Dutch bank ING has been doing a lot of business in Canada in the last few years too.
Yay socialism.
What does this have to do with socialism? Hell, if Canada were more capitalistic, the government would have allowed the major banks to merge, as they've been asking to for years, and there would
You missed some (Score:3, Informative)
As someone pointed out, you missed TD Canada Trust (until recently two banks). There's also National Bank, and Laurentian (which again was recently acquired by another bank, but still has some locations open under that name).
And we even get stuff like Ethical Funds [ethicalfunds.com]. Who aren't even the one I was looking for that offers a similar service.
Credit Union Networks (Score:5, Informative)
In my case one CU is an account I've had since I was in my teens but their offices are about 15 miles away, and the other is from my former employer and is two states away. I drop in at the local one sometimes since it's about a mile from my girlfriend's office, but if I'm not headed up that direction I can go to the local credit union about a mile and a half from my house.
Re:Their refusal to remove the fees... (Score:2)
Re:Oh no! (Score:2)
Re:Oh no! (Score:2, Insightful)
I know it's a common practice to float a check, you take in to account that the check will take X days via us mail, it will then take x days to post, my paycheck will go in to my account in X - y days so I'll be fine.
Just because it's common practice doesn't make it right. At some point you have to take responsibi
Re:Oh no! (Score:4, Informative)
I've got to defend him in this case. He said an addition mistake occurred and was made by the bank. Now, if he does his part, reconciles his account monthly, etc, then he has no reason to believe that things failed to occur as he expected them to. For instance, consider this:
June 1, he has a balance of $1,200 ready to pay his bills. In fact, he even confirms with the bank that he has this much money, and no outstanding checks/debits to his account. He does so ($700 rent, $150 phone, $50 cable, $75 electric, and $200 groceries to get the whole month's staples). That means he's spent $1,175. Now, the bank does an addition error. Let's say the first check to get to the bank is the cable bill, and their error is to add a 0 at the end of the check, meaning he just paid for $500 worth of cable. Second, in comes the rent, at $700. There's $1,200 out of his account. He has made zero errors here, but he will now be hit with insufficient funds charges for the groceries, and for the electric. And the elctric company and grocery store will also add their own charges on top of it. Some companies will run checks through repeatedly until they clear, incurring further insufficient funds charges from the bank and from the company. Furthermore, the grocery store is likely to stop taking his checks because of this.
No errors on his part, the bank screwed up, but he gets hit with all the penalties. He did what he was supposed to do, what more could he do?
Re:Oh no! (Score:3, Informative)
Re:Oh no! (Score:5, Interesting)
After spending 15 minutes tracking down the error.. the bank refunded the NSF fee it charged me and asked me to present reciept for the fee charged by my landlord. The refunded my landlord's fee and provided a letter of appology stating that it was all their fault.
If your bank refused to do that then I suggest you find another bank.
Somebody should get fired (Score:2, Interesting)
Re:Somebody should get fired (Score:4, Interesting)
I hope their programmers ARE unionized. If not, you can bet who the scapegoats will be, regardless of whether they are actually to blame.
wtf are you talking about. (Score:4, Interesting)
Really, there's all kinds of blame to go around, and programmers deserve some of it, the system never should have been so brittle as to cause these kinds of problems in the first place.
Re:Somebody should get fired (Score:5, Insightful)
If bad code makes it into the wild, then somebody signed off on it. Somebody cut corners on testing. Somebody decided deadline is more important than quality. Somebody insisted it had to run the newest Microsoft code.
That somebody is the programmer's boss.
Re:Somebody should get fired (Score:2, Insightful)
Re:Somebody should get fired (Score:5, Insightful)
But are the "correct" heads going to roll?
>In cases like this, you should be lucky if you aren't held 100% liable.
If you were suppose to be held liable, do you think anything would change? Were any Professional Engineers held liable for the big blackout last year?
Ah, Nostalgia... (Score:5, Insightful)
Geez, I'm showing my age again...
Re:Ah, Nostalgia... (Score:2)
Sure, some big company gets burnt trying to cut some corners; I'd be willing to bet that the vast majority of companies out there that demand high availability (also the gov't) still maintain policies and procedures for their upgrades.
Heck, I do. And my employers are rather happy about that.
Re:Ah, Nostalgia... (Score:4, Insightful)
It's called "Risk Management"
Sure, some big company gets burnt trying to cut some corners; I'd be willing to bet that the vast majority of companies out there that demand high availability (also the gov't) still maintain policies and procedures for their upgrades.
The deal is you have companies which now asses the costs of proper testing verses the cost of defending themsevles against their product blowing up and opt for whichever is cheapest.
There are companies which must maintain a higher standard, by law or existing contract. Unfortunately the trend I've been watching over the years is an acceptable level of incompetence or defects. Manufacturers of PC parts, f'rinstance, are fine with a 15% failure rate off the line. I couldn't imagine such being acceptable with pacemakers.
Ironically, most of the PC's in the world run on some version of Windows and even XP still loses its marbles on a regular basis. Thanks to the complexity of some products, some companies simply weigh the rist and make a financial decision and some CYA plan for Image Damage Control -- Gee, sounds just like the war on Iraq, come to think of it, it's a pervasive attitude.
Certainly Explains (Score:5, Funny)
Re:Certainly Explains (Score:3, Informative)
Re:Certainly Explains (Score:3, Informative)
As for OS/2, that was only on ATMs. And ATMs are running anything from VxWorks to System 7 unix to OS/2 to embedded windows.
Finally, if you're going to cite a comment on a web site, as reliable as that is, please give a reference to it that is more then the URL to the whole site...
What sytems, what upgrade? (Score:4, Insightful)
Re:What sytems, what upgrade? (Score:5, Funny)
"I honestly don't know. As I say, I mean, it's one of those tech things."
Source [globetechnology.com]
SCO Unix (Score:2, Funny)
Re:What sytems, what upgrade? (Score:5, Informative)
Internet on every PC (Score:3, Interesting)
I Need My Pay (Score:3, Interesting)
Honestly though. Being a software developer and knowing the development cycle like the back of my computer leads me to wonder how in the world they didnt test it fully. I mean... comeon guys. And that kind of institution using SCO's brand of UNIX? face + palm
Oh well... i dont care anymore... i close the accounts and visa card and when they asked me why, I just said: "I can't trust a bank that can't deal with this kind of glitch."
Just like the suits (Score:3, Informative)
It's their fault that these people aren't recieving paychecks and they're still hassling them aobut paying bills?
Re:Just like the suits (Score:3, Insightful)
Re:Just like the suits (Score:4, Insightful)
No, that's just somebody who thinks the world owes them everything taking the opportunity to complain because it might get them something they don't really deserve.
Re:Just like the suits (Score:2)
This is why (Score:5, Funny)
Sure, I don't earn interest on it, but at least I have some in an emergency
Re:This is why (Score:5, Funny)
"I don't keep all of my money in a bank, I've never had the hole in my back yard, under the oak, next to the stream, 5 steps from the bush refuse to give me a withdrawal."
Is that the blackberry bush or the raspberry bush? Never mind, I think I found it.
Re:This is why (Score:2)
apt-get install bank-upgrade (Score:5, Funny)
Double Withdrawl (Score:3, Interesting)
Partial Deployment Possible? (Score:2, Insightful)
Banking Hazards (Score:4, Interesting)
I'm sure if this had been their money they would've gotten it back in less than 7 days, and levied some hefty time and inconvenience charges.
Re:Banking Hazards (Score:5, Insightful)
Re:Banking Hazards (Score:4, Insightful)
Re:Banking Hazards (Score:3, Funny)
I can see it now... (Score:5, Funny)
junioradmin@rbchost:/> rm -rf core *
waiting
waiting
thinking...this is taking longer than it should
phone rings.
ctrl-| ctrl-| ctrl-| ctrl-|
Re:I can see it now... (Score:2, Funny)
thats a fun game to play as an admin at Canada's largest bank.
Credit damage (Score:5, Interesting)
Re:Credit damage (Score:5, Funny)
Or you could just print out one of the many press reports and official news releases on this subject.
Oh wait, this is slashdot. Sorry. Carry-on with the overacting dramatics.
16bit! (Score:3, Funny)
Who Scheduled this upgrade? (Score:4, Insightful)
Maybe they thought they would broaden their QA testing base to, say 20,000,000.
I wonder... (Score:2, Interesting)
Re:I wonder... (Score:5, Insightful)
It must be M$'s fault. (Score:3, Funny)
As one who is just making it by I offer this advic (Score:5, Insightful)
Anyway, my tip is, next time you get a bonus, tax returns, some lump sum of money, spend it on next months rent before you can do anything else with it. Trust me on this. If you put it in your savings you can too easily transfer it to checking when you see Wizz-Bang4000 on pricewatch for only $499! I do this every chance I get and it really helps out a lot.
Now if I could only figure out what to do with the SO.
Re:As one who is just making it by I offer this ad (Score:3, Insightful)
Better still, this plan doesn't have us up Shit Creek when a paycheck (or twenty) is missed.
If you have such a serious
One-time problem vs. tanking company (Score:2)
What are they running? UNIX. (Score:3, Informative)
See nextcraft: http://uptime.netcraft.com/up/graph?site=www.rbc.
MW
this just in..... (Score:5, Funny)
What happened to redundancy? (Score:4, Informative)
But when it happened a second time, in less than a year, I got a little frustrated and switched to president's choice bank.
My wife uses Royal Bank, and her pay has not gone through yet and it is now more than several days late. I certainly hope they work later than 4pm, monday to friday, to fix this. Some people who were supposed to be paid on the 31st have bounced their rent cheque!
is this not your greatest fear? (Score:4, Interesting)
Re:is this not your greatest fear? (Score:3, Insightful)
Those poor members of the IT department (Score:5, Insightful)
I know I always sweat when releasing new software, at least I don't have to worry about effecting the bank accounts of millions of people. That would truly be scary!
Big Questions (Score:4, Interesting)
1. What OS(s) were they running before this happened?
2. Were they really doing an upgrade or a crossgrade, that is, switching to a new system altogether?
3. Was this being handled by in house IT or was it being outsourced half way around the world?
Re:Big Questions (Score:5, Interesting)
RBC is probably an IBM mainframe environment. OS/390 or similar OS. They probably have AS/400 and AIX hosts as well for other functions (printing, data transfer, etc.) IBM sells you a whole shop and then you're stuck with it.
The OS is irrelevant, this is most likely an application (transactional database) error. Read the Globe And Mail article.
You call in the vendor (IBM) to do system upgrades and the like. They are VERY careful. If it went wrong, RBC would be laying the blame on them right now, and publicly.
Switching between (physical) systems for a mainframe environment is commonplace enough and would be fully supported by IBM. Parallel Sysplex and all that. One of my vendors did it over a weekend without incident - except they didn't wait until month-end to do it! Doh! That was dumb.
I don't do anything around month-end or quarter-end if I can help it. It's asking for trouble.
I'm pretty sure RBC does their IT in-house. These tend to be large, customized or internally developed systems, so outsourcing would mean almost certain death to the company. Even moving/consolidating operations between different groups in the same company is very painful.
Unless they were trying to
Re:Big Questions (Score:5, Informative)
- May 31st = month end. new month end code gets run first time in prod (JCL), there's problems - not sure of name of system, but the general bank ledger for customers is fubared.
- RBC rolls back all changes, so RBC is 1 day behind on Tuesday.
- re-run batch with old code on Wednesday am... unfortunately recovery procedures are flawed/human error, batch is screwed up again.
- now bank is 2-3 days behind...can't process transactions effectively, can't catch up with sequential batches in evenings because there's too much to run.
- RBC departments start running independently based on May 31st data...can't afford to be down more than 2 days. now the roadmap is a mess for recovery, general ledger is still at May 31st state (might be June 1st after a successful batch run last night???)
Apparently about 80-100 IT staff are living at the Skydome hotel in Toronto working 16 hour shifts (16 on, 8 off) to try and get caught up. Everything i've heard suggests that they know they can't get done during the business week...they're relying on 2 days of 24hr downtime on the weekend to reload the batches and get systems back in sync.
Based on my experience in Canadian banking (7 years) plus stories of old timers, this is the worst outage in close to 10 years, maybe the worst in 20 if my rumours prove to be true. I have no direct mainframe experience, though, so take my descripton of the problems with a grain of salt...
Re:Big Questions (Score:3, Insightful)
This comes down to poor testing. The manager(s) involved should (and probably will
Re:Big Questions (Score:3, Informative)
This question has "newbie" written all over. Bank applications run on true and tested OSes written decades ago. They run on large mainframes with uptime that often goes back to the day the computer was turned on sometime in the 70s or 80s.
This problem is, in all likelihood, an application problem.
Expert guesses? (Score:3, Interesting)
"George Geczy, a software developer and computer consultant based in Ancaster, Ont., guessed that the problem involves identification numbers assigned to transactions"
Thousands of different reasons why their system cratered and some guy running a consulting firm from his basement nailed it for us! Guess his experience in installing MySQL a couple times helped him diagnose their massively huge database issue.
Just because you have a IT job and a bank card, doesn't make you an expert.
Yeah, I got paided - Finally (Score:3, Interesting)
my take: (Score:3, Insightful)
(/former field-support rep for a vendor, who got *burned* by the incompetence and mendacity of RBC IT personnel who lied to their manager, and my manager, when THEY screwed up their evaluation of our product - AFTER they had dragged the evaluation out past the 12-month mark. . . how the hell do you justify evaluating a product for 12 months?).
Bank Computers and Office Hours (Score:3, Interesting)
Sure I can make a deposit at an ATM and have it instantly accessable, but what I am talking about is between bank transfers and such, for some reason these seem to take place only on weekdays.
Is there really some peon sitting in front of a terminal approving every transaction?
What peeves me is when I have something being transferred from say my merchant account, it can take 5 business days to get to my account, I mean these are computers we're talking about here and that type of delay really makes no sense when we live in a world where instant transactions are available.
I've seen stuff start transfer on a Wednesday and take until the following Tuesday to show in the account, that is just sad.
Re: (Score:2)
Comment removed (Score:5, Insightful)
Re:May be that will teach you (Score:4, Insightful)
Re:May be that will teach you (Score:3, Insightful)
Re:May be that will teach you-Expert testi-money (Score:3, Informative)
I finally got sick and tired of living paycheck to paycheck and started checking out personal finance websites, such as The Motley Fool [fool.com]. I got myself a free 30-day membership and visited forums to learn about living below your means, cutting credit card interest rates and playing their "bala
Re:Eh (Score:5, Funny)
Programmer 2: And welcome to day 3.
Programmer 1: "make install" hosed it.
Programmer 2: Hosed it down, eh, like backbacon at a Bah Mitzvah.
Programmer 1: But it's okay, eh. I got my thinking touque on and the beer and pizza are on the way.
Programmer 2: Yeah, we should have it back up by tomorrow, eh. Only, we're gonna need some more vacation after this.
Programmer 1: And beer, eh.
Programmer 2: Yeah, more beer.