Royal Bank of Canada Software Upgrade Goes Awry 602
Reader mks113 writes "Many Canadians living payday to payday have been in for a shock this week. Canada.com along with many other sources is reporting how thousands of customers have been inconvenienced following an unsuccessful software upgrade at the Royal Bank of Canada on Monday. All government employees (including me) in several provinces had their direct deposits delayed by a day or more." RBC has a comment on the mess.
Coincidence? (Score:2, Interesting)
Paranoia keeps you healthy!
Oh no! (Score:5, Interesting)
Wait till your bank holds onto your payroll checks for 2 weeks.
Once a bank of mine made an addition mistake, i wrote a pile of checks that all bounced. The bank acknowledged their mistake, and restored funds in my account, but refused to help out with all the check-bouncing fees.
$25 X 17 Hurray.
Somebody should get fired (Score:2, Interesting)
Internet on every PC (Score:3, Interesting)
I Need My Pay (Score:3, Interesting)
Honestly though. Being a software developer and knowing the development cycle like the back of my computer leads me to wonder how in the world they didnt test it fully. I mean... comeon guys. And that kind of institution using SCO's brand of UNIX? face + palm
Oh well... i dont care anymore... i close the accounts and visa card and when they asked me why, I just said: "I can't trust a bank that can't deal with this kind of glitch."
Double Withdrawl (Score:3, Interesting)
Re:Somebody should get fired (Score:4, Interesting)
I hope their programmers ARE unionized. If not, you can bet who the scapegoats will be, regardless of whether they are actually to blame.
Banking Hazards (Score:4, Interesting)
I'm sure if this had been their money they would've gotten it back in less than 7 days, and levied some hefty time and inconvenience charges.
Credit damage (Score:5, Interesting)
As a Canadian and a RBC customer... (Score:1, Interesting)
Re:Affects not just RB customers (Score:2, Interesting)
I deposit my cheque May 31st. I make two transfers from my chequing account to my Visa and my Savings. Come today, there's no money in my Chequing account! I look at the transaction history, and there's no trace of the deposit I made, or the transfers. Strangely enough, the money I transfered out of the savings account is still in the chequing account. They were supposed to fix this when? two days ago? I went into the main branch here in Vancouver and they told me they wouldn't be finished with their transaction backlog until next week. I hope they get sued. Canada's largest bank, and crappiest. I'm taking my money elsewhere!!
wtf are you talking about. (Score:4, Interesting)
Really, there's all kinds of blame to go around, and programmers deserve some of it, the system never should have been so brittle as to cause these kinds of problems in the first place.
I wonder... (Score:2, Interesting)
is this not your greatest fear? (Score:4, Interesting)
North American bank system is BACKWARD! (Score:2, Interesting)
1. Cheques? HAHAHHAHAHAH we didn't use them anymore for 15 years. NOT NECESSARY in Europe: you can just 'direct deposit' to everyone. And: NOT necessary to create 'bill payee' lists first - what's the use - just fill in the receivers' details if you want to pay (and yes, of course you can create lists of regular receivers). Also, people can pay ANYONE AT ANY BANK - not just payees who happen to be in YOUR bank's list.
2. Costs: Canadian banks are utterly expensive. Reason: no competition and he - why innovate or be more efficient when we can RAPE customers by charging outrageous fees for CRAP service. In Holland: crap service as well but you DON'T get charged! Just a very modest yearly FIXED fee.
3. Savings interest: in Europe it still pays to have savings account (beats inflation). Here in Canada I get 0.0000000001% (MINUS of course all kinds of 'service' fees because (a) I'm blond (b) I go biking 3 times a week (c) I just happen to be a customer who they can rape.
To give ONE good example that explains the difference. I still have Dutch bank accounts. From here (Canada) - I can use my computer to transfer money DIRECTLY from my Dutch bank account to - let's say - ANYONE in - let's say Greece; and WITHOUT costs. Now try doing that with Any north-American bank; they'll charge you with bizarre fees and the money is NOT in the account of the receiver the same day but instead you will have to wait a week at least. Yea you got a looong way to go (keep on dreaming that USA is the best in everything - take a vacation outside USA some day).
Re:Credit damage (Score:2, Interesting)
Big Questions (Score:4, Interesting)
1. What OS(s) were they running before this happened?
2. Were they really doing an upgrade or a crossgrade, that is, switching to a new system altogether?
3. Was this being handled by in house IT or was it being outsourced half way around the world?
Re:Ah, Nostalgia... (Score:2, Interesting)
It's called software release management (SRM) and infrastructure configuration management (ICM). Both are parts of general configuation management.
According to a Gartner study (I don't have it at my fingertips), over 90% of all failures in mission critical systems are due to poor problem or configuration management. When this study was done, Gartner estimated that only 6% of the major corporations would have industry best practices in place by 2004.
Hmm - sounds like both the Bank of Canada and the English FAA got bit by this big time.
This discipline sounds like a great business opportunity. Unfortunately my previous employer disagreed.
My previous employer also shut down a (formerly) successful consulting organization and laid off 1100 talented individuals.
I wonder how much the Bank of Canada would have paid for this type of expertise. Of course, the check might be a day or two late . . .
Re:Those poor members of the IT department (Score:1, Interesting)
Re:Those poor members of the IT department (Score:2, Interesting)
Re:Oh no! (Score:5, Interesting)
After spending 15 minutes tracking down the error.. the bank refunded the NSF fee it charged me and asked me to present reciept for the fee charged by my landlord. The refunded my landlord's fee and provided a letter of appology stating that it was all their fault.
If your bank refused to do that then I suggest you find another bank.
Expert guesses? (Score:3, Interesting)
"George Geczy, a software developer and computer consultant based in Ancaster, Ont., guessed that the problem involves identification numbers assigned to transactions"
Thousands of different reasons why their system cratered and some guy running a consulting firm from his basement nailed it for us! Guess his experience in installing MySQL a couple times helped him diagnose their massively huge database issue.
Just because you have a IT job and a bank card, doesn't make you an expert.
Re:Wait a minute... (Score:1, Interesting)
"Fun" is generally shared by all parties involved and shouldn't come at the expense of others. But, then, my innate Canuckness is showing.
Yeah, I got paided - Finally (Score:3, Interesting)
It happened to me ... (Score:2, Interesting)
(Background info: I had told the managers at the site that the site admin needed three classes of training, hands on work with me while I installed and implemented the systems, and some other experience before he went solo. The managers agreed to this but they never came through: He got zero training, and "was too busy" to work hands-on with me).
True enough, the system was down, and I had an appointment that night (Friday night, of course), but I would come in over the weekend to see what I could do. Of course, the guy hadn't backed up this system ever according to the backup procedures I had handed him.
I spend three hours on a Saturday trying to get this station up (it had design part data, and that data couldn't be permanently lost), and finally told the managers at the site that they needed to get the vendor out as it appeared to be a hardware problem based on what I was seeing (bios type messages, but once it hit the hard drive it died hard).
Vendor came out, checked out the hardware, and reported that nothing was wrong with the CPU, memory, SCSI cards, busses, disk drive, etc. The site administrator then remembers that the day before everything hit the fan that he created a
Immediately after he told me this little blurb (and I was red hot, Why didn't you tell me when I asked!) he informed me that it was time for him to leave and he did! Luckily for him he did leave, otherwise I was going to strangle him.
Fortunately, I was able to move the disk drive to piggyback off of a similar system, copy the device files from that system to the munged drive, and then recreate the couple of device files that were specific to this system. End result was that the system was back 100% again, and fully backed up (since I had zero confidence in the jerk). I told the managers what had happened, and what the actions of their site admin had cost me both personally and in terms of my work hours. I got blamed by the site admin for "not training him enough", for not being responsive enough, and for accepting his initial story and not digging into it to find out the root causes.
Re:Big Questions (Score:5, Interesting)
RBC is probably an IBM mainframe environment. OS/390 or similar OS. They probably have AS/400 and AIX hosts as well for other functions (printing, data transfer, etc.) IBM sells you a whole shop and then you're stuck with it.
The OS is irrelevant, this is most likely an application (transactional database) error. Read the Globe And Mail article.
You call in the vendor (IBM) to do system upgrades and the like. They are VERY careful. If it went wrong, RBC would be laying the blame on them right now, and publicly.
Switching between (physical) systems for a mainframe environment is commonplace enough and would be fully supported by IBM. Parallel Sysplex and all that. One of my vendors did it over a weekend without incident - except they didn't wait until month-end to do it! Doh! That was dumb.
I don't do anything around month-end or quarter-end if I can help it. It's asking for trouble.
I'm pretty sure RBC does their IT in-house. These tend to be large, customized or internally developed systems, so outsourcing would mean almost certain death to the company. Even moving/consolidating operations between different groups in the same company is very painful.
Unless they were trying to
Re:Somebody should get fired (Score:2, Interesting)
But back on topic.
Frankly, whoever is responsible for this, should be held accountable at least to the level of losing their job. In any PROFESSION (programming is NOT a profession, it's a job), the person responsible would be held completely liable. Yes, insurance would probably be there to cover the monetary costs (and if not, well, expect personal bankruptcy), but that person would never be able to work in that profession again, at least not in the same province/state/country/however far up the ladder the profession is administered. THAT is realistic, as it DOES exist already.
Doctor found liable for a patient's death? Can't practice medicine there anymore. Lawyer disbarred for something? Can't pratice law there anymore. Get it?
That kind of system should be in place for any "critical" system, including anything that deals with life and death (not applicable directly to this discussion, but worth mentioning for completeness), or is (as in this case) somewhat central to the smooth (or at least as smooth as usual) functioning of society. SOMEONE should have had to sign off on that upgrade and take responsibility for it. That someone SHOULD have been someone who was actually involved in the process, and not a PHB.
Re:wtf are you talking about. (Score:2, Interesting)
Irksome no end.
On blame,
QA deserves some of the blame, they should not have blessed it, and their tests should have caught major problems.
The development team deserves some of the blame, they should have unit tested for this, and should have made sure that QA had whatever information they needed to check it before blessing it.
Management probably deserves most of the blame. I can only imagine that it was they that developed the schedules ( probably not realistic ) that the development and QA teams had to work within. Probably they that insisted that all features be developed within their timeframe, ignoring whatever pushback came from dev and QA. And probably they that decided to shorten the QA schedule to allow a bit more dev time ( seen it happen so many times... And I am a dev... ).
1 day's interest (Score:1, Interesting)
They should make the software company pay for that and then they will think harder next time
Bank Computers and Office Hours (Score:3, Interesting)
Sure I can make a deposit at an ATM and have it instantly accessable, but what I am talking about is between bank transfers and such, for some reason these seem to take place only on weekdays.
Is there really some peon sitting in front of a terminal approving every transaction?
What peeves me is when I have something being transferred from say my merchant account, it can take 5 business days to get to my account, I mean these are computers we're talking about here and that type of delay really makes no sense when we live in a world where instant transactions are available.
I've seen stuff start transfer on a Wednesday and take until the following Tuesday to show in the account, that is just sad.
Re:Bank Computers and Office Hours (Score:2, Interesting)
"I've seen stuff start transfer on a Wednesday and take until the following Tuesday to show in the account, that is just sad."
It would be sadder if the transfer was instant, but turned out 5 days later to have been bogus, and they take the money out of your account. Oh, you emptied your account already? Don't you see the problem from the bank's point of view? Maybe everybody who has ever written a check to YOU has been good for it, but the bank hasn't been that lucky, and neither have I.
The alternative is to take responsibility for what is deposited into your account. If you get a bad check, it's your problem, not the bank's.
Would you rather wait 5 days and have a guarantee of your cash, or would you rather vouch for every instrument presented in order to get your cash earlier? You can probably arrange it like that with your bank.