Bank's IT Failure Loses 600,000 Payments 96
An anonymous reader writes: The Royal Bank of Scotland had an IT glitch last night that prevented some 600,000 payments from reaching the accounts of its customers. This included bill payments, wages, tax credits, and benefits payments. RBS apologized for the delay, and claims to have fixed the underlying problem. They hope to have all the missing payments sorted by the weekend. This isn't the first major IT screwup for RBS; in 2012, the company was fined £56 million after a software upgrade prevented about 6.5 million customers from logging into their accounts.
Bank admits error? (Score:2, Insightful)
What madness is this??
When my bank makes a mistake, they blame me and penalize me for the problem, then when I complain, they charge me for investigating their error.
Re: (Score:2)
The real question: Did they get The Low Price Always[TM] for their IT services?
Re: (Score:2)
But that doesn't impact bonuses for the top layer.
The top layer doesn't get bonuses [sky.com]. RBS is owned by the UK taxpayers, is continuing to bleed cash, and will likely eventually be shut down.
Re: (Score:1)
No, the govenment stake is about to be sold off at a very low rate to force it back into private hands. Someone will make a killing.
Re:Bank admits error? (Score:5, Insightful)
Maybe you should switch banks. I can't speak for the UK, but it never ceases to astound me how many people whine about banking in the United States when there are thousands of small community banks you could be doing business with. It's a tough industry and the little guys are facing setbacks on a daily basis, but they're still there if people are willing to look for and do business with them.
In the day and age of remote deposit [wikipedia.org] there's no reason to do business with a large national bank. I get waived ATM fees worldwide, no account fees of any sort, and competitive loan and deposit rates, all from a little regional bank that you've probably never heard of unless you're from my small hometown.
For the life of me I don't understand why Chase, Capital One, or Bank of America have any retail customers at all. They bend people over on fees, structure your transactions to obtain yet more fees, and generally do all sorts of nefarious things while offering no real advantage over their smaller competitors.
Re: (Score:2)
Re: (Score:2)
Re: Bank admits error? (Score:2)
Really, I had assumed that France would have gone the way of several Scandinavian countries and gotten rid of cheques. I'm looking forward to the day they're obsoleted in the UK so we don't have to do a banking run for the few cheques we get a week. I shredded my cheque book in about 2008 and haven't missed it.
I think we need a Slashdot poll - "when was the last time you wrote out a cheque"
Re: (Score:2)
You can eke out another week or two of money not actually leaving your bank quite easily.
Re: (Score:2)
What a superb way of doing business. Reminds me of a speaker at a business conference in the 80s who got a standing ovation. He described how he increased company revenue by refusing to pay his creditors until they went out of business. For some reason those lauding him didn't seem to think that the same thing could happen to them.
Any small businesses we've dealt with recently have refused to take cheques - probably for that exact reason.
Re: (Score:2)
On May 31, 2004 Royal Bank Canada (unrelated) had a similar massive failure of a software upgrade. [nytimes.com] As a result none of their customers had transactions processed, and it took almost a week to fix and clear the backlog. Not only did this impact individuals banking with RBC, but if an employer's payroll was through RBC, none of their employees got paid. Mortgage payments, car payments and other bill payments were bouncing all over the place.
As far as where people bank, I'm surprised how many people willingly
Re: (Score:2)
For the life of me I don't understand why Chase, Capital One, or Bank of America have any retail customers at all.
They seem to be buying the small banks. At least that keeps happening to every bank I've been a customer of.
So I joined a credit union. but it is just a vassal of one of the mega banks. Found that out while reviewing a transaction summary from the web hosting company I used to use. My payments were listed as coming from one of the mega banks. When I asked at the credit union, they told me that the CU had a master account with the mega bank and all the members' CU accounts were actually accounts at the mega
Re:Bank admits error? (Score:4, Interesting)
Re:Bank admits error? (Score:5, Interesting)
They probably will, but the last time a banking error caused a missed payment and penalties, I took the notice from my landlord to my bank and they gave me a refund for the amount and a letter of apology addressed to my landlord..
I'm guessing that cost cutting RBS did by laying off their staff and rehiring everyone in India is going to cost them dearly. This isn't their first major outage since they did that.
Re: Bank admits error? (Score:2)
I gave up on NatWest when the bank manager refused to allow me to withdraw the £500 deposit that I had to pay my new landlord - incredibly embarrassing and had to deal with the concern that he would find me untrustworthy. Apparently having a bank card and driving licence want good enough ID.
I got a phone call apologising after I wrote a letter of complaint explaining that she was likely costing the bank business as I was now moving banks and they'd be losing my new self-employed business account
Re: (Score:3)
You're right, and the question is - why should they? I mean, wouldn't you be pissed off if your workplace couldn't pay you on time because of this? It was their bank being the problem, but your creditors don't care that you didn't get paid today. Etc. etc.
And while credit cards not working or ATMs and su
Re: (Score:2)
Re: (Score:2)
You need a new bank.
Re: Bank admits error? (Score:1)
Tin foil speculation (Score:1)
It's waiting for the day that a solar storm disrupts grid power world-wide for a few days, and digital money will cease to exist. Although personally i'd bet bitcoin would survive such event.
Deliberate mismanagement (Score:4, Interesting)
Re: (Score:1)
The offshoring decisions regarding IT were taken several years before the bank was taken into public ownership.
#Source - I and several of my colleagues used to work for them in Edinburgh
Re:Deliberate mismanagement (Score:5, Informative)
Don't be an ass. Its not in any employees interest including the CEO to run it into the ground when a lot of them are on performance related pay. And its certainly not in the governments interest.
I suggest you put your tin foil hat away. This is just plain old fashioned incompetance.
Re: (Score:1)
Re:Deliberate mismanagement (Score:4, Interesting)
Wrong, it's not a conspiracy if it's false. A conspiracy is two or more parties working together for an illegal, unethical, or otherwise undesirable means. A conspiracy is NOT the incoherent ramblings of a wild-eyed mental patient. That's what we call a crackpot theory.
However, the powerful elites and controlled media outlets have worked tirelessly for decades to further the narrative that anyone who accuses someone else of a conspiracy is a wild-eyed mental patient. When, based on the simple definition of the word conspiracy, everyone can admit that they surely happen every single day in every country on Earth.
Hence, the "conspiracy theory" becoming a label that can be easily applied to anyone who is calling out a group of two or more people who are up to no good and immediately discredit them. For the sociopaths that run the world, this is the best thing that could have ever happened to them. Anyone who can see that 5+5=10 is now a nutter! How convenient for them!
Re: (Score:1)
Many CEOs have golden parachutes so large they are better off when the company ousts them. They get a huge windfall and can retire in comfort and wealth.
Re: (Score:3)
Greed knows no bounds.
Re: (Score:2)
Re: (Score:2)
I'm not sure how I can see it would be in the chancellors interest either...
He is a Tory with an aversion to public ownership, and he has a lot of Tory friends eager to buy shares in the bank cheaply. What's so hard to understand?
Icing on the cake is that it's all Gordon Brown's fault for being so evil as to take the bank into public ownership to stop it collapsing in the first place.
Re:Deliberate mismanagement (Score:4, Funny)
Don't be an ass. Its not in any employees interest including the CEO to run it into the ground when a lot of them are on performance related pay. And its certainly not in the governments interest.
I suggest you put your tin foil hat away. This is just plain old fashioned incompetance.
Incompetence - a bad day when you spell that word wrong.
Re: (Score:2)
CEO's "performance bonus" seems to generally be paid out even if the CEO navigates the company directly into the ice berg.
The people who make a fortune on the deal will all be sure to pull out a chair for him to land in when has gadzillion dollar (pound) golden parachute deploys.
Re: (Score:2)
That doesn't make sense. Why would they try to run down the share price and get less money when they sell it? They already got a kicking in the press for underpricing the Post Office when they sold it off .
It's about ideology. The Conservatives don't approve of businesses being in public ownership. Whether you agree with them or not, that is a fact.
The Royal Mail thing barely caused a ripple in the media that I remember, and besides it was Vince Cable's fault and he was a LibDem.
Re: (Score:2)
Simply not true.
I worked there as a contractor during that period. It was fairly grim, but don't ascribe to malice what can be explained by incompetence for example. It was also a better outcome than for my previous client, Lehman Brothers, in that it wasn't dropped on the floor for no particularly good reason, throwing away even more value and confidence.
Please don't quote conspiracy theories in the absence of any actual facts.
It is true that reducing the global markets operation is stated policy, but th
Re: (Score:2)
Since the taxpayer took a large stake in RBS, it has been deliberately run into the ground to lower it's share price ready to be sold back into private ownership. It ends up looking like a windfall for the current chancellor, when in fact it's transferred a big pile of taxpayer's money into private hands while wrecking a financial institution. Banksters.
The most amusing thing is the current Chancellor blaming Gordon Brown for paying too much in the first place during the banking crisis. Frankly, they should just have nationalised the fucking thing without compensation since it had no value.
Michael Bolton (Score:4, Funny)
He must have misplaced a decimal point. He always screws up some mundane detail.
Trimming the fat (Score:2)
Re: (Score:1)
RBS, in common with most of the banks, have been cutting IT staff and investments and offshoring what they can.
This is what then happens.
Re: (Score:2)
Forget about paying top dollar.
Banks spend huge amount of money on IT.
The problem is that it is really poorly allocated.
They'll spend massive amounts of money buying expensive products from IBM. Spend massive amount of money on IT contractors. Spend massive amounts of money on IT security scanning...
The problem is largely in the small space with dev and IT where the people who actually make the damn thing work day in and day out.
When it comes to banks, I'm sorry, I've worked in the industry. They have no sh
Is someone trying to replace COBOL? (Score:3)
I wonder if some lead IT whizz kid decided it would be really "kool" and look good on his CV to replace all that crufty old cobol running on the cobweb covered mainframe in the corner with this months latest hot language?
I'm exaggerating a bit, but it does seem to me having worked in the dev industry for > 20 years that a lot of the younger devs really don't understand the serious reasons behind the phrase If It Ain't Broke... If you mention it they either think you're being ironic or just too old to "get it". Sadly its them who don't get it but they're too arrogant/naive to realise it.
Re: (Score:1)
This was not new code
Re: (Score:2)
I'd imagine stopping any batch system halfway through is going to cause a mess. There's a reason it has the word "batch" in it.
Re: (Score:2)
Hence basic shit like rollback capabilities and error recovery. You know, fundamental it practices that have been common sense for decades.
Shit, assume your batch always runs to completion and you've pretty much guaranteed it's going to fuck things up massively.
Re: (Score:2)
There's a difference between re-starting and re-running. With a batch the first is normal, the 2nd is a big no no.
Re: (Score:1)
This was widely reported at the time. It wasn't a bad batch job, it was a screwed up update of batch software, when it failed they tried to back it out and made a complete balls up
http://www.theregister.co.uk/2... [theregister.co.uk]
Re: (Score:1)
AC or not... If it was told to you in confidence, why are you sharing it?
Re: (Score:2)
Comment removed (Score:3)
Re:Too big to care. (Score:4, Insightful)
That is just nonsense. Banks certainly can and do fail, and the FDIC does NOTHING to protect them. The FDIC is protecting the DEPOSITORS (ie, you). If a bank fails, the owners of the bank have lost their investment. The owners do not get to keep their investment just because the assets of the bank were bought by someone else. There is no reason employees or customers of a bank should suffer because the owners installed bad management. If the new owners keep that same bad management then they risk losing their investment too.
Re: (Score:2)
PS. If it is a credit union, as you state, then YOU are one of the owners. Did you do due diligence and vote for the proper board members, who would hire the management you want, or do you just let someone else do all that and whine when things go wrong?
Re: (Score:2)
Does your credit union use Intuit's web platform for online banking access per chance?
My credit union (in RI) recently went through a similar upgrade (including adding an extra digit to accounts), but to their credit, it seemed to go much smoother.
Re: (Score:1)
Re: (Score:2)
>> Extremely dated infrastructure and mainframes that handle all those payments / systems, are all IBM.
You're partially right, but there's also a lot of Unisys mainframes that power this stuff.
Re: (Score:1)
Whoever modded my comment flamebait must work for IBM, did I hit a nerve for speaking the truth? I've had to deal with IBM on so many occasions directly at work and time and time again they've let us down.
RBS are building a new system [telegraph.co.uk], with IBM ironically, while other banks are trying to move away from them.
Re: (Score:2)
they could have processed the 600,000 transactions to the wrong account
Ah, I wondered why I suddenly had a GBP 600,000,000 overdraft.
Seen it elsewhere (Score:1)
Posting anon, because customer data... Circa 5 years ago I worked in support for a commercial software vendor. A big bank in US used several of our software, in a somewhat customized way. One of those products was for network file transfer, one for data manipulation. They called with an issue of a small percentage of inbound transfers consistently failing and after manually reprocessing them a few minutes later they always succeeded. Long story short - they've customized the transfer product to drop a file
I hope it wasn't my stuff... (Score:3)
I installed some transaction processing software at RBS in the late 1990s. Here's hoping they retired it a while back...I haven't heard from these guys in over a decade.
They chose the cheaper, worse IT system (Score:2)
By far the most interesting statement on this comes from Iain Martin at the Daily Telegraph, who says that at the time RBS (a fairly small bank) took over the much-larger and more established NatWest bank:
> His team worked out quickly that the NatWest system was superior to the RBS computer system.... Then they crunched the numbers and
> confirmed that sticking with plan A and migrating NatWest's customers onto RBS's inferior and cheaper to run system would save more money.
http://blogs.telegraph.co.uk/
Re: H1B visas (Score:3)
Not quite. A large Indian outsourcer won a large part of work to maintain the CA7 schedule. Anyone has worked at a bank knows that the batch schedule is somewhat mystical and was put together by guys who had been at the bank for 20 yrs - replacing them with bright, but fresh out of uni, offshore outsourcers was a large part of the cause of the 2012 outage during an upgrade.
This outage hit the NatWest batches. The NatWest batches are far bigger than the RBS, Ulster(s) and IoM/Courts and are the first to r
Soon to be Trading under New Name... (Score:2)
Lack of mainframe skills maybe? (Score:2)
One of the things about banks is that most of their core transaction processing is done on mainframes. There's tons of stuff layered on top of it to do fancier things, but the day to day moving of funds from account to account is usually batched and run after business hours on a mainframe. One of the reasons for this is the sheer amount of business logic tied up in these systems, the massive transaction volume. and the fact that you can't easily swap out a working process with something untested.
The problem
Re: (Score:2)
This depends on a number of issues.
There's a few parts to a banking system, and the 'core' - is usually on a mainframe. The name 'core' in this context means "The software system used to store financial account data" - a database, for the most part. It stores the account data, and occasionally a few other pieces of information, but mostly the account data. Account data never really changes, so neither have these "core" systems. They're also like 20-30 years old, and so rarely break anymore. They're tim
Happens all the time. (Score:4, Informative)
So, I used to write banking software for a living, and that included being in the damage control team when things went south. Which they did. A lot. Which were brought to the technical staff's attention, without fail, every friday at 4 PM - but I digress.
This isn't really a big deal.
This sort of thing happens on almost a daily basis. I'd say that 1 in every 500 banks "loses" a day's transactions each week. From hardware failure to networking problems, to someone entering bad data to simple bugs in the software (usually data-dependent and hard to source), and even an occasional overwrite from a backup. Something breaks.
Once we had an FI lie about their data center setup - it wasn't redundant, offsite, certified, or even a datacenter. They just had some machines running in their basement. Imagine our surprise when they called up wanting us to (remotely) fix their servers which were under 6 feet of water during a flood. ... again, I digress.
The thing about these transactions is they're not really lost. Nothing really goes missing.
See, all financial software is super keen on accounting. I don't mean that in a strictly 'add up the numbers' way, but in an auditing way. There are logs upon logs. Using your ATM card to withdraw $20 probably generates around, oh, I dunno, 30-40 log messages, depending on how it's routed. There's a log of the transmission and response in each node along the way, using a standardized protocol. It's a severe pain in the butt, but even if a system goes down forever, we can regenerate those logs from the other systems. It's a great deal of effort, especially to preserve the sequence order, and it's tedious, but it can be done.
This could easily account for the initial delay in fixing things.
Not only that, all these sorts of systems are very keen on sequence-order processing, so if it's just the case that the end of day processing (clearance) system went down/had a bug/etc and none of the transactions were finalized, then they'll just stack up. Once they get that system up again, it'll start processing them again. It might take longer due to a large backload, but it'll eventually complete.
98% of the time, the only people that notice are companies waiting on a payroll to go out, and most of them are happy enough to accept the financial institution's admission of fault. For a day or two at least. Why this one made the paper, I can't tell you. Probably a customer was savvy enough with social media and decided to burn the bank. I guess they should just be happy the public in general doesn't realize how many problems and how much actual work goes into making banking seem reliable and secure.
Re: (Score:1)
Why this one made the paper, I can't tell you. Probably a customer was savvy enough with social media and decided to burn the bank. I guess they should just be happy the public in general doesn't realize how many problems and how much actual work goes into making banking seem reliable and secure.
A lot of benefit payments and tax credits were due, as these are expected payments from the Government to people who may be relying on that money it was quicly decided to get the story out. As someone mentioned above, at the poorer ends of society, those little handouts midway through the month can mean a lot to a working family.
Re: (Score:2)
I manage a few enterprise level applications and databases, and it is much the same, to a lesser degree. I recently discovered a bug in our code which a contractor inserted that didn't understand the business rules which no one caught and had been generating bad data for about 6 months. It was done in such a way that unless you were actually looking for it (I discovered by accident while running some statistics that didn't add up which prompted me to investigate). Anyway how this all ties into the topic is
The perils of downsizing the IT department .. (Score:2)
I read somewhere that RBS downsized their UK CA-7 batch processing department and then imported the one man from India to take over. Not being experienced enough he botched an overnight job and in the attempt to roll back a days worth of transactions accidentally rolled it back a months worth of transactions. Soon
So they didn't actually "lose" the payments? (Score:1)
Re: (Score:1)
That would have been a different story altogether and maybe one worth reading.
Sorry for the typos, posting on my phone.
This is why you don't use staffing agency labor. (Score:1)
When you cut corners with the very people responsible for your IT infrastructure, things like this will happen. Treat them well, even hire them directly, things like this tend not to happen.
Unfortunately, the UK is rife with this kind of corner-cutting.