Become a fan of Slashdot on Facebook

 



Forgot your password?
typodupeerror
Bug The Almighty Buck United Kingdom IT

Bank's IT Failure Loses 600,000 Payments 96

An anonymous reader writes: The Royal Bank of Scotland had an IT glitch last night that prevented some 600,000 payments from reaching the accounts of its customers. This included bill payments, wages, tax credits, and benefits payments. RBS apologized for the delay, and claims to have fixed the underlying problem. They hope to have all the missing payments sorted by the weekend. This isn't the first major IT screwup for RBS; in 2012, the company was fined £56 million after a software upgrade prevented about 6.5 million customers from logging into their accounts.
This discussion has been archived. No new comments can be posted.

Bank's IT Failure Loses 600,000 Payments

Comments Filter:
  • Bank admits error? (Score:2, Insightful)

    by Anonymous Coward

    What madness is this??

    When my bank makes a mistake, they blame me and penalize me for the problem, then when I complain, they charge me for investigating their error.

    • by Shakrai ( 717556 ) on Wednesday June 17, 2015 @09:37AM (#49929139) Journal

      Maybe you should switch banks. I can't speak for the UK, but it never ceases to astound me how many people whine about banking in the United States when there are thousands of small community banks you could be doing business with. It's a tough industry and the little guys are facing setbacks on a daily basis, but they're still there if people are willing to look for and do business with them.

      In the day and age of remote deposit [wikipedia.org] there's no reason to do business with a large national bank. I get waived ATM fees worldwide, no account fees of any sort, and competitive loan and deposit rates, all from a little regional bank that you've probably never heard of unless you're from my small hometown.

      For the life of me I don't understand why Chase, Capital One, or Bank of America have any retail customers at all. They bend people over on fees, structure your transactions to obtain yet more fees, and generally do all sorts of nefarious things while offering no real advantage over their smaller competitors.

      • by houghi ( 78078 )

        I don't get checks. Seriously, I don't get it. In Europe, we don't use them since several years.

        I just log on into my bank, I type in the number of the other bank account. I type in the amount and (after some security stuff) it is done.

        I can do this to every bank account in Europe easily. Probably also to the rest of the world.

        This means no handling of any paper. 24/7 access. I never payed any fees for this. Between personal accounts in Belgium it happens on the same day. When writing it to a business acou

        • by houghi ( 78078 )

          OK, perhaps a bit sparce on the details of the transaction.
          Beside the bankaccount numer and the amount, you also can enter a message. There are several option. e.g. "free" where you write anything you like. Some characters are forbidden. SO you can write something like "Hotel reservation for houghi on 31/12/2015".

          Next you have a structured message for Belgium. This is intended for ciomputers. The receiving company will read it and know what the payment was for. This is in the form (In Belgium) as 123/4567/8

      • On May 31, 2004 Royal Bank Canada (unrelated) had a similar massive failure of a software upgrade. [nytimes.com] As a result none of their customers had transactions processed, and it took almost a week to fix and clear the backlog. Not only did this impact individuals banking with RBC, but if an employer's payroll was through RBC, none of their employees got paid. Mortgage payments, car payments and other bill payments were bouncing all over the place.

        As far as where people bank, I'm surprised how many people willingly

      • For the life of me I don't understand why Chase, Capital One, or Bank of America have any retail customers at all.

        They seem to be buying the small banks. At least that keeps happening to every bank I've been a customer of.

        So I joined a credit union. but it is just a vassal of one of the mega banks. Found that out while reviewing a transaction summary from the web hosting company I used to use. My payments were listed as coming from one of the mega banks. When I asked at the credit union, they told me that the CU had a master account with the mega bank and all the members' CU accounts were actually accounts at the mega

    • by cyberchondriac ( 456626 ) on Wednesday June 17, 2015 @09:56AM (#49929295) Journal
      I wouldn't be remotely shocked if the businesses that didn't receive payments still find it reasonable to ascribe late penalties to their customers and say "Hey, it wasn't our fault your payments were late".
      • by gmack ( 197796 ) <gmack@inn[ ]ire.net ['erf' in gap]> on Wednesday June 17, 2015 @10:38AM (#49929639) Homepage Journal

        They probably will, but the last time a banking error caused a missed payment and penalties, I took the notice from my landlord to my bank and they gave me a refund for the amount and a letter of apology addressed to my landlord..

        I'm guessing that cost cutting RBS did by laying off their staff and rehiring everyone in India is going to cost them dearly. This isn't their first major outage since they did that.

        • I gave up on NatWest when the bank manager refused to allow me to withdraw the £500 deposit that I had to pay my new landlord - incredibly embarrassing and had to deal with the concern that he would find me untrustworthy. Apparently having a bank card and driving licence want good enough ID.

          I got a phone call apologising after I wrote a letter of complaint explaining that she was likely costing the bank business as I was now moving banks and they'd be losing my new self-employed business account

      • by tlhIngan ( 30335 )

        I wouldn't be remotely shocked if the businesses that didn't receive payments still find it reasonable to ascribe late penalties to their customers and say "Hey, it wasn't our fault your payments were late".

        You're right, and the question is - why should they? I mean, wouldn't you be pissed off if your workplace couldn't pay you on time because of this? It was their bank being the problem, but your creditors don't care that you didn't get paid today. Etc. etc.

        And while credit cards not working or ATMs and su

    • Mod parent up please!
    • You need a new bank.

    • Then, after all that the bank goes bust and the government bails them out. As long as rich bankers sit bathing in lives of luxury who cares?
  • by Anonymous Coward

    It's waiting for the day that a solar storm disrupts grid power world-wide for a few days, and digital money will cease to exist. Although personally i'd bet bitcoin would survive such event.

  • by ikoleverhate ( 607286 ) on Wednesday June 17, 2015 @09:08AM (#49928933)
    Since the taxpayer took a large stake in RBS, it has been deliberately run into the ground to lower it's share price ready to be sold back into private ownership. It ends up looking like a windfall for the current chancellor, when in fact it's transferred a big pile of taxpayer's money into private hands while wrecking a financial institution. Banksters.
    • by Viol8 ( 599362 ) on Wednesday June 17, 2015 @09:15AM (#49928979) Homepage

      Don't be an ass. Its not in any employees interest including the CEO to run it into the ground when a lot of them are on performance related pay. And its certainly not in the governments interest.

      I suggest you put your tin foil hat away. This is just plain old fashioned incompetance.

      • It's not a conspiracy if it's true!
        • by clonehappy ( 655530 ) on Wednesday June 17, 2015 @10:10AM (#49929413)

          Wrong, it's not a conspiracy if it's false. A conspiracy is two or more parties working together for an illegal, unethical, or otherwise undesirable means. A conspiracy is NOT the incoherent ramblings of a wild-eyed mental patient. That's what we call a crackpot theory.

          However, the powerful elites and controlled media outlets have worked tirelessly for decades to further the narrative that anyone who accuses someone else of a conspiracy is a wild-eyed mental patient. When, based on the simple definition of the word conspiracy, everyone can admit that they surely happen every single day in every country on Earth.

          Hence, the "conspiracy theory" becoming a label that can be easily applied to anyone who is calling out a group of two or more people who are up to no good and immediately discredit them. For the sociopaths that run the world, this is the best thing that could have ever happened to them. Anyone who can see that 5+5=10 is now a nutter! How convenient for them!

      • by Anonymous Coward

        Many CEOs have golden parachutes so large they are better off when the company ousts them. They get a huge windfall and can retire in comfort and wealth.

      • I'm not sure how I can see it would be in the chancellors interest either...
        • I'm not sure how I can see it would be in the chancellors interest either...

          He is a Tory with an aversion to public ownership, and he has a lot of Tory friends eager to buy shares in the bank cheaply. What's so hard to understand?

          Icing on the cake is that it's all Gordon Brown's fault for being so evil as to take the bank into public ownership to stop it collapsing in the first place.

      • by kencurry ( 471519 ) on Wednesday June 17, 2015 @10:14AM (#49929459)

        Don't be an ass. Its not in any employees interest including the CEO to run it into the ground when a lot of them are on performance related pay. And its certainly not in the governments interest.

        I suggest you put your tin foil hat away. This is just plain old fashioned incompetance.

        Incompetence - a bad day when you spell that word wrong.

      • by sjames ( 1099 )

        CEO's "performance bonus" seems to generally be paid out even if the CEO navigates the company directly into the ice berg.

        The people who make a fortune on the deal will all be sure to pull out a chair for him to land in when has gadzillion dollar (pound) golden parachute deploys.

    • by DamonHD ( 794830 )

      Simply not true.

      I worked there as a contractor during that period. It was fairly grim, but don't ascribe to malice what can be explained by incompetence for example. It was also a better outcome than for my previous client, Lehman Brothers, in that it wasn't dropped on the floor for no particularly good reason, throwing away even more value and confidence.

      Please don't quote conspiracy theories in the absence of any actual facts.

      It is true that reducing the global markets operation is stated policy, but th

    • Since the taxpayer took a large stake in RBS, it has been deliberately run into the ground to lower it's share price ready to be sold back into private ownership. It ends up looking like a windfall for the current chancellor, when in fact it's transferred a big pile of taxpayer's money into private hands while wrecking a financial institution. Banksters.

      The most amusing thing is the current Chancellor blaming Gordon Brown for paying too much in the first place during the banking crisis. Frankly, they should just have nationalised the fucking thing without compensation since it had no value.

  • by Major Blud ( 789630 ) on Wednesday June 17, 2015 @09:08AM (#49928935) Homepage

    He must have misplaced a decimal point. He always screws up some mundane detail.

  • Never worked for RBS, so I'm not claiming to have any knowledge into their IT dept, but I've seen this trend in IT locally too many times. Execs don't see the need paying top dollar for IT staff and wind up getting people that do "just enough." That's when snafus like this one happen, and it costs the company more in the long run than paying top $$ for good talent.
    • by Anonymous Coward

      RBS, in common with most of the banks, have been cutting IT staff and investments and offshoring what they can.

      This is what then happens.

    • Forget about paying top dollar.

      Banks spend huge amount of money on IT.
      The problem is that it is really poorly allocated.

      They'll spend massive amounts of money buying expensive products from IBM. Spend massive amount of money on IT contractors. Spend massive amounts of money on IT security scanning...

      The problem is largely in the small space with dev and IT where the people who actually make the damn thing work day in and day out.

      When it comes to banks, I'm sorry, I've worked in the industry. They have no sh

  • by Viol8 ( 599362 ) on Wednesday June 17, 2015 @09:20AM (#49929017) Homepage

    I wonder if some lead IT whizz kid decided it would be really "kool" and look good on his CV to replace all that crufty old cobol running on the cobweb covered mainframe in the corner with this months latest hot language?

    I'm exaggerating a bit, but it does seem to me having worked in the dev industry for > 20 years that a lot of the younger devs really don't understand the serious reasons behind the phrase If It Ain't Broke... If you mention it they either think you're being ironic or just too old to "get it". Sadly its them who don't get it but they're too arrogant/naive to realise it.

    • by Anonymous Coward
      I'm currently working with a project manager that was around at RBS during their last screwup, and he's told us that the last mess was caused when somebody kicked off a bunch of batch jobs, then panicked and stopped them half way through. These jobs were poorly documented, and not robust enough to be able to be started and stopped like that, so it caused a massive mess and they had to go through by hand and try and put everything back to where it had been before the jobs were started.

      This was not new code
      • by Viol8 ( 599362 )

        I'd imagine stopping any batch system halfway through is going to cause a mess. There's a reason it has the word "batch" in it.

        • by Cederic ( 9623 )

          Hence basic shit like rollback capabilities and error recovery. You know, fundamental it practices that have been common sense for decades.

          Shit, assume your batch always runs to completion and you've pretty much guaranteed it's going to fuck things up massively.

          • by Viol8 ( 599362 )

            There's a difference between re-starting and re-running. With a batch the first is normal, the 2nd is a big no no.

      • by Peil ( 549875 )

        This was widely reported at the time. It wasn't a bad batch job, it was a screwed up update of batch software, when it failed they tried to back it out and made a complete balls up

        http://www.theregister.co.uk/2... [theregister.co.uk]

      • by KGIII ( 973947 )

        AC or not... If it was told to you in confidence, why are you sharing it?

    • by u38cg ( 607297 )
      No. You don't get to just make changes like that in that envirnoment. It's probably more likely to do with the fact the crusty old COBOL is running on several layers of emulation (really) because the hardware is generations old, and someone somewhere slipped a unicode character into the message stream, or something equally daft.
  • by nimbius ( 983462 ) on Wednesday June 17, 2015 @09:24AM (#49929053) Homepage
    I experienced the same chicanery with my credit union recently. A system upgrade took everything (offices, ATM's, network transactions, credit card access and online sites) offline for 3 days. On monday the issues still werent resolved. Credit cards didnt work, online transfers failed, networks that normally recognized the payment cards suddenly didnt, and the on-hold wait time ballooned to nearly 40 minutes. My direct deposits stopped working due to the fact that my accounts had, without explanation, been suffixed with an additional integer. No one cared.

    No one cared for a very important reason: there are no repercussions. Banks dont fail in america as im sure they dont fail in europe. theyre secured by the FDIC which raids and reassigns bank ownership quickly and quietly to prevent runs because the very concept of a bank operates on the principle of too big to fail. During the 2008 financial crisis hundreds of banks across the country failed, but were quietly never reported and instead gobbled up by larger institutions. most of the staff, management, and customers just changed letterhead. no one lost a job and the old bank fines became a pittance; a frosting the larger banks kindly paid.

    so if a run happened after 600,000 payments were bungled here in the states, the bank manager would keep his job and just assume a new name like 'ally bank.'
    • by bws111 ( 1216812 ) on Wednesday June 17, 2015 @10:45AM (#49929695)

      That is just nonsense. Banks certainly can and do fail, and the FDIC does NOTHING to protect them. The FDIC is protecting the DEPOSITORS (ie, you). If a bank fails, the owners of the bank have lost their investment. The owners do not get to keep their investment just because the assets of the bank were bought by someone else. There is no reason employees or customers of a bank should suffer because the owners installed bad management. If the new owners keep that same bad management then they risk losing their investment too.

    • by bws111 ( 1216812 )

      PS. If it is a credit union, as you state, then YOU are one of the owners. Did you do due diligence and vote for the proper board members, who would hire the management you want, or do you just let someone else do all that and whine when things go wrong?

    • by mystik ( 38627 )

      Does your credit union use Intuit's web platform for online banking access per chance?

      My credit union (in RI) recently went through a similar upgrade (including adding an extra digit to accounts), but to their credit, it seemed to go much smoother.

    • A credit union isn't a bank. They are regulated, and deposits are insured, by the NCUA. My company is well aware of the fact that the NCUA regularly audits our clients, so I assure you that such a debacle will not go unnoticed. If you haven't received redress for this incident, you should pursue it.
  • by Anonymous Coward

    Posting anon, because customer data... Circa 5 years ago I worked in support for a commercial software vendor. A big bank in US used several of our software, in a somewhat customized way. One of those products was for network file transfer, one for data manipulation. They called with an issue of a small percentage of inbound transfers consistently failing and after manually reprocessing them a few minutes later they always succeeded. Long story short - they've customized the transfer product to drop a file

  • by xxxJonBoyxxx ( 565205 ) on Wednesday June 17, 2015 @10:44AM (#49929681)

    I installed some transaction processing software at RBS in the late 1990s. Here's hoping they retired it a while back...I haven't heard from these guys in over a decade.

  • By far the most interesting statement on this comes from Iain Martin at the Daily Telegraph, who says that at the time RBS (a fairly small bank) took over the much-larger and more established NatWest bank:

    > His team worked out quickly that the NatWest system was superior to the RBS computer system.... Then they crunched the numbers and
    > confirmed that sticking with plan A and migrating NatWest's customers onto RBS's inferior and cheaper to run system would save more money.

    http://blogs.telegraph.co.uk/

  • One of the things about banks is that most of their core transaction processing is done on mainframes. There's tons of stuff layered on top of it to do fancier things, but the day to day moving of funds from account to account is usually batched and run after business hours on a mainframe. One of the reasons for this is the sheer amount of business logic tied up in these systems, the massive transaction volume. and the fact that you can't easily swap out a working process with something untested.

    The problem

    • This depends on a number of issues.

      There's a few parts to a banking system, and the 'core' - is usually on a mainframe. The name 'core' in this context means "The software system used to store financial account data" - a database, for the most part. It stores the account data, and occasionally a few other pieces of information, but mostly the account data. Account data never really changes, so neither have these "core" systems. They're also like 20-30 years old, and so rarely break anymore. They're tim

  • by quietwalker ( 969769 ) <pdughi@gmail.com> on Wednesday June 17, 2015 @12:49PM (#49930687)

    So, I used to write banking software for a living, and that included being in the damage control team when things went south. Which they did. A lot. Which were brought to the technical staff's attention, without fail, every friday at 4 PM - but I digress.

    This isn't really a big deal.

    This sort of thing happens on almost a daily basis. I'd say that 1 in every 500 banks "loses" a day's transactions each week. From hardware failure to networking problems, to someone entering bad data to simple bugs in the software (usually data-dependent and hard to source), and even an occasional overwrite from a backup. Something breaks.

    Once we had an FI lie about their data center setup - it wasn't redundant, offsite, certified, or even a datacenter. They just had some machines running in their basement. Imagine our surprise when they called up wanting us to (remotely) fix their servers which were under 6 feet of water during a flood. ... again, I digress.

    The thing about these transactions is they're not really lost. Nothing really goes missing.

    See, all financial software is super keen on accounting. I don't mean that in a strictly 'add up the numbers' way, but in an auditing way. There are logs upon logs. Using your ATM card to withdraw $20 probably generates around, oh, I dunno, 30-40 log messages, depending on how it's routed. There's a log of the transmission and response in each node along the way, using a standardized protocol. It's a severe pain in the butt, but even if a system goes down forever, we can regenerate those logs from the other systems. It's a great deal of effort, especially to preserve the sequence order, and it's tedious, but it can be done.

    This could easily account for the initial delay in fixing things.

    Not only that, all these sorts of systems are very keen on sequence-order processing, so if it's just the case that the end of day processing (clearance) system went down/had a bug/etc and none of the transactions were finalized, then they'll just stack up. Once they get that system up again, it'll start processing them again. It might take longer due to a large backload, but it'll eventually complete.

    98% of the time, the only people that notice are companies waiting on a payroll to go out, and most of them are happy enough to accept the financial institution's admission of fault. For a day or two at least. Why this one made the paper, I can't tell you. Probably a customer was savvy enough with social media and decided to burn the bank. I guess they should just be happy the public in general doesn't realize how many problems and how much actual work goes into making banking seem reliable and secure.

    • by Peil ( 549875 )

      Why this one made the paper, I can't tell you. Probably a customer was savvy enough with social media and decided to burn the bank. I guess they should just be happy the public in general doesn't realize how many problems and how much actual work goes into making banking seem reliable and secure.

      A lot of benefit payments and tax credits were due, as these are expected payments from the Government to people who may be relying on that money it was quicly decided to get the story out. As someone mentioned above, at the poorer ends of society, those little handouts midway through the month can mean a lot to a working family.

    • I manage a few enterprise level applications and databases, and it is much the same, to a lesser degree. I recently discovered a bug in our code which a contractor inserted that didn't understand the business rules which no one caught and had been generating bad data for about 6 months. It was done in such a way that unless you were actually looking for it (I discovered by accident while running some statistics that didn't add up which prompted me to investigate). Anyway how this all ties into the topic is

  • "This isn't the first major IT screwup for RBS; in 2012, the company was fined £56 million after a software upgrade prevented about 6.5 million customers from logging into their accounts"

    I read somewhere that RBS downsized their UK CA-7 batch processing department and then imported the one man from India to take over. Not being experienced enough he botched an overnight job and in the attempt to roll back a days worth of transactions accidentally rolled it back a months worth of transactions. Soon
  • That would have been a different sorry l story altogether and maybe one worth reading.
  • When you cut corners with the very people responsible for your IT infrastructure, things like this will happen. Treat them well, even hire them directly, things like this tend not to happen.

    Unfortunately, the UK is rife with this kind of corner-cutting.

Testing can show the presense of bugs, but not their absence. -- Dijkstra

Working...