Become a fan of Slashdot on Facebook

 


Forgot your password?
Close
typodupeerror
×
Bug

Programming Bug Costs Citigroup $7M After Legit Transactions Mistaken For Test Data For 15 Years (theregister.co.uk) 135

Posted by msmash from the field-day dept.
An anonymous reader shares a report on The Register:A programming blunder in its reporting software has led to Citigroup being fined $7m. According to the US Securities and Exchange Commission (SEC), that error [PDF] resulted in the financial regulator being sent incomplete "blue sheet" information for a remarkable 15 years -- from May 1999 to April 2014. The mistake was discovered by Citigroup itself when it was asked to send a large but precise chunk of trading data to the SEC in April 2014 and asked its technical support team to help identify which internal ID numbers they should run a request on. That team quickly noticed that some branches' trades were not being included in the automated system and alerted those above them. Four days later a patch was in place, but it wasn't until eight months later that the company received a formal report noting that the error had affected SEC reports going back more than a decade. The next month, January 2015, Citigroup fessed up to the SEC.The glitch resided in new alphanumeric branch codes that the bank had introduced in the mid-1990s. The program code filtered out any transactions that were given three-digit branch codes from 089 to 100 and used those prefixes for testing purposes. The report adds, "But in 1998, the company started using alphanumeric branch codes as it expanded its business. Among them were the codes 10B, 10C and so on, which the system treated as being within the excluded range, and so their transactions were removed from any reports sent to the SEC."
This discussion has been archived. No new comments can be posted.

Programming Bug Costs Citigroup $7M After Legit Transactions Mistaken For Test Data For 15 Years More

Programming Bug Costs Citigroup $7M After Legit Transactions Mistaken For Test Data For 15 Years

Comments Filter:
  • I wouldn't call it "remarkable" that it wasn't caught for nearly 15 years. It actually makes sense, as the assumption was that 089 to 100 wouldn't include 10B, 10C, etc. Those kinds of mistakes can happen, and very easily. Just goes to show that you should be more explicit with how you filter data, in many cases.

    • I wouldn't call it "remarkable" that it wasn't caught for nearly 15 years. It actually makes sense, as the assumption was that 089 to 100 wouldn't include 10B, 10C, etc. Those kinds of mistakes can happen, and very easily. Just goes to show that you should be more explicit with how you filter data, in many cases.

      Also, if a business rule changes previous assumptions there is a need to understand the impact. The issue there, though, is for whatever reason this may have been overlooked either because the original coders had moved on or the is a belief that it is just a set of new values that couldn't possibly impact anything?

    • So what type of character set considers 10B to be less than 100?

      • Re:15 Years (Score:5, Informative)

        by Stormy Dragon ( 800799 ) on Wednesday July 13, 2016 @04:50PM (#52506537)

        EBCIDIC [wikipedia.org]

        Extended Binary Coded Decimal Interchange Code[1] (EBCDIC[1]) is an eight-bit character encoding used mainly on IBM mainframe and IBM midrange computer operating systems.

        • Beat me to it.

        • Re: (Score:1)

          by Anonymous Coward

          "but IBM's own description of the EBCDIC variants and how to convert between them is still internally classified top-secret, burn-before-reading. Hackers blanch at the very name of EBCDIC and consider it a manifestation of purest evil."

          Well there's your problem!

        • Re:15 Years (Score:5, Funny)

          by dgatwood ( 11270 ) on Wednesday July 13, 2016 @05:35PM (#52506785) Homepage Journal

          EBCDIC [wikipedia.org]

          Extended Binary Coded Decimal Interchange Code[1] (EBCDIC[1]) is an eight-bit character encoding used mainly on IBM mainframe and IBM midrange computer operating systems.

          I have a feeling that the whole "Nobody ever got fired for choosing IBM" thing is about to become very untrue at Citigroup....

          • No worries there, the one person that wrote the code in the first place has long since been laid off and replaced with 10,000 outsourced Indian contractors.

      • Re: (Score:2)

        by Kjella ( 173770 )

        That might not have been the actual check, for example it might have been:

        WHERE branch_code <= '088' OR branch_code >= '101'

        Just to make it explicit that yes, we did intend to exclude '089' AND '100'. I tend to prefer it that way if the boundaries are somewhat arbitrary, unless it's about dates where <= '2015-12-31' and < '2016-01-01' aren't the same when implicitly converting to a time stamp. Like '2015-12-31T12:34:56' <= '2015-12-31' = false, '2015-12-31T12:34:56' < '2016-01-01' = true.

    • It actually makes sense, as the assumption was that 089 to 100 wouldn't include 10B, 10C, etc.

      That only makes sense to software engineers who are terrible at their jobs.

  • Sounds like it worked exactly as designed. Should have consulted your dev team before changing the way you name things. Maybe have IT in these meetings?
  • Proper unit tests would have prevented this problem in the first place.

    • Dear Rich Bastard (Score:4, Informative)

      by Hognoxious ( 631665 ) on Wednesday July 13, 2016 @05:00PM (#52506585) Homepage Journal

      As would having a separate system for testing, rather than trying to create a test island within your actual real live [that's enough - Ed] production system.

    • Would they have? When the code was written, if there had been such a thing as automated unit tests, one would have been written that checked three-digit numeric strings (9(3) or X(3)). (COBOL was not designed to support automated tests, so I doubt there ever were any.) When the branch code format was changed, if someone had realized the appropriate unit test needed to be changed that someone would have realized the program needed to be changed.

  • Not surprising (Score:5, Interesting)

    by somenickname ( 1270442 ) on Wednesday July 13, 2016 @05:47PM (#52506845)

    Anyone who has worked in the finance industry on the tech side of things has probably seen eye-searing levels of problems like this. It's clusterfucks all the way down. It always surprised me that something that seems like such a natural fit for software was always, without fail, so riddled with glaring bugs that it's almost unfathomable that you are the first person to notice them. At a lot of shops, the bugs are so ingrained in the process that they can't even be fixed. Working in the finance industry certainly doesn't inspire confidence in the finance industry.

    • I have seen a fair bit of banking code in my day, and I agree. Much of what I saw was ludicrously awful and barely working. But that it was working at all seemed a miracle. Certainly, I've never seen another industry that was saddled with as much crap.

    • Anyone who has worked in the finance industry on the tech side of things has probably seen eye-searing levels of problems like this. It's clusterfucks all the way down. It always surprised me that something that seems like such a natural fit for software was always, without fail, so riddled with glaring bugs that it's almost unfathomable that you are the first person to notice them. At a lot of shops, the bugs are so ingrained in the process that they can't even be fixed. Working in the finance industry certainly doesn't inspire confidence in the finance industry.

      I tell you, every time I have to interact with the software people at a new bank I often contemplate becoming a cash only enterprise.

    • I used to do electronic trading support for a large investment bank and yeah, their applications and processes tended to be disasters.

      Part of the problem was bad developers; I saw some very basic mistakes like case-sensitive columns in a database that were supposed to be case-insensitive.

      Poor QA process; management would have the team simply not report bugs because it made "the numbers" look bad and they were trying to get the software out the door.

      Poor training. Support staff were never really trained

    • Partly it's because COBOL has some unique ways to screw up. At least when I used it last, control structures were holdovers from the dark ages, and data types had to be ridiculously overspecified. Take a money amount that handles six digits (9(6)V99), decide you have to increase the number of digits (say to 9(8)V99), and you have to make absolutely sure you change all places where the amount is handled (and you have to keep track of file operations, MOVE CORRESPONDING, everything) or it will silently dro

  • I guess this week we're punishing people for "unintentional" failures to comply with regulations again?

  • If I have a software glich in my brain and enter made up data in my tax return, for 15 years, will IRS be as easy on me as SEC has been to Citi?

    The banksters are simply too big to jail and too big to even question. Break up the banks.

    "Honey I shrunk the government. And the banksters drowned it in the bathtub!"

    • IRS wouldnt have gone as easy as SEC, in case the tax citibank filled had a glitch. Whats your point anyway? IRS is tougher than SEC?

      If you could prove your brain had a glitch for 15 years, I bet the SEC would have gone as easy on you as it had on citibank. Good luck proving it though.

    • Are you planning to screw up your income taxes intentionally, or do you just make mistakes? I've made mistakes. I've gotten reasonably polite letters from the IRS saying that their information and/or calculation differs from mine, and that I can either send them a check for the difference plus a small amount of interest or tell them why not. The year I put dividends into the interest income instead, I explained what must have happened, and they accepted my explanation and I owed nothing. The IRS is int

  • if they were serious, it would have at least been a $100 billion fine.

    • If it was intentional non-compliance, yes. An unintentional oversight by an organization acting in good faith shouldn't be punished as harshly.

  • ...given an alphanumeric sort, 10B and 10C fall outside the range of 089 to 100. So TFS got it wrong.

    I'd like to know which favored customers had accounts in branches 10B and 10C. And what sorts of trades slipped by SEC scrutiny for 15 years.

    • Well, there would have been no point in coding it to handle alphanumeric input as a valid ID, so they probably had it handle non-numeric input by setting the ID to one of the test numbers to keep bad data from going to the SEC.

  • As a beta tester for AppleLink PE... (Score:3)

    by jpellino ( 202698 ) on Wednesday July 13, 2016 @09:40PM (#52507821)
    which eventually became AOL, we were routinely sent CDs with patches on them. Eventually we got the CDs that would patch our beta releases to become public release apps. As beta testers the service was charged at half price. Almost a year into the public release, I got a phone call from Steve, the boss at Quantum, letting me know that the one thing they forgot to patch in the upgrade CDs was the switch to full price. So would you please cut us a check for everything you paid us already for the past year.? Um, no... by the way how many users did this affect? We're not sure. Dozens? Well yeah. Hundreds? Yeah. Thousands? Look, that's not the important part. I believe I offered to pay double the monthly bill until I was caught up. Never heard back, next release placed us at full charges. I bailed once it was AOL, and it was back to Delphi and The WELL.

  • Why "smart" IDs are a bad idea (Score:5, Insightful)

    by Tony Isaac ( 1301187 ) on Wednesday July 13, 2016 @11:51PM (#52508173) Homepage

    This is a perfect illustration of why "smart" IDs are a bad idea. Any time you encode attributes (like "this is a test transaction") into an ID (like a range of bank branch IDs) you are asking for trouble. Everybody does it, but it's usually just plain lazy and careless. DON'T! Add an attribute that marks the transaction as a test transaction! Then anybody who sees it will instantly know the difference.

    • Re: (Score:2)

      by thewils ( 463314 )

      Amen to that one!

      Any time you have one piece of information serving two roles you are opening yourself up to problems down the line.

    • Re: (Score:2)

      by Kiuas ( 1084567 )

      Completely agreed. I don't get why anyone would do it the way they did to begin with?

      I mean, it's not being actually used for anything during testing other than testing, so why can't they just create the number of test transactions they need, and then wipe the history after testing is done and it moves to production? That's what we're doing here (working on building an ERP for hospital/patient logistics): the test-environment is a sandbox in which we just create any types of orders we need, and prior to lau

      • Completely agreed. I don't get why anyone would do it the way they did to begin with?

        I'm going to take a guess that you haven't been in the software field for more than twenty or thirty years, and have no feel as to how things were done in the old days. They may have wanted to keep the test data around for regression testing, which actually would show more forethought than usual back then. In any case, keeping the data around wouldn't hurt much of anything since the system would ignore it except for tes

        • Re: (Score:2)

          by Kiuas ( 1084567 )

          I'm going to take a guess that you haven't been in the software field for more than twenty or thirty years

          Correct, just a couple years so far so I'm a padawan, hence my question.

          Thanks for the answer, makes more sense now!

    • The greater sin here was that they were using a live system for testing. That's extreme negligence. If they followed best practices, then there would be no need for special "testing" IDs at all.

      If you have a need for special "testing" IDs at all, then you are already in dangerous territory and, at a minimum, need to be much more cautious than would ordinarily be called for. A good rule of thumb: the need for testing IDs is a big red flag that something is likely very wrong in the production process.

      • The greater sin here was that they were using a live system for testing

        I don't know about that. There are many situations where it is impossible, or infeasible, to reproduce an entire production environment with test equipment. For example, if you are writing software to use a credit card processor, you have no choice but to connect with the "real" credit card processor to do your testing, since you don't have access to the processor's system. Sure the processor might have a test environment for you to use, but there are bugs that don't show up in a test environment, only i

        • You are correct, there are circumstances where testing in a production environment to some degree is not possible. But in my experience, those circumstances are really very rare. More often, companies decide that "impossible" means "expensive or very inconvenient."

    • There's an excellent chance that this data was on punch cards at one time, and punch card columns are valuable and scarce real estate. Adding another column to denote test data would have seemed wasteful when they could just set aside a range of branch numbers for the purpose. This sounds like forty-year-old software to me, and that's not how things were done back then.

      • Oh how well I know! I'm old enough to have used punch cards. In a sense, it's like the Y2K problem. It's a legacy technique that had a purpose in the beginning, and was never corrected when it became obvious that there was a real cost. The biggest difference is that the "smart key" problem is still in wide use in NEW applications today, while the Y2K problem has largely been obliterated.

        • Back in the stone ages of business software, a lot of it was in assembly language, and the standard idea was that you'd rewrite your stuff when you got a new system (it wasn't universal, which is why at least some 360s came with 709x emulators). I don't think many people had the idea that all this software would last for decades. We at least should know better today.

          Y2K was a serious problem with a hard deadline, which is why software was modified to deal with it. (I don't expect to be around for the

    • Re: (Score:1)

      by healyp ( 1260440 )
      Same idea but, this is my pet peeve with the license plates in Connecticut.

      They're designed to be human readable and have special codes for everything.

      School Buses have a custom plate with a picture of a school bus, Interstate Taxis are prefixed and suffixed with "Z", In-state Taxis are prefixed and suffixed with "T", Trucks have a special plate, Combination passenger/work trucks have another plate. Passenger cars have three styles, XXX{Space}XXX, XXXXXX and a new one XX-XXXX. Municipal cars have a number

  • It's a management bug. The programming was fine, but somebody failed to make sure it was updated for the branch ID change. It was never intended to handle alphanumeric input, so management should have made sure the programmers knew about the change and thoroughly tested how the software handled it.

  • This sounds suspiciously like they probably had older developers who likely knew what they were doing and the history of the data/application/business who retired/fired/left were replaced by younger cheaper models, who were given a task, did it as best they could without all the prior experience and knowledge (and likely little or no documentation). Having no one else in the organization that understands or sees what is going on, fast forward 15 years, and presto a big problem (though 7 million for a corpor

Slashdot Top Deals

Technology is dominated by those who manage what they do not understand.

Close