Forgot your password?
typodupeerror
Bug Government

Xerox "Routine Backup Test" Leave 17 States Without Food Stamps 305

Posted by samzenpus
from the time-to-revert dept.
An anonymous reader writes "People in Ohio, Michigan and 15 other states found themselves temporarily unable to use their food stamp debit-style cards on Saturday, after a routine test of backup systems by vendor Xerox Corp. resulted in a system failure. Xerox announced late in the evening that access has been restored for users in the 17 states affected by the outage, hours after the first problems were reported. 'Restarting the EBT system required time to ensure service was back at full functionality,' spokeswoman Jennifer Wasmer said in an email. An emergency voucher process was available in some of the areas while the problems were occurring, she said. U.S. Department of Agriculture spokeswoman Courtney Rowe underscored that the outage was not related to the government shutdown."
This discussion has been archived. No new comments can be posted.

Xerox "Routine Backup Test" Leave 17 States Without Food Stamps

Comments Filter:
  • by ThatAblaze (1723456) on Sunday October 13, 2013 @04:00PM (#45115621)

    "[A politician] underscored that the outage was not related to the government shutdown."

    These words should never have to be said.

    • by Anonymous Coward on Sunday October 13, 2013 @04:16PM (#45115737)

      Let me remind you all of Senator Obama's words from 2006 regarding the raising of the debt ceiling. He voted against raising the debt ceiling at that time.

      "The fact that we are here today to debate raising America's debt limit is a sign of leadership failure. It is a sign that the US Government can not pay its own bills. It is a sign that we now depend on ongoing financial assistance from foreign countries to finance our Government's reckless fiscal policies. Increasing America's debt weakens us domestically and internationally."

      Source [snopes.com]

      How true are those words? I only wish President Obama still believed what he did as Senator.

      • That statement is no longer operative.

      • Re: (Score:3, Informative)

        by Ksevio (865461)
        However, when Obama was a senator, the administration had turned a budget surplus into a budget deficit. That was the result of reckless spending on tax cuts and wars.

        That contrasts with the current administration which was given a large deficit to start with (made worse by declining tax revenues due to the recession) that has cut government spending.
        • by clarkkent09 (1104833) on Sunday October 13, 2013 @08:28PM (#45117139)

          |Congress controls the spending. Btw, "Bush's wars" propaganda is getting boring. Democrats overwhelmingly supported them (unanimously in case of Afghanistan) and it was Clinton admin that set the stage for Iraq war with regime change policy (Iraq Liberation Act 1998). You can argue whether it was right or wrong but you can't blame just one side for it - they all had the same intelligence. Same applies with the current administration. Would they really cut the spending if Republicans weren't fighting for it all along.

      • Let me remind you all of Senator Obama's words from 2006 regarding the raising of the debt ceiling. He voted against raising the debt ceiling at that time.

        The debt ceiling really isn't related to the government shutdown. The media (and certain politicians) are trying to conflate the two -- and it seems that they're succeeding, because most people don't seem to realize that there's a difference.

    • "[A politician] underscored that the entry of the US into WW2 was not related to the attack on Pearl Harbor."

  • by dwhitaker (1500855) on Sunday October 13, 2013 @04:02PM (#45115637) Homepage
    Backups don't always work - that's why you test them. This time they did not work - much better that you experience problems when you anticipate them than when everything else is going wrong, too. It's unfortunate that the system was down, but it seems they got it back up in a reasonably quick time frame. Moreover, merchants are supposed to have manual means of recording EBT payments for just such a scenario.
    • by amiga3D (567632)

      The local Walmart was lacking in any backup method. They had at least 50 buggies packed full of food sitting around the registers and a lot of pissed off customers. Glad they got it back up, I don't look forward to that riot.

    • by plover (150551)

      Moreover, merchants are supposed to have manual means of recording EBT payments for just such a scenario.

      Those lead to fraud loopholes, and not just EBT. Someone can claim, "oh, my card doesn't work because the system is down, just fill out the paperwork for me, please." Thats more problematic when nothing distinguishes a "DECLINE-card has no funds" from a "DECLINE-system is broken" to the cashier.

      • Did you just pull random speculation out of your ass? If the system works, the card gets used. Most people on the registers are not going to know about any backup system. Want to use your card? swipe it first. Doesn't work? swipe it. I need to know what the problem is so I can ask my manager.

        Manager comes over. What happens when you swipe it?

        Now, if you're talking about friend of the cashier, that would raise lots of red flags to have piles of swipes work, followed by a single transaction by the fri

    • by Rick Zeman (15628)

      Backups don't always work - that's why you test them. This time they did not work - much better that you experience problems when you anticipate them than when everything else is going wrong, too. It's unfortunate that the system was down, but it seems they got it back up in a reasonably quick time frame. Moreover, merchants are supposed to have manual means of recording EBT payments for just such a scenario.

      Exactly. Imagine a more catastrophic meltdown down the road and all of the Nancy Naysayers saying, "WhyoWhy didn't anyone test it?"

    • ... much better that you experience problems when you anticipate them than when everything else is going wrong, too.

      So what you're saying is: if you make a 1 million dollar mistake, your response should be "Phew! At least I didn't lose 5 million!".

      An outside observer might suggest that losing the $1 million is bad on its face. Mitigating the outrage by making false comparisons is the sort of thing politicians do, as a dodge for responsibility.

      Should we be sanguine about these sorts of problems because they're not the worst possible scenario? Is that an acceptable excuse?

      • Risk exists, and appropriate management of it is how the world moves forward. I'm not privy to the inner-workings of Xerox, and it is entirely possible that they were not following best practices and that a substantial portion of this is due to operator error. However, even routine tests have risk associated with them. My point is that immediately blaming Xerox is not a good reaction: making and testing backups is an effort to mitigate risks (with much worse outcomes). If minor problems arise from time to t
      • by Dunbal (464142) *
        Errare humanum est
      • by DarkOx (621550)

        Should we be sanguine about these sorts of problems because they're not the worst possible scenario? Is that an acceptable excuse?

        To some degree yes, mistakes happen, especially with large complex systems. We should count having avoided the worst cases scenario as a success and see what can be done to mitigate the failure mode that did occur in the future, and the answer to that question might very well be nothing or nothing less costly than the future number of anticipated similar failures.

    • by sjames (1099)

      When i test failover, I announce the test in advance so people can have a plan ready in case the failover doesn't. It sounds like this was un-announced and nobody had any sort of plan to deal with a failure.

    • "Backups don't always work - that's why you test them."

      ... and it's also why, when you do test them, you make sure to test them on the data affecting 17 different states!

  • Hahaha I was walking thru Kroger's yesterday and they kept announcing over the speakers "We cannot accept EBT today because our computers are having problems."

  • Fail-safe (Score:4, Insightful)

    by girlintraining (1395911) on Sunday October 13, 2013 @04:04PM (#45115649)

    People in Ohio, Michigan and 15 other states found themselves temporarily unable to use their food stamp debit-style cards on Saturday,

    Why is it that a convenience -- our credit cards, are able to weather a failure like this by simply allowing all purchases, but our food stamp cards simply stop working? Credit card systems are, at every level, designed to cope with a failure by simply authorizing the purchase. Only a very small number of transactions would have been failed anyway for insufficient funds, etc., and these are reconciled when that part of the system is restored to service... meaning there's very little loss to the provider for this.

    For that matter, if they've decided to design the system in this fashion, where were the redundancies? If a routine backup can result in failure on this scale, then it begs the question of where and how the backup of the actual systems, not just the data, got overlooked.

    • Re:Fail-safe (Score:4, Informative)

      by dwhitaker (1500855) on Sunday October 13, 2013 @04:07PM (#45115681) Homepage
      One of the news articles mentioned that merchants were supposed to record transactions manually and allow purchases up to $50. Not ideal, and not the same as allowing all purchases, but it is a provision of the system that is supposed to ensure people aren't deprived food and necessities during a short outage. Now, whether retailers actually followed protocol is an entirely different matter (and one that does have implications for the way the system is run).
      • Re:Fail-safe (Score:4, Insightful)

        by girlintraining (1395911) on Sunday October 13, 2013 @04:58PM (#45115945)

        One of the news articles mentioned that merchants were supposed to record transactions manually and allow purchases up to $50

        Due to the government shutdown, I cannot provide primary source data such as would be normally available from the USDA, etc. In lieu of that, the links provided represent the best non-authoritative sources available at this time.

        The average household size is 2.48. Source [usatoday.com].
        The average person spends about $70 a week on food Source [loweryourspending.com]
        76% of people on food stamps are disabled, elderly, or children. Source [feedingamerica.org]
        Around 44 million Americans are on food stamps now*

        * [Couldn't find credible source; Estimated from multiple sources]

        This would mean that the average weekly trip to the grocery store, for an average household, would be $173.60. If your number is correct, then the government has opted to allow vendors to 28% of a family's food to be processed. Also according to the article, this outage may last up to three days.

        Now here's the thing; A lot of those families live 'paycheck to paycheck'. Even if it is welfare; They don't have a fully stocked pantry. If they don't buy food today, a lot of them don't eat. And most people go shopping on the weekend. Your quoted $50 means the average family runs out of food in just under two days. I was unable to find any citation to back your assertion that they were allowing purchases as long as they were under $50 as well, so I have my doubts as to its validity. Anecdotally, two of my friends who have food stamps in the midwestern area reported being unable to purchase any food or remove any amount of cash benefits from their accounts.

        So either the situation is 'rather bad' -- 1 in 8 Americans will be going hungry for at least one day this week on average. Or it's 'very bad', in that 1 in 8 Americans will be going hungry for three days. And possibly longer -- many of those people use public transportation or arranged rides to get to the grocery store every week. Especially the elderly and disabled. These rides are picked out weeks ahead of time. For them, they could be looking at not eating for a week or more.

        So I return to my original point: Why is it that credit card companies, who offer a convenience, do this, but our government, which provides something that in a very literal sense is life or death to some people, does not? There is no answer to that question that I come away with that makes this look like anything other than criminal neglect of a vulnerable population.

        • by sumdumass (711423)

          I need to challenge your premise a bit.

          Living paycheck to paycheck should have no bearing on how stocked a pantry is when the pantry is being stocked by welfare benefits cards. It is literally like me saying here is $174.00, buy food with it and only food and you waiting until your paycheck comes in to purchase the bare minimum and save the rest for your next paycheck.

          Furthermore, and I understand this is anecdotal, but it is representative of most of the people I know getting food stamps via EBT cards, mos

        • Thank you for bringing numbers into this. My $50 number is from one of several articles like this: Source [nbcwashington.com].

          My thought was that allowing up to $50 is certainly not enough for a typically grocery trip, but that if the outage was of a short duration (measured in hours rather than days), that it might be sufficient. It's at least a system in place today rather than one that would ostensibly need to clear numerous political hurdles.

          I think some of the other threads have covered that EBT is run more like debit
      • by Slugster (635830)

        One of the news articles mentioned that merchants were supposed to record transactions manually and allow purchases up to $50. ...

        The one I work for didn't. But then, they couldn't. There is no mode programmed into the (computerized) cash register system in the stores to allow such a thing (at least at the chain I work at).

        Also, the lack of such a capability may be no accident. With no way to verify that a card was still good or had money on it, well, -the people might, um, forget how much they're really

    • That's because the EBT cards function like Debit cards, not credit cards. So they need to contact the account to verify funds every time.

      They could make it some other way, but we wouldn't want people cheating the government by getting one extra cart of groceries early, would we. All because we didn't program the computer to check the cards balance every time.

    • Only a very small number of transactions would have been failed anyway for insufficient funds, etc., and these are reconciled when that part of the system is restored to service... meaning there's very little loss to the provider for this.

      Right. And if EBT payments did likewise, it'd be a headline in some newspaper that Sen. Local voted for allowing welfare recipients to receive benefits to which they weren't otherwise entitled.

      Not a commentary or criticism; simply a statement of fact.

    • by Hartree (191324)

      That's the system they went to in New Mexico when I was working for a card processor there. When the system first started, there was no default approval.

      That was bad. Try telling someone "Sorry. Your kids are going to go hungry tonight."

      So, they allowed store level approvals up to a certain amount when the system was down. It got abused some, but was a much better option than a blanket denial.

      This was in the late 1990s, and would have varied from state to state, obviously. I would expect most states would h

    • by Shavano (2541114)

      Credit cards are taken as a promise to pay. If you have the card, either you are good for it or the credit company is the one who has to come after you for payment.

      Debit cards don't work that way. If they can't verify the funds in your account, they don't get paid.

  • This can get scary: (Score:5, Interesting)

    by Hartree (191324) on Sunday October 13, 2013 @04:37PM (#45115843)

    In the late 1990s, the company I worked for was one of many processing EBT card transactions for grocery stores in New Mexico when they first switched to it from paper food stamps. The bank that was the approving authority for them (the next higher link up the chain from us) had a system problem and had been down for about 45 minutes.

    I got a call from a very stressed sounding manager at a store in a bad neighborhood of Albuquerque and explained that the outage was statewide, and I'd already called the next highest level.

    His response: "You don't understand! These people carry guns."

    I really didn't have a good answer for that one, but certainly sympathized.

    They later changed the rules so that when the statewide system was down, they could approve it at the store and then take out any overuse from later payments. That got abused, but it made some store managers a lot less nervous.

  • by Jayfar (630313) on Sunday October 13, 2013 @04:52PM (#45115917)

    With various reports referring to it as a power outage and others as a test of backup systems, I'd guess this was a generator load test where something went wrong with the transfer switch. We do those off-hours monthly at the data center where I work and, being the nervous sort, I'm grateful they usually coincide with one of my days off, although ours have gone smoothly.

  • I actually feel for the Xerox people working on this. I know what it is like to have a system with intense and broad functionality and you are frozen in fear every time you make the slightest change. Testing ahead of time is great but can you be 100% certain that you have 100% test coverage? If you are then you are a fool.

    If I had to point any fingers it would be that they should have a multi-layered deployment system where they deploy to the test center, a small random group, a larger group, and then na
  • The article is vague as to what Xerox provides. I gather they're not just the hardware vendor. Do they supply the software? Do they handle clearance, like a credit card company? What exactly is their role?
  • by J'raxis (248192) on Sunday October 13, 2013 @11:19PM (#45118151) Homepage

    Maybe they can blame their buggy copiers [theverge.com]. Didn't this used to be a quality company?

"The vast majority of successful major crimes against property are perpetrated by individuals abusing positions of trust." -- Lawrence Dalzell

Working...