Please create an account to participate in the Slashdot moderation system

 


Forgot your password?
Close
typodupeerror
×
Microsoft IT

MS Excel Data Files Exceeding the Maximum Size Resulted in Nearly 16,000 Covid-19 Cases Go Unreported in England (bbc.com) 142

Posted by msmash from the closer-look dept.
rastos1 shares a report: The health secretary has said a technical glitch that saw nearly 16,000 Covid-19 cases go unreported in England "should never have happened." The error meant that although those who tested positive were told about their results, their close contacts were not traced. By Monday afternoon, around half of those who tested positive had yet to be asked about their close contacts. Labour said the missing results were "putting lives at risk." Experts advise that ideally contacts should be tracked down within 48 hours. The technical error was caused by some Microsoft Excel data files exceeding the maximum size after they were sent from NHS Test and Trace to Public Health England. It meant 15,841 cases between 25 September and 2 October were left out of the UK daily case figures. PHE said the error itself, discovered overnight on Friday, has been fixed, and outstanding cases had been passed on to tracers by 01:00 BST on Saturday. But Health Secretary Matt Hancock told MPs the incident as a whole had not yet been resolved - with only 51% of those whose positive results were caught up in the glitch now reached by contact tracers.
This discussion has been archived. No new comments can be posted.

MS Excel Data Files Exceeding the Maximum Size Resulted in Nearly 16,000 Covid-19 Cases Go Unreported in England More

MS Excel Data Files Exceeding the Maximum Size Resulted in Nearly 16,000 Covid-19 Cases Go Unreported in England

Comments Filter:

  • Literally a matter of life and death... (Score:5, Insightful)

    by Bearhouse ( 1034238 ) on Monday October 05, 2020 @04:14PM (#60575050)

    So of course no need for "expensive", experienced IT professionals, we can just hack it together with Excel, right guys?
    Your tax dollars at work.

    • Re:Literally a matter of life and death... (Score:5, Interesting)

      by BrainJunkie ( 6219718 ) on Monday October 05, 2020 @04:20PM (#60575086)
      But that is what typically happens with all sorts of organizations that implement "expensive" IT solutions.

      I've done maybe 50 integrations for SAP deployments of various sizes and one of the consistent questions from the business users is "how do I get this into Excel". Nearly everyone configures the thing to get reports into Excel in a couple of clicks, and most of the analysis and reporting happens from there, even after they spend $10MM+ on a BI/BW HANA system. Most of the big SQL deployments I've been on ended up just putting SSRS table dumps out there, in place of fully developed reports. The only replacement I've seen consistent success with is Tableau (and Tableau-like systems), and only then when the company has Tableau expertise working in outside IT.

      • Well yeah. You're dealing with SAP. Who the hell wants to use that pile of yak manure?

        Considering it used to be a German company, it is truly amazing how inefficient and non-userfriendly it is.

        • Re:Literally a matter of life and death... (Score:5, Informative)

          by BrainJunkie ( 6219718 ) on Monday October 05, 2020 @05:17PM (#60575286)
          It is quite user unfriendly, but it is so large I don't blame them for declining to do the regression testing that would be need to change it entirely. Their BI/BW system is much better in that regard.

          But I've done Oracle ERP, and Dynamics, and Salesforce, and a few others...and the story is always the same. Either how do we get this into Excel or how do I connect my Access database to this. It is what people know so it is what they look to and I don't blame them. Business reporting and data analysis moves pretty fast and I've never seen an IT support team at a large company that can keep up. Nor have I seen a leadership team that understands any of this well enough to do any better.

          • ...Business reporting and data analysis moves pretty fast and I've never seen an IT support team at a large company that can keep up. Nor have I seen a leadership team that understands any of this well enough to do any better.

            That, I think, is the essence of the problem. The large company does not have enough competent people in IT to keep up with the changing business needs. The solution is more people but that means more money, and the "leadership team" doesn't think the additional money is justified.

            • Or that the people that need to interface with the data need it to be flexible enough to do ad hoc analysis because their bosses keep asking for random ways of aggregating the data. Or because getting data into a database in bulk is surprisingly difficult. Or because editing data in a database without admin permissions is typically done via a form allowing for access to only a single record at a time. There are a lot of reasons companies still use Excel, but mostly it comes down to cost in developing adequ

              • Re: (Score:3)

                by Bert64 ( 520050 )

                Excel is the hammer that makes everything else look like a nail...
                It's probably all the staff have ever been given, and thus all they know how to use.
                So you end up with lots of flakey crap.

                There's also the lack of urgency that results from having a flakey temporary kludge...
                If you have nothing, there will be a drive to fix it.
                Once you have a flakey temporary kludge, that will quickly become permanent and any efforts to develop a proper long term solution will be cancelled.

                • Re: (Score:2)

                  by Bongo ( 13261 )

                  This is true. There's other ways, but Excel is all people know.

                • I'm holding out hope that in another decade we'll be saying that about R instead of excel. I'm very guilty of doing the same thing with R.

                  The difference is that it's organized into projects and all of the scripts are commented. And since it's open source, there's minimal chance of shit suddenly becoming incompatible when the cloud version gets updated.

                  Office 365 auto-updating breaking macros is the absolute fucking devil, and it's the stake through the heart, as far as I'm concerned. Although I do gain life

                  • Re: (Score:2)

                    by imidan ( 559239 )
                    I tend to work on projects in different languages pretty frequently over the course of a year, and at one point, I needed to script the creation of a bunch of SQL statements for a one-off batch update run. The language I'd been using for other projects at the time was R, so I just did this string processing script there instead of some language more obviously appropriate. So, yeah, at that moment, R was my hammer. It worked out great, spat out all of my SQL, and I ran it against the server. It felt a little

            • Re: (Score:2)

              by imidan ( 559239 )

              The solution is more people but that means more money, and the "leadership team" doesn't think the additional money is justified.

              I applied for a job with a group in a large organization that did analytics related to their Contact Management System, organizing fundraising, grants, estate gifts, and so forth. The group was super excited because they'd been given a decent budget to hire a senior developer who could help with migrating legacy reports and processes to the new system, help generate automated repo

      • This is an expensive IT solution. The whole track and trace thing, that this is part of, is costing the UK taxpayer £2bn.

        I think the real problem is that its all under the charge of Dido Harding. ex CEO of TalkTalk who happily ignored all the IT teams who repeatedly warned of security problems, and only woke up enough to get a PR team on the case.

    • for the type of data they are putting together for a bulk upload Excel is fine, just not the ancient XLS file version they picked. Though the way the UK is going even a million rows may not be enough.

    • Re: (Score:2)

      by AmiMoJo ( 196126 )

      It cost 12 BILLION Pounds!

      • That's six million tons!

        • Well, well, well!

          Slow down your horse cowboy!!

          Is that US pounds or UK pounds or even German pounds?

          Is that a gross ton, metric ton or a long ton or a short ton or is it even a tonne?

          You are confusing me!

          Ah! I got it, you mean the megagram!

    • Well to be fair to them if they had competent leadership they wouldn't need to hire experienced IT professionals and the COVID cases wouldn't have exceeded the maximum Excel file size.

      Instead we get a country that was rivalling the USA in ignoring the virus. Although they definitely were ahead of the game when it comes to sending their respective moron running the place to hospital.

  • Desktop spreadsheet for this? (Score:5, Insightful)

    by Philotomy ( 1635267 ) on Monday October 05, 2020 @04:15PM (#60575054)

    I can't believe someone made the decision to use a desktop spreadsheet for this kind of data. Wait...no, actually I *can* believe it...

    • Re: (Score:3, Informative)

      by Jarwulf ( 530523 )
      Actually excel is the defacto standard for this sort of thing both in government and corporate settings. What you think some done is going to bother banging out R code?

      • Re:Desktop spreadsheet for this? (Score:4, Insightful)

        by Philotomy ( 1635267 ) on Monday October 05, 2020 @04:29PM (#60575116)

        That sad fact is "why I can believe it." But I still say: use the right tool for the job. Desktop spreadsheets used like databases are...a plague.

      • Actually excel is the defacto standard for this sort of thing both in government and corporate settings. What you think some done is going to bother banging out R code?

        Databases exist for a reason.

        "MS Excel Data Files Exceeding the Maximum Size Resulted in (data loss/corruption)"

        In fact databases exist for that exact fucking reason.

        So yeah, this is the "defacto" standard...if you're dumb enough to assume you're not going to eventually lose data, and in the most preventable way.

        • Re: (Score:3)

          by vakuona ( 788200 )

          This spreadsheet probably started life as a very simple tool pulling together a small number of tests and result, and grew as the COVID crisis grew. Over summer, as COVID seemed to come under control, it would have probably seemed unnecessary to spend time and effort and money developing the tool further until cases started increasing quite quickly and PHE was overtaken by events.

          Yes, someone probably should have realised the issue sooner, but they found the issue, and they have a short term fix and probabl

        • Re: (Score:2)

          by HiThere ( 15173 )

          Actually, that's only one of the reasons databases exist. Another is to do sanity checks on the input. There are others.

          A spreadsheet is a truly horrendous thing to choose for this application, even if it will handle enough data.

    • Re: (Score:2)

      by khchung ( 462899 )

      I can't believe someone made the decision to use a desktop spreadsheet for this kind of data. Wait...no, actually I *can* believe it...

      The right tool for the right job.

      Don't forget that when they started doing this in March, the country with the most covid case was China, with ~80k confirmed cases for 1.4 billion population. *Surely* an advanced western democratic country like UK with just 1/20 population *will* do much better than Communist China, right? RIGHT? Who in the their right mind would think UK needed to track more than a few thousand cases or so?

      Sending xls file was the right decision based on the expected volume *at that time

  • Duh, when the installer asks if you want the 64-bit version of office, you say "yes."

    • I know you're joking, but actual the problem was apparently using xls files, so it wouldn't help!

    • Re: (Score:2)

      by xlsior ( 524145 )

      Duh, when the installer asks if you want the 64-bit version of office, you say "yes."

      The big problem with that is that there are a crapton of other applications and plugins that can interface/integrate with Office as well, and many of those are still 32 bit and won't work together with 64-bit office. Heck, even the download in Microsoft's volume license portal for Office tells you explicitly to use the 32-bit version of office unless you have special needs for the 64-bit version.

  • Why not use a Database that doesn't have a stupid limit? Ignoring the privacy invasion of having your data collected and having other people contacted, why use Excel, it's a really stupid system for large data.

    • It's just crazy from a security perspective too. You just know this giant covid database is being passed around on thumb drives with no access controls. If you used a real database with some kind of front end in front of it you could at least pretend to try and limit how much of the data any single person saw. But using Excel just seems like it's inevitable that the entire thing will be made public.

      • 100% agree, and how long would it take to developer a good frontend / backend for this kind of system? I would think two months at most and you could have a very nice system in place, it would only take a day or two to get a working prototype up using GO, JavaScript, RUST, etc... so I don't see the excuse.

        • One person, two months for a system with proper control of access rights? It will take two months before you have agreed-upon draft requirements for how to manage access to and storage of sensitive medical records.

          Sure, writing a front-end for entering and querying data is easy. The hard part is ensuring that a single rogue employee (including IT staff) can only do a limited amount of damage while not unnecessarily interfering with their ability to do their job.

          I'll admit that I've never set up such a datab

          • Re: (Score:2)

            by Bert64 ( 520050 )

            Well their current system does not have proper control of access rights and no way for a single employee to do damage, so even if they totally screw that up it won't be any worse than what they have now.

            No, it's not that difficult.
            Yes you can have transaction logs that can be rolled back if someone does something stupid.

      • Re: (Score:2)

        by AmiMoJo ( 196126 )

        Copy of Trak and Trace spreadsheet V2 (copy).xls.exe
        1.99GB

  • Excel (Score:4, Funny)

    by Impy the Impiuos Imp ( 442658 ) on Monday October 05, 2020 @04:20PM (#60575082) Journal

    65535 rows is enough for anybody!

    Or

    1 million rows is enough for anybody!

    Just generate the .csv from the dbc and import it into Excel. It's all good.

  • The the usual case (Score:5, Insightful)

    by nospam007 ( 722110 ) * on Monday October 05, 2020 @04:23PM (#60575096)

    When morons use a spreadsheet as a database.

    • So true! Why would those dummies use Excel when Access is clearly the right tool for the job!

      • Re: (Score:2)

        by hey! ( 33014 )

        I've seen Access as a medium of exchange in government and public health datasets. The one thing you can say in favor of that is that at least it's better than Excel.

      • You joke, but Access would've been a better choice. Not a GOOD choice, but a better choice.

    • They needed something they could pass around I imagine. When was the last time you emailed a fucking postgresql database to someone, and what are the odds a normal office worker would know what to do with it.

  • I think covid-19 is not the biggest problem!

    • I think covid-19 is not the biggest problem!

      Isn't the NHS is the same government agency who couldn't get rid of Windows 7 before Microsoft ended support for it, even having almost a decade's worth of notice? Why are we surprised they use Excel for ad hoc stuff like this?

      • This is Public Health England, not technically part of the NHS, but definitely part of the problem - its yet another quango stuffed full of useless cronies who are trying hard to justify having their excessively well paid jobs without having either knowledge, skill or talent.

        But yes, the NHS is just the same - stuffed full of the same type of "managers", often whose main job is counting penises and what colour they are.

      • It's not a matter of this or that office or agency.
        It's more a matter of stupidity: you won't sail the ocean with a nutshell. And if you don;t know, you can ask skilled sailormen.

        Using "personal productivity" tools for enterprise/agency/government grade objectives looks like a deliberate stupid choice.

        • Oh but it IS about it being funded by the government..

          In the "for profit" world there is a strong motive to not be stupid. Those who do stupid things, don't survive. Their competitors will force them to adapt, stop doing stupid things or end up out of business. It's a self correcting problem. So all those "Pay what you can afford" cafes never make it very long. It's a stupid idea, and the marketplace deals with stupidity by getting rid of it. So there is a strong motive to "not be stupid" when you are i

  • SPSS, R, Anything Else (Score:3)

    by medv4380 ( 1604309 ) on Monday October 05, 2020 @04:48PM (#60575200)
    This is why we don't do professional analysis in excel for anything of any significant size. Excel is nice for a nice tinny dataset, but after that, R, SPSS, or writing actual code is required.
  • The future looks very bright... or does that bit mean dark? Every no-coder aced their basic introduction to boolean logic, right?

  • I like spreadsheets as much as the next guy... (Score:3)

    by satanicat ( 239025 ) on Monday October 05, 2020 @05:17PM (#60575290)

    But spreadsheets aren't a great tool for everything... why hasn't then been handled using a database!? :)

  • NHS Test and Trace and Public Heath England (Score:5, Informative)

    by close_wait ( 697035 ) on Monday October 05, 2020 @05:19PM (#60575294)
    Despite having 'NHS' attached to its name, the test and trace service is run by a private company who have had billions of pounds of public money thrown at them. Public Health England used to be part of the NHS until they were split off and attached to local councils about 8 years ago. PHE ran the original tracing scheme at the start of the outbreak, but were soon overwhelmed and tracing was aborted. (PHE haven't received billions in extra funding.) Some months later "NHS" Test and Trace was set up as a bunch of massive regional call centres with no involvement from PHE.

    NHS T&T has been widely criticised for, amongst other things, having no local knowledge - unlike PHE. Any hard tracing tends to get handed over to PHE by NHS T&T - hence the spreadsheets.

    • Test and Trace is Serco run by Dido Harding and overseen by Matt Hancock
      PHE is run by Dido Harding and overseen by Matt Hancock
      NHS has very little to do with either ...

  • What is/was it, I remember 19,000 something lines, then poof!

  • "Any tech caught using an Excel spreadsheet as a production database will be shot."

  • Who need experts? (Score:3)

    by Dr.Saeuerlich ( 27313 ) on Monday October 05, 2020 @10:48PM (#60576064) Homepage

    "People in this country have had enough of experts" (Michael Gove). The government really followed through on that promise at every step. "Promises kept"

  • At this point, it is just a money making thing for certain companies. The positive test results are mostly false. Either that, or the disease is not nearly as dangerous as it was made out to be, since the hospitals are empty, despite the high positive results. The whole thing has turned into a BS farce.

    • Either that, or the disease is not nearly as dangerous as it was made out to be, since the hospitals are empty, despite the high positive results.

      You are forgetting about
      .
      .
      .
      the lag.

      Check the numbers again in 2 or 3 weeks.
      (and keep checking after that)

      Btw, hospital numbers are already increasing slowly, like exponential growth actually starts before it takes off.

    • At this point, it is just a money making thing for certain companies. The positive test results are mostly false.

      [citation needed]

      Either that, or the disease is not nearly as dangerous as it was made out to be, since the hospitals are empty, despite the high positive results.

      Distancing works.

  • Gunshot deaths get called corona deaths all the time.

    These numbers are way exaggerated.

    https://www.freedomfoundation.... [freedomfoundation.com]

  • AWK (Score:3)

    by NewYork ( 1602285 ) <4thaugust1932@gmail.com> on Wednesday October 07, 2020 @12:40AM (#60579938) Homepage
    I use AWK to do floating point calculations and copy the results to Excel due to http://0.30000000000000004.com... [30000000000000004.com]

Slashdot Top Deals

The optimum committee has no members. -- Norman Augustine

Close