Slashdot is powered by your submissions, so send in your scoop

 


Forgot your password?
Close
typodupeerror
×
Google IT

Google Pushed a One-Character Typo To Production, Bricking Chrome OS Devices (arstechnica.com) 75

Posted by msmash from the closer-look dept.
Google says it has fixed a major Chrome OS bug that locked users out of their devices. Google's bulletin says that Chrome OS version 91.0.4472.165, which was briefly available this week, renders users unable to log in to their devices, essentially bricking them. From a report: Chrome OS automatically downloads updates and switches to the new version after a reboot, so users who reboot their devices are suddenly locked out them. The go-to advice while this broken update is out there is to not reboot. The bulletin says that a new build, version 91.0.4472.167, is rolling out now to fix the issue, but it could take a "few days" to hit everyone. Users affected by the bad update can either wait for the device to update again or "powerwash" their device -- meaning wipe all the local data -- to get logged in. Chrome OS is primarily cloud-based, so if you're not doing something advanced like running Linux apps, this solution presents less of an inconvenience than it would on other operating systems. Still, some users are complaining about lost data.
This discussion has been archived. No new comments can be posted.

Google Pushed a One-Character Typo To Production, Bricking Chrome OS Devices More

Google Pushed a One-Character Typo To Production, Bricking Chrome OS Devices

Comments Filter:

  • What a joke. (Score:2, Funny)

    by Anonymous Coward

    Fiascos like this are a cautionary tale to avoid open source hobby OSes and using industry standards products made by professionals like OSX, Windows, etc.

  • Testing? (Score:3, Interesting)

    by Fly Swatter ( 30498 ) on Thursday July 22, 2021 @03:17PM (#61609147) Homepage
    I guess that no longer exists. I can't wait to see the next OS where we can see the real time changes in the code as the 'developers' type in the changes and add features.

    Please note I left fix bugs out intentionally, who does that?!?

    • Re:Testing? (Score:5, Funny)

      by fahrbot-bot ( 874524 ) on Thursday July 22, 2021 @03:27PM (#61609195)

      I guess that no longer exists.

      Google's software development cycle is a bit different. It goes Alpha (limit users), Beta (all users), Discontinued. :-)

    • Re:Testing? (Score:5, Insightful)

      by The MAZZTer ( 911996 ) <megazzt.gmail@com> on Thursday July 22, 2021 @06:09PM (#61609807) Homepage
      Google has extensive automatic and manual testing for Chrome. Even the alpha (dev) channel builds have been solid enough to use as a daily driver pretty much for the entire lifetime of Chrome with a few builds here and there as exceptions. I assume they have similar testing for Chrome OS. My guess is someone made "a quick fix" and bypassed this testing to get it into the next release on time.

    • Basically the problem seems to be that they have quite stupid programmers and processes, so I would guess that the testing department is overwhelmed.

  • Inconvenience Rating (Score:4, Insightful)

    by SuperKendall ( 25149 ) on Thursday July 22, 2021 @03:18PM (#61609151)

    Chrome OS is primarily cloud-based, so if you're not doing something advanced like running Linux apps, this solution presents less of an inconvenience than it would on other operating systems.

    I'm going to have to say that locking my out of login on device for several days, is WAY more inconvenient than any previous desktop OS snafus I have seen.

    • Re: (Score:1)

      by AmiMoJo ( 196126 )

      It's actually better than most other operating systems. Take iOS for example, the advice is always too wipe and restore from backup, the assumption being that you make regular backups or pay for iCloud.

      Remember when Windows 10 upgrades trashed installs? Wipe and start over.

      The fact that it can fix itself with an update even when the user can't log in is significantly better than most rivals.

      • Take iOS for example, the advice is always too wipe and restore from backup,

        Whose advice? I have literally never had to do that, nor has anyone else I know - and I develop on IOS systems, sometimes with beta OS versions.

        The iCloud backup is more for if you need to get a whole new device....

        Remember when Windows 10 upgrades trashed installs? Wipe and start over.

        Still faster than what is going on with the Chromebook issue.

        • Re: (Score:2)

          by AmiMoJo ( 196126 )

          My wife has been given that advice for her overheating iPhone multiple times. Any issues the advice is always backup, wipe and see if the issue is still there. Same when sending the phone off for repair, wipe and restore when you get it back.

    • Im sure there is some exceptions but usually google drive is where you store your docs and app data. There local storage is fairly small compared to most devices. That leaves removable media as the non cloud storage. Power wash took me 5min, and once I logged my son back into his account it pushed most things back to his desktop except for a few apps i had to fetch from his google play library.

      • Im sure there is some exceptions but usually google drive is where you store your docs and app data.

        Yeah you can get to the data right away but what if your only device was a Chromebook? Then you have to find something else to access that data, even if it is all online...

        That's why I think of blocking access to the actual device for any length of time is such an issue.

        • I guess if you dont want to do a powerwash on it. Just as soon as it was power washed it worked again. So if your data wasnt on the internal drive, your inaccessible period is just 10min from the time you take that action. For me it was a couple hours since I was convinced at the time it had something to do with my login. 2hrs later I threw in the towel and just reset the thing. Also the 'connect as guest' also worked to let you access a web browser.

  • Windows is better with update rollback.
    Wipe all local data to back out of an bad update??

    • A lot of devices I've worked on have two flash ROMs to hold the firmware. You flash it, and if the flashing fails, you can always roll back to the other ROM.

  • Another reason why auto updates are a joke.

    I suppose your device really is secure if it won't let you sign in!

  • Google Beta. (Score:4, Insightful)

    by geekmux ( 1040042 ) on Thursday July 22, 2021 @03:25PM (#61609187)

    Mom and Pop shop not testing a patch before Production? Shit happens all the time.

    Google maintaining a damn army of personnel and still can't ensure the same is done properly? Fucking pathetic.

    Ah, nothing like trusting a corporation that loves to excuse their incompetence by classifying damn near every product offering, as Beta.

    • Re:Google Beta. (Score:5, Interesting)

      by scamper_22 ( 1073470 ) on Thursday July 22, 2021 @03:34PM (#61609215)

      And this one is very weird as well.

      Based on my reading, this is a bug that WOULD occur 100% of the time on a BASIC login. This is not some obscure bug somewhere. This would be caught by literal sanity testing of the build as in.

      1. Log in
      2. Launch an app or two
      3. Looks good.

      This would fail in Step 1. You can't login.

      • If you have no password, does it still lock you out?
        • Every chrome account I have ever added required a google account to access. That is your login. I know of no password-less google accounts. I think all you can setup is an auto-login.

          • Chrome account and Chrome OS login aren't the same thing. You can log in to Chrome OS via PIN and also via SmartLock - tapping your Android device on your Chromebook.

            Now as to whether these other two methods call the same erroneous function to check the credentials against the keystore, that I don't know.

      • Re:Google Beta. (Score:5, Insightful)

        by AmiMoJo ( 196126 ) on Thursday July 22, 2021 @04:09PM (#61609321) Homepage Journal

        My guess would be that someone pushed a change at the last minute when they shouldn't have, but it must have been someone with the credentials to do that.

      • It was caught by QA, which for Google Beta products apparently consist solely of customers. "Ship now, fix it later via OTA update" is the moto of most Silicon Valley companies, no?

      • Re: (Score:2)

        by tlhIngan ( 30335 )

        And this one is very weird as well.

        Based on my reading, this is a bug that WOULD occur 100% of the time on a BASIC login. This is not some obscure bug somewhere. This would be caught by literal sanity testing of the build as in.

        1. Log in
        2. Launch an app or two
        3. Looks good.

        This would fail in Step 1. You can't login.

        So Google checked in code without testing it and pushed it to production as a "it should work".

        Not exactly an unusual occurrence in software development - after all, who hasn't said "It was a min

        • I've worked for a variety of companies both big and small.
          Testing varied for them all.

          Yet, I honestly can't think of a time that a build was sent to a customer/production without at least running it as a sanity test. This is what is so surprising.

    • Ah, nothing like trusting a corporation that loves to excuse their incompetence by classifying damn near every product offering, as Beta.

      Nearly everything as Beta?! That's not true and you should give them a break because Google Competence is still Beta! ;)

  • This is not the first time I've heard of a major bug due to the fact that both "|" and "||" or "&" and "&&" are syntactically legal in C. At some point we'll realize "C syntax is harmful" because stupid typos like this produce code that compiles. (The BTSJ articles on 'faults in System V' are full of these kinds of problems" See "An Empirical Study of Software Faults Preventable at a Personal Level in a Very Large Software Development Environment", Weider D. Yu, Alvin Barshefsky, and Steel

    • It wouldn't help here, the code is bad. Look at it:

      if(key_data_.has_value() && !key_data_->label().empty())

      This is a single concept written out in code. A concept should be encapsulated in a function, otherwise you're going to mess it up. Maybe not with a typo, but with something.

      Also the _ at the end of the variable name is weird, but not necessarily bad.

      • Contrast to Ada syntax for such comparisons, for example:

        "If (key_data_.has_value() and not key_data_->label().empty() ) then ... "

        (Although I suspect the actual intent is probably "if (key_data_.has_value() and then not key_data_->label().empty() ) then ... " )

      • Are we sure its C? I havent seen the entire section of code but that one line shares a lot of similarities with others.

        • Unfortunately, a lot of languages have copied C's (terrible) syntax, with the argument "it's what programmers are used to."

          I personally find that an absurd argument. (1) Errors like this show that 'programmers are used to' is not a good recommendation for perpetuating potential errors. (2) any reasonable programmer can learn new syntax.

        • I know no one ever reads the article but it's clearly C++

        • Presumably it's C++ from the file extension.

    • So Google doesn't look at (or even worse, disables) compiler warnings? Google doesn't use a static code checker to flag suspect code?

      Warnings are there for a reason, folks. Static checks, including for situations dependent on operator precedence, are minimum expectations in any shop I've worked at.

      • The code is perfectly legal in C/C++. The compiler would argue that the programmer meant a bitwise comparison!

        Java, without the burden of legacy code, solved this a quarter of a century ago by requiring that if expressions evaluate to the boolean data type, true or false.

        Surely there's a compiler flag since the introduction of bool in C99 to enforce similar?

  • Bricking (Score:3, Informative)

    by Anonymous Coward on Thursday July 22, 2021 @03:39PM (#61609237)

    If you can recover from it it's not bricked

    • Re: (Score:2)

      by TWX ( 665546 )

      Exactly. This is a severe inconvenience but if the vendor actually understands what they screwed up and fixes it in a timely manner then for most users it's merely inconvenient as opposed to the end of the device as we know it.

    • Re:Bricking (Score:5, Funny)

      by Waffle Iron ( 339739 ) on Thursday July 22, 2021 @04:08PM (#61609313)

      If you can recover from it it's not bricked

      Well, mine *is* bricked now.

      When I saw their instruction to "powerwash" the device, I went ahead and put it on the driveway, then gave it a thorough once-over on all sides with full pressure.

      After that, when I tried to power it on, all I got was a wisp of foul-smelling white smoke.

  • Chrome OS is primarily cloud-based, so if you're not doing something advanced like running Linux apps, this solution presents less of an inconvenience than it would on other operating systems

    This is a major advantage over iOS in my opinion - if your device breaks, it's much easier to move to another one/restore it than anything Apple has come up with; usually it's just a matter of picking up another device, logging in, and you're back to doing what you need. With Apple this is likely a multi-hour process (1hr min. in my experience).

    • Re: (Score:2)

      by DRJlaw ( 946416 )

      This is a major advantage over iOS in my opinion - if your device breaks, it's much easier to move to another one/restore it than anything Apple has come up with

      Restore from iCloud backup begs to differ.

      usually it's just a matter of picking up another device, logging in, and you're back to doing what you need.

      So I have to have two devices in case Google bricks one, and make sure that I haven't updated both at the same time.

      With Apple this is likely a multi-hour process (1hr min. in my experience).

      Having don

  • The important question, to which we would be fool to assume the answer, is did they learn their lesson and put into place robust measure to prevent it from ever happening again?

    • You mean push it to a device and test logging in, browsing sites, and installing apps from the play store; all before publishing to revision to production? I thought that was software management 101?

      • Yes exactly. The way I reckon it if they weren't the type to have done that before .. they will probably F it up going forward too unless they are forced to. I mean, is the culture such that they could fuck that off?

    • They are having a post-mortem as we speak.
  • I had to power wash the kids chromebook. Bricked is the wrong term. I could boot the damn thing. It just said it couldnt verify my password. Control-Alt-Shift-R
  • Back in the day, "bricked" meant permanently and irrevocably useless - rendered somewhat like a brick, in fact - rather than something that would fix itself if you left it to update.

    • Maybe they should call it "Jesus bricked" since it gets resurrected.

    • Back in the day, "bricked" meant permanently and irrevocably useless - rendered somewhat like a brick, in fact - rather than something that would fix itself if you left it to update.

      Given the sheer addiction factor, they'll be throwing it like a brick in frustration after being offline for more than 12 hours anyway.

      Technically you're correct, but like it really matters.

    • It still does. The people "editing" Slashdot simply have zero technical chops now. I don't know what dumpster they found them living in, but they clearly didn't have internet access there.

  • Enjoy our forced updates, enjoy our spying on you, and enjoy all the restrictions we put on you which continue to grow as time goes on.

      We are Trusted(TM) and you are a filthy criminal. Now bow to your king and lick his feet, you filthy animal!

    • Enjoy our forced updates

      I do. It's great living in a world without heavily wormable exploits thanks to some idiot not being allowed to run buggy outdated crap. I also enjoy not having to support my parents who wouldn't know a CVE from a salt shaker much less what security implications are.

      I don't look back fondly on installing Windows only for the OS to get properfucked before it even has the opportunity to install security patches via Windows updates. You may think custom rolling a Windows XP install disk with a pre-applied patch

      • "I do. It's great living in a world without heavily wormable exploits thanks to some idiot not being allowed to run buggy outdated crap"

        And then your machine gets bricked with a screwy update, and you have no recourse. Hope it does not happen when you are about to give that slam dunk presentation.

        I don't do forced updates, and I disable that crap every time I can. I can live with the security risks. I can't live with downtime, and constant updating. And frankly, while I have some degree of Trust(TM)

        • And for those who are about to pull the "what about other people" card for whatever reason here (which I feel will happen), I don't give a rats ass about other people. Really I don't, and plenty of others share the same sentiment.

        • And then your machine gets bricked with a screwy update

          I'd take a Chromebook bricked for a couple of days over having to reinstall Windows any day. You have some very rose coloured glasses on.

          Unsorry, but I refuse to give up control of my life.

          And yet, that control is precisely something you never had in the security nightmare of the past. You can thank the anti-vaxxers ... errr anti-updaters.

          • "#And then your machine gets bricked with a screwy update#

            I'd take a Chromebook bricked for a couple of days over having to reinstall Windows any day. You have some very rose coloured glasses on."

            A couple days? Sure, if you are using your laptop as a 'Wintendo', you can do with a couple days downtime. Meanwhile, some of us actually need our machines for work.

            Having to reinstall Windows was a problem in the past, and the last time I had to do this was ironically because of a botched forced Windows 10

  • Well thats one good reason to never buy a chromebook, if it can be bricked remotely, no thank you.

    • Re: (Score:2)

      by acroyear ( 5882 )

      any update to any operating system can do this. the only way to avoid it is to never upgrade, with all the inherent security risks that come with it. I've been 'bricked' by windows and linux upgrades quite a few times (though the only permanent brick was Android TV's 8.0 upgrade that toasted a LOT of people's nexus tv set-tops JUST after they announced no more support)

Slashdot Top Deals

Keep up the good work! But please don't ask me to help.

Close