Catch up on stories from the past week (and beyond) at the Slashdot story archive

 


Forgot your password?
Close
typodupeerror
×
Bug Microsoft

'Year 2022' Bug Breaks Email Delivery For Microsoft Exchange On-Premise Servers (bleepingcomputer.com) 146

Posted by EditorDavid from the big-bug-theory dept.
Kalper (Slashdot reader #57,281) shares news from Bleeping Computer: Microsoft Exchange on-premise servers cannot deliver email starting on January 1st, 2022, due to a "Year 2022" bug in the FIP-FS anti-malware scanning engine.

Starting with Exchange Server 2013, Microsoft enabled the FIP-FS anti-spam and anti-malware scanning engine by default to protect users from malicious email. According to numerous reports from Microsoft Exchange admins worldwide, a bug in the FIP-FS engine is blocking email delivery with on-premise servers starting at midnight on January 1st, 2022.

Security researcher and Exchange admin Joseph Roosen said that this is caused by Microsoft using a signed int32 variable to store the value of a date, which has a maximum value of 2,147,483,647. However, dates in 2022 have a minimum value of 2,201,010,001 or larger, which is greater than the maximum value that can be stored in the signed int32 variable, causing the scanning engine to fail and not release mail for delivery. When this bug is triggered, an 1106 error will appear in the Exchange Server's Event Log stating, "The FIP-FS Scan Process failed initialization. Error: 0x8004005. Error Details: Unspecified Error" or "Error Code: 0x80004005. Error Description: Can't convert "2201010001" to long." Microsoft will need to release an Exchange Server update that uses a larger variable to hold the date to officially fix this bug.

However, for on-premise Exchange Servers currently affected, admins have found that you can disable the FIP-FS scanning engine to allow email to start delivering again... Unfortunately, with this unofficial fix, delivered mail will no longer be scanned by Microsoft's scanning engine, leading to more malicious emails and spam getting through to users.
This discussion has been archived. No new comments can be posted.

'Year 2022' Bug Breaks Email Delivery For Microsoft Exchange On-Premise Servers More

'Year 2022' Bug Breaks Email Delivery For Microsoft Exchange On-Premise Servers

Comments Filter:

  • Good grief (Score:5, Funny)

    by J. L. Tympanum ( 39265 ) on Saturday January 01, 2022 @07:38PM (#62134743)

    Could Microsoft be any stupider?

    • Re:Good grief (Score:5, Funny)

      by sqlrob ( 173498 ) on Saturday January 01, 2022 @07:52PM (#62134771)

      The patch could just change it to a float.

    • Re: Good grief (Score:5, Insightful)

      by Jeremi ( 14640 ) on Saturday January 01, 2022 @08:09PM (#62134803) Homepage

      Whatâ(TM)s unnerving is that they are working on a fix just hours before the shit hits the fan. This problem wasnâ(TM)t difficult to test for; does nobody at Microsoft SQA ever set their test computerâ(TM)s clock forward a few years to see if anything breaks?

      • Microsoft often seems poorly managed. It seems that there is an overall problem of lack of healthy social connections inside the company. Is that one of the problems?

        • Re:Microsoft: Poor social health? (Score:5, Informative)

          by JustAnotherOldGuy ( 4145623 ) on Sunday January 02, 2022 @12:47AM (#62135179) Journal

          Microsoft often seems poorly managed. It seems that there is an overall problem of lack of healthy social connections inside the company. Is that one of the problems?

          Take it from someone who's worked there- you could write a 12-volume set of books on the problems at Microsoft and you'd barely be scratching the surface.

          The problems are deep and wide, vast and directionless. They start just below the floor tiles and go all the way up to low-earth orbit.

          You could literally teach a college course called "What's Wrong With Microsoft 101".

          But anyway, to answer your question, "Yes."

          • Re:Microsoft: Poor social health? (Score:4, Insightful)

            by gweihir ( 88907 ) on Sunday January 02, 2022 @03:19AM (#62135353)

            Good to know. I long since suspected that the success of MS is basically a gross market failure. They just cannot do things right. And they should never have gotten this important.

            • I had a conversation with a friend several years ago and the subject turned to Microsoft software. In the course of that conversation I commented that I thought Microsoft was the biggest thing holding back technology, only to discover that he actually worked for them. However, I didn't feel any need to amend my comment. He did end up quitting a year or two later because he was "tired of lying to customers".

              I lived through Y2K as a professional software developer, and I can say that this bug is stupider th

              • Re: (Score:2)

                by gweihir ( 88907 )

                I lived through Y2K as a professional software developer, and I can say that this bug is stupider than any date-related stupidity I ever saw back then.

                I can say that I have seen a more stupid bug in supposedly professional software. But only once and it was not date-related.

          • Take it from someone who's worked there- you could write a 12-volume set of books on the problems at Microsoft and you'd barely be scratching the surface.

            The problems are deep and wide, vast and directionless. They start just below the floor tiles and go all the way up to low-earth orbit.

            You could literally teach a college course called "What's Wrong With Microsoft 101".

            But anyway, to answer your question, "Yes."

            They really have no need to correct their problems either.

            Their customers either happily accept whatever shit Microsoft hands them, or at worst grumbles a bit, then console themselves with the idea that as the industry standard, it is not possible for anything other than Microsoft to perform work.

            Your 101 level class is a good idea. The follow on class might be Microsoft 201 - "Why do Microsoft Customers happily accept the problems caused by Microsoft?

            • At some point Microsoft will become, or has already become, a cargo cult maintaining software that no one understands, because there is no one left at the company that actually made any of it, and they're just going through the motions.

              • At some point Microsoft will become, or has already become, a cargo cult maintaining software that no one understands, because there is no one left at the company that actually made any of it, and they're just going through the motions.

                I kinda think they indeed have reached that point. Mailing it into the installed user base, to infinity and beyond, the never ending standard.

      • Microsoft, where quality is job number two.

        And where we go number one directly in the face of anyone who gives us money.

      • Re: (Score:2)

        by AmiMoJo ( 196126 )

        Reminds me of a recent test that Google started where they set the Chrome version number to 100. I think it's mostly to test what websites break, rather than if the browser can handle it.

      • This problem wasnâ(TM)t difficult to test for; does nobody at Microsoft SQA ever set their test computerâ(TM)s clock forward a few years to see if anything breaks?

        You and I both know the answer - no they do not.

        And why should they? Their customers have shown that there is nothing that Microsoft can do that will turn them away.

    • Re:Good grief (Score:4, Insightful)

      by Rosco P. Coltrane ( 209368 ) on Saturday January 01, 2022 @08:12PM (#62134813)

      Stupid or not.

      You see, I can't help noticing that on-prem Exchange servers hurt Microsoft's cloud offerings. I bet they'd love nothing better than to convert all those customers to their new SaaS lock-in scheme, and a few zero-days and other tetchnical issues might just be what they need to be convinced to switch.

      I'm not saying Microsoft introduced the bug themselves, because that's conspiracy theory stuff and I don't bite. But I wouldn't put it past them to have known about the issue for a long time and sat on it in order to reveal it at the most "convenient" moment - i.e. to introduce a strong sense of urgency in the customer's mind to rethink their IT.

      • If you are unlucky enough to have chosen a hybrid exchange setup, you are stuck with a On Premise Exchange even if you are using their SaaS offering. It might have changed with Exchange 2019, I haven't check recently. Looks more like a "damned if you do, damned if you don't" situation to me!
        • As it stands you will always need an on prem exchange server because of a quirk with provisioning mailboxes and the AD Schema

      • Stupid or not.

        You see, I can't help noticing that on-prem Exchange servers hurt Microsoft's cloud offerings. I bet they'd love nothing better than to convert all those customers to their new SaaS lock-in scheme

        Well, a couple pieces of pragmatism here. First, the rumor mill seems pretty settled that the next version of Exchange will require a subscription [redmondmag.com], so as far as recurring revenue goes, MS seems to be coming for it whether mailboxes live on one's own server, or in a datacenter in Redmond.

        Second, Office365 isn't new; it's been around for about a decade now. Zero people who deployed Exchange 2016 or 2019 did so unaware that O365/M365 existed; these were intentional choices. There are plenty of reasons for that

      • But I wouldn't put it past them to have known about the issue for a long time and sat on it in order to reveal it at the most "convenient" moment - i.e. to introduce a strong sense of urgency in the customer's mind to rethink their IT.

        Nobody needs to make a conscious decision like that for a particular bug. Just squeeze the budget for maintaining this piece of software in favor of the cloud offerings. Sooner or later a bad bug like this will appear.

        Someone making a conscious decision to leave a time-bomb bug in has some options:

        - don't tell anyone. Then no personal gain and risk getting fired if it is discovered.
        - agree to do this with people above you. Now they risk getting fired if it turns out to have too much fallout. And they could

      • That's also a conspiracy.

        A non-conspiracy would be that they know the Exchange market is secure and has little room for growth, so they devote only minimal efforts to maintaining Exchange.

    • Re: (Score:2)

      by gweihir ( 88907 )

      Could Microsoft be any stupider?

      They are obviously trying really hard to be. Will be interesting which amateur mistake they make next.

    • They keep trying to get worse by firing everyone experienced and replacing them with morons to save money.

    • Could Microsoft be any stupider?

      I don't know that this is more stupid, but my company's email is hosted by Microsoft, and we have properly set up DMARC, DKIM and SPF records. Our SPF records include the appropriate Microsoft records and are fully valid including not exceeding the DNS lookup limit.

      I regularly get DMARC reports of SPF failures from Microsoft owned domains. When Microsoft sends emails to outlook.com or linkedin.com, it sends them from IP addresses that are not in Microsoft's own SPF lists.

  • Exchange doesn't even do actual e-mail. (Score:4, Insightful)

    by Anonymous Coward on Saturday January 01, 2022 @07:40PM (#62134747)

    Microsoft[r] Exchange[tm] message delivery broken by bug in malware detection mechanism.

    Malware detection mechanism required by decades of extremely shoddy programming work courtesy that very same company.

    O irony.

  • Rookie move there Microsoft...

  • Technical Details (Score:5, Informative)

    by crow ( 16139 ) on Saturday January 01, 2022 @07:56PM (#62134777) Homepage Journal

    In case the number in the summary didn't make sense, Microsoft is storing a timestamp in the format yymmddhhmm. They're using two-digit years, which we should know by now is completely stupid, and they're then using a BCD (binary coded decimal) storage to cram that into a 32-bit value, which they happened to set as signed for no good reason.

    I hope the fix isn't just to make the value unsigned, but I expect it will be. If they must stick with BCD, then switch to a 64-bit value and use a four-digit year. At least they're using a format that always increases as time moves forward, so they can use the values when sorting, but they can't subtract values to get the difference in any meaningful way.

    • Re: (Score:3, Insightful)

      by Anonymous Coward

      Usually 'binary coded decimal implies' a fixed number of bits for each digit:
      https://en.wikipedia.org/wiki/... [wikipedia.org]

      This is more like 'converting a string representation of the date to an integer', which is so useless it doesn't have a name.

    • When they change it to an unsigned int and carry on the same BCD scheme they'll only buy themselves another 21 years... it will fail again at [20]4301010001

    • Re: (Score:2)

      by gweihir ( 88907 )

      It is hard to see how they could have made a worse choice that still mostly works.

      • > a worse choice that still mostly works.

        It's like you've reverse engineered Microsoft's mission statement.

        All of my open source majl is working today, yay. 2038 I'll pay attention.

        • Re: (Score:3)

          by gweihir ( 88907 )

          > a worse choice that still mostly works.

          It's like you've reverse engineered Microsoft's mission statement.

          All of my open source majl is working today, yay. 2038 I'll pay attention.

          Same here. Although I do not think Postfix has a year 2038 problem.

  • It's working! (Score:4, Funny)

    by virtig01 ( 414328 ) on Saturday January 01, 2022 @07:58PM (#62134783)

    Microsoft enabled the FIP-FS anti-spam and anti-malware scanning engine by default to protect users from malicious email. According to numerous reports from Microsoft Exchange admins worldwide, a bug in the FIP-FS engine is blocking email delivery

    Well, users are certainly being protected from receiving malicious mail....

  • Could they be anymore obvious? (Score:4, Insightful)

    by darkjedi521 ( 744526 ) on Saturday January 01, 2022 @08:08PM (#62134801)
    A bug that breaks email for self-hosted/on premise exchange only? If that's not a move to push hold outs to their cloud, I've got a bridge for sale.

  • Can't wait till 2050 (Score:3)

    by scdeimos ( 632778 ) on Saturday January 01, 2022 @08:10PM (#62134807)
    There are so many pieces of software out there kludging two digit years into four digit years - most commonly converting 00-49 to 2000-2049 and 50-99 to 1950-1999.

    • Re: (Score:2, Interesting)

      by Anonymous Coward

      And don't forget the 2038 'Unix time' issue. Only an issue for 32-bit systems? Nope, some software, particularly closed source, still uses 32-bit timestamps internally. So your system may run fine at the OS level, but your database will fuck up. And in the world beyond servers and desktops, there are loads of 32-bit devices, many still being produced, that will fail in a variety of interesting ways.

      2038 used to seem a long time away. Suddenly, it's only 16 years, and various systems using future dates are a

      • >The good news is that the 64 bit timestamp that replaces the 32 bit original gives enough capacity to last about 292 billion years, so once dealt with it's unlikely to be of further concern. :)

        Indeed as the universe is only about 14 billion years old and earth only about 4.5 billion.

  • Exchange Spamfilter (Score:5, Interesting)

    by xlsior ( 524145 ) on Saturday January 01, 2022 @08:12PM (#62134809) Homepage

    Exchange spamfilter has had other bone-headed design choices in years past: After their in-house testing showed that email send/received outside of typical business hours was statistically more likely to be spam, Microsoft made the timestamp part of their weighing algorithm to classify spam.

    However, not only did they fail to account for companies that uses non-traditional office hours, they also failed to account for timezones and centered everything around Pacific Time, where the Microsoft headquarters were located. That meant that if you were in Europe or Asia, the spam detection algorithm would to the exact opposite of what it was trying to accomplish and penalize mail received while you were open and promote mail that was received while you were closed.

    • Re:Exchange Spamfilter (Score:5, Interesting)

      by MightyMartian ( 840721 ) on Saturday January 01, 2022 @08:33PM (#62134843) Journal

      When I was still administering an Exchange server (the last version I worked with was 2013), I put the whole damned thing behind a Postfix mail proxy running on a separate Linix box. There was no access to Exchange at all on port 25, and all spam functionality ran through the Postfix daemon with all the nifty anti spam tricks, along with Spamassassin. There were multiple reasons. One of them was that Postfix, even when being directly assaulted by massive dictionary attacks, never became unresponsive. In part this was, I think, just a superior TCP stack on Linux, and in part the fact that Postfix, like all good *nix tools, does a singular job, being an MTA, very well.

      Frankly I long ago came to detest Exchange.

      • But Exchange has what plants crave!

      • Re:Exchange Spamfilter (Score:5, Interesting)

        by stabiesoft ( 733417 ) on Saturday January 01, 2022 @10:24PM (#62134997) Homepage
        I've looked over the postfix code. Careful programming is an understatement.

        • I've looked over the postfix code. Careful programming is an understatement.

          Do you remember any bits that stood out? I'd like to read them and learn from them.

          • It is the work as a whole really. Modular like unix where pieces can be plugged in/out. Memory is careful, interfaces are clean, code is readable with helpful comments. Readability is an under rated thing. I've seen code where people want to demonstrate how superior they are by making it indecipherable. Several people have commented to me that my code is readable. I take that as the quite a compliment.
      • > I put the whole damned thing behind a Postfix mail proxy running on a separate Linix box.

        Actually, I just did this the other day - works great and I get nice graphing with mailgraph.

        • Re: (Score:2)

          by ls671 ( 1122017 )

          Have a look at proxmox mailgateway. It uses postfix and includes all anti spam tools, DKIM, etc. etc. configured out of the box with a graphical interface on top of it.

      • Re: (Score:2)

        by ksw_92 ( 5249207 )

        I think a lot of us Exchange admins did this kind of "roll your own" MTA proxy back in the day, giving rise to a whole class of products. Linux, Postfix, Spam Assassin and ClamAV on some cast-away hardware did a good job "hardening" Exchange. Plug in the various DNS-based blacklists to suit your needs and you had a pretty good solution up until the mid 2010s when you had to go with a commercial solution to keep up with all the changes in attack techniques.

        • Re: (Score:2)

          by ls671 ( 1122017 )

          All of what you mentioned is included in proxmox mailgateway without any need to mess around with complicated installation and configuration. You can even use it for free if you change the update repository to the dev branch instead of the enterprise one.

        • Ayup, I also did the whole Postfix Spamassassin thing way back yoinks and even invented greylisting with a recompiled Postfix, before greylisting was officially invented. Then I moved to another country and got my life back.
    • Well, MS sure is the centre of the flat earth.

  • You can say they deliver email... (Score:4, Informative)

    by kriston ( 7886 ) on Saturday January 01, 2022 @08:12PM (#62134811) Homepage Journal

    ...on premise!

    Seriously, we call it "on premises," not "on premise."

  • Happy New Year from Microsoft (Score:3)

    by awwshit ( 6214476 ) on Saturday January 01, 2022 @08:22PM (#62134821)

    Thank you for being a Microsoft customer. Be sure to ring in the new year by renewing maintenance, at our new 2022 rate (20% increase), so that you can continue to enjoy working on holidays.

  • Probably deliberately. (Score:3, Interesting)

    by Qbertino ( 265505 ) <moiraNO@SPAMmodparlor.com> on Saturday January 01, 2022 @08:22PM (#62134825)

    They are coming up with all kinds of b*llsh*t to force customers into their subscriptions. Including an artificially initiated ca. 124GB RAM requirement for the recent versions of on-prem SharePoint.

    I wouldn't be surprised if they deliberately go lax on development and maintenance of on-prem Exchange for the same reasons.

    It's Microsoft after all, what would you expect?

    • Re:Probably deliberately. (Score:5, Informative)

      by goonerw ( 99408 ) on Saturday January 01, 2022 @08:45PM (#62134863) Homepage

      They are coming up with all kinds of b*llsh*t to force customers into their subscriptions. Including an artificially initiated ca. 124GB RAM requirement for the recent versions of on-prem SharePoint.

      Huh?
      Requirements [microsoft.com] haven't changed for On-Prem Sharepoint for over a decade. The max for a single server is still 24GB and that's only if it's running everything, including SQL Server.

  • Well to be fair (Score:5, Funny)

    by JustAnotherOldGuy ( 4145623 ) on Saturday January 01, 2022 @08:31PM (#62134841) Journal

    "'Year 2022' Bug Breaks Email Delivery For Microsoft Exchange On-Premise Servers"

    Cut Microsoft some slack, who could have predicted that "2022" would have followed "2021"?

  • .... strikes again. Apparently, yet another thoroughly avoidable problem, if only Microsoft had good QA.
  • My ASSP/Courier-MTA/Courier-IMAP system serving 200 users of a local government worked just fine through the new year .
  • campaign from Microsoft. After all, everyone should have dumped their own Paid for On Site Microsoft products and go monthly O365 subscriptions in the cloud anyway,

  • It's apparent that nobody at MSFT does code reviews. Or at least, nobody competent.

    Or maybe crap like this is deliberate to make it such a pain to run on-prem Exchange servers that everyone throws in the towel and goes with MSFT's hosted Exchange service.

  • In the long run, will MS make more money of off on-premise servers or SAS? Especially if MS can frighten everyone onto SAS by being "extra" sloppy with on-premises software. This had been tried before. Never worked, never will.

  • >Microsoft Exchange on-premise servers cannot deliver email starting on January 1st, 2022, due to a "Year 2022" bug in the FIP-FS anti-malware scanning engine.

    Productivity will soar without constant email interruptions.

  • Microsoft should have seen that one coming, however they were probably consuming a few too many mushrooms.

  • https://en.wikipedia.org/wiki/... [wikipedia.org]
    https://en.wikipedia.org/wiki/... [wikipedia.org]

    This is a native English speaking website isn't it?
    Ignorance is a virus.

  • Anyone would think someone would have learned from that? Apparently not...

  • Hail the greatest software company of all time.

Slashdot Top Deals

Get hold of portable property. -- Charles Dickens, "Great Expectations"

Close