'Year 2022' Bug Breaks Email Delivery For Microsoft Exchange On-Premise Servers (bleepingcomputer.com) 146
Kalper (Slashdot reader #57,281) shares news from Bleeping Computer:
Microsoft Exchange on-premise servers cannot deliver email starting on January 1st, 2022, due to a "Year 2022" bug in the FIP-FS anti-malware scanning engine.
Starting with Exchange Server 2013, Microsoft enabled the FIP-FS anti-spam and anti-malware scanning engine by default to protect users from malicious email. According to numerous reports from Microsoft Exchange admins worldwide, a bug in the FIP-FS engine is blocking email delivery with on-premise servers starting at midnight on January 1st, 2022.
Security researcher and Exchange admin Joseph Roosen said that this is caused by Microsoft using a signed int32 variable to store the value of a date, which has a maximum value of 2,147,483,647. However, dates in 2022 have a minimum value of 2,201,010,001 or larger, which is greater than the maximum value that can be stored in the signed int32 variable, causing the scanning engine to fail and not release mail for delivery. When this bug is triggered, an 1106 error will appear in the Exchange Server's Event Log stating, "The FIP-FS Scan Process failed initialization. Error: 0x8004005. Error Details: Unspecified Error" or "Error Code: 0x80004005. Error Description: Can't convert "2201010001" to long." Microsoft will need to release an Exchange Server update that uses a larger variable to hold the date to officially fix this bug.
However, for on-premise Exchange Servers currently affected, admins have found that you can disable the FIP-FS scanning engine to allow email to start delivering again... Unfortunately, with this unofficial fix, delivered mail will no longer be scanned by Microsoft's scanning engine, leading to more malicious emails and spam getting through to users.
Starting with Exchange Server 2013, Microsoft enabled the FIP-FS anti-spam and anti-malware scanning engine by default to protect users from malicious email. According to numerous reports from Microsoft Exchange admins worldwide, a bug in the FIP-FS engine is blocking email delivery with on-premise servers starting at midnight on January 1st, 2022.
Security researcher and Exchange admin Joseph Roosen said that this is caused by Microsoft using a signed int32 variable to store the value of a date, which has a maximum value of 2,147,483,647. However, dates in 2022 have a minimum value of 2,201,010,001 or larger, which is greater than the maximum value that can be stored in the signed int32 variable, causing the scanning engine to fail and not release mail for delivery. When this bug is triggered, an 1106 error will appear in the Exchange Server's Event Log stating, "The FIP-FS Scan Process failed initialization. Error: 0x8004005. Error Details: Unspecified Error" or "Error Code: 0x80004005. Error Description: Can't convert "2201010001" to long." Microsoft will need to release an Exchange Server update that uses a larger variable to hold the date to officially fix this bug.
However, for on-premise Exchange Servers currently affected, admins have found that you can disable the FIP-FS scanning engine to allow email to start delivering again... Unfortunately, with this unofficial fix, delivered mail will no longer be scanned by Microsoft's scanning engine, leading to more malicious emails and spam getting through to users.
Good grief (Score:5, Funny)
Could Microsoft be any stupider?
Re:Good grief (Score:5, Funny)
The patch could just change it to a float.
Re:Good grief (Score:5, Funny)
you also work at Microsoft, I can tell
Re: (Score:2)
Re:Good grief (Score:5, Insightful)
Wow, talk about amateur.
Double, it needs to be a double.
Re: Good grief (Score:2)
Re: (Score:2)
Re: (Score:2)
Friends do not let friends create new date-time encodings!
Re:Good grief (Score:5, Funny)
Make it a triple then. That'll fix it for years and years to come.
Re: Good grief (Score:3)
Re: (Score:2)
Re: (Score:2)
It's not a floating point number. It's a fixed-point number.
Fixed point numbers are regular discretely counted integers, though often with a division factor to give you a fixed decimal point.
So for money, you'd use a fixed-point format, which you might know as storing money
Re: (Score:2)
An unsigned int should be enough until 2042.
Re: Good grief (Score:5, Insightful)
Whatâ(TM)s unnerving is that they are working on a fix just hours before the shit hits the fan. This problem wasnâ(TM)t difficult to test for; does nobody at Microsoft SQA ever set their test computerâ(TM)s clock forward a few years to see if anything breaks?
Microsoft: Poor social health? (Score:2, Troll)
Re:Microsoft: Poor social health? (Score:5, Informative)
Microsoft often seems poorly managed. It seems that there is an overall problem of lack of healthy social connections inside the company. Is that one of the problems?
Take it from someone who's worked there- you could write a 12-volume set of books on the problems at Microsoft and you'd barely be scratching the surface.
The problems are deep and wide, vast and directionless. They start just below the floor tiles and go all the way up to low-earth orbit.
You could literally teach a college course called "What's Wrong With Microsoft 101".
But anyway, to answer your question, "Yes."
Re:Microsoft: Poor social health? (Score:4, Insightful)
Good to know. I long since suspected that the success of MS is basically a gross market failure. They just cannot do things right. And they should never have gotten this important.
Re: (Score:2)
I had a conversation with a friend several years ago and the subject turned to Microsoft software. In the course of that conversation I commented that I thought Microsoft was the biggest thing holding back technology, only to discover that he actually worked for them. However, I didn't feel any need to amend my comment. He did end up quitting a year or two later because he was "tired of lying to customers".
I lived through Y2K as a professional software developer, and I can say that this bug is stupider th
Re: (Score:2)
I lived through Y2K as a professional software developer, and I can say that this bug is stupider than any date-related stupidity I ever saw back then.
I can say that I have seen a more stupid bug in supposedly professional software. But only once and it was not date-related.
Re: (Score:3)
They do a lot of things right. This whole Corporation bad hurr durr nonsense kinda needs to stop. It's obvious that there's not a billion nerds overlooking every facet of a tiny piece of code they decide they want to do the honours of maintaining for free either
The Stockholm Syndrome is strong in this one, Obi-Wan!
Re: (Score:2)
They do a lot of things right. This whole Corporation bad hurr durr nonsense kinda needs to stop. It's obvious that there's not a billion nerds overlooking every facet of a tiny piece of code they decide they want to do the honours of maintaining for free either
The Stockholm Syndrome is strong in this one, Obi-Wan!
Pretty much. Completely blind to the grossly obvious facts.
Re: (Score:2)
They do a lot of things right. This whole Corporation bad hurr durr nonsense kinda needs to stop. It's obvious that there's not a billion nerds overlooking every facet of a tiny piece of code they decide they want to do the honours of maintaining for free either
The Stockholm Syndrome is strong in this one, Obi-Wan!
Pretty much. Completely blind to the grossly obvious facts.
The sad thing is that if they held Microsoft's nose to the grindstone, they might not have to eat what Microsoft serves up. That was the biggest issue I had with MS. I wouldn't put up with the crap, but I was more the outlier. Their regular users would just assume that all computers had those problems.
Meanwhile, my experience with MacOS and their overnighting me new parts under warranty, as well as how they instantly responded with help when dealing with a Big Sur Bluetooth problem when I let them know I
Re: (Score:2)
Indeed. I do hope they have overdone it with the requirements for Win11 and more people will notice how they get screwed by MS. It used to be that Linux was somewhat difficult to install and Windows was easy. That situation seems to be pretty much reversed today and getting worse on the Windows side.
Re: (Score:2)
Indeed. I do hope they have overdone it with the requirements for Win11 and more people will notice how they get screwed by MS. It used to be that Linux was somewhat difficult to install and Windows was easy. That situation seems to be pretty much reversed today and getting worse on the Windows side.
That business of installation - as long as you have an internet connection. Linux installs are exceptionally easy. They go out and get what you need. And now that driver support is demonstrably better than Windows, since Windows drops drivers and even removes them. One time I accidentally installed an ancient USB-RS232 adapter on a dual boot machine, I went along fat dumb and happy until I worked on the Windows side. Wouldn't install after doing research, Windows deprecated and wasn't going to have it for
Re: (Score:2)
They do a lot of things right.
They do some things right, but the truth is that large swaths of the company are hopelessly broken, and they're too big to effectively do anything about it.
The built-in inertia of the culture and the many vested interests in keeping things the way they are effectively sabotage any attempts to right the ship.
Re:Microsoft: Poor social health? (Score:4, Insightful)
they are effectively sabotag[ing] any attempts to right the ship.
Why would they need to right the ship? From their perspective, nothing's wrong:
1) Their customers won't leave them, no matter how often or how badly they royally fuck up.
2) World governments have no desire to punish them for their misdeeds -- ever.*
3) They make money hand over fist with ten thousand percent markups that never fail to deliver extraordinary revenues.
* The antitrust trial was entirely ineffective on all counts, and accomplished nothing. The current antitrust clown-show in the U.S. is barking up the wrong tree entirely, focusing almost exclusively on disposable social media companies like Facebook and Twitter (Google and Apple are relatively minor clown-shows within a clown-show), and completely ignoring the continuing Microsoft monopoly threat.
Re: (Score:3)
Microsoft clearly does something right: their lobbying and influencing.
Re: (Score:2)
Re: (Score:2)
Take it from someone who's worked there- you could write a 12-volume set of books on the problems at Microsoft and you'd barely be scratching the surface.
The problems are deep and wide, vast and directionless. They start just below the floor tiles and go all the way up to low-earth orbit.
You could literally teach a college course called "What's Wrong With Microsoft 101".
But anyway, to answer your question, "Yes."
They really have no need to correct their problems either.
Their customers either happily accept whatever shit Microsoft hands them, or at worst grumbles a bit, then console themselves with the idea that as the industry standard, it is not possible for anything other than Microsoft to perform work.
Your 101 level class is a good idea. The follow on class might be Microsoft 201 - "Why do Microsoft Customers happily accept the problems caused by Microsoft?
Re: (Score:2)
At some point Microsoft will become, or has already become, a cargo cult maintaining software that no one understands, because there is no one left at the company that actually made any of it, and they're just going through the motions.
Re: (Score:2)
At some point Microsoft will become, or has already become, a cargo cult maintaining software that no one understands, because there is no one left at the company that actually made any of it, and they're just going through the motions.
I kinda think they indeed have reached that point. Mailing it into the installed user base, to infinity and beyond, the never ending standard.
Re: (Score:2)
I can pay a cranky know-it-all like you to cobble together a non-microsoft solution,
The best part of your attitude and the attitude of the Microsoft uber alles outlook is watching y'all struggle in your Sisyphian task, having problems that should have been solved in 1999, yet accepting them because the amount of shit you will accept from Microsoft is the amount of shit they will happily send you, as well as pushing the boundaries. And over and over again. Because as you and your ilk have shown, you're always willing to accept more shit. Kinda the sunk cost fallacy meeds Ford versus Chevy S
Re: (Score:2)
Microsoft, where quality is job number two.
And where we go number one directly in the face of anyone who gives us money.
Re: (Score:2)
Reminds me of a recent test that Google started where they set the Chrome version number to 100. I think it's mostly to test what websites break, rather than if the browser can handle it.
Re: (Score:2)
This problem wasnâ(TM)t difficult to test for; does nobody at Microsoft SQA ever set their test computerâ(TM)s clock forward a few years to see if anything breaks?
You and I both know the answer - no they do not.
And why should they? Their customers have shown that there is nothing that Microsoft can do that will turn them away.
Re:Good grief (Score:4, Insightful)
Stupid or not.
You see, I can't help noticing that on-prem Exchange servers hurt Microsoft's cloud offerings. I bet they'd love nothing better than to convert all those customers to their new SaaS lock-in scheme, and a few zero-days and other tetchnical issues might just be what they need to be convinced to switch.
I'm not saying Microsoft introduced the bug themselves, because that's conspiracy theory stuff and I don't bite. But I wouldn't put it past them to have known about the issue for a long time and sat on it in order to reveal it at the most "convenient" moment - i.e. to introduce a strong sense of urgency in the customer's mind to rethink their IT.
Re: (Score:2)
Re: (Score:2)
Re: (Score:3)
Stupid or not.
You see, I can't help noticing that on-prem Exchange servers hurt Microsoft's cloud offerings. I bet they'd love nothing better than to convert all those customers to their new SaaS lock-in scheme
Well, a couple pieces of pragmatism here. First, the rumor mill seems pretty settled that the next version of Exchange will require a subscription [redmondmag.com], so as far as recurring revenue goes, MS seems to be coming for it whether mailboxes live on one's own server, or in a datacenter in Redmond.
Second, Office365 isn't new; it's been around for about a decade now. Zero people who deployed Exchange 2016 or 2019 did so unaware that O365/M365 existed; these were intentional choices. There are plenty of reasons for that
Re: (Score:2)
But I wouldn't put it past them to have known about the issue for a long time and sat on it in order to reveal it at the most "convenient" moment - i.e. to introduce a strong sense of urgency in the customer's mind to rethink their IT.
Nobody needs to make a conscious decision like that for a particular bug. Just squeeze the budget for maintaining this piece of software in favor of the cloud offerings. Sooner or later a bad bug like this will appear.
Someone making a conscious decision to leave a time-bomb bug in has some options:
- don't tell anyone. Then no personal gain and risk getting fired if it is discovered.
- agree to do this with people above you. Now they risk getting fired if it turns out to have too much fallout. And they could
Re: (Score:2)
That's also a conspiracy.
A non-conspiracy would be that they know the Exchange market is secure and has little room for growth, so they devote only minimal efforts to maintaining Exchange.
Re: (Score:2)
Could Microsoft be any stupider?
They are obviously trying really hard to be. Will be interesting which amateur mistake they make next.
Re: Good grief (Score:2)
They keep trying to get worse by firing everyone experienced and replacing them with morons to save money.
Re: (Score:3)
I don't know that this is more stupid, but my company's email is hosted by Microsoft, and we have properly set up DMARC, DKIM and SPF records. Our SPF records include the appropriate Microsoft records and are fully valid including not exceeding the DNS lookup limit.
I regularly get DMARC reports of SPF failures from Microsoft owned domains. When Microsoft sends emails to outlook.com or linkedin.com, it sends them from IP addresses that are not in Microsoft's own SPF lists.
Re: Good grief (Score:2)
And low level IT employees that get paid per hour also love Microsoft.
Exchange doesn't even do actual e-mail. (Score:4, Insightful)
Microsoft[r] Exchange[tm] message delivery broken by bug in malware detection mechanism.
Malware detection mechanism required by decades of extremely shoddy programming work courtesy that very same company.
O irony.
unshocking (Score:2)
Rookie move there Microsoft...
Re: unshocking (Score:2)
Technical Details (Score:5, Informative)
In case the number in the summary didn't make sense, Microsoft is storing a timestamp in the format yymmddhhmm. They're using two-digit years, which we should know by now is completely stupid, and they're then using a BCD (binary coded decimal) storage to cram that into a 32-bit value, which they happened to set as signed for no good reason.
I hope the fix isn't just to make the value unsigned, but I expect it will be. If they must stick with BCD, then switch to a 64-bit value and use a four-digit year. At least they're using a format that always increases as time moves forward, so they can use the values when sorting, but they can't subtract values to get the difference in any meaningful way.
Re: (Score:3, Insightful)
Usually 'binary coded decimal implies' a fixed number of bits for each digit:
https://en.wikipedia.org/wiki/... [wikipedia.org]
This is more like 'converting a string representation of the date to an integer', which is so useless it doesn't have a name.
Re:Technical Details (Score:5, Informative)
Yes, you're right. Normal BCD uses four bits for each decimal number. If you display the raw value in hex, it comes out to the decimal value. This isn't BCD; it's just taking the decimal value and shoving it into a variable that isn't always big enough for it.
Re:Technical Details... 2043 (Score:2)
Re: (Score:2)
21 more years? Then, I am sure, it will be off support and, as Douglas Adams says, it will be invisible 'cause it is an SEP (Somebody Else's Problem).
Re: Technical Details... 2043 (Score:2)
Re: (Score:2)
It is hard to see how they could have made a worse choice that still mostly works.
Re: (Score:2)
> a worse choice that still mostly works.
It's like you've reverse engineered Microsoft's mission statement.
All of my open source majl is working today, yay. 2038 I'll pay attention.
Re: (Score:3)
> a worse choice that still mostly works.
It's like you've reverse engineered Microsoft's mission statement.
All of my open source majl is working today, yay. 2038 I'll pay attention.
Same here. Although I do not think Postfix has a year 2038 problem.
It's working! (Score:4, Funny)
Well, users are certainly being protected from receiving malicious mail....
Could they be anymore obvious? (Score:4, Insightful)
Re: (Score:2)
Mod parent up. I came here for this.
Can't wait till 2050 (Score:3)
Re: (Score:2, Interesting)
And don't forget the 2038 'Unix time' issue. Only an issue for 32-bit systems? Nope, some software, particularly closed source, still uses 32-bit timestamps internally. So your system may run fine at the OS level, but your database will fuck up. And in the world beyond servers and desktops, there are loads of 32-bit devices, many still being produced, that will fail in a variety of interesting ways.
2038 used to seem a long time away. Suddenly, it's only 16 years, and various systems using future dates are a
Re: (Score:2)
>The good news is that the 64 bit timestamp that replaces the 32 bit original gives enough capacity to last about 292 billion years, so once dealt with it's unlikely to be of further concern. :)
Indeed as the universe is only about 14 billion years old and earth only about 4.5 billion.
Exchange Spamfilter (Score:5, Interesting)
Exchange spamfilter has had other bone-headed design choices in years past: After their in-house testing showed that email send/received outside of typical business hours was statistically more likely to be spam, Microsoft made the timestamp part of their weighing algorithm to classify spam.
However, not only did they fail to account for companies that uses non-traditional office hours, they also failed to account for timezones and centered everything around Pacific Time, where the Microsoft headquarters were located. That meant that if you were in Europe or Asia, the spam detection algorithm would to the exact opposite of what it was trying to accomplish and penalize mail received while you were open and promote mail that was received while you were closed.
Re:Exchange Spamfilter (Score:5, Interesting)
When I was still administering an Exchange server (the last version I worked with was 2013), I put the whole damned thing behind a Postfix mail proxy running on a separate Linix box. There was no access to Exchange at all on port 25, and all spam functionality ran through the Postfix daemon with all the nifty anti spam tricks, along with Spamassassin. There were multiple reasons. One of them was that Postfix, even when being directly assaulted by massive dictionary attacks, never became unresponsive. In part this was, I think, just a superior TCP stack on Linux, and in part the fact that Postfix, like all good *nix tools, does a singular job, being an MTA, very well.
Frankly I long ago came to detest Exchange.
Re: Exchange Spamfilter (Score:2, Funny)
But Exchange has what plants crave!
Re:Exchange Spamfilter (Score:5, Interesting)
Re: (Score:2)
I've looked over the postfix code. Careful programming is an understatement.
Do you remember any bits that stood out? I'd like to read them and learn from them.
Re: (Score:2)
Re: (Score:2)
Actually, I just did this the other day - works great and I get nice graphing with mailgraph.
Re: (Score:2)
Have a look at proxmox mailgateway. It uses postfix and includes all anti spam tools, DKIM, etc. etc. configured out of the box with a graphical interface on top of it.
Re: (Score:2)
I think a lot of us Exchange admins did this kind of "roll your own" MTA proxy back in the day, giving rise to a whole class of products. Linux, Postfix, Spam Assassin and ClamAV on some cast-away hardware did a good job "hardening" Exchange. Plug in the various DNS-based blacklists to suit your needs and you had a pretty good solution up until the mid 2010s when you had to go with a commercial solution to keep up with all the changes in attack techniques.
Re: (Score:2)
All of what you mentioned is included in proxmox mailgateway without any need to mess around with complicated installation and configuration. You can even use it for free if you change the update repository to the dev branch instead of the enterprise one.
Re: Exchange Spamfilter (Score:2)
Re: Exchange Spamfilter (Score:3)
You can say they deliver email... (Score:4, Informative)
...on premise!
Seriously, we call it "on premises," not "on premise."
Happy New Year from Microsoft (Score:3)
Thank you for being a Microsoft customer. Be sure to ring in the new year by renewing maintenance, at our new 2022 rate (20% increase), so that you can continue to enjoy working on holidays.
Probably deliberately. (Score:3, Interesting)
They are coming up with all kinds of b*llsh*t to force customers into their subscriptions. Including an artificially initiated ca. 124GB RAM requirement for the recent versions of on-prem SharePoint.
I wouldn't be surprised if they deliberately go lax on development and maintenance of on-prem Exchange for the same reasons.
It's Microsoft after all, what would you expect?
Re:Probably deliberately. (Score:5, Informative)
They are coming up with all kinds of b*llsh*t to force customers into their subscriptions. Including an artificially initiated ca. 124GB RAM requirement for the recent versions of on-prem SharePoint.
Huh?
Requirements [microsoft.com] haven't changed for On-Prem Sharepoint for over a decade. The max for a single server is still 24GB and that's only if it's running everything, including SQL Server.
Well to be fair (Score:5, Funny)
"'Year 2022' Bug Breaks Email Delivery For Microsoft Exchange On-Premise Servers"
Cut Microsoft some slack, who could have predicted that "2022" would have followed "2021"?
Re: (Score:2)
It's been 22 years after 2000. That's a long time to remember why this was a bad idea. Right? Right?
Re: (Score:2)
**WHOOSH**
Did YOU know that the number "2022" comes after the number "2021"?
It seems like it should be kinda obvious, but maybe it's a lot trickier than I thought.
Re: Well to be fair (Score:2)
Re: (Score:2)
Re: (Score:2)
The last Y2K two digit year encoding doubled the H1B fro 65000 a year to 130,000 a year.
I'm sorry, I don't understand the point you're trying to make. (??)
Inconceivable! (Score:2)
Cut Microsoft some slack, who could have predicted that "2022" would have followed "2021"?
Yeah, that would be inconceivable! [youtube.com]
The quality of Microsoft software ... (Score:2)
Re: (Score:2)
.
Phases in a Tester's Mental Life
Phase 0 - There's no difference between testing and debugging. Other than in support of debugging, testing has no purpose.
Phase 1 - The purpose of t
Re: The quality of Microsoft software ... (Score:2)
How is exchange such a pos (Score:2)
Sounds like a new Marketing (Score:2)
WTF to the power of 6... (Score:2)
It's apparent that nobody at MSFT does code reviews. Or at least, nobody competent.
Or maybe crap like this is deliberate to make it such a pain to run on-prem Exchange servers that everyone throws in the towel and goes with MSFT's hosted Exchange service.
Follow the feedbag (Score:2)
Yay! (Score:2)
>Microsoft Exchange on-premise servers cannot deliver email starting on January 1st, 2022, due to a "Year 2022" bug in the FIP-FS anti-malware scanning engine.
Productivity will soar without constant email interruptions.
Ok then (Score:2)
On Premises (Score:2)
https://en.wikipedia.org/wiki/... [wikipedia.org]
https://en.wikipedia.org/wiki/... [wikipedia.org]
This is a native English speaking website isn't it?
Ignorance is a virus.
Did all the people who worked on Y2K retire since? (Score:2)
Anyone would think someone would have learned from that? Apparently not...
Hail to the great (Score:2)
Hail the greatest software company of all time.
Re: (Score:2)
Microsoft Spokesperson #1: "Numberz be confoozin"
Microsoft Spokesperson #2: "Look, '2022' doesn't always follow '2021' but we appreciate your input"
Microsoft Spokesperson #3: "We wuz too busy innonvatin' to think about integers 'n shit, dawg!"
Microsoft Spokesperson #4: "We hope to issue a patch soon that will make things even worse."
Re: Exchange has probably cost... (Score:2)