SpamAssassin 2010 Bug 115
SEWilco writes "You might want to check your spam folder, as SpamAssassin has a rule which is tending to mark email sent in 2010 as spam. There is some discussion in a bug report. The SpamAssassin Wiki FH_DATE_PAST_20XX page doesn't have discussion, but it was updated today with a different date rule."
Re: (Score:3, Insightful)
First, it learned that 90% of mail is spam. Then, it got lazy. Now, it just stamps everything.
Millenium bug, how I have missed thee (Score:2, Insightful)
Re:Millenium bug, how I have missed thee (Score:5, Informative)
A lot of gross little hacks look like (and are) great ideas when hardware costs a fortune and you don't yet know how persistent legacy stuff is going to be.
Re: (Score:2)
You're almost right, but actually you miss the point. Back in 1970, RAM cost around $1/byte, so using 32 bits instead of 64 saved you $4/record. However, that wasn't what caused the millennium bug, it was choosing a silly representation to use to store the date. If you wanted to, you could have stored dates as a 32-bit quantity in seconds since some fixed time and been able to handle any date in a 140 year period. You could also have performed calculations on these dates very quickly (just simple arithm
Re: (Score:2)
I'm actually thankful that they ran this one as a story. After seeing it, I remembered that rule. I just hopped in and fixed my servers.
The better rule would have been to watch for dates more than X days from today. But, they're using regular expressions, so that makes it a bit harder.
Re: (Score:2)
> I'm actually thankful that they ran this one as a story.
Yep, not that /. is a security list, but it occurred to me a few times that I first learn about similar issues on /.
Re: (Score:2)
I think 2038 is a nonissue. Technological progress is accelerating. It took us 30 years to upgrade from 2 decimal digits to 4 for storing years. It will not take 30 years to upgrade from 32 bits to 256 bits for time.
FX_DATE_20XX rule due to be obsoleted (Score:2, Informative)
I think 2038 is a nonissue.
In this case it really is as this rule is due to be removed in future releases of SpamAssassin, for details see: https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6271 [apache.org]
Re: (Score:2)
If they had decided to start using 64-bit time on the 1st of January, 1970 none of these problems would have happened
The pattern:
/20[1-9][0-9]/ [if-unset: 2006]
##{ FH_DATE_PAST_20XX
header FH_DATE_PAST_20XX Date =~
describe FH_DATE_PAST_20XX The date is grossly in the future.
##} FH_DATE_PAST_20XX
Will match 2010 no matter how many bits are used to keep time. This is a problem that has been know for years; it's a total embarrassment to OSS practices that it wasn't fixed before 1/1/10, before becoming critical. It's not based on a bad architectural decision, or even particularly bad code - it's just a
crapola (Score:4, Interesting)
My provider runs spamassassin, and given their track record in updating their other software, I rather doubt that they'll update spamassassin anytime soon. Is there any way around this that doesn't involve root access? (I love helpful responses from idiots that start with "first, edit the /etc/spamassassin.conf file" or whatever.)
Oh yeah, the other wonderfully helpful stock response "stop using the software if you don't like it". Sure, I'd love to go back to getting 500 spams a day.
Re: (Score:2, Informative)
Is there any way around this that doesn't involve root access?
No.
Re:crapola (Score:4, Funny)
I love helpful responses from idiots that start with "first, edit the /etc/spamassassin.conf file"
Yes. That's easy! First edit the /etc/spamassassin.conf file!
Wait a sec...
Re:crapola (Score:4, Interesting)
Re:crapola (Score:5, Funny)
I think you missed the "My provider runs spamassassin" part of the parent post.
Re: (Score:2, Informative)
OpenBSD spamd(8) is wholly unrelated from spamassassin spamd. FYI.
OpenBSD spamd(8) has no code from any other project. Its similarity in appellation is name deep.
OpenBSD spamd(8) approach is different and was created by deraadt@.
Re: (Score:2)
OpenBSD spamd(8) is wholly unrelated from spamassassin spamd
Not entirely. You can combine the two quite well. The spamdb tool can be configured quite easily to take input from SpamAssassin, so if someone sends you spam they then get blacklisted or more aggressively greylisted.
Re: (Score:2)
Re:crapola (Score:5, Informative)
Yes, but it isn't a good way. Check your scores file for the scores associated with the FH_DATE_PAST_20XX. This indicates the number of points added to the spam score of every message that fails this test. Basically, increase your spam threshold by this amount until you can apply this patch.
Good for a quick-n-dirty fix.
Re: (Score:3, Informative)
Re: (Score:1, Offtopic)
My provider runs spamassassin, and given their track record in updating their other software, I rather doubt that they'll update spamassassin anytime soon. Is there any way around this that doesn't involve root access? (I love helpful responses from idiots that start with "first, edit the /etc/spamassassin.conf file" or whatever.)
Oh yeah, the other wonderfully helpful stock response "stop using the software if you don't like it". Sure, I'd love to go back to getting 500 spams a day.
My car is so old it doesn't have an airbag. I heard about this glitch where I have a higher chance of dying because my car doesn't have an airbag. Given my track record in buying new cards, I doubt I'll update my gremlin anytime soon. Is there a way around this that doesn't involve require spending money? (I love helpful responses from idiots that start with "first go to NAPA" or whatever.) Oh yeah, the other wonderfully helpful stock response "stop driving the car if you're worried about dying in a wr
Re:crapola (Score:4, Informative)
The new rule gets picked up when "sa-update" is run. spamassassin deployments should run sa-update automatically on a regular basis, for example every day via a cronjob. Thus, most deployments will pick up the update automatically tonight if a sysadmin doesn't do it first.
Re: (Score:2)
Simplest non-root way (assuming they have it allowed): Edit the ~/.spamassassin/user_prefs file.
Re:crapola (Score:5, Informative)
My provider runs spamassassin, and given their track record in updating their other software, I rather doubt that they'll update spamassassin anytime soon. Is there any way around this that doesn't involve root access?
If you have shell access, it should be trivial, although you do have to edit a file.
Add the following to ~/.spamassassin/user_prefs:
score FH_DATE_PAST_20XX 0.0
Re: (Score:2, Informative)
Re: (Score:3, Interesting)
Re: (Score:2)
My provider runs spamassassin, and given their track record in updating their other software, I rather doubt that they'll update spamassassin anytime soon
If they are slow enough updating, it's possible you don't have the bug. I haven't bothered to update my home mail server past Ubuntu 6.06 LTS, since it works fine. The version of Spamassassin it uses, 3.1.7, does not appear to have this bug.
Re: (Score:1, Troll)
> (I love helpful responses from idiots that start with "first, edit the /etc/spamassassin.conf file" or whatever.)
>Oh yeah, the other wonderfully helpful stock response "stop using the software if you don't like it". Sure, I'd love to go back to getting 500 spams a day.
"Idiots?"
You might have a point if the ONLY step began with "first". But here you see a bunch of people answered you.. and yes, ALL of them began with editing the "whatever" file.
You know... paid support for SpamAssassin IS available.
I
Re: (Score:1)
Oh yeah, the other wonderfully helpful stock response "stop using the software if you don't like it". Sure, I'd love to go back to getting 500 spams a day.
Is the alternative of receiving no mail more palatable?
Re: (Score:1)
My provider runs spamassassin, and given their track record in updating their other software, I rather doubt that they'll update spamassassin anytime soon. Is there any way around this that doesn't involve root access?
Not that I know of, but perhaps you might have access to a local.cf file in which you can set the score of the rule to a lower value or ignore it by setting the score to 0. Before the update I did this (in my case) by adding the following line to /etc/mail/spamassassin/local.cf:
score FH_DATE_PAST_20XX 0.0
The new rule however is not released in a new spamassassin release yet, but the rules are pushed through the udpate channels, so perhaps your provider might have received the update. Othwerwise it is w
"I'll just use a regex!" (Score:2, Interesting)
Re: (Score:3, Interesting)
Last I checked a timezone was mandatory in mail dates, allowing you to handle this correctly - for example by converting your current date and the date specified in the email to GMT...
Re: (Score:3, Funny)
Re: (Score:2)
I'm guessing the obvious solution is to make sure $current_year >= $header_year + 1
That would only hurt you when receiving mail from a machine with a clock wildly out of sync, and as all major OSes do NTP out of the box by default, I don't see how that's an excuse.
Re: (Score:2)
I agree, that would be the better solution.
I've seen lots of people's machines that have their dates wrong though. NTP is a wonderful thing, if it works. It's possible that corp firewalls can block unexpected traffic, including NTP though.
I just got an embedded machine, which doesn't keep it's time right. I have to force an update at boot time, which makes the logs look real funny.
There was a Y2K bug on some of my servers (like, a decade ago), so
Re: (Score:2)
I agree totally.
When I set up those servers, ntpd didn't cooperate very well for huge changes, it would just try to skew the clock. I wrote my own little program to check in with our local time server. Our time server would hit a real time server once a day, and the servers would all update hourly. In the case of those, I put it in at boot time, before any important services started that needed to know the time.
Re: (Score:2)
That's actually what I've been doing at boot time for several years now. When I did my own program, it was quick and dirty to run my own.
Re: (Score:2)
...(like, a decade ago)...
Hopefully nobody had trouble inferring that, since it is now 2010, Y2k would have been ten years ago...
Re:"I'll just use a regex!" (Score:4, Insightful)
Your solution doesn't work.
It fails on new years eve if someone is in a different time zone or if their clock is slightly off.
I'd suggest that any message sent more than seven (pick your favorite number) days in the future is spam.
Re: (Score:3, Interesting)
Your solution doesn't work.
It fails on new years eve if someone is in a different time zone or if their clock is slightly off.
You need to talk to Microsoft - they could have used someone like you when they were building their 'stampinf' tool that I can't use until 10am in the morning (11am during DST) because it stamps UTC which then confuses another tool in the build chain...
Re: (Score:1)
FIX details: (Score:5, Informative)
this is also happening on Ubuntu server, running Spamassassin 3.2.5
The linked article references a workaround: /etc/spamassassin/local.cf
add this line to the "local.cf" spamassassin config file, on this system is was
score FH_DATE_PAST_20XX 0.0
If you're running spamassassin as a daemon, you *may* also want to restart spamd
with something like:
sudo /etc/init.d/spamassassin restart
This solution simply removes the rule by setting the score for that rule to 0.
You'll want to undo this once a solution is deployed.
Re:FIX details: (Score:5, Informative)
Since nearly 14 hours ago, you can simply run "sa-update".
It is in cron.daily in the default install, too.
Re: (Score:2)
score FH_DATE_PAST_20XX 0.0
You'd probably be better off changing the rule instead of the score. Putting something like:
in your local SA config would probably be better (unless you really want to just drop the rule entirely).
That said, you'd probably be better off updating everything with sa-update, but this f
Re: (Score:2)
If you are running amavis, sa-update alone is not enough. You also need to restart amavis so that it picks up the new rules. On a Debian system, that would be something like:
sa-update && /etc/init.d/amavis restart
The spamassassin daily cron job does sa-update, and reloads spamassassin, but doesn't reload amavis.
Re: (Score:1)
Re: (Score:2, Funny)
Y2KX has a better ring to it.
One hack replaced by another (Score:3, Informative)
From the "fix"
> FH_DATE_PAST_20XX
> change '/20[1-9][0-9]/' to '/20[2-9][0-9]/'
That's no fix, it just puts the problem off for another 10 years. Why call the rule FH_DATE_PAST_20XX, shouldn't it be FH_DATE_PAST_201X? At least then the hack would be documented.
Re: (Score:3, Funny)
I just fixed a mail server for someone (after reading the article), and told them to remind me in 10 years to fix it again. :) Hopefully by then, they'll have rewritten the rule so it behaves better.
Re:One hack replaced by another (Score:4, Insightful)
Exactly. This is the right "fix" right now - the bug is out there causing real problems, and the fix (while definitely a filthy hack) is well-understood and can be pushed out immediately. If the same thing were to happen ten years from now (or the threshold quietly got pushed back to 2030), that would be nothing short of negligent.
Re: (Score:2)
That's no fix, it just puts the problem off for another 10 years.
I had that thought too. Then I had another thought which was 'Meh'.
Re: (Score:1)
Why does this rule exist anyway? (Score:2)
I can only guess that it's a rule to flag e-mails as being spammy (all of my new years' wishes to brits and americans I got replies to with the subject tagged "{Spam?}") when they are 'sent from the future'.
I can also only guess that this is done via a regex on the date with the date being somewhere far into the future (but that future kinda crept up on us the other day, right?) so as not to burden CPUs with actual date conversions to universal time and doing comparisons on that, and flagging
Re: (Score:1)
2. spammers will be readily aware of this rule and really why would they make an effort to inject a future date anyway when their mail daemons will happily use the -current date-
They will use dates from the future to have their offerings shown at the top of the mailbox.
Re: (Score:2)
Then there's the other date-mangling tactic, setting the date weeks or months in the past, I'm guessing they're doing this to make recepients think that the spam is some old legit mail they forgot about, it's annoying when one of those slip through though...
/Mikael
Re: (Score:1)
Why call the rule FH_DATE_PAST_20XX, shouldn't it be FH_DATE_PAST_201X? At least then the hack would be documented.
Changing the rule name would break existing configuration files changing the rule score and/or description (like in that comment [slashdot.org]).
Implicit whitelisting? (Score:1)
Nice.... (Score:5, Funny)
[url]https://issues.apache.org/SpamAssassin/show_bug.cgi?id=5852[/url]
Noticed 14 months ago. Fixed 5 months ago. Released today.
Re: (Score:3, Insightful)
Hey! Nice to see open source software gets fixed so ultra fast! :P
To be fair though, at least they released it the day it broke things, why didn't they release it by yesterday? Then the default cron job would have picked it up on most servers and nobody would have noticed.
Re: (Score:1)
Hey! Nice to see open source software gets fixed so ultra fast! :P
My sarcasm-o-meter is broken, so I cannot tell whether you are kidding. But it is indeed impressive to have a fix within hours, on a holiday.
To be fair though, at least they released it the day it broke things, why didn't they release it by yesterday? Then the default cron job would have picked it up on most servers and nobody would have noticed.
Why didn't they release it before someone noticed? Well, that's a good question. Maybe it was because nobody had noticed?
Re: (Score:1, Insightful)
Why didn't they release this when it was fixed.
Fixed in spamassassin 3.2.5-7 in Debian/Unstable (Score:2)
n/t
Re: (Score:2)
spamassassin: /var/lib/dpkg/status
Installed: 3.2.5-6
Candidate: 3.2.5-6
Version table:
*** 3.2.5-6 0
701 http://ftp.us.debian.org/ [debian.org] testing/main Packages
70 http://ftp.us.debian.org/ [debian.org] unstable/main Packages
100
apt-get updated 30 seconds ago.
Re: (Score:3, Informative)
toncho/~ sudo apt-get install spamassassin
Reading package lists... Done
Building dependency tree
Reading state information... Done
Suggested packages:
libmail-dkim-perl
Recommended packages:
re2c
The following packages will be upgraded:
spamassassin
1 upgraded, 0 newly installed, 0 to remove and 1086 not upgraded.
Need to get 1097kB of archives.
After this operation, 0B of additional disk space will be used.
Get:1 http://ftp.us.debian.org/ [debian.org] unstable/main spamassassin 3.2.5-7
Re: (Score:2)
I know y'all are all proud about how fast it got fixed, but in reality it was discovered over a year ago, fixed half a year ago, and was only released today.
Why? Why not at least release it yesterday so most people's daily update would pick it up and make the whole thing invisible?
Re: (Score:3, Informative)
It was an oversight. The rule fix got committed but never added to the update channel. Nobody noticed before it was too late.
Re: (Score:2)
> I know y'all are all proud about how fast it got fixed...
Where do you get that? I posted to inform people running Debian that they need not apply the patch. That's all. I expressed no opinion about the speed of the fix.
Aha! (Score:2)
So this is why my Schwab Stock Alert was marked as Spam this morning.
Now it shows up (Score:1, Funny)
Ahh, the y2k.01 bug, naturally.
holy crap (Score:2, Informative)
Thanks for the heads up, my kid's birthday party is next weekend and when I look at the spam folder it turns out 3 more people have replied that I hadn't seen.
Great workaround (Score:4, Informative)
The suggested fix is just silly... They postpone the problem to 2020-01-01:
3) change '/20[1-9][0-9]/' to '/20[2-9][0-9]/'
Re: (Score:2)
We're likely going to drop the rule all-together in the future. In the interest of getting the fix out fast, with the least confusion of users, we decided to just make the change to 2020 for now.
Although... maybe a little visibility for the project every 10 years wouldn't hurt. :)
Re: (Score:2)
Ideally it should use an offset from the current date (which assumes the system clock is correct, but on a mail server it really REALLY should be).
Re: (Score:1)
Have you ever done production operations before?
Goal 1: Get People Working Again.
Goal much-farther-down-the-list: Document, Debug, and get people on the issue to properly patch/ensure it never happens again.
End User (Score:3, Interesting)
As an end user, with no real ability to configure SpamAssassin other than through cPanel, what can I do about this? (Unless my webhost is right on top of it.)
I've disabled it, for now.
Re: (Score:2)
sa-update, assuming you can run that through cpanel.
Re: (Score:3, Informative)
Re: (Score:1)
If you can't get the rule to stop firing (by changing the rule or zeroing the score) you could increase your spam threshold by the amount of the rule's score (anywhere from 2 to 3.6 or more depending on your config). This would essentially cancel out the rule.
Re: (Score:1)
I almost missed some important mail! (Score:5, Funny)
"You might want to check your spam folder, as SpamAssassin has a rule ...
Thanks for the heads-up. There was a very important e-mail from the Internet Lottery people telling me my e-mail address had been picked as the winner of the EUR 20,000 prize. All I have to do is send them $200 by Western Union to cover the processing fees. And to think I almost missed it!
It's terrible that SpamAssassin flags such important messages as spam.
Re: (Score:1)
I hate it when I make a joke and I get modded "+5 insightful". Mod the stupid comments "funny", not "insightful", pleas
Moderators, please note: he's not being funny --- he's being insightful [xkcd.com].
It would be 90% right! (Score:2)
Only a small fraction of email I get is legit, so if it dumps all messages into the spam folder, it is pretty much doing the right thing
Note from the VP, Apache SpamAssassin (Score:5, Informative)
Clearly we dropped the ball on this one. As far as I know it's our first big rule screw up in the project's 10 years. If you're going to screw up you might as well do it well.
I posted the following note to the Apache SpamAssassin website (http://spamassassin.apache.org/). Updates are available via sa-update, please run sa-update immediately. It's included in all versions of 3.2.x (the affected version of SpamAssassin). Alternatively zero the rule's score in your local.cf file if you have access to it. If you don't, increase your spam threshold by 3.6 points if your mail provider allows you to do that.
Y2K10 Rule Bug - Update Your Rules Now!
2010-01-01:
Versions of the FH_DATE_PAST_20XX rule released with versions of Apache SpamAssassin 3.2.0 thru 3.2.5 will trigger on most mail with a Date header that includes the year 2010 or later. The rule will add a score of up to 3.6 towards the spam classification of all email. You should take corrective action immediately; there are two easy ways to correct the problem:
* If your system is configured to use sa-update run sa-update now. An update is available that will correct the rule. No further action is necessary (other than restarting spamd or any service that uses SpamAssassin directly).
* Add "score FH_DATE_PAST_20XX 0" without the quotes to the end of your local.cf file to disable the rule.
If you require help updating your rules to correct this issue you are encouraged to ask for assistance on the Apache SpamAssassin Users' list. Users' mailing list info is here.
On behalf of the Apache SpamAssassin project I apologize for this error and the grief it may have caused you.
Regards,
Daryl C. W. O'Shea
VP, Apache SpamAssassin
Re: (Score:3, Interesting)
Thanks. SpamAssassin has saved a lot of time on my life.
I was using Spamcop and got a bit zealous on reporting so spammers flooded my email address out of vengeance. It was then when I learned that reporting spam to admins only works in a few cases while the majority of spam is sent either through botnets or from countries with relaxed or no spam laws.
SpamAssassin removed 99% of the spam without ever giving me problems of blocking wanted messages...
Respectfully, thanks.
Re: (Score:2)
Working great over here - all I had to do was run sa-update.
Thanks for the hard work!
2K10 is not 2010 (Score:1)
Re: (Score:2)
Re: (Score:1)
Unless you are used to the electronics way:
1K2 = 1.2 k = 1,200
M47 = 0.47 M = 470,000
68R = 68
Fine grain rule to stop anything after 2014 (Score:2)
This rule is a little bit more fine-grained. It will not allow anything to be sent after 2014.
edit the file /usr/share/spamassassin/72_active.cf
##{ FH_DATE_PAST_20XX /20[1-9][4-9]/ [if-unset: 2006]
header FH_DATE_PAST_20XX Date =~
describe FH_DATE_PAST_20XX The date is grossly in the future.
##} FH_DATE_PAST_20XX
I wish variables could be used in regexps like these, like $year + 2 would be a nice start....
Re: (Score:1)
I recommend against the above suggestion for a number of reasons:
- /usr/share/spamassassin is not the place to edit your rules. Edit your local.cf file.
- We will be dropping this rule or changing it to a non-regex rule that actually takes the current date into account. We'll do whichever our statistical data suggests would be most beneficial. Once we've done this, we'll push another update via sa-update. I wouldn't risk forgetting that you've modified a rule that's going to hit you again down the road.
B
Re: (Score:2)
Actually a better rule taking that in account is:
##{ FH_DATE_PAST_20XX /20(1[4-9]|[2-9][0-9])/ [if-unset: 2006]
header FH_DATE_PAST_20XX Date =~
describe FH_DATE_PAST_20XX The date is grossly in the future.
##} FH_DATE_PAST_20XX
I've made a post entry about this on my blog at http://development.artistblog.me/2010/01/spamassassin-2010-bugfix/ [artistblog.me] ; although I've heard cases where sa-update didn't work, so I wouldn't know which way would be the best since there are more ways to skin a cat ...
Many thanks to the poster! (Score:1)