Another Look At Mozilla's BugFix Rate 174
An anonymous reader writes "Washingtonpost.com's Security Fix blog has published the results of a look back at three years worth of critical patches from Mozilla, and found that Mozilla typically ships updates for critical flaws in about three weeks, though in more than a third of the cases it pushed out a fix in ten days or less. The data comes just a few weeks after The Post published data from a similar study that found Microsoft averaged 130+ days to fix critical flaws. Slashdot also covered that study in a previous post."
Just one previous post? (Score:3, Funny)
It is just your imagination. (Score:3, Funny)
Re:It is just your imagination. (Score:4, Funny)
Let's work on preventing bugs. (Score:4, Insightful)
Together they could come up with a development system that focuses on writing solid code. Such a methodology won't prevent all security bugs by any means, but it could go a long way towards vastly increasing the quality of the Mozilla project's software.
Re:Let's work on preventing bugs. (Score:2)
--
Q
The eyes are looking at the edges (Score:4, Informative)
I used to think it was just poor management, but now you have open source projects with thousands of eyes looking at every line of code.
IMHO, I believe that the reason why is because most of the developers are looking "at the edges" - where new functionality is being added. For example, how many of those developers are looking at the JPEG decompress routine? Turns out that wound up being important exploit-wise recently. And there it sat for years, unnoticed.
from what I remember in taking computer science, if you follow some simple procedures, the code is robust.
Well, robust doesn't just come from simple procedures. It's also design and style. You can't come up with excellent procedures and guarantee good software. You have to design well, communicate well, and implement ideas correctly. A lot is also owed to experience - sometimes, the only way to find out you've screwed up is after the fact. A good example is strcpy(). We know unbounded copy is a bad idea now, but how many years went by before we did?
Re:The eyes are looking at the edges (Score:3, Interesting)
Memory-related exploits should never be an issue. A proper garbage collection system, like that offered by most implementations of LISP, Smalltalk and OCaml, for instance, eliminates memory leaks. It lets the developer focus on writing solid code, without having to worry about mundane issues, without affecting performance to any extent.
And no, don't us
Re:The eyes are looking at the edges (Score:3, Interesting)
You are talking about things that _you_ want, and I can even mostly agree. However it has been proven repeatedly that only a very small minority are willing to put security/quality before other design trade offs ... things might well get better in the next 10 years due to python/C#/etc. not having as significant downsides. But even so we are going to be stuck with C based programs for a long time, and there are still very few people who want to pay to do even the minimal fixes [and.org].
Again, I'm not saying I wo
Re:Honest question (Score:3, Informative)
Re:Honest question (Score:3, Insightful)
What you're talking about is one of the oldest pieces of programming advice ever: never trust input. One of the most common human cognitive failings is to focus on intended outcomes (in this case processing valid data) and not focusing on unintended outcomes (having to handle invalid data). Probably 90% of the bugs in the world woul
Re:Honest question (Score:2)
The specification needn't be complex to reap many of the benefits of verified code; in some programs, you might just want a proof that the
Re:Honest question (Score:2)
There's no doubt in my mind that software bugs are mainly a social problem. The vast majority of software is nowhere good as the current state of the art permits. There is little reason to expect that as the state of the art progresses, that this situation will change.
That's not say
Re:Honest question (Score:2)
Naturally. You did this in first year CS because every experienced programmer knows : never trust input. And the reason this maxim is still importan
Re:Honest question (Score:2)
In a theoretical situation where you're working on a simple program, that might be a valid strategy. Another approach might be to say, "what are all the possible states of the program." You could then put the program into all those states and test that they're all stable.
If you take a program like FireFox, though, you don't get off t
So much for "fixed within hours" (Score:4, Funny)
"from the must-go-faster dept." (Score:5, Insightful)
They might be fixing critical security bugs, but they certainly don't seem to be fixing memleaks and such.
Re:"from the must-go-faster dept." (Score:3, Insightful)
http://it.slashdot.org/comments.pl?sid=176459&cid= 14655214 [slashdot.org]
The last paragraph is perhaps the most telling. It is apparently felt that issues such as memory consumption just don't matter to the average user. Of course, that's a very incorrect assertion to make. When even a normal user finds that Firefox has consumed 400+ MB of their 512 MB of RAM
Re:"from the must-go-faster dept." (Score:2)
Re:"from the must-go-faster dept." (Score:2)
I'm going to start watching this thing. There are times when I'm working in Photoshop or Illustrator when I have to start shutting things down to free up memory. I never really bothered to close out browsers though because I didn't think they'd use much.
Re:"from the must-go-faster dept." (Score:2)
Re:"from the must-go-faster dept." (Score:2)
Re:"from the must-go-faster dept." (Score:2)
Re:"from the must-go-faster dept." (Score:2)
I do think it's related to the pages I visit -- for example, I notice it more after browsing a dozen photoshop forums on Fark. And a simple shutdown/startup does clean it up, of course.
I had thought it was PDF related, so I installed the PDF Download exte
Re:"from the must-go-faster dept." (Score:3, Informative)
I couldn't believe it (I am a die hard Opera 'fanboy', although all things said Opera still wins hands down in terms of rendering performance, application startup time and general UI responsiveness).
On
Re:"from the must-go-faster dept." (Score:2)
IMHO it seems to vary alot from system to system on how well that naughty fox behaves.
Re:"from the must-go-faster dept." (Score:2)
Re:"from the must-go-faster dept." (Score:5, Informative)
CyricZ, please stop trying to get attention by being dramatic and twisting words. Your criticism is not contructive, just uninformed and inflamatory. [slashdot.org]
P.S. Re: "the attitude of the Firefox developers" - I am only one Firefox developer. I am not speaking for any other devs.
Re:Best solution! (Score:2)
Buy more RAM!
Re:"from the must-go-faster dept." (Score:2)
That, I think, is wrong. Finite resources mean setting priorities. If you set hardware conservation above all else, it will come at a cost to conceptual integrity, security, usability, etc.
Re:"from the must-go-faster dept." (Score:2)
Re:"from the must-go-faster dept." (Score:2)
Users sure do notice! (Score:2)
Even if they don't care about the numerical values in question, they still do care about getting work done. And when Firefox (or any other piece of software) is directly responsible for such slowdowns, it will be remembered.
Recall, exce
Re:Users sure do notice! (Score:2)
Re:Users sure do notice! (Score:2)
Oh, yes it does. Because it does. Granted, I'm running a bunch of extensions so the problem could be anything :) but every so often Firefox will stop responding because of some webpage, and eat up 300 or 400 MB and refuse to die until I kill the process (not just the task.) This is really fucking annoying and if I didn't know how to use the task manager I'd have to reboot, because the firefox.exe process runni
Re:"from the must-go-faster dept." (Score:2)
No kidding, this article [dbaron.org] is also from a well-know firefox developer, they're fixing lots of memory leaks.
Apparently the top poster is one of those people who lovse to get karma for apparently-insightful articles. "Cyricz said that firefox developers don't care about memory leaks!". I mean, who wouldn't believe someone who says that a software developer don't want to fix a serious bug in his software?
I wonder what paragraph is the "most te
Re:"from the must-go-faster dept." (Score:2)
I now use Opera, and I'm extremely happy with it. That's not to say I wouldn't go back to Firefox permanently if they make further improvements, but I haven't really seen too much in recent releases.
Re:"from the must-go-faster dept." (Score:2)
Already 1.5.0.1 has incorporated two memory leak fixes and more are
Re:"from the must-go-faster dept." (Score:2)
I created a
But npsr.log is always 0 bytes when I close or kill Firefox.
Re:"from the must-go-faster dept." (Score:2)
Time to play "Choose your own adventure"!
If you would like to make fun of the poster's spelling, go to item 1. If you would like to imply the poster cannot speak well, go to item 2.
1. Apparently you also need to close Firefox to run your spellchecker. (Go to item 3.)
2. Try not to talk while playing video games. (Go to item 3.)
3. You fall in a hole and die.
What about Opera, Safari, OmniWeb, etc? (Score:3, Insightful)
If so, how do they compare to Mozilla and Internet Explorer in not only the time it takes to address security problems, but also in terms of the number of incidents per release?
From TFA (Score:2)
MS Release Cycle (Score:5, Insightful)
That's a result of their past decisions. (Score:3, Insightful)
If they had maintained a rigid distinction between OS & apps, they wouldn't have those problems.
This was predicted back when MS first "integrated" their browser.
Re:That's a result of their past decisions. (Score:2)
Re:That's a result of their past decisions. (Score:2)
The problem is that the regression testbed for IE is "everything". every website ever, and almost every windows app that uses HTML rendering (which is about all of them). What you define as "welded to the OS" is irrelevant - IE's "welding" to the OS is more or less "the HTML renderer and transport layers are available as shared libraries, and are used in a few spots".
It's no more severe than a fix being made to glibc - except that windows users expect software to keep running
Re:That's a result of their past decisions. (Score:3, Funny)
Not all linux users are on gentoo. When was the last time you recompiled your entire system because of glibc fixes? I haven't had to do that once in 9 years of using GNU/linux.
except that windows users expect software to keep running no matter what
That has to be the funniest thing I read all day.
Re:That's a result of their past decisions. (Score:2)
Never.
I stopped using linux for anything meaningful prior to glibc being widely used - my last linux machine still had libc.4.so and libc5.so on it. I "survived" the a.out->elf transition, etc. That is of course an exceptional case, but it illustrates the "either rebuild world, or take an entire new binary set (from a new version
Re:That's a result of their past decisions. (Score:3, Informative)
there have certainly been library patches for freeware unix systems that have required significant rebuilding-from-sources, or installing-over your machine with a newer vendor supplied binary set.
Certainly "there have been". Probably on some special-case distro with 50 users. But I still haven't seen such a thing
Re:MS Release Cycle (Score:2)
Perhaps the reason for this is that everything in Windows is so grotesquely interdependent that MS is forced to do a full regression test on the entire OS for every little fix. When's the last time replacing your windshield wipers caused your car not to start up?
It's just numerology. (Score:5, Interesting)
No problem, they just made the shovels REALLY HEAVY, so they only had to make a few of them.
Software metrics are very slippery things.
Be Fair (Score:5, Funny)
<laugh\>
Mozilla Adoption (Score:2)
Count me as one of those... (Score:2)
I tried to search for a way to report the problem, and found the Firefox bug reporting page to be a fricking maze.
Opera conversion on its way here... (Score:2)
I'm doing with Opera what I did with Firefox (then Phoenix just prior to the name change to Firebird) when IE was pissing me off.
I downloaded Phoenix and would use it when I thought of it or when IE did something specific to piss me off. As time went on Mozilla's browser became the default browser a
Re:Opera conversion on its way here... (Score:2)
Firefox is indeed starting to show the same poor planning, poor implementation, numerous security issues, and poor resource usage that Internet Explorer is well known for.
A bug ignored? (Score:4, Interesting)
Re:A bug ignored? (Score:2)
Now, Firefox with three or four HUNDRED open tabs is kind of beastly, but then, what else was I going to do with that second gigabyte of RAM, anyway?
Re:A bug ignored? (Score:2)
Many of us who were developing software in the 1970s, and even up until the mid-1990s, are quite horrified at how poorly software today manages memory. Whatever gains have been made in terms of increased memory capacity are quickly lost due to a subsequent decr
Re:A bug ignored? (Score:2)
Now you're just being silly! (Score:2)
Re:A bug ignored? (Score:2)
Re:A bug ignored? (Score:2)
Re:A bug ignored? (Score:3, Interesting)
I think it must be adblock. I've heard so many people complaining about this, I wish one of them would try and narrow it down to a single cause. I have Firefox 1.5.0.1 on XP, with 26 tabs, over two windows, and my memory usage is at 130 MB. This includes several comics and sites with Java. I have All-in-One-Gestures, SessionSaver, undoclosetab and FLST as extensions. It's been running for about a week (I think that's the last
Re:A bug ignored? (Score:2, Interesting)
Re:A bug ignored? (Score:2)
Re:A bug ignored? (Score:2)
Re:A bug ignored? (Score:2, Troll)
One of the main indicators of over-engineering in the software world is a high level of memory consumption. Even without using an in-RAM cache, the memory usage of Mozilla is still excessive. A quick glance at the code or developers documentation will show you why: it's a poo
Re:A bug ignored? (Score:2)
Re:A bug ignored? (Score:2)
I've seen you repeatedly make this claim. Can you back it up with some examples? Perhaps a link to LXR showing some bad code?
Re:A bug ignored? (Score:2)
Right.
Konqueror is slow as a dog on my system. And after only opening three of norways major newspapers (www.vg.no, www.dagbladet.no, www.aftenposten.no) this is the memory consumption:
runevi 28997 21.3 6.4 157080 65476 ? S 20:36 0:09 konqueror [kdeinit] --silent
Plus all the kio_http proocesses, of course. Why the fsck does konq need to fork out a dozen children just for http?
Re:A bug ignored? (Score:2)
Re:A bug ignored? (Score:3, Informative)
Re:A bug ignored? (Score:3, Informative)
No, it is extensions causing the problem. Specifically adblock. Adblock appears to be a horribly written piece of shit -- it leaks memory all over the damn place. Use Adblock Plus [mozdev.org] instead -- I think it still leaks memory a bit, but I can surf for several days without reloading Firefox and be at <200M usage, while I'd hit 400M with Adblock in a day.
And I've actually added a couple extensions since switching to Adblock Plus, so if
Re:A bug ignored? (Score:2)
Re:A bug ignored? (Score:2)
Re:A bug ignored? (Score:2)
I'm looking for something just like adblock. Not an extra program.
Re:A bug ignored? (Score:2)
Re:A bug ignored? (Score:2)
Re:A bug ignored? (Score:2)
Re:A bug ignored? (Score:2)
Only publicly known bugs (Score:2)
Don't just rely on averages (Score:4, Interesting)
For example, the standard deviation for 2005 had Microsoft with a 80.87 stdev and Firefox with a 97.5 stdev.
Firefox had one flaw that took 674 days to fix, nearly twice the max of Microsoft's 357 days. Does that make up for such a larger average? Dunno. I suppose you could look at the issue [mozilla.org] and decide for yourself.
Averages are important, but it's not always the single most important thing to consider.
Re:Don't just rely on averages (Score:2)
Ok... *click*
Sorry, links to Bugzilla from Slashdot are disabled.
Ook! I suppose not.
Re:Don't just rely on averages (Score:2)
The median would be a better measure of the location of the "center" of this dataset.
The median is: 18 days
Lower bound of 95% confidence interval for median: 11.137
Lower bound of 95% confidence interval for median: 21.621
This means that you can be
Moderate Spin, but still embarassing. (Score:3, Interesting)
So if you cut the days-to-fix time up by year, for 2004, the avg is 65 days. In 2003 they used the "fixed" date in the bug DB. For 2005, its 37 days, and for 2004/2005 combined, its 42 days.
The 2004/2005 # is the interesting one, because that measures how long until the patch actually makes it into a shipping build. The availability date of source-code patches is irrelevant because most organizations are not equipped to deal with them; this is especially the case with firefox!
None of this is an excuse, however. As an MS employee, the data on our speed of patch delivery is pretty upsetting - the numbers are much worse than I would have expected. They're so bad that I am suspicious and wonder if there isn't some deeper story somewhere, but in any case, I'd like to think the maximum time anyone would have to wait would be ~1 month (flaw reported on the wednesday after "patch tuesday"), but according to this data we're not hitting that at all. I can't speak for the IE or the MSRC teams but those numbers really appear to suck. Sorry about that, everyone
Not just Microsoft (Score:2)
Not really fair... (Score:5, Insightful)
Granted, that's no small task, but it still isn't on the level of fixing an O.S., in my opinion. It's like comparing apples and pumpkins.
It would be better to compare Windows patch release time with Linux patch release time, which I believe has been done before (and then covered on Slashdot- Linux probably had the shorter time.)
Regardless, how much does market share factor into this? With Linux, if a patch breaks a program, most people can just shrug it off and rewrite the program to work with the patch. So mass testing isn't as big of an issue. With Windows, if a patch breaks a program, a user doesn't have a lot they can do except to sit there and weep until Company X releases their own patch or next version.
Re:Not really fair... (Score:2)
Re:Not really fair... (Score:2)
Except for one small fact: IE is part of (has been rolled into) the Microsoft kernel, according to Bill Gates and friends at the DOJ trial. Remember that faked video where they tried to prove it wasn't so?
So any discussion of IE vunerabilities must of necessity include the Windows kernel. Countless are the times IE or Windows explorer crashed, only to bring down W
Re:Not really fair... (Score:2)
It seems as much development model as market share (Score:2)
Personally I would have thought that this was more a development model and documentation issue than a market share i
Difficult bugs simply aren't fixed. (Score:4, Interesting)
However, other bugs simply aren't fixed. For about 3 years many, many people have reported the CPU hogging bug which is unique to Firefox and Mozilla browsers. For a small example of the reports of problems see Firefox is the most unstable program in common use [slashdot.org].
Now the problems are beginning to be reported in technical magazines [cmp.com], newsletters [scotsnewsletter.com], bloggers [slyerfox.com], and even the mainstream media.
Under the conditions mentioned in the bug reports, I'm not able to make the CPU hogging bug fail; it is always there. I've tried Linux, Windows XP SP2, and Windows 98 SE. I've tried Intel and Via chipset motherboards. For about 3 years, in all versions, the CPU and memory hogging bug has always been there. Firefox version 1.5.0.1 is worse than Firefox version 1.5, and those versions are worse than earlier ones. This is with a clean profile and no extensions except DOM Inspector, which is a menu choice on the installation program.
In 3 years, I've never had any evidence that any Firefox or Mozilla developer has reproduced the conditions that cause the problem.
The problem with Firefox and Mozilla developers not fixing difficult bugs seems to be a social one, not primarily a technical one. The developers keep asking for the problem to be made easier, but it appears to me that there is already plenty of evidence that would allow further investigation.
Perhaps the developers do not understand that there is a class of bugs that can only be found using the methods of scientific research. Many people like programming, but only people who accept the biggest challenges truly have programming in their hearts and minds:
Three biggest challenges of programming
Here are programming's three biggest challenges. Coding is relatively easy. It is these challenges which separate a true professional from an average programmer:
Instead there are excuses:
Mozilla Top 12 Excuses
Top 12 things Firefox and Mozilla developers say about those who report difficult bugs, collected during the last 3 years:
Re:Difficult bugs simply aren't fixed. (Score:4, Insightful)
It usually is. The yahoo.com crashes in 1.5 were one prime example - they were already fixed on both the MOZILLA_1_8_BRANCH and the MOZILLA_1_8_0_BRANCH.
> 2. Yes, this bug exists, but other things are more important.
While this is a rather contentious thing to say, it's usually true - there often _are_ bugs that are more important, and very little (except getting more people to hack Firefox and fix the unglamourous bugs) is going to change that.
> 3. No one has posted a TalkBack report. [If they had read the bug report, they would know that there is never a TalkBack report, because the bug crashes TalkBack, too, or a TalkBack report is not generated.]
This is rare - TalkBack usually runs for most crashes, and for ones that don't generate a report, someone can usually apply some debugger-fu to make it happen.
> 4. If you would just give us more information, we would fix this bug.
This is an excuse?
> 5. This bug report is a composite of other bugs, so this bug report is invalid. [The other bugs aren't specified.]
So? If you're reporting more than one bug in a single report, then it gets much harder to fix each of them. Separate bugs, separate reports - that's the way Bugzilla works.
> 6. You are using Firefox in a way that would crash any software. [But the same use does not crash Opera.]
Can you give an example?
> 7. I don't like the way you worded your report. (So, I didn't read it or think about it.)
If the bug report is crap, who's going to read it? Bugs don't get fixed if you can't properly explain what the bug is.
> 8. You should run a debugger and find what causes this problem yourself. (Then when you have done most of the work, tell us what causes the problem, and we may fix it.)
Why is this a Bad Thing? Some users are more than happy to do this - I personally haven't seen more than a few bugs in Bugzilla where this was even requested.
> 9. Many bugs that are filed aren't important to 99.99% of the users.
Sad, but sometimes, true.
> 10. If you are saying bad things about Mozilla and Firefox, you must be trolling. [They say this even though Firefox and Mozilla instability is beginning to be reported in media such as Information Week.]
This sort of thing is subjective.
> 11. Your problem is probably caused by using extensions. [These are extensions advertised on the Firefox and Mozilla web site.]
Advertisement on website != Well-written.
> 12. Your problem is probably caused a corrupt profile.
This is almost always the primary, major source of just about all of the problems experienced by most users. That's why there's been so much effort with mechanisms like the Extension Manager modifications, the extension versioning mechanism, the bookmarks-backup code, and the general depreciation of profiles in order to prevent users from misusing them and potentially breaking their Firefox installation.
Re:Difficult bugs simply aren't fixed. (Score:4, Insightful)
And I was intrigued by the bug you mentioned, as it seems like it would be great fun to figure out. I have never had any stability issues with firefox (any version), and I am a pretty heavy user, 8 tabs open right now, and thats really light usage for me. Normally I'll have 2-3 windows with 10-15 tabs each... I tried to use the gnu libc page, I opened 20 tabs of it, and yeah firefox was using quite a bit of RAM (about 35MB per tab), but I opened 10 windows if IE to that page and each IE window was using 45MB of RAM... so firefox was 10MB per page better as far as RAM usage is concerned... Firefox used more CPU each time I opened a new tab, but also rendered each new page faster than IE, which used less CPU for a longer period of time... I am running Win XP SP2 all patches applied, Firefox 1.5...
The only time I've seen firefox die has been on pages with that really annoying smiley face animated GIF or flash I don't know which banner ad. However, that is not the bug you are describing, so they are most likely not related... and I haven't had that bug crash FF since 1.5 came out. In fact I haven't had FF crash at all since 1.5 was released...
In short if you are having a problem, and people can't recreate it, the only option really is to attach a debugger and get to the bottom of the problem that way. As I explained above I tried to recreate your bug, I'm actually trying it on a 3rd computer right now (Dell latitude d810 512mb ram, white box winxp sp2 1gb ram, mac os 10.3 512MB ram powerbook) none of these computers exhibit the problem.. I would love to help, show me how to recreate it, and I'll gladly try to figure it out.
Re:Difficult bugs simply aren't fixed. (Score:2)
Part of the problem is that it doesn't seem to happen the same way each time. Even if I revisit something
Re:Difficult bugs simply aren't fixed. (Score:2)
I think that's a bit of an over-simplification. It's the biggest issue for you, and undoubtedly for a number of others - but to say that nothing else should be done until it's fixed is a stretch.
It should be given a fair priority, in the context of all of the bugs outstanding. (FWIW, I know a huge number of FF users, none have seen this problem AFAICT). There are curr
Re:Difficult bugs simply aren't fixed. (Score:3, Insightful)
I have nothing against people saying bad things about Mozilla and Firefox. However, in your infamous Firefox is the most unstable program in common use [slashdot.org] post, you state:
Re:Flash + Mozilla = CPU on a treadmill (Score:2, Insightful)
Re:Mozilla Bug Fixes (Score:2)
This of course turns off the people capable of fixing the original issue, and it thus gets ignored. A good first step would be creating a new Bugzilla bug, containing a distilled report of the pertinent issues surrounding the problem, and ensuring that it does not get spammed.
A perfect example
Re:Integration will just kill those turnaround tim (Score:2)