Slashdot Log In
Firefox Analyzed for Bugs by Software
Posted by
CowboyNeal
on Sat Aug 12, 2006 10:47 AM
from the killing-bugs-dead dept.
from the killing-bugs-dead dept.
eldavojohn writes "In a brief article on CNet, a company named Coverity announced that Firefox is using software to detect flaws in Firefox's source code. Even more interesting is the DHS initiative for Coverity to use this same bug detection software on 40 open source projects." An interesting tidbit from the article: "Most of the 40 programs tested averaged less than one defect per thousand lines of code. The cleanest program was XMMS, a Unix-based multimedia application. It had only six bugs in its 116,899 lines of code, or .51 bugs per thousands lines of code. The buggiest program is the Advanced Maryland Automatic Network Disk Archiver, or AMANDA, a Linux backup application first developed at the University of Maryland. Coverity found 108 bugs in its 88,950 lines of code, or about 1.214 bugs per thousand lines of code." We've covered this before, only now Firefox is actually licensing the Coverity software and using it directly.
Related Stories
[+]
Developers: Programmers Learn to Check Code Earlier for Holes 212 comments
Carl Bialik from WSJ writes "Many companies are teaching programmers to write safer code and test their security as software is built, not afterward, the Wall Street Journal reports. This stands in contrast to an earlier ethos to rush to beat rivals with new software, and, of course, brings tradeoffs: 'Revamping the software-development process creates a Catch 22: being more careful can mean missing deadlines.' The WSJ focuses on RIM and Herb Little, its security director, who 'uses Coverity every night to scan the code turned in by engineers. The tool sends Mr. Little an email listing potential red flags. He figures out which problems are real and tracks down each offending programmer, who has to fix the flaw before moving on. Mr. Little has also ramped up security training and requires programmers to double-check each others' code more regularly.'"
This discussion has been archived.
No new comments can be posted.
Firefox Analyzed for Bugs by Software
|
Log In/Create an Account
| Top
| 226 comments
| Search Discussion
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
Math (Score:5, Informative)
Re:Math (Score:5, Informative)
(http://www.fishface.g.la/)
If this is the same (Score:3, Interesting)
Re:If this is the same (Score:4, Informative)
(http://www.carva.org/charles-henri.gros)
Corrected link [coverity.com]. Unfortunately there are only 2 examples since there are trade secrets involved with bug reports.
This might look like a slashvertisement, but I didn't submit the original story (which does pick up on a press release)
That would tend to reccomend it to me (Score:2, Insightful)
(Last Journal: Saturday January 01 2005, @08:16AM)
Errr... (Score:1, Redundant)
(http://blog.mzzt.net/)
I hope these Coverity guys aren't pompous enough to think that their tool can find ALL bugs in a program with... magic...
Hmm, they should run their tool on its own source code, that would be fun.
Re:Errr... (Score:5, Interesting)
Re:Errr... (Score:5, Interesting)
Re:Errr... (Score:5, Insightful)
Re:Errr... (Score:5, Informative)
(http://www.carva.org/charles-henri.gros)
We aren't (I'm a Coverity employee). We find real bugs, and we find false positives (but not too many of those).
Hmm, they should run their tool on its own source code, that would be fun.
We do that regularly.
this slashdot news is already outdated (Score:5, Informative)
Re:this slashdot news is already outdated (Score:5, Insightful)
(http://www.livejournal.com/users/strawberryfrog/ | Last Journal: Wednesday April 27 2005, @06:28AM)
You mean "who have brought down the count of their bugs that this tool can detect down to zero." I'm sure they will have other bugs in code and design.
How does this tool compare to tools that do analysis by introspection on bytecode from languages like C# and Java. I use FxCop [gotdotnet.com] on C# code, and while it is very cool, using it is not newsworthy at all. Does this tool do more? Is is the news that it's used in a high-profile C++ program?
Integrating tools like this into your build process may be cutting-edge best-practice at present, but give it a while.
Re:this slashdot news is already outdated (Score:5, Interesting)
(http://www.lightandmatter.com/)
You mean "who have brought down the count of their bugs that this tool can detect down to zero." I'm sure they will have other bugs in code and design.
Yeah, if they could make a program that would detect all bugs in a program, it would violate Turing's proof that the halting probelm is undecidable. [wikipedia.org]
From the articles, it sounds like they're basically looking for mistakes that could lead to security flaws, e.g., buffer overflows. If AMANDA is particularly buggy by their metric (detectable bugs per thousand lines of code), it's probably because AMANDA doesn't interface to the web, so the people coding it knew that certain classes of buffer overflow "bugs" wouldn't be a problem, because they wouldn't be exploited through an internet-facing interface. If you went back and ran this program on Unix apps written in C from the 1980's, you'd probably find zillions of bugs, but it wouldn't indicate low quality, it would just mean that the programs weren't written for an internet-facing environment in the year 2006, when the internet has become a battle zone for evil spammers, botnets, etc. If the only way such a bug can show up is for the user to supply carefully tailored input, and the result is simply that the program dumps core, then that's not a bug for a program that isn't facing the modern internet.
Interesting... (Score:3, Interesting)
Bug/Lines of Code (Score:5, Funny)
(http://hysdeals.blogspot.com/ | Last Journal: Tuesday January 11 2005, @11:30PM)
Sounds like someone needs to run this debugger on their calculator.
Once detected can they also fix the flaws? (Score:3, Funny)
AMANDA is cross-platform (Score:1, Interesting)
Amanda IS, however being very actively developed right now, lots of new features -> lots of new bugs. Other issue is that it's a componenty, plugin architecture, made of a few processes communicating over pipes and sockets. A failure in one component won't necessarily be a security risk or take the whole system down, it's extremely robust in normal operation in my experience, despite this "high bug count". Unlike XMMS, various contributed plugins (e.g. tape changer robot drivers) are redistributed in the source tarball but only used by very small numbers of people with outlandish hardware.
I suspect if you included various XMMS plugins in the XMMS count, things would be different...
None of that *really* excuses a high bug count - but what really pisses me off is coverity's "we've found X bugs, but we're not going to tell you what they are or substantiate our claims (some of amanda is quite old code, has a lot of strcpys, I know that some automated security checkers will treat a strcpy as a "bug" even if it's safe), just FUD your project in various public fora...
Which type of bugs? (Score:3, Informative)
"Meh. So much for the 'many eyes' theory" (Score:5, Insightful)
(http://robotterror.com/slashdot | Last Journal: Thursday November 04 2004, @05:48PM)
In this age of SarbOx and risk management there is a real competitive advantage to F/OSS over proprietary code to large companies: audit-ability. In previous roles I've had to attest under HIPPA::Security that proprietary code was "secure" -- how? All I could do was obtain a vendor statement that was as non-commital and burden-shifting as possible. Yet, with a true ability to audit the code my pharmaceutical company depended on it would tilt the balance between similar-featured Closed vs Open source solutions. Especially today.
Ok, maybe nobody really cares about the 'many eyes' theory anymore. Regardless, the "open the hood" theory still applies, perhaps more than ever.
Meanwhile... (Score:3, Funny)
(http://www.linicks.net/)
For those who are interested in Firefox' results (Score:5, Interesting)
Open Coverity Bugs [mozilla.org]
All Coverity Bugs [mozilla.org]
Why AMANDA is buggy (Score:3, Insightful)
(Last Journal: Monday May 05 2003, @06:46PM)
AMANDA (and others) have fixed the defects (Score:2)
(http://arc.nucapt.northwestern.edu/F/OSS)
Those that follow amanda-hackers will know that there was less than a week [yahoo.com] between when coverity released the report on March 6th and it was announced that all bugs were fixed in AMANDA on March 12th.
I dislike the idea of Coverity (Score:1, Interesting)
I would find heuristic analysis annoying. I'd get quite annoyed if the program says "fix this buffer overflow" 1000 times because I use "strcpy" somewhere - even though I'm very careful and only use it when I know it can't overflow.
I should write a program that searches for odd perfect numbers [wikipedia.org] and terminates if it finds one. I wonder whether Coverity would say it is an infinite loop.
Coverity sounds like scare tactics to make money by claiming to do the impossible. They won't even disclose what their algorithm is. I would never trust them, especially on closed-source programs. Firefox doesn't have that risk, but they are wasting money.
Microsoft's PREfast is simpler but seems like a much more realistic solution: mark up your code to say how things are supposed to be used and the compiler can decidably sense problems. I'd just get tired of typing 2 underscores a million times.
Melissa
Re:I dislike the idea of Coverity (Score:5, Insightful)
(http://www.animats.com)
It is not possible for a program to analyze another program and find all the bugs; see halting problem .
Wrong. It is quite possible to analyze a program and find all the bugs that violate the language constraints (null pointers, buffer overflows, etc.). That's what program verification is for. For some programs, you can't tell whether a bug condition will occur, so you treat that as a bug.
Automated program verification is a good idea that went away because C and C++ have such ambiguous semantics. It's hopeless for those languages. The "pointer equals array" concept alone makes it very tough, because the language has no idea how big an array is. Worst idea in the language, and the root cause of buffer overflows.
Good verifiers were written for Pascal (I headed one of those projects [animats.com]), a good one was written for Java [dec.com] (at DEC, just before DEC went under), and Microsoft is working on one for C#. [microsoft.com]
The halting problem is not an issue (Score:5, Informative)
(http://www.animats.com)
The halting problem is not an issue for program verification. This claim is raised repeatedly by the clueless, and it just isn't an issue.
Yes, you can construct a program that's formally undecideable. It's a hard way to write a bad program. It takes some work, and the resulting program is unlikely to be useful.
Most crash-type and security-hole problems in programs are entirely decidable. This is because almost all subscript calculations are composed from addition, multiplication by constants, and logic operations. Those are totally decideable, and there are good decision algorithms for that problem. Only when multiplication of two variables (both non-constant) is introduced can formal undecidability appear. See Presburger arithmetic [wikipedia.org].
In fact, halting is decidable for all deterministic machines with finite memory. Either you repeat a previous state, or halt within a finite number of cycles. The decision process may be made arbitrarily hard, but that's not undecidability. True undecidability in the Turing sense requires infinite memory.
Most of the practical problems with program verification come from dealing with interactions between various parts of the program. Containing those interactions well enough that you can localize problems is constraining on the programmer. "Design by contract" languages like Eiffel try to do that, but they're not popular. Retrofitting design by contract into C and C++ has been discussed, but the proposed schemes all have holes you could drive a truck through. A big truck.
Although software work seldom uses proof of correctness techniques, there's a whole industry doing it for hardware. There was a machine-generated formal proof of correctness for the FPU in AMD's K7 processor. [onr.com] AMD thus avoided the "Pentium division bug".
Re:I dislike the idea of Coverity (Score:5, Informative)
(http://open-news.net/)
The fact that it is impossible to solve the whole problem of program correctness and that false positives will come up doesn't mean that the problem Coverity is adressing isn't usefull.
Regards,
No rsync? (Score:3, Interesting)
In defense of amanda (Score:3, Informative)
Thats not to say that as new features are added, new bugs haven't been too, but to actually call amanda a truely buggy application does stretch this users belief a wee bit. I'm currently running a 20060424 dated snapshot of the 2.5.0 tree, with no hiccups at all.
--
Cheers, Gene
So then ... (Score:1)
(http://www.iknowcss.com/)
VMware Uses It Too (Score:1, Informative)
Firefox is again the most unstable program... (Score:2)
(http://www.futurepower.net/)
The 1.5.0.4 version of Firefox was quite stable, if the Flashblock extension was installed. The 1.5.0.6 version is unstable again. The CPU-hogging bug is back!
This comment posted from a copy of Firefox that is constantly using 2.8% of the CPU, even when all pages have been loaded, and there is no active content. That's 2.8% on the way to 70% or more, making it necessary to close Firefox and reboot Windows XP.
There are some bugs found by Coverity [mozilla.org] left unfixed, but so far things have gotten worse since 1.5.0.4, not better.
Types of bugs (Score:3, Interesting)
(http://dexplor.com/)
It looks like most of the real bugs consist of not checking return values, the worst being routines that act upon an object allocated by another routine without checking for null pointer.
Dan East
Homeland Security Tested XMMS?! (Score:3, Insightful)
40 programs were tested. 40 open source programs. Not even all the programs installed by, or regularly used on, a default install of a particular distro or two; just 40 programs. I thought maybe these 40 were just the first 40 tested, but the original announcement of the award of the grant states that 40 programs would be tested.
And yet they didn't test BIND? ssh? Also, PostgreSQL is on the results list, but MySQL isn't? Did Homeland Security put this list together?! Using a dartboard and a list of open source applications, or what?!
This seems like a great software package, and I'm glad that Homeland Security acknowledges that "much of the critical infrastructure runs on open source", but I could think of a few other ways they could've spent $1.2 million, or at least a few other applications they should've tested before they got to XMMS.
Managed Code? (Score:1)
Why are we in the IT world still causing ourselves problems by using C/C++ in any situation except those which call for the strengths of C/C++ -- strengths which are quickly being matched by their managed counterparts.
Realtime? Embedded? Video game? Ok, use C/C++ (though even video games you could make an argument...). For everything else, there's managed code. No more memory pointer leaks (well, the hard-to-find kind caused by poor pointer management in C/C++), no more buffer overruns that aren't immediately fixable in one place, etc, etc.
C'mon, we, as the IT profession, have evolved past that. Why are we still trying to work around these already-solved problems?
Is this supposed to be news? (Score:2)
(http://www.adkap.com/ | Last Journal: Thursday August 10 2006, @04:10PM)
Hiring dozens of QA people to discover discover a broken error path that a developer could have fixed in 5 minutes is inefficant and slow. Instead, developers can find many bugs quickly by excerising the data paths of their own code. After (or even better, before) writing a new snippet of code, a motivated developer writes a unit test for it. The test is simple: given certain specific conditions, your code should return some predictable result.
Example- I need to write code that tell me which of two numbers between 1 and 10 is larger. I would need to write at least 6 simple tests that would guarantee this code will work in virtually every scenario. I would need to three tests to make sure it works when the input in valid, and another 3 tests to make sure it works when the input is invalid. Sample inputs:
After a while there will be dozens, hundreds of even thousands of these simple tests. They will not mean much alone, but chained together they can provide a very stable testing bed. By running all the tests every time you release another version, developers can be pretty sure that everything that worked in the past will still work. (see regression testing).
Anyway, I didn't mean to start a lecture on basic testing principles... This post was motivated by the claims in the article regarding code quality. I would take the claim that 'XMMS is the most bug free software' with a grain of salt; a big grain. The results of any testing procedures are only as good as the individual tests. Lets say that the above example was one of the tests that Coverity was using in XMMS. Maybe it passed, great! But what where to happen if a user input -13 instead of 12? or maybe -5,000,000,000 and 1? How about the letter A as an argument?
They don't even give us code coverage numbers, or a count of tests... just a claim. I would bet that the AMANDA project had 10 times as many tests per line of code as XMMS. That would make AMANDA the most bug free software. Oh, and on another note, i noticed some comments about IE testing. I don't use IE, but I am 100% certain that Microsoft has an internal testing framework that puts this one to shame.
But how much does it cost? (Score:1, Interesting)
The less up-front anybody is about costs, the less worthwhile their product usually is. And the more variable the cost usually is (ie: as they figure out how much they can overcharge you). And no, I will not register with them for the "honor" of finding out more information. I'm guessing that it's something stupidly outrageous since the cost of running their application on a bunch of Open Source programs cost $1.2 million - which anyone with a single copy and a free weekend probably could have done for themselves.
They also don't disclose what their product actually does. So I'll join with the other voices here in calling for the need of an open-source alternative to this project - an alternative that has full disclosure about what the product is capable of and what it's going to cost you to use.
Coverity on Windows? (Score:4, Insightful)
From the Amanda Wiki: (Score:1)
(http://amanda.zmanda.com/)
http://wiki.zmanda.com/index.php/Developer_documen tation [zmanda.com]
coverity (Score:1)
Patch gratitude (Score:2)
(Last Journal: Thursday March 30 2006, @10:04PM)
An example from one of the submissions for the Coverity bugs in Mozilla where the coder who made the patch actually had the balls to set the arrogant committer prick right.
Re:not just any software? (Score:3, Informative)
How does this bug detection software work anyway?
Re:not just any software? (Score:3, Informative)
Bugzilla is a issue tracking software; it's useful only after you've already found a bug. The only other bug-related tool they use is the FullCircle crash reporter thingy, again, after-the-fact thing. This is different - this tool finds flaws from the source code automatically.
Re:GNAA (Score:4, Funny)
Firefox is a browser (Score:2, Insightful)
(http://clusty.com/search?query=google)
And I'm assuming that they mean "Mozilla is using Coverity..." or "Firefox developers are using Coverity...". After all you don't hear about what Internet Explorer is doing, but rather what MS are doing with it.
Wouldn't it be great if the summary was clearer and neither of us had to make mental amendments?
Re:Check the checker (Score:2)
Re:not just any software? (Score:1, Funny)
Re:not just any software? (Score:1)
Re:not just any software? (Score:1, Informative)