US DHS Testing FOSS Security

Stony Stevenson alerts us to a US Department of Homeland Security program in which subcontractors have been examining FOSS source code for security vulnerabilities. InformationWeek.com takes a glass-half-empty approach to reporting the story, saying that for FOSS code on average 1 line in 1000 contains a security bug. From the article: 'A total of 7,826 open source project defects have been fixed through the Homeland Security review, or one every two hours since it was launched in 2006 ...' ZDNet Australia prefers to emphasize those FOSS projects that fixed every reported bug, thus achieving a clean bill of health according to DHS. These include PHP, Perl, Python, Postfix, and Samba.

False positives

[clem.dickey]

The article did not seem to give any data on false positives. A story here [internetnews.com] has Coverity claiming a 10% false positive rate. But there is no independent confirmation. It would also be interesting to know how hard it is to prove a false positive vs. how hard to fix a true positive. In other words, it it worth Coverity's time to further reduce the false positive rate.

L, A and P, but where's M?

[ThreeGigs]

From TFA:
The popular MySQL open source database was not included in the scans for reasons that were not immediately evident.

Any suggestions as to why MySQL has no results? I'm stumped and wondering why one whole corner of a LAMP foundation was left unchecked.

Re:What about MS?

[filbranden]

Actually, it would be really nice if it was possible to do it with Microsoft. Microsoft (or most other companies that produce proprietary software) certainly can't do better than what the open source projects do, and certainly their code contains at least as much issues as the ones found in open source projects.

The ability to do code audits has always been one great advantage of open source software, but until now, it was mostly in theory. Now we start to see big code audit projects such as this one, showing that the advantage is real and that the results of the audit are good, since some of the projects have alread patched all of the issues, and certainly most of others will finish patching them soon. This shows that open source is here to stay, is going mainstream, and will not be stopped by any company's interests.

All issues that currently exist on Microsoft's code, on the other hand, will be unpatched. Unless they hire some consultant company (why not the same?) to do the audit on their code (certainly under NDA). But you can be sure that, if they do, for one, they won't publish the results of how many issues were found. No transparency there. And also, probably many issues won't be fixed as promptly as all of them were fixed in many of the audited open source projects. This is not a speculation, if you only look at how long it takes for them to fix issues for which there are security vulnerability reports issued, then you realise that the ones only they know about will certainly take much longer.

some notes on the article

[ehovland]

First off, prevent is not strictly a security flaw static-analysis checker. It is a static-analysis checker that checks for all sorts of defects. Some of which are directly related to security. Second, I have used prevent extensively over the past year and have found it to be an invaluable tool. It has a pretty low false positive rate and fixing the defects it finds means your code is better. On the code I work on, I find that we have a much lower defect count. But we also have pretty mature code and we really do attempt to make it as bullet proof as possible. But we still have defects.

My experience is with the C/C++ version of tool. We have also been evaluating the java version of the tool and it is good. But some of the free alternatives like findbugs are still better. I would use findbugs w/ prevent for java if I wanted good coverage.

Re:Looking good, too bad the press didn't understa

[civilizedINTENSITY]

"For example, MacOS and Windows had a similar number of critical security patches last year."

Willing to stipulate for the purpose of this discussion.

However, there were dozens of Windows viruses and hundreds of thousands of compromised machines, and zero MacOS viruses.

Likewise willing to stipulate.

Thus, while a certain measure of vulnerability is comparable, the likelihood of actually being attacked is infinitely highder with Windows.

I would suggest this doesn't necessarily follow. It could be. It could also be that while both fixed the same number of holes, the percentage of holes fixed was different. It could be that x holes represented 85% of the mac holes, whereas the same exact number x was only 13% of the available windows holes.

Not saying one or the other interpretation is true. Just that the facts don't necessarily lead to the conclusion posited.

Well THAT IS TOTALLY COOL!

[killmofasta]

Thank you DHS for the contribution to FOSS!
We get all the bug fixes, and it will become that much more robust.
Too bad that Windows will never get this kind of review.

It probibly has a few less bugs per line,
but not much hope of getting those fixed.

On second thought, Mr Allen, I challenge you to compare!
I am willing to bet that FOSS software,
just because of its nature of peer review,
and from my experence of reading ALen Cox's work on the Kernel,
that it has less bugs than Windows.

Re:PHP - no security bugs!

[ajs318]

The biggest "security problem" in PHP is related to file upload by HTTP and the way Windows handles permissions. And it's not a problem with PHP per se, but with the way some people (mis)use it. Blaming PHP for insecure scripts is a bit like blaming Severn Trent for drownings!

Every file on a Windows box has execute permission set. This appears to be a designed behaviour of Windows. If you do not perform a chmod on it after upload, it keeps its execute bit. This is entirely to be expected, and any other behaviour would violate the Principle of Least Surprise. And if you transfer the uploaded file to a directory from which the web server can serve pages directly to the outside world, it becomes a CGI script. This is a designed behaviour of the web server: on a UNIX system, files with execute set are executed and their STDOUT stream is served up. In short, by uploading a crafted script from a Windows host using a badly-written PHP script on a webserver, you can execute arbitrary code.

A naïve developer testing PHP scripts on a Linux desktop machine with Apache + PHP installed (a very common environment for pre-deployment testing; always used to be a Windows desktop with Apache + PHP, but it's easier nowadays to get Linux up and running than it is to get Apache + PHP on Windows up and running) probably will not spot this, because files uploaded from Linux hosts usually do not have the execute bit set.

One possible fix would be to add a third parameter to move_uploaded_file() allowing you to set the permissions on the destination file, and make this default to 644 if absent. Until then, don't forget to chmod() uploaded files -- and it probably wouldn't hurt to put a .htaccess file in your upload directory, blocking all HTTP transfers from there.

Re:Wow... FOSS looks pretty pathetic

[mr_mischief]

You're right. I forgot to weight them based on the portion of the installation they'd each represent.

It's also unlikely that any real installation would have exactly those packages installed, BTW. Almost any installation will have packages from CPAN, PEAR, whatever Python's central repository is called, some extra stuff like syslog, logrotate, bash, and at least one text editor at the very minimum.

Let's be a little more accurate than multiplying by defects per thousand lines to make up for my previous late-night gaffe. Let's use the actual defect numbers of verified but unfixed and unverified defects.

Apache has 19 defects in 135,916 LOC.
glibc has 0 defects in 588,931 LOC.
Linux has 461 defects in 3,639,322 LOC.
Perl has 12 defects in 496,517 LOC.
PHP has 0 defects in 474,988 LOC.
PostgreSQL has 37 defects in 909,148 LOC.
Python has 0 defects in 282,444 LOC.

That's 6,527,266 LOC and 529 defects. That's 6527.266 TLOC. I get 0.081 defects per TLOC. That's still pretty damn good.

As I said, there's probably some other software on that server, but it starts from a pretty strong base.

Re:Wow... FOSS looks pretty pathetic

[Anonymous Coward]

Are you even paying attention? Did you even read the article?
The article itself stated that
"

Open source code, much like its commercial counterpart, tends to contain one security exposure for every 1,000 lines of code

" (emphasis added)...
How does this make an argument that FOSS is a buggy mess? As others have stated, similar firms have rated the average flaws per 1000 lines higher (from 2 to 7), making 1 per 1000 equivalent to proprietary software at worst, or better depending on whose average you use.

Further it's not DHS that did the analysis, Coverity did - and they specialize in just this sort of analysis.

Now if you want to actually back up your B.S. with numbers, you should start by demonstrating how many users of FOSS actually know what to do with source code.

The numbers relevant for "MS-Haters" as you seem to like to classify Linux users are in the article.
"Linux kernel had a security bug rate of .127 per thousand lines of code."
Just over a tenth of the average for both FOSS and closed-source software.

OK, so now that you know less than 1% of FOSS's users actually know what to do with source code, you can now start looking at the quality of that 1% to see how many are actually good coders. You will likely find about 1% of that.

You haven't demonstrated this, you've just asserted it blindly as if it were a well-established fact.

Another issue is that a majority of FOSS projects have no, as in zero, code management. Nobody checks for errors, nobody worries about design, nobody cares about writing quality code. They are just cranking the stuff out and clunking it all together so they can go back to working on making another text editor for teh Lunix.

Same here. You just make vast generalizations without the numbers you are so fond of to back them up.

But when you have a multi-billion dollar software company which writes the most successful operating system ever created, do you think Microsoft has code management policies in place? I'm also willing to be MS is hiring people far more qualified than some DHS hacks.

Assumptions galore. I'd be willing to accept that Microsoft does indeed have code management in place, but your assertions about the quality of MS programmers versus DHS 'hacks' is unsupported (not to mention spurious since DHS just paid for the analysis, and did not perform it).
You are clearly either a troll, or don't know what you are talking about. You cry foul for a lack of support, but bring none of your own, and obviously either A) did not read, or B) did not understand the article in its entirety. Even as stated by Coverity, the major difference here is that we get to know the number of bugs, which we don't get for closed source apps.

Until you can give me the Coverity reports on MS products, your comments comparing MS to FOSS based on this article are pointless. The only comparison between Closed Source and FOSS that can be drawn here is "

Open source code, much like its commercial counterpart, tends to contain one security exposure for every 1,000 lines of code

"
and even that does not account for severity or exploitability of a flaw - which could come out in favor of FOSS or Closed-Source. Come back with some facts, and maybe we can have an actual discussion instead of ill-informed ranting.