Vulnerability Numerology - Defective by Design?

rdmreader writes "RDM has a point by point disassembly of the security vulnerability story phenomenon. We regularly see these, comparing various vulnerability lists for different operating systems. ZDNet's George Ou, for example, condemns Linux and Mac OS X by tallying up reported flaws and comparing them against Microsoft's. What he doesn't note is that his source, Secunia, only lists what vendors and researchers report. Results selectively include or exclude component software seemingly at random, and backhandedly claims its data is evidence of what it now tells journalists they shouldn't report. Is Secunia presenting slanted information with the expectation it will be misused?"

Numerology?

[RyanFenton]

<Skeptical Nitpick>
Did the guy who titled this know what the term Numerology [wikipedia.org] means? It's usually associated with wild "magical thinking" about numbers, and is at best a rather silly form of pseudomathematics.
</Skeptical Nitpick>

Ryan Fenton

Re:Numerology?

[Spy der Mann]

Did the guy who titled this know what the term Numerology means?

Exactly. IMHO, he's saying that Secunia vulnerability comparisons aren't any more reliable than numerology predictions.

# of Vulnerabilities!=Acknowledged Vulnerabilities

[Foofoobar]

Number of vulnerabilities in a product is not the same thing as the acknowledged number of vulnerabilities in a product. Secunia reports on the number of acknowledged vulnerabilities. Microsoft is known for NOT acknowledging vulnerabilities even though they have been reported to the company and then SUDDENLY fixing them in a patch.

And then unfortunately, their supporters like to bash Linux and Mac for actually working with security agencies and fixing their bugs as well as reporting them. This will forever be the bane of open source and it's benefit... that everyone gets to see its flaws but at the same time, everyone gets to contribute to fix them.

Again and again and again

[RAMMS+EIN]

We keep hearing this again and again and again.

It's very simple, really.

You can _never_ know the relative security of two systems. There simply isn't any way to measure it fairly.

Count disclosed vulnerabilities? What about the vulnerabilities that weren't disclosed?

Have teams search for vulnerabilities and compare the results? What does that tell you? Was one team equally good at finding vulnerabilities in one system as the other was at finding them in the other system? What if one system had many easy to find vulnerabilities, and the other had a couple of severe but harder to find vulnerabilities?

Count actual break-ins? Well, was that due to the system being vulnerable the way the vendor left it, or because of the administrator? What about break-ins you don't know about?

It's always a matter of what you don't know about. You don't know the vulnerabilities that weren't reported. You don't know the vulnerabilities that weren't found. You don't know the relative skills of the teams you used. You don't know if you tested for all possible classes of vulnerability.

And I haven't even mentioned the severity of vulnerabilities, the availability of exploit code, the way vulnerabilities are dealt with by the vendor, and a host of other issues.

The take home message is that you just _can't_ know. It's a hard pill to swallow, but you will just never know which system is more secure. All you have is flawed metrics and your gut feeling.

Re:Fishing for vulnerabilities

[Bill, Shooter of Bul]

Yeah, but if the htmlspecialchars was exploitable in geshi, then it was a vulnerability in geshi. You can't ignore vulnerabilities inherit in the language you use. If it was exploitable in geshi, then you in turn exposed the users of geshi to the vulnerability by incorporating the function into your implementation. I mean imagine microsoft claiming that buffer overflows were not its fault, as they were really vulnerabilities in C, not windows/explorer/office ect.

Nothing to see there, move along

[Aaron Isotton]

When I read the summary, I thought TFA could actually be interesting. But it's not any better than what it is criticizing.

Long story short:

ZDnet published an article comparing Secunia vulnerability counts in Mac OS X and Windows Vista/XP. They spun it the Microsoft way, so Mac OS X loses big time. A mac fanboy wrote a reply spinning it the Apple way.

TFA starts with a long-winded attack against the author of the ZDnet article without ever getting to the point. Let's just say that it talks about Zunes, XBoxes, train wrecks, ballet dancing and many more things.

Then it explains what Secunia does (in about two pages): they track software vulnerabilities which are - among others - reported by the vendors. So "honest" vendors get higher vulnerability counts. Who would have thought.

On it goes by saying that the "border" of an operating system is nowadays blurry; should the vulnerabilities in bundled applications be counted? Even if they are by another vendor?

Then he babbles about how most of the cited vulnerabilites in Mac OS X are related to what he calls "external software" - things such as python, java, perl, samba, tcpdump etc and that those same programs have the the same (or a similar) amount of vulnerabilities on other platforms. What he fails to point out is that Mac OS X *consists* of such "external software" for a big part, and that they are *part* of Mac OS X and cannot be removed easily.

Conclusion: a pointless (and extremely long-winded) article full of Microsoft bashing, as reply to an equally pointless article full of Apple bashing.

Re:Fishing for vulnerabilities

[pongo000]

Then using this logic, it would be appropriate and fair for Secunia to list every project that is using PHP with the tainted function. Hundreds? Thousands? Tens of thousands? Where are those vulnerability reports?

Again, this goes back to my argument that Secunia simply cherry-picks its reports, penalizing those projects that are most open with their changelogs and issue tracking, often listing so-called "vulnerabilities" after said vulnerabilities have already been addressed (as in this case).

Re:About Secunia

[sumdumass]

No, Not perfectly. Earlier versions of windows could be exploited without any interaction of any user at all outside the author of an automated virus. Even things in Linux could have the same types of vulnerabilities. Although, it is rare to see automated programs that could exploit them with no user interaction and then replicate and launch another attack somewhere else. This seems to be a windows only thing.

If you said that removing the user removes a significant portion of the vulnerabilities, then you would likely be correct.

Re:Numerology?

[irenaeous]

. . . that Secunia vulnerability comparisons aren't any more reliable than numerology predictions.

I RTFA. He is not critical of Secunia per se. He quotes a lot from Secunia's advisories and claims that George Ou has misused the data. In other words, Ou is practicing Numerology with Secunia's numbers. Presumably then, Secunia's numbers can be used intelligently by others who know how to correctly interpret the data. His criticisms of Ou sound correct to me, but I don't care for all the extremely harsh ad hominem. It makes him look angry does not help.

Re:Vulnerability Counts: Humorous, Not Useful

[51mon]

It is a common trait to want to reduce everything down to a single number, or something easily compared, especially when most folks have only a very vague definition of the area being compared.

Everyone wants to validate their own prejudices (and some are paid to support other folks interests).

Security is a process, the goal of which is to protect something (usually your data - maybe your hardware - maybe availability or even user sanity!) and (usually at least) to minimize the resources it takes to do it. You can only meaningfully produce numbers when you are more specific than "security" or even "vulnerability".

So it might be possible to say discover the number of bugs that allow arbitrary remote code execution through web surfing (although in some cases the answer might be "may be" for some bugs), using the bog standard install of the OS, installing all the latest patches as soon as they are available, using the vendor preferred web browser. But even then this is only listing discovered vulnerability, so all you have is a number that is almost meaningless to real security, although it is comparable, if that you can use it to compared how safe browsing was. The IE/Firefox days vulnerable is a good example of such a metric, but again it depends on known vulnerabilities.

If someone produced a range of such tests, not just covering vulnerability counts, but covering other things (for example - some one mentioned that users don't always patch - thus the proportion of users who are patched up to date could make a useful metric about how usable the softwares update mechanism is, which I'd suggest is a key security metric).

One might be able to make a case for a rigorous methodology for using a selection of such tests, but that requires serious research and effort, and we already know the result will be; -- most Desktop OSes are less secure than most end users would like if they only understood what all the techie blurb meant --