Study Finds Bug Bounty Programs Extremely Cost-Effective 95
itwbennett writes "U.C. Berkeley researchers have determined that crowdsourcing bug-finding is a far better investment than hiring employees to do the job. Here's the math: Over the last three years, Google has paid $580,000 and Mozilla has paid $570,000 for bugs found in their Chrome and Firefox browsers — and hundreds of vulnerabilities have been fixed. Compare that to the average annual cost of a single North American developer (about $100,000, plus 50% overhead), 'we see that the cost of either of these VRPs (vulnerability reward programs) is comparable to the cost of just one member of the browser security team,' the researchers wrote (PDF). And the crowdsourcing also uncovered more bugs than a single full-time developer could find."
Incentives (Score:5, Informative)
The major problem is that on-staff developers are usually discouraged from going on bug-hunts. Management would rather have them developing new features, so they won't allocate time towards finding bugs. When what the company policy towards finding bugs is conflicts with how your manager assigns you tasks, guess which one wins. Worse, most of the time an employee who ignores his to-do list to go find problems ends up penalized either explicitly (by bad reviews) or implicitly (negative impact from people being annoyed that he made work for them). Outsiders in these bounty programs don't have to worry about a manager assigning them 100% to new features and 0% to finding vulnerabilities and they don't have to worry about the impact of bad reviews or negative comments by managers about the extra work they created for everybody.
Re: (Score:2)
Hmm, just like every major software company is doing: sell first version of software, let the paying crowd find the bugs, sell second version of software with (some) bug fixed. Rinse, repeat. Profit!
Re:Incentives (Score:5, Informative)
This.
And not just bug hunts. I have a laundry list of things that need to be refactored, but every time we think we might have a chance to do so, project management decides something else is more important. We have people complaining about things being slow, but when told that we need to spend time to make it faster, we instead get directed at new features or, worse, tweaks for the sake of a single non-representative customer that happens to have the ear of the project owner.
Re: (Score:1)
Do you work at the same middleware company as me?
Re: (Score:2)
Yes. At work we have a long-outstanding issue open about performance. I nailed down where the bottleneck was, and put in the library module needed to fix it. Tests using the new module were showing a minimum order-of-magnitude speedup compared to the old (minimum because the test cases were biased to be as favorable to the old method as possible). Yet we've been directed to postpone actually switching to the new module (I suspect politics).
Re:Incentives (Score:5, Insightful)
Re: (Score:3)
Wow. Aren't you worried that by posting this online that you might get fired from your position on the Microsoft SQL Server dev team?
Re: (Score:2)
Yes this happens. The trick to get around it is to sell the bug fixing or refactoring as a new feature. i.e. you refactor the code in order to fix the longstanding bug and you add a new feature to boot. If your manager is too stupid to figure out the bug really, really needs to be fixed, you can always convince the client to "persuade" them instead of you.
Re: (Score:2)
I've done this as well - but I don't like effectively misleading management by saying that we need to do X in order to achieve what is only a loosely related Y. Yes, X would make Y easier, or improve Z and W, but it isn't truly essential.
That's not to say things are entirely bad. I've also had management see the true value of some stuff that isn't technically being requested at that very moment, and push things upwards by tying the work to something else in just this manner.
I guess the moral is that I don
Re: (Score:2)
I do not like to mislead project management either. The less information people have about the actual de facto state of the project the more likely it is that things will go off track and development will end up in failure because you either blew up the deadline or funding. However it is also important that projects have a certain degree of quality in them. Otherwise the clients will end up not renewing their contracts and going somewhere else next time. Balancing these needs is certainly not easy.
I once ha
Re: (Score:2)
that depends (Score:2)
Re: (Score:2)
Nonsense. The outside world can only fuzz against the product looking for a security vulnerability and they are only paid for security vulnerabilities.
Bug hunts reveal architecture missteps that will break the product during upgrades or other usage. Internal developers are aware of the architecture, so they are more able to focus on searching and finding both security vulnerabilities as well as general bugs. A testing matrix cannot predict all the
Re: (Score:2)
As the Firefox Security Manager I completely and vehemently disagree. I employ a team that spends 100% of their time "going on bug-hunts" looking for security bugs in Firefox, and I know my counter-part at Google is doing the same for Chrome. Our Bug Bounty programs (VRP? ugh, so very corporate) are an incentive for people who stumble on neat stuff to pass it on, not a substitute for doing the work ourselves.
What's an interview have to do with it? (Score:2)
Re: (Score:2)
And you don't have to worry about employment laws.
math problem? (Score:2)
isn't $570,000 / $150,000 about 3.8 people? (articles numbers.) Still probably a good deal, but not quite as good.
Re: (Score:1)
Re: (Score:2)
Re: (Score:3)
No. The developer just reports the bugs to the development team. Perhaps we should give that developer a special title like "Quality Assurance Engineer".
Re: (Score:1)
isn't $570,000 / $150,000 about 3.8 people? (articles numbers.) Still probably a good deal, but not quite as good.
$570,000 was over 3 years. $150,000 was salary plus overhead for 1 year.
Instead of 570K/150K = 3.8 developers, you get 570K/450K = 1.27 developers
Re: (Score:2)
That number is over 3 years according to the summary, so you need to divide by 3.
dilbert (Score:5, Funny)
http://dilbert.com/strips/comic/1995-11-13/ [dilbert.com]
Re:dilbert (Score:5, Interesting)
Re: (Score:3)
For a lot of these people, it might be a hobby. If it weren't for bug hunting for a bounty, they might be working on open source software instead with no payout at all. For those people, the payout is infinitely greater even if it amounts to $2.50/hr. Most people are just happy to have a hobby that breaks even, nevermind nets a profit.
Re: (Score:3)
Re: (Score:2)
That works well until the guy gets caught red handed passing off vulnerabilities to an outsider and not only canned but possibly jailed.
Google has shown itself to be very strict about enforcing its NDA policy, and divulging exploits to outsiders in general is a major legal risk.
Re: (Score:1)
Re: (Score:2)
I don't understand that comic, is the funny part where the salaried employees (dumb slaves) realize that the money is in consulting?
No the funny part is that the boss doesn't realize that he's just written the software developers a blank check that they can write any amount they want in.
Re: (Score:2)
Re: (Score:2)
VRPs are the new sweatshops (Score:3, Interesting)
This is indeed true specially for popular companies with rather mature SecOps that pay minimum wages for vulnerabilities that are indeed hard to find or require a pretty darn good skill level to discover. Some of them even only offer swag in exchange of finding serious threats such as persistent XSS or authentication bypass. They maybe feature the researcher in some blog post to publicly thank him and attract the wannabe crowds.
Having said that, I myself have participated in several of these programs (with varying success) and come to realize that probably Google and Facebook are the only VRPs currently paying reasonable wages for bugs in terms of cost efficiency for the researcher.
On the other hand, some of us just enjoy from time to time trying to find security bugs for fun (maybe because we are huge nerds) so these programs offer a great opportunity to test things and not risking ending up in jail.
Re: (Score:2)
Then dont spend your time finding vulnerabilities for those companies?
Im not seeing the comparison to a sweatshop here.
Re: (Score:2)
A Ferengi would find an exploit, sell it on the black market, and then shortly there after report it. Profit!
Re: (Score:2)
Re: (Score:2)
Morals are expensive. If you can get the same work for less money (by using a VRP) then you're doing it right, as far as the organization is concerned. They don't care about anything else.
Re: (Score:1)
QA Finds Bugs, Devs Fix Them (Score:1)
Maybe they should have compared the salary of a QA person instead of a developer. As a developer, I find lots of bugs, and then fix them. I also fix the bugs that QA finds, but usually spend a lot of time trying to figure out how to reproduce the issue ("uhh, first I clicked on this and then I clicked on that and then something weird happened").
Anywhile, it's hard to crowd source a product that has not been released yet and most companies don't have the fan-bois and gurls to even consider this strategy
Re: (Score:2)
but usually spend a lot of time trying to figure out how to reproduce the issue
You need better test employees.
Re: (Score:2)
Wait, you're saying QA isn't writing bugs that have detailed steps to reproduce? If they are ones that can be easily reproduced, then they're not writing good bugs. (Obviously there are lots of bugs that are worth writing up that DON'T yet have reproducible cases yet.. sometimes a conglomeration of those not-reproducible cases can lead to a reproducible case too...)
BTW, while I definitely think that 'bug bounty' isn't as good as a company finding its own bugs, I wish ALL companies had an official way to r
It does not work if Wally is in the team. (Score:1, Redundant)
Re: (Score:1)
I believe it was a minvan
Cost of fiunding bugs != cost of fixing them. (Score:5, Insightful)
this is very bad (Score:2)
That means that there is a strong incentive for companies to create insecure, crappy software and then let so-called "white hat hackers" fix their bugs at a discount. And because any other form of disclosure is illegal, the companies are pretty well protected from negative consequences of their bugs and deflect from their own negligence by blaming "black hat hackers".
...at creating mystery meat. (Score:2)
Given that disclosure is also at the terms of the payer, you also get less transparency versus independent disclosure.
Why employees don't find these bugs (Score:3, Interesting)
Because the sort of programmer that's good at finding/fixing these bugs...is not the sort of programmer that the interview process determines would be a "good fit" for the organization.
Ineffective, unfortunately (Score:4, Interesting)
This is effective for the low-hanging fruit, i.e. the easy (relatively) to find security-related bugs. For things that require advanced techniques or expensive tools (like Fortify), it fails. Unfortunately, the harder to find bugs are still well within reach of spy agencies of all kind, including a number that is allowed to do industrial espionage (like the US or France).
So while this looks good on the surface, it is really just making the problem worse. The only exception is software that has very low security needs.
For reliability, it is about as ineffective, as only easy to identify bugs will be tracked down.
$100,000 bug finder (Score:2)
No shit Sherlock (Score:3)
What I'm really shocked about is that you need a university to figure this out. Or rather do research on this. Companies figured this out quite some time ago and anyone with a functioning brain can see why. :)
What I'm more interested in is that king of people spend their time in participating in programs like this. The chances that you find a bug are not that big. The financial reward, given the amount of time you will spend on finding a bug is probably also relatively small.
From a company's point of view on the other hand, it's great. Many people working for you. For free. A job well done
Even better if the checks aren't cashed (Score:1)
or one instead offers ``certificates of deposit'' in the (fictional) ``Bank of San Seriffe'': http://en.wikipedia.org/wiki/Knuth_reward_check [wikipedia.org]
William
(who is quite bummed that he didn't get his reward check back when Dr. Knuth was using Wells Fargo as his bank: http://www.truetex.com/knuthchk.htm [truetex.com] )
Not a replacement (Score:3)
Average? Where? (Score:1)
It sure isn't the average in Canada.
Re: (Score:2)
So what? You've got 3 developers. Not like your going to move the average.
Re: (Score:3)
High-tech regions tend to be high cost of living, too. In Silicon Valley, 100K USD may well be better than an entry-level salary for a dev with a 4-year degree. The cost of living is so high that this is less impressive than it sounds, though. It's a little less bad up around Seattle ("Silicon Forest") where starting salaries are more commonly in the 70-90k range, but people break six digits very quickly. I haven't job-hunted anywhere else, but at least on the west coast, a 100k estimated average might actu
Typical business-centric bullshit reporting (Score:2)
It's not surprising at all that piecemeal work, with no provision for healthcare, vacation etc. - much less reliable, ongoing income - is more profitable for business.
Why should technology workers be intrigued or inspired by this? Why is this information presented to technology workers as another avenue to praise Google's or Mozilla's cleverness? And why do technology workers so consistently dig their own graves by latching onto this kind of ideology and failing to fight for labor rights?
Well, yes, it'll work for browsers... (Score:3)
where you have millions of folks looking at your free software for long periods of time. If you're a commercial software vendor, however, with a $10,000 non web-based package and at most a few thousand users (There are still a *lot* of these), then this approach is very unlikely to succeed. Commercial software users are rarely interested enough to report a bug that doesn't actively interfere with their daily work.
Re: (Score:2)
get outsourced to india or cut entirely?
Probably not a replacement for full time employees (Score:2)
I get paid to audit code, so I'm biased.
The article says that no one employee could find hundreds of bugs and that's true. But when you hire employees you are building a process. Improving the process by writing a new QC script can eliminate hundreds of bugs over a couple years. These are not attributed to one employee and since the offending code is not committed then they aren't even counted as bug fixes.
Offering a bug bounty, on the other hand, is a unpredictable thing and you'll get random fixes. It
developers don't create bugs........(kinda) (Score:1)
developers don't "create" bugs, we don't sit down and say, hey, lets create a bug! and then go about making one, most of the time, we believe our code doesn't have bugs, cause if we thought it had bugs, we'd write it a different way, or we'd know about the bug in the first place and it'd be in our list of things to fix, normally those things are fixed quickly because we knew it was there, but things we don't know about, well, how do you expect us to find it? I didn't find it whilst I was writing the code an
Researchers should compare vs QA (Score:2)
So UC Berkeley should compare the number of bugs found by researchers vs the number that Google's Internal QA Dept has found.
Then we'll know if it was really worth while. Since Google would never publish the number of bugs they find internally, all this data is worthless.
It is nice though getting people to QA your projects for basically free.