AI Spots Critical Microsoft Security Bugs 97% of the Time (venturebeat.com) 41
Microsoft claims to have developed a system that correctly distinguishes between security and non-security software bugs 99% of the time, and that accurately identifies the critical, high-priority security bugs on average 97% of the time. From a report: In the coming months, it plans to open-source the methodology on GitHub, along with example models and other resources. Their work suggests that such a system, which was trained on a data set of 13 million work items and bugs from 47,000 developers at Microsoft stored across AzureDevOps and GitHub repositories, could be used to support human experts. It's estimated that developers create 70 bugs per 1,000 lines of code and that fixing a bug takes 30 times longer than writing a line of code, and that in the U.S., $113 billion is spent annually on identifying and fixing product defects. In the course of architecting the model, Microsoft says that security experts approved the training data and that statistical sampling was used to provide those experts a manageable amount of data to review. The data was then encoded into representations called feature vectors and Microsoft researchers designed the system using a two-step process, in which the model first learned to classify security and non-security bugs and then to apply severity labels -- critical, important, low-impact -- to the security bugs.
AI determines 97% of bugs are Microsoft products (Score:2, Funny)
Re: (Score:2)
ftfy
Boy, that's the truth!
Microsoft products are the most consistently-buggy, inconsistently-improved software products on the planet.
I have to Develop around their many giant turds on a daily basis; and the sheer mind-numbing quantity of their bugs, both big and small, both in the products and the tools for working with them, is both appalling, and a source of daily frustration for me,
Re: (Score:1)
This can be applied to the healthcare industry as well. "YES THE PATIENT WILL DIE". (Eventually. Maybe even 40 years later.)
AI determines 97% of MS products code has bugs (Score:2)
50% of them are gonna be: (Score:3)
"You were using C or C++ and did not intend to write optomized low-level routines."
The others will mostly be: :)
"You are a script kiddie. Go back to your homepage!" and
"You wrote an enterprisey abomination. Are you perchance a consultant?"
Re: (Score:2)
If you're using C you should be switching to C++.
But ... one robo-detected bug every 14 lines of code? How low did they stoop to find those code samples? Did they just download the whole of GitHub for analysis or something?
Re: (Score:3)
from the article, this thing doesn't seem to be detecting bugs, but categorizing text descriptions of known ones, so as to prioritize them.
Re: (Score:2)
sorry, responded to wrong post. ignore.
Re: (Score:2)
from the article, this thing doesn't seem to be detecting bugs, but categorizing text descriptions of known ones, so as to prioritize them.
If this tool could actually detect 97% of bugs (or even 10% of bugs) it would be an astounding advance toward strong-AI.
What it actually does is rather trivial and uninteresting.
Re: (Score:2)
something something Rust (Score:2)
I'm just waiting for the flood of "But if you'd just used Rust, then..." posts.
Re: (Score:3)
Hour and a half later you're the only one to have chimed in about Rust... perhaps the insufferable Rust advocates you were hoping for aren't really the problem you think they are. It may even be that people bitching about Rust advocates are the actual problem now.
It has been half a decade since Rust 1.0. The initial hype has died away and the posers and evangelists are off hyping something else now. What we're left with is a maturing language and serious people making stuff, and they don't spend a lot
How it works (Score:2)
If (author == "Microsoft") { is_critial_bug = true;}
Not too suprising. (Score:4, Interesting)
Many bugs and security glitches are from often simple mistakes, made when the developer is tired (either short term, from physical exhaustion, or long term from just doing that big project for such a long time) shortcuts are made, Proof of concepts are put in and forgotten to harden. Specs changed mid-project so half-written code is still there, debugging backdoors are not closed...
Even the best developer isn't perfect. AI isn't as good as a person is, but it is tireless and will catch a lot of the simple things people do.
Re: (Score:2)
AI isn't as good as a person is, but it is tireless and will catch a lot of the simple things people do.
from the article, this thing doesn't seem to be detecting bugs, but categorizing text descriptions of known ones, so as to prioritize them.
Re: (Score:1)
What? (Score:2)
97% of Microsoft code has critical security flaws?
We knew that since Windows 2.03, no AI needed for that.
Maybe (Score:3, Insightful)
Re: (Score:2)
I know. Why doesn't Microsoft just hire perfect people. They only seen to hire crap coders who produce buggy software.
In case you can't tell I'm mocking you for how you think programming and people work.
Zoom (Score:2)
Look, the only kinds of articles about security bugs I want to hear about are articles about Zoom security holes. It's been almost 24 hours since the last Zoom security hole article. WTF is up with the Editors? We need to kill that company's stock ASAP, so it should wall to wall "Zoom Sucks!" articles until the company goes broke.
MS Devs are terrible... (Score:2)
Re: (Score:2)
Over the course of my life I've developed systems that were thousands of lines long and had maybe one or two bugs!
So what you're saying is you've worked for places without proper QA/testing and all your projects kept getting cancelled before they were deployed?
Re: (Score:1)
Oh, it gets even better if you keep reading and run the numbers:
It's estimated that developers create 70 bugs per 1,000 lines of code and that fixing a bug takes 30 times longer than writing a line of code...
Note that 70 * 30 = 2100 > 1000. Apparently one must allocate 3.1 times as much development time as expected from the code alone in order to also fix the bugs; this also assumes that finding bugs is a zero cost operation.
Choice quote (Score:2)
Itâ(TM)s estimated that developers create 70 bugs per 1,000 lines of code and that fixing a bug takes 30 times longer than writing a line of code
Who estimated this? Microsoft? If so, that explains quite a bit.
I'm not trying to troll here, but that number is about 5 times what it should be. Without naming names, I know for a fact that it is possible for large organizations to achieve less than 15 defects per kloc through formal testing and code reviews. If you bring in standards-based approa
Re: (Score:3)
If you read TFA, the number isn't from microsoft. It's just sourced from this random article (https://coralogix.com/log-analytics-blog/this-is-what-your-developers-are-doing-75-of-the-time-and-this-is-the-cost-you-pay/) that claims (totally unsourced) that 70 bugs/1000 lines is before QA, and 15 bugs/1000 lines is what get to customers
Re: (Score:2)
Really going to depend on the complexity of the lines.
Re: (Score:2)
Itâ(TM)s estimated that developers create 70 bugs per 1,000 lines of code and that fixing a bug takes 30 times longer than writing a line of code
Who estimated this? Microsoft? If so, that explains quite a bit.
I'm not trying to troll here, but that number is about 5 times what it should be.
If you read the article, you'd find that those numbers come from Ariel Assaraf writing for coralogix.com [coralogix.com], and the next line reads:
15 bugs per 1,000 lines of code find their way to the customers
Re: (Score:2)
15 bugs per 1,000 lines of code find their way to the customers
that's orders of magnitude worse! :'-) sounds like a really shitty sw shop to me, and not just the junior programmers ...
that said, measuring bugs per lines of code is kind of nonsense, unless used only in a closed and very specific environment (language, field, expertise, ...) over time to read 'trends'. even so ...
Re: (Score:2)
Something doesn't add up: if they spend 75% of their time fixing bugs, that means the cost of fixing the bug is only 3x the cost of writing the code, not 30x. To get a 30x number, they'd have to spend only 2.5% of their time writing code, 22.5% on something else.
The 15/1000 number is in an article linked from the original. I suppose I should have looked at that, but the linked article appears to be a marketing pitch. Granted, that's not a bad number, but it's still on the high side of my experience.
Re: (Score:2)
The number of bugs in code is highly dependant upon the developer producing the code. Additionally, some developers have bad habits that are too ingrained to fix. I once worked with a "consultant" who just could not understand the concept of freeing a resource at the same level of acquiring it in a C project. What he would consistently do is malloc() a piece of memory in a function in the middle of the call tree, and then free() that memory in all the leaf functions that could potentially be reached from th
Great! (Score:2)
As much as I dislike Microsoft, this is an excellent development. A reduction in programming errors is always a good thing, no matter who is to credit because bad code needs fixing. However, I do wonder if this will catch too many non-bugs that are really just shitty/non-deterministic code. Too many false positives and it's worthless.
Re:Great! (Score:4, Informative)
read again. this is not about bug detection. this is about evaluating bug reports, likely raw user input or first level report, and highlighting those that might raise security concerns, which are "must fix. asap". it can probably do so in real time, so you'd expect an alarm bell going off when one of those is comes in.
Re: (Score:2)
This is Microsoft we're talking about. I doubt there will be many false positives.
You're Welcome (Score:2)
My invoice is in the mail.
The CCP will crack *you* up (Score:2)
And the state planners and computer scientists of a certain very large communist dictatorship with massive investment in AI were shocked, SHOCKED, upon learning this.