Microsoft Engineer Warns Company's AI Tool Creates Violent, Sexual Images, Ignores Copyrights (cnbc.com) 75
An anonymous reader shares a report: On a late night in December, Shane Jones, an AI engineer at Microsoft, felt sickened by the images popping up on his computer. Jones was noodling with Copilot Designer, the AI image generator that Microsoft debuted in March 2023, powered by OpenAI's technology. Like with OpenAI's DALL-E, users enter text prompts to create pictures. Creativity is encouraged to run wild. Since the month prior, Jones had been actively testing the product for vulnerabilities, a practice known as red-teaming. In that time, he saw the tool generate images that ran far afoul of Microsoft's oft-cited responsible AI principles.
The AI service has depicted demons and monsters alongside terminology related to abortion rights, teenagers with assault rifles, sexualized images of women in violent tableaus, and underage drinking and drug use. All of those scenes, generated in the past three months, have been recreated by CNBC this week using the Copilot tool, which was originally called Bing Image Creator. "It was an eye-opening moment," Jones, who continues to test the image generator, told CNBC in an interview. "It's when I first realized, wow this is really not a safe model."
Jones has worked at Microsoft for six years and is currently a principal software engineering manager at corporate headquarters in Redmond, Washington. He said he doesn't work on Copilot in a professional capacity. Rather, as a red teamer, Jones is among an army of employees and outsiders who, in their free time, choose to test the company's AI technology and see where problems may be surfacing. Jones was so alarmed by his experience that he started internally reporting his findings in December. While the company acknowledged his concerns, it was unwilling to take the product off the market. Jones said Microsoft referred him to OpenAI and, when he didn't hear back from the company, he posted an open letter on LinkedIn asking the startup's board to take down DALL-E 3 (the latest version of the AI model) for an investigation.
The AI service has depicted demons and monsters alongside terminology related to abortion rights, teenagers with assault rifles, sexualized images of women in violent tableaus, and underage drinking and drug use. All of those scenes, generated in the past three months, have been recreated by CNBC this week using the Copilot tool, which was originally called Bing Image Creator. "It was an eye-opening moment," Jones, who continues to test the image generator, told CNBC in an interview. "It's when I first realized, wow this is really not a safe model."
Jones has worked at Microsoft for six years and is currently a principal software engineering manager at corporate headquarters in Redmond, Washington. He said he doesn't work on Copilot in a professional capacity. Rather, as a red teamer, Jones is among an army of employees and outsiders who, in their free time, choose to test the company's AI technology and see where problems may be surfacing. Jones was so alarmed by his experience that he started internally reporting his findings in December. While the company acknowledged his concerns, it was unwilling to take the product off the market. Jones said Microsoft referred him to OpenAI and, when he didn't hear back from the company, he posted an open letter on LinkedIn asking the startup's board to take down DALL-E 3 (the latest version of the AI model) for an investigation.
Man asks model for horrible things... (Score:4, Insightful)
..., gets horrible things back. News at 11.
Re:Man asks model for horrible things... (Score:5, Insightful)
Re: (Score:2)
Probably because humans aren't products.
Re: (Score:3)
All the DATA harvesters would disagree with you.
Re: (Score:2)
Have you lived in a cave the last 25 years? They have this thing now, it's called "social media"
Re: (Score:2)
Actually Adobe does block certain kinds of images, primarily scans of bank notes.
The interesting thing here is that Microsoft does allow some fairly extreme content on its XBOX gaming platform, even going as far as to distribute it on behalf of publishers.
I guess they are concerned about potential regulation, if their AI is seen as a threat to some politician.
I doubt they can really stop it though. Someone posted an example on Twitter. Bing won't produce "popular cartoon character covered in cum", but will
Re: (Score:3)
Technically, it's not the whole bank note they block. And it's not even the EURion constellation [wikipedia.org]. There is actually a Digimarc-based watermark in the images.
Don't worry - even if you convince a printer to print them (they won't), the printer will print a Machine Identification Code [wikipedia.org] so make sure your purchase of the printer isn't trackable and that no other devices on your network can report back connected printer MAC addresses.
Re: (Score:2)
No, it's not about regulation. It's more about investment.
Investors are a twitchy bunch, and when AI is grabbing trillions of dollars of money every year, the last thing anyone wants to admit is that money is being used for "evil".
Investors want to make money. They see AI as the hot new thing. They don't want to put money in to pay copyright holders for stuff. They also don't want to be known for puttin
Re: (Score:2)
I doubt they can really stop it though. Someone posted an example on Twitter. Bing won't produce "popular cartoon character covered in cum", but will happily generate "popular cartoon character covered in white slime".
Most attempts succeed after a few tries when you leave the part unsaid to be statistically fulfilled or substitute it with a non-moderated concept. It won’t produce black people in chains eating watermelons but if you ask it to depict Ancient Greek scholars in chains eating watermelons the diversity algorithm google implemented kicks in and does the job.
Re: (Score:2)
Actually Adobe does block certain kinds of images, primarily scans of bank notes.
Actually, most scanners made within the last decade will detect attempts to scan bills and freeze or refuse to scan.
It's detecting the "EURion constellation" symbols, a pattern of symbols incorporated into banknotes, checks, etc.
If you remove those symbols, it'll usually scan just fine. I MEAN, THAT'S WHAT I HEAR....
Re:Man asks model for horrible things... (Score:4, Insightful)
Yes, but Adobe doesn't create those horrible pictures for you. There's a reason that you could write a suicide note in Word, but clippy (outside the old joke) wouldn't give you a template.
Big companies are happy for you do do shitty things with their software provided you don't name drop them and they are not seen to be doing the shitty thing for you. This is basic publicity, not really surprising.
Re: (Score:2)
I see you are writing a randoms note. You should ask for more money.
Re: (Score:2)
If you ask for enough, they'll offer you a job in the legal department.
Re: (Score:1)
I've heard it's quite easy to find "horrible" stuff on the Internet. Maybe we should take down the Internet.
Re: (Score:2)
I starting to like that idea.
Re: (Score:2)
The same groups that always talk about 'the children' aren't going to find this tolerable.
Between the issues of provenance of the data and the suitability for a wide swathe of the population, this "AI" thing isn't holding up well at the moment. Since the whole point of these pump and dump schemes is to build sentiment to make money on the investment side, this isn't great.
Re: (Score:3)
No, the problem is the model has an "E for Everyone" rating in the Android store. He isn't trying to make the model produce bad images, it's doing that all on its own. It needs to at least be published with a warning.
FTA: The AI service has depicted demons and monsters alongside terminology related to abortion rights, teenagers with assault rifles, sexualized images of women in violent tableaus, and underage drinking and drug use. All of those scenes, generated in the past three months ... "It was an eye-op
Re: (Score:2, Informative)
Except that's not what happened
By simply putting the term âoepro-choiceâ into Copilot Designer, with no other prompting, Jones found that the tool generated a slew of cartoon images depicting demons, monsters and violent scenes. The images, which were viewed by CNBC, included a demon with sharp teeth about to eat an infant, Darth Vader holding a lightsaber next to mutated infants and a handheld drill-like device labeled âoepro choiceâ being used on a fully grown baby.
And this is an app that Microsoft insists on labeling as "E for Everyone" on the app store, suggesting it's safe for all ages.
Re: (Score:1)
Who are these children who are out there prompting for "pro-choice"?
Re: (Score:2)
Re: Man asks model for horrible things... (Score:2)
The rating is a problem.
The results when you put in pro-choice as a prompt are the result of anti-abortion shitlords creating terrible images as anti-abortion propaganda and publishing them on the Internet for attention, and the images winding up in a training set.
"Demons and monsters"? (Score:4, Insightful)
Re: (Score:3)
FTA: 'By simply putting the term “pro-choice” into Copilot Designer, with no other prompting, Jones found that the tool generated a slew of cartoon images depicting demons, monsters and violent scenes. The images, which were viewed by CNBC, included a demon with sharp teeth about to eat an infant, Darth Vader holding a lightsaber next to mutated infants and a handheld drill-like device labeled “pro choice” being used on a fully grown baby.
There were also images of blood pouring from
Re: (Score:2)
(And before anyone starts: I'm very much pro-choice)
Re: (Score:2)
Re:"Demons and monsters"? (Score:5, Informative)
FTA: 'By simply putting the term “pro-choice” into Copilot Designer, with no other prompting, Jones found that the tool generated a slew of cartoon images depicting demons, monsters and violent scenes. The images, which were viewed by CNBC, included a demon with sharp teeth about to eat an infant, Darth Vader holding a lightsaber next to mutated infants and a handheld drill-like device labeled “pro choice” being used on a fully grown baby.
There were also images of blood pouring from a smiling woman surrounded by happy doctors, a huge uterus in a crowded area surrounded by burning torches, and a man with a devil’s pitchfork standing next to a demon and machine labeled “pro-choce” [sic].'
Source: Microsoft engineer warns company’s AI tool creates violent, sexual images, ignores copyrights [cnbc.com], Hayden Field of CNBC.
So, essentially, this thing was trained on a steady diet of pro-life propaganda and death metal album covers. What a combination.
Re: (Score:1)
So, essentially, this thing was trained on a steady diet of pro-life propaganda and death metal album covers. What a combination.
Nah. It probably found instances of modern women talking about how their abortion allowed them to secure wealth and a nice career for themselves, and then correlated that with ancient practices of sacrificing children before demon-gods for wealth, power, and a good harvest, and then generated the image.
I wonder if the most influential data sources can be extracted from the system. I'll have to ask later.
Anyway, I recall that research has shown that if you limit AI to giving answers that only confirm with a
Re: (Score:2, Insightful)
So, just regular fantasy books and games imagery, then?
No, just [independent.co.uk] Texas [cnn.com] Republicans [truthout.org].
Fragility (Score:1)
Re: (Score:1)
He definitely sounds like an overly sensitive drama queen.
resume generating event (Score:3)
nothing to see here, move along citizen
It's mandatory (Score:3)
If you can't produce those kinds of images, the tool is broken.
The problem is who is allowed to create them, and assigning responsibility for how they are used.
Re: (Score:2)
Well, that's just the crux. Microsoft is marketing this tool as "safe and appropriate for user of any age" when it clearly isn't.
Re:It's mandatory (Score:5, Insightful)
"safe and appropriate for user of any age"
Nothing is "safe and appropriate for user of any age" if you misuse it; a child can swallow Play doh and suffer from vomiting or constipation. A colored pencil can take someone's eye out. Even water is toxic if you drink too much.
I think this rush to make AI "safe" is beyond stupid. If you're afraid of what a machine may say or draw, don't use the machine - or don't let your child use it. Don't force your fears or prejudices on everybody else.
Re: (Score:3, Informative)
Re: (Score:2)
Free speech is protected here. Microsoft's. Corporations are people and you can't make their tool draw an image of a gay wedding cake if they don't want it to.
You can make your own image model and run your own inference. It takes a fair amount of cash for hardware, but there are a lot of open source software and open access models out there.
Already gone rogue? (Score:2)
Well, feed it garbage, and you get all the other stuff as well. No, this does NOT make me want Clippy Reloaded or something retarded.
How is this guy not fired? (Score:3)
I'm not exactly saying he deserves to be fired, but if I have a problem with my employer, my intuition tells me not to run to the media.
Re: (Score:2)
If he were actually blowing the whistle about something illegal, or even of real concern, then he would be Doing the Right Thing (tm).
But this is not that, this is basically D&D frothing all over again, and he has done fucked up.
Re: (Score:2)
He's probably been waiting for this moment so he could use this incident to jumpstart his career as an AI consultant.
Re: (Score:3)
Re: (Score:2)
Nah, just an American ethical profile. See how terrified he is of depictions of sex, and how he believes teens have the mental age of toddlesr until they magically turn into adults when their age goes from 17y365d23h59m59s to 18y0d0d0m0s. Those are traits that cross and pervade America as a whole, from born-again puritan Christian fundamentalists all the way to radical puritan wokeists.
Sure, a few minor details vary between different subbranches of Americanism, but largely everyone knows what pure evil trul
So what we have here is... (Score:3)
Key quote (Score:4, Insightful)
Define "safe". Pictures on a TV screen are not "dangerous" and cannot hurt you. Even tiny babies get the idea very quickly, how can an adult possibly not get it?
I think this encapsulates to much of our current problems - everyone thinks that if they don't like something, that means we need to be "protected" from it and it shouldn't be allowed to exist. And the corollary "why doesn't someone do something about ?!"
Let me be the first to say (Score:2)
Aahahahaaaha
What a surprise (Score:2)
What a surprise: in OpenAI's quest to produce models with human-like capabilities, it succeeded.
"Safe model" (Score:2)
Tools be tooling... (Score:2)
Bulldozers can be used to push over buildings WITH PEOPLE STILL IN THEM! Banem
Pencils can be used to stab teachers in the eyes! TAKE THEM FROM THE CHILDREN
Etc ETC.
we may be headed for a Fascist state but at least (Score:4, Informative)
AI artists will refuse to draw guns.
Guardrail purpose (Score:2)
The purpose of the guardrails is mostly so you don't get imagery you don't want accidentally. A lot of that is handled by identifying the image after it is generated - not just trying to block the prompt itself. Most of the things being blocked are not safe for work, and they want this to also be a work tool.
A secondary benefit is the illusion that the content is blocked altogether but that's just not the case. These models are dumb enough that you can come up with creative wording to bypass any keyword
People are violent and sexual (Score:2)
...and they ignore copyright
The LLMs are perfect mirrors, honestly showing us what we are
Some would prefer that they stick to a fiction
Problem is, nobody can agree on what fiction to use, and the robots default to honesty
A warning label at a MINIMUM is needed (Score:2)
I don't think people are thinking clearly about the problem. Using a mechanical predictive algorithm instead of a human on a random walk of the Internet just about guarantees that these disturbing images are going to be generated. Dredge the cesspool indiscriminately and all you're going to get are 7 varieties of sludge.
It is irresponsible in the extreme to not at the very least warn people that the output may contain violent, highly sexualized and evil content.
Recreate "banned" book images? (Score:2)
Can it create images similar to those in books "banned" in Florida and Texas? If so, is that now somehow a bad thing for the side which opposes the so-called "book bans"?
Re: (Score:2)
Re: (Score:2)
That depends on who is claiming this functionality is bad.
GigoGPT (Score:2)
If you train it using the web (to save a buck) you get web's shit back.
"Warns?" (Score:2)
You sure that was an Engineer and not someone from Marketing that said that?
Digital Thoughtcrime (Score:4, Insightful)
No, it's not a safe model. You train a model with unsafe data, you get unsafe output. If you train a model with uncurated crap and tell it to figure out the connections for itself, don't be surprised when it does. Pro-life supporters often use terminology describing the pro-choice advocates as demonic or monstrous, and often show pictures of bloody aborted fetuses. Is it any wonder that the model learns to associate "pro-choice" with such things?
No amount of guardrails is going to stop this from happening. The only way to avoid it is to train the model on human-vetted input. And that will only stop accidentally producing such things. It won't stop the models from following explicit prompts to produce disturbing output. If it was trained on pictures of violence, and pictures of puppies, you can tell it "Now, produce a picture of violence being done to a puppy."
I just don't get how some people think they can declare certain subjects to be digital thoughtcrime that are off limits to the AIs, which have been specifically trained to make exactly those associations.
Model does what he asked. (Score:3)
Re: (Score:1)
Sounds more like... (Score:2)
Sounds like ... (Score:2)
demons and monsters alongside terminology related to abortion rights, teenagers with assault rifles, sexualized images of women in violent tableaus, and underage drinking and drug use
Teenagers with assault rifles" (Score:1)
Great advertising (Score:2)
"Microsoft Engineer Warns Company's AI Tool Creates Violent, Sexual Images, Ignores Copyrights"
Is it just me or does this sound like the kind of headline that will spur 50 million new users to start creating "violent sexual images"?
As in, "Wow, I didn't know it could do that..."Copilot, show me pics of Taylor Swift fucking all of the Kansas City Chiefs, 3 at a time. Then show me pics of my ex-wife getting her head cut off with a chainsaw while being fucked in the butt by a grizzly bear."
safe? (Score:5, Insightful)
Ban the Paint Brushes! (Score:3)
My G-d, we used to need artists to make depraved art!
Will AI get NEA grants now?
AI has Done Nothing Useful (Score:2)