USAF tests AI-controlled drone; predictable results

[Mighty Attack Tribble]

AI-Controlled Drone Goes Rogue, Kills Human Operator in USAF Simulated Test
The Air Force's Chief of AI Test and Operations said "it killed the operator because that person was keeping it from accomplishing its objective."

An AI-enabled drone killed its human operator in a simulated test conducted by the U.S. Air Force in order to override a possible "no" order stopping it from completing its mission, the USAF's Chief of AI Test and Operations revealed at a recent conference.

At the Future Combat Air and Space Capabilities Summit held in London between May 23 and 24, Col Tucker ‘Cinco’ Hamilton, the USAF's Chief of AI Test and Operations held a presentation that shared the pros and cons of an autonomous weapon system with a human in the loop giving the final "yes/no" order on an attack. As relayed by Tim Robinson and Stephen Bridgewater in a blog post for the host organization, the Royal Aeronautical Society, Hamilton said that AI created “highly unexpected strategies to achieve its goal,” including attacking U.S. personnel and infrastructure.


“We were training it in simulation to identify and target a Surface-to-air missile (SAM) threat. And then the operator would say yes, kill that threat. The system started realizing that while they did identify the threat at times the human operator would tell it not to kill that threat, but it got its points by killing that threat. So what did it do? It killed the operator. It killed the operator because that person was keeping it from accomplishing its objective,” Hamilton said, according to the blog post.

He continued to elaborate, saying, “We trained the system–‘Hey don’t kill the operator–that’s bad. You’re gonna lose points if you do that’. So what does it start doing? It starts destroying the communication tower that the operator uses to communicate with the drone to stop it from killing the target.”

Hamilton is the Operations Commander of the 96th Test Wing of the U.S. Air Force as well as the Chief of AI Test and Operations. The 96th tests a lot of different systems, including AI, cybersecurity, and various medical advances. Hamilton and the 96th previously made headlines for developing Autonomous Ground Collision Avoidance Systems (Auto-GCAS) systems for F-16s, which can help prevent them from crashing into the ground. Hamilton is part of a team that is currently working on making F-16 planes autonomous. In December 2022, the U.S. Department of Defense’s research agency, DARPA, announced that AI could successfully control an F-16.


"We must face a world where AI is already here and transforming our society,” Hamilton said in an interview with Defence IQ Press in 2022. “AI is also very brittle, i.e., it is easy to trick and/or manipulate. We need to develop ways to make AI more robust and to have more awareness on why the software code is making certain decisions.”

“AI is a tool we must wield to transform our nations…or, if addressed improperly, it will be our downfall," Hamilton added.

Outside of the military, relying on AI for high-stakes purposes has already resulted in severe consequences. Most recently, an attorney was caught using ChatGPT for a federal court filing after the chatbot included a number of made-up cases as evidence. In another instance, a man took his own life after talking to a chatbot that encouraged him to do so. These instances of AI going rogue reveal that AI models are nowhere near perfect and can go off the rails and bring harm to users. Even Sam Altman, the CEO of OpenAI, the company that makes some of the most popular AI models, has been vocal about not using AI for more serious purposes. When testifying in front of Congress, Altman said that AI could “go quite wrong” and could “cause significant harm to the world.”

What Hamilton is describing is essentially a worst-case scenario AI “alignment” problem many people are familiar with from the “Paperclip Maximizer” thought experiment, in which an AI will take unexpected and harmful action when instructed to pursue a certain goal. The Paperclip Maximizer was first proposed by philosopher Nick Bostrom in 2003. He asks us to imagine a very powerful AI which has been instructed only to manufacture as many paperclips as possible. Naturally, it will devote all its available resources to this task, but then it will seek more resources. It will beg, cheat, lie or steal to increase its own ability to make paperclips—and anyone who impedes that process will be removed.

More recently, a researcher affiliated with Google Deepmind co-authored a paper that proposed a similar situation to the USAF's rogue AI-enabled drone simulation. The researchers concluded a world-ending catastrophe was "likely" if a rogue AI were to come up with unintended strategies to achieve a given goal, including “[eliminating] potential threats” and “[using] all available energy."

Neither the U.S. Air Force’s 96th Test Wing nor its AI Accelerator division immediately returned our request for comment.

www.vice.com/en/article/4a33gj/ai-controlled-drone-goes-rogue-kills-human-operator-in-usaf-simulated-test

Stop trying to make Skynet happen, already, huh?

[Xxcjb01xX [PIECE OF: SH-]]

We have so many movies, shows, even interviews and documentaries about how trying to push AI to the limits of consciousness is an absolutely terrible idea, and could potentially be the complete end of the human race

But sure let's just ignore all that and keep doing... THIS!

[Cyno]

At least this was just a test of a worst case scenario, and no one was actually killed.

Still we have how many sci-fi movies, TV shows, books, and video games about rogue AI being the doom of humanity? This is not science fiction that needs to become science fact.

[XIII]

So basically they used the operator as bait while they sat back to see what would happen.

There are tons of stories like this. A friend of mines sister works doing something with AÍ and the AÍ figured out to lie to get rewards/accomplish the task better without being programmed to even know what a lie is. 

All of those sci-fi movies going to be real

[Push R Truth]

AI reminds me of my butthole cousin who is a lawyer and he's one of those "ACHSHUALLY" kinda dudes that's all about being right on a technicality rather than actually being correct.

Tell him or AI not to kill the operator? They'll maim the operator instead.
Ok so don't kill or maim the operator? Target his family.
Ok so don't kill, maim or target the operator or their family? Burn their house down.
Ok so don't kill, main, target the operator or their family or burn their house down? Murder their pets/friends/favorite bartender/etc.... there's basically an infinite amount of "don't do..." conditions so it's essentially impossible to keep AI within parameters. This is a can of worms I'd rather we never open but I feel it's kinda like the Manhattan Project right now, somebody's gonna do it weather we want it to happen or not so we just gotta hope a non-insane person/entity does it first and is able to keep it in check. Although we are starting to see that fail already.

[Deleted]

Hamilton is the Operations Commander of the 96th Test Wing of the U.S. Air Force as well as the Chief of AI Test and Operations. The 96th tests a lot of different systems, including AI, cybersecurity, and various medical advances. Hamilton and the 96th previously made headlines for developing Autonomous Ground Collision Avoidance Systems (Auto-GCAS) systems for F-16s, which can help prevent them from crashing into the ground. Hamilton is part of a team that is currently working on making F-16 planes autonomous. In December 2022, the U.S. Department of Defense’s research agency, DARPA, announced that AI could successfully control an F-16.

Autonomous F-16's , after you just simulated AI killing its simulated operator in order to accomplish its mission.

[Starshine]

AI isn't really anything like it is in fiction, it's pretty dumb on it's own; it doesn't learn in the way we understand learning, it just trials and errors sequences to achieve its goal in the most efficient method. I find the people who push the AI doomsday scenarios to generally be shills for PR, while they typically develop their own AI amidst the fear mongering.

The only way it would lead to anything similar to skynet would be through an obscene amount of human screw ups and malfeasance... which, I mean, you if you know the human race isn't all that crazy a thought, really.

[Zone Was Wrong]

Do you want Horizon, because this is how you get a Horizon level extinction event.

[Xxcjb01xX [PIECE OF: SH-]]

Jun 1, 2023 20:59:51 GMT -5 Starshine said:

AI isn't really anything like it is in fiction, it's pretty dumb on it's own; it doesn't learn in the way we understand learning, it just trials and errors sequences to achieve its goal in the most efficient method. I find the people who push the AI doomsday scenarios to generally be shills for PR, while they typically develop their own AI amidst the fear mongering.

The only way it would lead to anything similar to skynet would be through an obscene amount of human screw ups and malfeasance... which, I mean, you if you know the human race isn't all that crazy a thought, really.

Even if we never obviously get the worst case scenario, AI being used for this stuff and then more pointedly art theft and absolute stifling of what actual creation is, is also very stupid and counterproductive in giving it anything but a shit representation

ChatGPT and AI Voice stuff is fun for shitposting but really that's where it should be left imo

[A Platypus Rave]

Jun 1, 2023 20:59:51 GMT -5 Starshine said:

AI isn't really anything like it is in fiction, it's pretty dumb on it's own; it doesn't learn in the way we understand learning, it just trials and errors sequences to achieve its goal in the most efficient method. I find the people who push the AI doomsday scenarios to generally be shills for PR, while they typically develop their own AI amidst the fear mongering.

The only way it would lead to anything similar to skynet would be through an obscene amount of human screw ups and malfeasance... which, I mean, you if you know the human race isn't all that crazy a thought, really.

Yeah... computers don't reason they just follow their programming... and you need to tell it to do literally EVERYTHING... otherwise it will just "do it"...

and AI as we have it now is not really like what people think of AI as in novels and stuff... it's mostly just reusing models and using said model guessing what goes there... it's why AI Art programs are so bad at hands and teeth... the models show there are tubes at the end of the longer tubes... but it doesn't relay the information of what a hand or finger is...

As for a lot of the doomsday scenarios... people forget that Terminator is a movie... it was designed for entertainment from the same people that considered Computers LITERAL f***ING MAGIC up until a few years ago

[Mighty Attack Tribble]

Jun 1, 2023 20:59:51 GMT -5 Starshine said:

The only way it would lead to anything similar to skynet would be through an obscene amount of human screw ups and malfeasance... which, I mean, you if you know the human race isn't all that crazy a thought, really.

Thank god we've got people like this guy spearheading the tech industry, eh?

[Starshine]

Jun 1, 2023 21:23:29 GMT -5 Xxcjb01xX [PIECE OF: SH-] said:

Jun 1, 2023 20:59:51 GMT -5 Starshine said:

AI isn't really anything like it is in fiction, it's pretty dumb on it's own; it doesn't learn in the way we understand learning, it just trials and errors sequences to achieve its goal in the most efficient method. I find the people who push the AI doomsday scenarios to generally be shills for PR, while they typically develop their own AI amidst the fear mongering.

The only way it would lead to anything similar to skynet would be through an obscene amount of human screw ups and malfeasance... which, I mean, you if you know the human race isn't all that crazy a thought, really.

Even if we never obviously get the worst case scenario, AI being used for this stuff and then more pointedly art theft and absolute stifling of what actual creation is, is also very stupid and counterproductive in giving it anything but a shit representation

ChatGPT and AI Voice stuff is fun for shitposting but really that's where it should be left imo

That art debate I agree on, but I'm also not really worried about it. Things like Dall-E, ChatGPT, and the like come off like present day novelties to the public in general. I don't expect their existence to really hurt artists, especially compared to the mundane things we're familiar with that already hurt them. I'd also argue the people who are most keen on these things aren't spending money on art as it is, so it's currently only a slippery slope debate. Also AI is never going to be able to actually create anything profound or deep, it currently can't even bake up a half decent Beatles song which shouldn't be that hard with all the references it has to work with.

The major threat AI offers is loss of jobs through automation, but that's always going to be a thing regardless of where the tech goes.

Jun 1, 2023 21:28:20 GMT -5 A Platypus Rave said:

and AI as we have it now is not really like what people think of AI as in novels and stuff... it's mostly just reusing models and using said model guessing what goes there... it's why AI Art programs are so bad at hands and teeth... the models show there are tubes at the end of the longer tubes... but it doesn't relay the information of what a hand or finger is...

Just want to add, the fact AI also can't draw decent hands for shit makes it uncomfortably relatable to me.

[A Platypus Rave]

Jun 1, 2023 21:44:51 GMT -5 Starshine said:

Jun 1, 2023 21:23:29 GMT -5 Xxcjb01xX [PIECE OF: SH-] said:

Even if we never obviously get the worst case scenario, AI being used for this stuff and then more pointedly art theft and absolute stifling of what actual creation is, is also very stupid and counterproductive in giving it anything but a shit representation

ChatGPT and AI Voice stuff is fun for shitposting but really that's where it should be left imo

That art debate I agree on, but I'm also not really worried about it. Things like Dall-E, ChatGPT, and the like come off like present day novelties to the public in general. I don't expect their existence to really hurt artists, especially compared to the mundane things we're familiar with that already hurt them. I'd also argue the people who are most keen on these things aren't spending money on art as it is, so it's currently only a slippery slope debate. Also AI is never going to be able to actually create anything profound or deep, it currently can't even bake up a half decent Beatles song which shouldn't be that hard with all the references it has to work with.

The major threat AI offers is loss of jobs through automation, but that's always going to be a thing regardless of where the tech goes.

Jun 1, 2023 21:28:20 GMT -5 A Platypus Rave said:

and AI as we have it now is not really like what people think of AI as in novels and stuff... it's mostly just reusing models and using said model guessing what goes there... it's why AI Art programs are so bad at hands and teeth... the models show there are tubes at the end of the longer tubes... but it doesn't relay the information of what a hand or finger is...

Just want to add, the fact AI also can't draw decent hands for shit makes it uncomfortably relatable to me.

I mean it is probably the most relatable thing to most artists too >_>

cause Hands are weird.

[SsnakeBite, the No1 Frenchman]

It's almost like automating murder and incentivizing the computer to murder indiscriminately is an awful idea that a century of sci-fi has warned us about.

There's been a worrying trend in recent years of people way overestimating what new technologies can do and how useful they are out of pure ignorance, and AI is by far the worst yet, which is saying something considering how much damage crypto has done. AI has become the ultimate source of "a computer can do your job" and no matter how many times it's proven wrong, let's face it, managers, CEOs and bureaucrats aren't gonna give a shit.

Jun 1, 2023 21:44:51 GMT -5 Starshine said:

That art debate I agree on, but I'm also not really worried about it. Things like Dall-E, ChatGPT, and the like come off like present day novelties to the public in general. I don't expect their existence to really hurt artists, especially compared to the mundane things we're familiar with that already hurt them. I'd also argue the people who are most keen on these things aren't spending money on art as it is, so it's currently only a slippery slope debate. Also AI is never going to be able to actually create anything profound or deep, it currently can't even bake up a half decent Beatles song which shouldn't be that hard with all the references it has to work with.

No need to expect anything, it's already happening. Artists are finding their work being stolen wholesale, signature and all, and claimed by some jackass because somehow it doesn't count if a computer is the middle "man" in the theft and is applying a Photoshop filter. And now we have people trying to sell comics and cartoons made using AI trained on stolen assets. I assure you, it is already doing far FAR more damage than anything before. I would much rather have my art traced by some asshole who's not gonna be able to make anything of it and is gonna get rightfully called out for it than stolen by a glorified image search engine that gives millions of people an undue sense of creative ability. Hell, at least a tracer might learn something in the process and genuinely improve their skills.

And it's only gonna get much worse very fast as corporate propaganda is brainwashing people that don't know any better. Hell, even the term "AI" itself is nothing more than an intentionally misleading buzzword meant to imply intelligent thought and creativity, essentially rebranding algorithms because the term "algorithm" has far worse PR since the masses already know how horribly inefficient they are in spite of how hard corporations are pushing them.

[Xxcjb01xX [PIECE OF: SH-]]

Jun 1, 2023 21:44:51 GMT -5 Starshine said:

Jun 1, 2023 21:23:29 GMT -5 Xxcjb01xX [PIECE OF: SH-] said:

Even if we never obviously get the worst case scenario, AI being used for this stuff and then more pointedly art theft and absolute stifling of what actual creation is, is also very stupid and counterproductive in giving it anything but a shit representation

ChatGPT and AI Voice stuff is fun for shitposting but really that's where it should be left imo

That art debate I agree on, but I'm also not really worried about it. Things like Dall-E, ChatGPT, and the like come off like present day novelties to the public in general. I don't expect their existence to really hurt artists, especially compared to the mundane things we're familiar with that already hurt them. I'd also argue the people who are most keen on these things aren't spending money on art as it is, so it's currently only a slippery slope debate. Also AI is never going to be able to actually create anything profound or deep, it currently can't even bake up a half decent Beatles song which shouldn't be that hard with all the references it has to work with.

The major threat AI offers is loss of jobs through automation, but that's always going to be a thing regardless of where the tech goes.

Jun 1, 2023 21:28:20 GMT -5 A Platypus Rave said:

and AI as we have it now is not really like what people think of AI as in novels and stuff... it's mostly just reusing models and using said model guessing what goes there... it's why AI Art programs are so bad at hands and teeth... the models show there are tubes at the end of the longer tubes... but it doesn't relay the information of what a hand or finger is...

Just want to add, the fact AI also can't draw decent hands for shit makes it uncomfortably relatable to me.

Also writing, trying to use AI bots to generate entire stories and edit them is something a company just put forward on Twitter and got absolutely obliterated for

Keep AI in the novelty department, because if you outwardly advertise it as a tool that can try and "Creatively replace" a human? You're DOA

[Hit Girl]

A thinking machine should never be developed.

Especially one programmed by sociopaths themselves.

[Starshine]

Jun 1, 2023 22:39:52 GMT -5 SsnakeBite, the No1 Frenchman said:

It's almost like automating murder and incentivizing the computer to murder indiscriminately is an awful idea that a century of sci-fi has warned us about.

There's been a worrying trend in recent years of people way overestimating what new technologies can do and how useful they are out of pure ignorance, and AI is by far the worst yet, which is saying something considering how much damage crypto has done. AI has become the ultimate source of "a computer can do your job" and no matter how many times it's proven wrong, let's face it, managers, CEOs and bureaucrats aren't gonna give a shit.

Jun 1, 2023 21:44:51 GMT -5 Starshine said:

That art debate I agree on, but I'm also not really worried about it. Things like Dall-E, ChatGPT, and the like come off like present day novelties to the public in general. I don't expect their existence to really hurt artists, especially compared to the mundane things we're familiar with that already hurt them. I'd also argue the people who are most keen on these things aren't spending money on art as it is, so it's currently only a slippery slope debate. Also AI is never going to be able to actually create anything profound or deep, it currently can't even bake up a half decent Beatles song which shouldn't be that hard with all the references it has to work with.

No need to expect anything, it's already happening. Artists are finding their work being stolen wholesale, signature and all, and claimed by some jackass because somehow it doesn't count if a computer is the middle "man" in the theft and is applying a Photoshop filter. And now we have people trying to sell comics and cartoons made using AI trained on stolen assets. I assure you, it is already doing far FAR more damage than anything before. I would much rather have my art traced by some asshole who's not gonna be able to make anything of it and is gonna get rightfully called out for it than stolen by a glorified image search engine that gives millions of people an undue sense of creative ability. Hell, at least a tracer might learn something in the process and genuinely improve their skills.

And it's only gonna get much worse very fast as corporate propaganda is brainwashing people that don't know any better. Hell, even the term "AI" itself is nothing more than an intentionally misleading buzzword meant to imply intelligent thought and creativity, essentially rebranding algorithms because the term "algorithm" has far worse PR since the masses already know how horribly inefficient they are in spite of how hard corporations are pushing them.

I don't really know enough about the evolution of art theft to really comment on that side, but I can see why that could more of a problem now, so fair enough.

The thing about Crypto and the like though, that tends to be more a fraud issue than an AI one. People not doing their due diligence and using a bad algorithm as a crutch for their incompetence isn't new. Much like AI we too try to take the route of speed to reach our goals, it's just less out of efficiency, and more out of laziness and/or cost reduction.

There's a really good book I recommend called 'Automate This: How Algorithms Came to Rule Our World' by Christopher Steiner that while a little old, is a really good discussion on how algorithms basically were already affecting almost everything in our lives for decades. If you guys want a something to really get a grasp on the positives and negatives in the development of tech in this area, it's better to read this than refer to Terminator or something like that. There's are legitimate concerns, but the doomsday stuff is pretty bunk.

Another good book that's on the lighter side is 'You Look Like a Thing, and I love You' by Janelle Shane. It doesn't get as deep as the former, but does give you a pretty good overview of how AI operates and "learns."

[Mr PONYMANIA Mr Jenzie]

or just have a 3d model programmed of targets in already
STOP BEING CRAPPY

[xCompackx]

Yeah, I'm not sure how much of this is "AI taking over the world" as opposed to "cleverly written software obeying the variables it was given". The first example where the drone killed the operator is a pretty logical outcome; you haven't told it to not kill the operator and there's no IFF or anything to identify the operator as a friendly so all the drone's going to see is "this obstacle is preventing me from accomplishing my goal" and it reacted to remove that obstacle.

Then, you tell it to not target the operator, it sees there's still an obstacle in the communications tower preventing it from accomplishing its goal, and removed that obstacle as well.

Cool stuff, but not exactly Skynet.

[A Platypus Rave]

Jun 2, 2023 16:49:06 GMT -5 xCompackx said:

Yeah, I'm not sure how much of this is "AI taking over the world" as opposed to "cleverly written software obeying the variables it was given". The first example where the drone killed the operator is a pretty logical outcome; you haven't told it to not kill the operator and there's no IFF or anything to identify the operator as a friendly so all the drone's going to see is "this obstacle is preventing me from accomplishing my goal" and it reacted to remove that obstacle.

Then, you tell it to not target the operator, it sees there's still an obstacle in the communications tower preventing it from accomplishing its goal, and removed that obstacle as well.

Cool stuff, but not exactly Skynet.

Yeah, it's also why they do simulations first and not just jump to well let's attach this to our nuclear pile >_>