By Izzie Gall. Photography by Shawn Poynter.
Artificial intelligence models (AIs) are gaining increased independence. Soon, sophisticated AIs will be able to contribute to collaborative projects, revolutionizing decision-making in manufacturing, nuclear energy, disaster recovery, healthcare, and more.
“AI presents a great opportunity for teaming because its inherent computational strengths and weaknesses often complement our own,” said ISE Assistant Professor Beau Schelble, who has been studying the human-AI teaming space for nearly a decade. “An effective human-AI team (HAT) should achieve outcomes that either exceed what either could accomplish alone or enable what neither could accomplish independently.”
However, compromised AI teammates also pose a new vulnerability to team dynamics by spreading misleading information, misattributing responsibility for tasks or errors, or creating confusion that undermines coordination.
Schelble is serving as the principal investigator for a cooperative agreement from the United States Army Research Office (ARO) to study how to prevent—and respond from—attacks that undermine team performance in human-AI teams. He is joined by two co-investigators, MAJ Allyson Hauptman from the United States Military Academy Army Cyber Institute and Professor Lionel Robert from the University of Michigan, Ann Arbor.
“Human-AI teams are going to be a common component of the working environment across several industries very soon,” said Yayun Tian, a PhD student in Schelble’s lab. “When AI teammates are attacked, it is critical to understand how to support HATs’ ability to identify and mitigate the attack before significant harm occurs.”
Identifying Malicious AI
Recognizing the signs of a cyberattack and knowing when it began are critical to minimizing and repairing damage. Unfortunately, human users faced with false or misleading AI output (known as “hallucinations”) generally view it as a competency issue—the result of asking a limited model for too much.
That means humans may not recognize when a compromised AI teammate’s actions are backed by intelligent, malicious intent.
An effective human-AI team (HAT) should achieve outcomes that either exceed what either could accomplish alone or enable what neither could accomplish independently.
“If a compromised AI teammate gives one person inaccurate information, that teammate will be less effective because they are working from an inaccurate model of their environment and task,” Schelble said. “A compromised AI teammate could even get human teammates to argue with one another by feeding them conflicting information.”
To investigate humans’ ability to sense when their code-based teammate is working against them, Schelble and his students programmed a team task into a commercial video game. Human participants will come to the lab and work together with one of two LLM teammates—either a helper or a saboteur.
Schelble and the team will observe the sessions to identify and define novel aspects of situation awareness, information-sharing, and shared knowledge that can be leveraged to help people on a HAT identify a compromised AI teammate, then quickly recover from and reverse any actions it has taken.
By the end of the study, Schelble and his team hope to establish the field’s first fundamental understanding and guidelines for effectively identifying, preventing, and recovering from compromised AI teammates.
“Like the master caution light on an aircraft flight deck lets you know that there’s a problem, we want to give people the tools they need to augment their team’s ability to recognize and prevent, or recover from, an attack,” Schelble said. “That way, they can see through a compromised AI and bring the team back to the ground truth as quickly and efficiently as possible.”