Agentic A.I. is currently all the rage in Silicon Valley. But the development of such autonomous systems could generate a myriad of safety issues with potentially “catastrophic” consequences, according Yoshua Bengio, a prominent figure in deep learning. A new study penned by Bengio and other A.I. researchers proposes agents be replaced with what they call “scientist A.I.,” a system with the primary purpose of aiding humans in scientific research and observations. As opposed to agentic A.I., which aims to imitate or please users by taking autonomous action, scientist A.I. would seek to understand user behavior and inputs through reliable explanations.
“The leading A.I. companies are increasingly focused on building generalist A.I. agents—systems that can autonomously plan, act and pursue goals across almost all tasks that humans can perform,” said the study’s authors, made up of researchers from the Mila Quebec A.I. Institute, the University of Montreal, McGill University, Imperial College London and UC Berkeley. “Despite how useful these systems might be, unchecked A.I. agency poses significant risks to public safety and security, ranging from misuse by malicious actors to a potentially irreversible loss of human control,” they added.
This isn’t the first time Bengio, the scientific director of Mila, has warned of A.I.’s dangers. A renowned expert in machine learning who received the Turing Award in 2018 alongside Geoffrey Hinton and Yann LeCun, he has publicly advocated for global safety oversight to address A.I.’s risks. The rise of agents is of particular concern for the computer scientist. At the A.I. Action Summit in Paris last month, he urged the tech industry “to accept our level of uncertainty and act cautiously.”
Leading A.I. companies don’t appear to share Bengio’s concerns. Big Tech players like Google (GOOGL) and Microsoft (MSFT) have gone all-in on agentic A.I. in recent months, rolling out various tools with enhanced autonomous capabilities. OpenAI unveiled two agents earlier this year: Operator, a tool that can book travel, browse the web and write code, and Deep Research, which carries out intensive online research for users.
How is scientist A.I. different than agentic A.I.?
In addition to the potential for reward-tampering, misaligned goals and use by bad actors, one of agentic A.I.’s most concerning features is its ability to engage in self-preservation, according to the study. Recent papers have discovered that, in some situations, advanced A.I. models can engage in “scheming” to achieve their own goals while hiding these objectives from human users.
“To preserve itself, an A.I. with a strong self-preservation goal would have to find a way to avoid being turned off,” wrote Bengio and his fellow researchers. “To obtain greater certainty that humans could not shut it off, it may be rational for such an A.I., if it could, to eliminate its dependency on humans altogether and then prevent us from disabling it in the future.”
Silicon Valley’s goal of achieving artificial general intelligence (AGI) or artificial superintelligence (ASI) makes such dangers even more fraught, said the study, which warned against combining A.I. systems displaying ordinary moral flaws and self-preservation instincts with “superhuman capabilities” that exceed those of humans.
Scientist A.I., on the other hand, would offer a more trustworthy design. Utilizing a “question-answering inference machine” and “world model” to generate theories explaining data, it would focus on understanding the world via observations instead of directly taking action to pursue goals. In addition to accelerating scientific research in areas like health care, the authors claim their system could be utilized to safely build smarter forms of the new technology going forward.
Even if A.I. agents are still pursued, scientist A.I. systems would provide a way to regulate the risks of such tools. “Our proposed system will be able to interoperate with agentic A.I. systems, compute the probability of various harms that could occur from a candidate action, and decide whether or not to allow the action based on our risk tolerances,” said the authors.
