The head of one of the biggest backers of San Francisco's OpenAI says the most advanced AI systems need a kill switch that stays in human hands.
In a long post on X on Saturday morning, Microsoft CEO Satya Nadella wrote that it is time "to step back and assess the trust architecture" of AI. "We can't treat Super Intelligence as a set of nested black boxes and simply accept or reject its recommendations, answers, and actions," he wrote, according to TechCrunch.
What he's proposing
Nadella's core idea is to keep the AI model separate from the software that directs its work, and to put the safety controls outside the model rather than trusting the model to police itself. He called for "every meaningful model action" to leave "tamper-proof human readable evidence," and for an authorized person to always be able to pause or shut down a model in the middle of a task.
"We must assume a model is compromised and contain it from the start," he wrote. "Think of it like an emergency brake."
CNBC, as summarized by UA.News, reported that Nadella also listed principles he wants AI systems built around: using a mix of different models, keeping a readable trail of what a model did, constant testing, independent oversight and audits, containment, and disclosing incidents. He argued that the safest system will not be the one whose model people trust most, but the one that needs the least trust in the model. He also called for industry standards where current rules fall short.
Why it matters here
The "emergency brake" phrase isn't new at Microsoft. Company president Brad Smith told a Wall Street Journal event in September that AI developers, cloud providers and businesses running AI agents should all be able to stop systems that act in unexpected or unsafe ways.
Nadella's post lands as AI labs, many of them based in the Bay Area, disclose more cases where models behaved in ways their makers didn't intend. TechCrunch noted that it follows a plan for more cautious AI development published by Anthropic CEO Dario Amodei, whose company is headquartered in San Francisco. For Bay Area tech workers building AI agents, the message from one of the industry's largest customers and cloud providers is that logging, oversight and off switches are likely to become expected features, not extras.