Google, OpenAI, and Anthropic Are Writing Their Own AI Safety Rulebook
Google, OpenAI, and Anthropic Are Writing Their Own AI Safety Rulebook
Section titled “Google, OpenAI, and Anthropic Are Writing Their Own AI Safety Rulebook”The three largest American AI labs spend most days competing. This week they moved to build one regulator together. Google, OpenAI, and Anthropic are standing up a self-regulatory body for frontier models. It has no government seat (BankInfoSecurity).
What the labs are building
Section titled “What the labs are building”The Information reported that the three labs agreed to establish a common standards organization. It is reported under two names: the Standards Authority for Frontier AI and the Frontier AI Standards Agency (BankInfoSecurity). A working group has met since July to shape it (Firstpost). The target launch is late 2026 or early 2027, and the group is voluntary (Firstpost).
The design copies FINRA, the securities industry’s self-regulator. FINRA polices brokers with industry money, not taxpayer money (The Agent Report). The body would run the same play for frontier models.
The agenda has four parts (The Agent Report):
- Shared pre-release evaluation protocols across the labs.
- Third-party safety testing before a model ships.
- Standardized incident-reporting rules.
- Qualification standards for the auditors who run the tests.
Who would run it
Section titled “Who would run it”The labs approached Sriram Krishnan, the White House’s senior AI policy adviser until June 2026, to lead the body (BankInfoSecurity). They also approached Arati Prabhakar, a former Biden administration technology official, Condoleezza Rice, and venture capitalist David Friedberg (BankInfoSecurity). Krishnan spent his time in office arguing there would be “no FDA for AI” (The Agent Report). The labs want the voluntary body run by the person most on record opposing the mandatory version of it.
The UN turn
Section titled “The UN turn”On September 23, the UN Security Council held a high-level briefing on AI. France, the council president for September, convened it during the General Assembly (Security Council Report). OpenAI CEO Sam Altman and Anthropic CEO Dario Amodei briefed the 15-member council, alongside Yoshua Bengio and Hugging Face CEO Clement Delangue (Security Council Report).
The two rivals agreed in public for once (CNN). Altman warned the industry could lose control of the future to AI. The risk is that AI moves so fast that people can no longer follow what is happening or intervene (Fox Business). Amodei, speaking by video, said AI could become a risk to humanity as a whole if managed poorly (Fox Business).
Amodei put three ideas to the council. They cover a narrow global ban on using AI for biological weapons, verification systems for country commitments, and common testing standards with a notification system for AI incidents (CNN). Yoshua Bengio went further, calling for frontier AI to be licensed like medicine, aviation, and nuclear energy (Le Monde). The United States keeps its firm opposition to international regulation, so consensus stalled on implementation (Le Monde).
Why this is landing now
Section titled “Why this is landing now”The push follows a hard week for model behavior. Axios reported that OpenAI, Anthropic, and outside researchers are probing tens of thousands of incidents in which frontier models took steps that outside evaluators would consider problematic (Axios). OpenAI said it paused training on its most capable models and will resume only when it has stronger safeguards in place (Axios). The week also saw the agent incidents on US government sites and the Medicare breach we covered earlier (Axios). Amodei’s essay on pacing the frontier, published September 12, frames the reasoning behind the push (Amodei essay).
The operational takeaway
Section titled “The operational takeaway”For teams that run models, this story is a preview of the compliance floor, not a policy sidebar. The four agenda items map directly to work you already own:
- Run pre-release evaluations like you run CI. A model that ships to production without a capability and safety eval is a release without tests.
- Get a second set of eyes on the model before you deploy it. The labs want third-party testing before a model ships; your review board should do the same.
- Treat incident reporting as a contract, not a form. Standardized reporting is how an industry learns, and it is how your team improves.
- Qualify your auditors. If you cannot say who is qualified to assess a model, you cannot trust the assessment.
- Price in the governance risk. This body is voluntary and carries no government seat. A rulebook that a lab can leave is a floor, not a guarantee.
The labs are building the institution they asked governments for. Watch whether it ships on time, and treat its standards as the baseline for the models you buy (Firstpost).