AI Industry Faces Tension Over Independent Reviews and Publication Rights
By Aditya Soni
Control, Independence, and Publication Rights in AI Evaluations
NEW YORK, Oct 13 (Reuters) - Anthropic and OpenAI have sought significant control over what outside evaluators can publish, concerned that findings could reveal trade secrets, according to two people familiar with the discussions.
Anthropic CEO Dario Amodei said last month that external evaluators should have the right to publish key findings "without editorial control.” OpenAI agreed days later, saying assessors should have “editorial independence.”
Concerns Over Proprietary Information
The AI companies are worried that evaluator staff working inside the companies' walls as part of so-called embedded evaluation could take proprietary information to rivals, according to the people, who declined to be named as the matter was private.
Measures under discussion to prevent intellectual-property leaks include having lab employees observe embedded teams as they work and preventing evaluators embedded at one lab from working at another simultaneously, one of the people added.
Balancing IP Protection and Risk Assessment
But what labs consider crucial intellectual property can overlap with information relevant to judging a model's risk, which can lead to arguments and redactions, said a separate person with knowledge of past dealings with the labs.
Experts have said independence of evaluators is a concern as Anthropic and OpenAI choose who gets to carry out the reviews. Reuters is reporting details of these negotiations between the evaluators and the AI labs for the first time.
Industry Tensions and Global Implications
The discussions highlight the tension between a need to make the world aware of powerful and potentially dangerous AI and demands to safeguard trade secrets by AI companies seeking trillion-dollar valuations in what could be major IPOs.
The outcome of the talks will help determine how the world scrutinizes a technology that some of its creators warn may be advancing beyond humanity's ability to manage.
Without agreed standards and legislation for independent reviews, some experts said the companies building the most powerful AI models could dictate how their systems are examined, leaving evaluators at the mercy of the labs themselves.
Negotiations Over Publication Rights
NEGOTIATIONS OVER PUBLICATION RIGHTS
Shifting Stances in Editorial Control
The nature of the discussions changed quickly. The two people with knowledge of the matter said that as recently as October 5, Anthropic and OpenAI were seeking strict editorial control over what the evaluators could publish. By October 8, talks between Anthropic and some evaluators appeared to have evolved, according to one person familiar with the matter.
In response to questions from Reuters, Anthropic said that seeking significant control over embedded evaluators' publications did not accurately characterize all of its discussions and work with independent evaluators.
The company behind the Claude AI models said that, in current conversations with evaluators it previously worked with, it had made clear that the outside reviewers control their methods, evaluation criteria, conclusions and what to publish.
Policies for New and Existing Evaluators
For new evaluator firms, Anthropic said it reserved the right to comment on and approve reports as the reviewers learn how its models operate and establish a working relationship.
OpenAI, for its part, said it was not seeking approval rights over the full text of reports produced by evaluators. It said its standard practice is to redact confidential information, and evaluators can disclose whether major redactions were made.
The ChatGPT maker said last month that AI labs and assessors should agree on the scope and access before testing, disclose methods and conflicts, and let assessors control their findings, while allowing labs to protect sensitive information.
The language AI labs already use in certain contracts with firms that evaluate their models can bar those firms from publishing confidential information in their reports without the companies' consent, according to a contract seen by Reuters.
Promises of Independence
PROMISES OF INDEPENDENCE
First Embedded Evaluators and Ongoing Negotiations
Anthropic last month said that Faculty, Accenture's specialist AI business, would be its first embedded evaluator, but said details on what evaluators can access and how they report findings were still being worked out.
“Evaluators are only useful if people can trust what they report, so independence and transparency are critical," Faculty CEO Marc Warner told Reuters.
Technology companies typically bar the firms they work with from sharing such information with others without their consent.
High Stakes for AI and Human Impact
But experts said the stakes are higher with AI, technology that, according to its developers, will have major implications for humans.
The labs could control the access evaluators depend on in the absence of standards, giving them leverage over what gets reported, Rumman Chowdhury, former US Science Envoy for AI under the Biden administration and CEO of Humane Intelligence PBC, told Reuters.
"Evaluators have this very privileged access, but whether explicitly or implicitly, they know that it comes at a cost," Chowdhury said.
White House Accord and Evaluator Demands
Independent evaluation was one of four safety measures in the one-page White House Accord on Super Intelligence, signed by President Donald Trump and leading AI executives late last month. But the accord, which Trump has called “morally binding,” did not say how those reviews should be conducted.
Evaluators also issued their own minimum conditions last month in a public letter that has been signed by more than 200 researchers, including Geoffrey Hinton, one of the godfathers of AI. The letter calls for independent control of findings, broad access to AI labs and protection from retaliation.
Redactions and Restricted Access
REDACTIONS AND RESTRICTED ACCESS
OpenAI redacted nonpublic information from a report by Model Evaluation and Threat Research, or METR, and Redwood Research on the Hugging Face incident, in which autonomous agents powered by OpenAI's advanced models went
