The U.S. government’s process for evaluating the safety of frontier artificial intelligence models, such as those released by OpenAI and Anthropic, remains largely opaque, leaving significant questions regarding the specific criteria used to greenlight high-capability systems. While federal agencies have established voluntary frameworks and collaborative testing agreements, the precise nature of the internal dialogues and the specific safety benchmarks required for public deployment are not fully disclosed to the public.
Following the Biden-Harris administration’s October 2023 Executive Order on the Safe, Secure, and Trustworthy Development and Use of Artificial Intelligence, the federal government has sought to formalize its oversight of AI developers. This mandate directed the National Institute of Standards and Technology (NIST) to establish rigorous safety standards and required developers of powerful AI systems to share the results of safety tests—or “red-teaming”—with the government. According to the White House fact sheet, these requirements apply to models that pose serious risks to national security, national economic security, or national public health and safety.
The Role of the U.S. AI Safety Institute
To implement these safety measures, the Department of Commerce launched the U.S. AI Safety Institute (AISI) within NIST. In August 2024, the Department of Commerce announced a formal memorandum of understanding with OpenAI, granting the AISI access to the company’s next-generation frontier models prior to their public release. A similar agreement was established with Anthropic, providing the government with a mechanism to evaluate models before they reach the consumer market.
These agreements are designed to facilitate “pre-deployment testing,” allowing government researchers to identify potential hazards, including risks related to cybersecurity, biological threats, or autonomous weapon development. Despite these arrangements, the actual “dialogue” between officials and company engineers—specifically how safety trade-offs are negotiated—is not publicly documented. The government does not have a formal “veto” power over commercial releases under current federal law, though the threat of regulatory action or public pressure often shapes these private interactions.
Challenges in Model Evaluation
The technical difficulty of auditing frontier models is a primary hurdle for regulators. Because these models are proprietary and exhibit emergent behaviors, static testing is often insufficient. According to a NIST operational report, the institute is currently building the infrastructure to conduct repeatable, standardized evaluations, but the field is moving faster than the development of these benchmarks. Researchers have noted that identifying what constitutes a “safe” model is a moving target, as capabilities increase with each successive release.
The lack of transparency regarding these safety evaluations has drawn criticism from civil society organizations and technology policy analysts. Critics argue that without clear, public-facing safety metrics, it is difficult to determine whether the government’s oversight is effective or merely performative. Currently, companies self-report much of their safety data, and while the new agreements with the AISI represent a shift toward independent verification, the results of these government-led audits remain protected by non-disclosure agreements to safeguard intellectual property and national security interests.
Looking Ahead to New Compliance Deadlines
The next major checkpoint for AI oversight involves the implementation of the reporting requirements established by the Department of Commerce in September 2024. Under these new rules, companies are required to report information regarding the development of large-scale AI models, including the computing power used and the results of safety tests, to the federal government. According to the Department of Commerce, this data collection is intended to provide federal agencies with the visibility needed to assess risks as models scale.
As the industry moves toward the release of even more powerful systems, the tension between proprietary development and public safety oversight is expected to increase. Future legislative efforts, such as potential updates to the AI Foundation Model Transparency Act, may eventually force more transparency into these private government-company dialogues. For now, the process remains a hybrid of voluntary industry cooperation and limited, high-level government inspection.
What do you think about the balance between innovation and regulation in the AI sector? Share your thoughts in the comments below or join the discussion on our social channels.