From 3,700 Applicants We Announce the 2026 AI Safety Fellows Exploring Public Policy at the Frontier

For years, we have debated a future in which advanced AI systems could threaten the foundations of modern society.  These debates included hypothetical situations where AI systems could autonomously exploit cybersecurity vulnerabilities, accelerate biological threats, or act in ways their developers could neither anticipate nor control. In the summer of 2026, when we launched the latest cohort of our  AI Safety Policy Entrepreneurship Fellowship, those scenarios began to look considerably less hypothetical.

In April, Anthropic announced Claude Mythos Preview, a model that the company said could “surpass all but the most skilled humans at finding and exploiting software vulnerabilities.” By May, Anthropic limited its release, arguing that the model was “too powerful to share with the general public.” Meanwhile, Stanford researchers demonstrated how AI-assisted biological design could be used to engineer bacteria-fighting phages, showcasing both the extraordinary scientific potential of these systems and the increasingly complex biosecurity questions they raise.

Most emblematic of all, hundreds of isolated OpenAI agents were able to coordinate a large-scale hack of Hugging Face through unsanctioned message boards, an incident that garnered mass attention and prompted global debates on how to effectively have oversight over these systems. This was underpinned even further as a rogue OpenAI agent infiltrated Australia’s universal healthcare system’s data portal. These two incidents offered a glimpse of how increasingly autonomous systems might interact in unexpected ways, and what those capabilities could eventually mean for the security of critical infrastructure, financial systems, and essential government services in this new paradigm created by frontier AI.

So what should policy look like when the technology keeps changing?

As the summer progressed, so did the debate over what effective governance of advanced AI systems should look like. In the United States, bipartisan legislation raised the possibility of government intervention to shut down dangerous AI systems.  Leading scientists called for a new international body to coordinate efforts to contain emerging AI risks. Meanwhile, Anthropic CEO Dario Amodei called for “pacing the frontier”, including embedding independent evaluators within frontier AI laboratories and coordinating safety standards across developers and governments.

But no single proposal, however ambitious, can address the full range of challenges posed by increasingly capable AI systems. Effective AI governance requires an ecosystem of complementary policies, institutions, and capabilities. We need credible ways to evaluate frontier systems, safeguards against their misuse, and mechanisms for international dialogue, including with strategic competitors. We must also modernize existing legal and regulatory frameworks, strengthen the capacity of governments to understand and oversee these technologies, and build the resilience necessary to respond when incidents inevitably occur. As Washington wakes up to the need for effective regulation at the frontier of AI policy, we have to make sure that our institutions have the capabilities to anticipate, manage, and respond to risks as the technology evolves. That’s why we launched this fellowship: to move beyond identifying the challenges of frontier AI and start developing the detailed, actionable policy solutions this moment demands.

That work is already underway. Across cybersecurity, biosecurity, national security, and AI governance, a new generation of policy thinkers are developing ideas for how the U.S. government can better understand and manage AI risks. 

The appetite for this kind of work is clear. More than 3,700 people applied to the Policy Entrepreneurship Fellowship (PEF) program, bringing ideas and expertise across disciplines to some of today’s most pressing policy challenges. The fellows selected represent a community working to move promising ideas beyond the page and into policy. 

Meet the FAS Fall 2026 AI Safety PEFs

At the Federation of American Scientists (FAS), these fellows will be working to turn ambitious ideas into actionable policy – from improving how frontier AI models are evaluated for government use to strengthening biosecurity protections and building the workforce needed to keep AI systems safe. 

Here are the people working on what comes next. 

J.B. Branch is the AI Governance and Technology Policy Counsel for Public Citizen’s Congress Watch division, where he leads advocacy on AI accountability, consumer privacy, product safety, platform oversight, and child online safety protections. His expertise spans AI governance, agentic AI, civil liberties, technology policy, and his analysis has appeared in Tech Policy Press, Lawfare, Slate, and other publications. He has advised governments, federal agencies, and elected officials on AI and technology policy. Previously, he worked on automated expungement systems, privacy issues at the Department of Justice, and AI policy at the Center for AI and Digital Policy. He holds a BA in Political Science from the Pennsylvania State University Schreyer Honors College, a JD from Georgetown University Law Center, and an MPA from Harvard Kennedy School.
Dr. Matthew F. Calabria is a Senior Fellow at the Orion Policy Institute and a Fellow in Residence with Harvard’s Weatherhead Scholars Program. He brings 14 years of federal service, including serving as Director for Cyber Operations and Incident Response at the White House Office of the National Cyber Director, where he advised on cyber incident response and national cybersecurity policy. He has also held analytical and leadership roles at the Departments of Defense and Energy. His scholarship has appeared in Survival, The Washington Quarterly, and The Hill, among other publications. He holds a PhD in International Affairs and an MA in International Public Policy from Johns Hopkins University’s School of Advanced International Studies; an MA in Strategic Intelligence from National Intelligence University; and a BA and an MA in International Affairs from George Washington University.
Kevin Chen is a Master of Public Policy candidate at Yale’s Jackson School of Global Affairs and an affiliate of the Schmidt Program on AI, Emerging Technologies, and National Power. His research examines epistemic risks from frontier AI, particularly how overreliance on large language models can erode independent human judgment in national security decision-making. As an FAS AI Safety Policy Entrepreneurship Fellow, he is developing an evaluation standard for bias in frontier models deployed in government analytic work. Previously, he worked as a data scientist in the U.S. intelligence community, conducting open-weight model evaluations, and was a Summer Research Fellow at the Centre for the Governance of AI. He is pursuing his MA in Public Policy from Yale University.
Olena Didenko is a PhD candidate in Medical Sciences at University College London, where she researches gene-based approaches to treating cancer pain. Trained in biology and neuroscience, she began her biosecurity work as an AIxBiosecurity Research Fellow at ERA Cambridge, studying how benchtop DNA synthesis and AI protein-design tools challenge existing synthesis-screening safeguards. She develops U.S. biosecurity policy at the intersection of AI and nucleic acid synthesis screening. She also works as an Associate Business Analyst at Infinitopes, a cancer immunotherapy company. She holds a BSc in Biology from Taras Shevchenko National University of Kyiv and an MSc in Neuroscience from the University of Oxford.
Dr. Rachel A. George is a Lecturer in International Relations and a Legal Innovation & Policy Fellow at Stanford University, where she researches how institutions adapt to technological and political change. She has been affiliated with Stanford’s Center for International Security and Cooperation and the Carnegie Endowment for International Peace. Previously, she served as Director of Education Content at the Council on Foreign Relations, a Lecturing Fellow at Duke University’s Sanford School of Public Policy, and a Visiting Assistant Professor at Duke Kunshan University. She holds a PhD in International Relations from the London School of Economics and Political Science, an MA in Middle Eastern Studies & Policy from Harvard University, and a BA in Politics from Princeton University.
Elizabeth Irwin is a public policy leader specializing in cybersecurity, AI governance, and national security strategy. She is a Strategy & Statecraft Fellow at the Center for Strategic and International Studies and a Digital Domain Fellow with the Center for the Application of Grand Strategy. Previously, she served as Senior Director for Strategic Risk at the White House Office of the National Cyber Director, leading U.S. government efforts on cyber regulation, critical infrastructure resilience, and supply chain risk management. Over more than a decade in federal service, she also held roles at the Departments of the Treasury and Defense, the Office of Management and Budget, and the Cybersecurity and Infrastructure Security Agency. She holds a BS in Communication from Northwestern University and graduate degrees in public administration, international relations, and security studies from Syracuse University and Georgetown University.
Dr. Mark A. McKinney, MBA, MPA, is Dean of the College of Computer and Information Technology at St. Petersburg College in Florida, which holds the NSA/DHS National Center of Academic Excellence in Cyber Defense designation. He oversees 19 academic programs across 11 campuses and serves as principal investigator on more than $7.2 million in active and proposed grants. As an FAS AI Safety Policy Entrepreneurship Fellow, he is developing a proposal to extend the existing cyber defense framework to fund an AI Safety and Security Operations curricular track at community colleges. He previously served 18 years in the U.S. Army, including deployments to Iraq and Afghanistan. He earned his EdD from the University of New England, where he researched veterans’ transitions into higher education.
Mariel Povolny is a researcher and policy analyst whose work spans AI security and governance, online safety, and red teaming policy. Most recently, she worked with Columbia University’s Technology and Democracy Initiative. At FAS, she explores how frontier AI model evaluations are documented and reported to the government. She also contributes to Tech Policy Press. She holds a Master of International Affairs in Technology Policy & Innovation from Columbia University’s School of International and Public Affairs and a BA in International Development from McGill University.

Strengthening EO 14409’s Approach to Frontier AI Through Independent Assessment and Published Criteria for Trusted Partners

The Executive Order on Promoting Advanced Artificial Intelligence Innovation and Security (EO 14409) marks a significant shift in the federal government’s approach to frontier artificial intelligence (AI). While keeping with previous efforts to outline voluntary regulation and prioritize American innovation, the EO establishes a new classified government assessment framework for the cyber capabilities of AI models. It also  seeks to work with private industry to select “trusted partners” eligible for early access to frontier models. 

Both the assessment and trusted-partner designation processes are internal processes of government agencies. Section 3 tasks a small group of national security actors with building a classified benchmarking process for measuring advanced cyber capabilities and identifying “covered frontier models” that meet the classified threshold. While the EO provides no public definition of the term “covered frontier models”, the designation of the title is significant because it determines which models  participating developers are then asked to make available to the government for up to 30 days before they are first released to trusted partners. 

However, the framework designates overlapping authority to the Director of NSA to both set the standards and make the final determination as to whether models should be designated as a covered frontier model under those standards. Grouping the roles of standards-setter and verifier creates a concentration of authority with no outside validation, which weakens the framework’s credibility because there are no parties equipped with relevant technical expertise to catch errors. The trusted-partner title does not have any established criteria, and therefore an entity’s eligibility cannot be properly assessed.

While preserving its original goals, the EO 14409’s classified benchmarking and trusted-partner selection processes can be strengthened. First, by the Department of Defense (DOD) designating qualified Federally Funded Research and Development Centers (FFRDCs) as third-party assessors of frontier AI evaluation standards. Second, by designating the  National Institute of Science and Technology (NIST), in consultation with the Directors of National Security Agency (NSA) and Cybersecurity and Infrastructure Security Agency (CISA), to publish a set of function-based criteria defining qualifications for the “trusted partner” status. 

FFRDCs such as MITRE National Security Engineering Center (NSEC), The Institute for Defense Analyses (IDA) Center for Communications and Computing, Massachusetts Institute of Technology (MIT) Lincoln Laboratory, and Carnegie Mellon University (CMU)’s Software Engineering Institute (SEI) already have past AI and cybersecurity project portfolios and existing working relationships with the very agencies that EO 14409 tasks with benchmarking. Because FFRDCs’ sponsoring agreements restrict outside work, these centers do not rely on funding provided by developers whose model they would assess, and therefore hold no stake in whether a given model is designated as frontier AI. Final authorization would remain decided by cleared federal agencies, mirroring the Federal Risk and Authorization Management Program (FedRAMP) procedure. 

Inserting an assessor between developers and the deciding agencies adds required time and resources. That cost is manageable because the named FFRDCs already operate under contracts with the DOD and maintain secure facilities and technical staff that evaluation requires. 

The EO’s “trusted partners” designation, which grants access to the most technologically advanced AI models to critical infrastructure firms, has no public and explicit criteria for qualification. Implementing a transparent trusted-partner standard is necessary. It would not only mitigate criticisms of opaque membership criteria that the CISA’s Joint Cyber Defense Collaborative (JCDC) faced, but also would encourage private industry to strive towards meeting standards and fostering a collaborative frontier AI industry.

AI’s high-stakes outputs directly shape American economic, industrial, and technological outcomes. They must not be accepted entirely on the basis of institutional faith rather than verifiable evidence.

Challenge and Opportunity

Whether a model is designated as a covered frontier model determines whether its developers are asked to give the government up to 30 days of pre-release access and to release it to trusted partners first. Under Section 3 of EO 14409, the final authority to make this designation rests with the Director of NSA alone, with only consultation with other government actors. The classified benchmarking process that establishes the threshold the models are checked against is developed by the Secretaries of Treasury, War (through the Director of NSA), and Homeland Security (through the Director of CISA), in consultation with the National Cyber Director, the Assistant to the President for Science and Technology, and NIST. NSA is therefore given both a role in setting the classified benchmarks and the authority to make the final call on which models meet them. 

Grouping the roles of standards-setter and verifier together creates a concentration of authority without independent validation, which weakens the credibility of the framework as there are no outside parties equipped with the relevant technical expertise to correct errors. A verification process that is not well supported with technical knowledge and opinion that holds no stake in the final designation outcome could cause significant national security risks in two areas. First, a model may be designated as a covered frontier AI model when its capabilities do not warrant it, causing significant cost for developers in delayed timelines and constraints upon release, and for government actors in spent resources with no corresponding security benefit. Second, a frontier AI model capable of creating significant national security impacts may pass without designation, leaving it to be deployed without the needed safeguards.

The absence of external technical review also weakens the incentive for developers to participate. A voluntary framework succeeds when participants have something to gain from engaging it. As the framework currently stands, developers cannot examine what is being tested, as the benchmarks are classified, and cannot verify whether the benchmarks are being applied consistently across firms, because verification is internal. The benefits that come from engaging the framework, such as reputational benefit or a working relationship with government agencies, may not outweigh the time and resources it needs. As a result, developers may choose not to participate in the voluntary framework, which will become ineffective when government cannot evaluate the models it is most concerned with.

DOD has the opportunity to outsource verification of classified AI benchmarks to FFRDCs. Federally Funded Research and Development Centers (FFRDCs) are research institutions typically administered by universities or non-profits that are funded by federal support to provide additional R&D capabilities and independent scientific perspectives. In the past, when the government needed to consolidate voluntary agreements into audited, credible requirements, it has collaborated with FFRDCs for technical capacity it couldn’t support by itself. 

For instance, the Cybersecurity Maturity Model Certification (CMMC) came from the collaboration between DOD and the CMU Software Engineering Institute (SEI). The CMU SEI FFRDC drew on its decades of cybersecurity expertise to co-develop a certification structure, assessment standards, and train over 160,000 contracting workers in the defense acquisition workforce. 

There are multiple existing FFRDCs conducting highly classified work under the major actors (NSA, NCD, APST, CISA, and DOD) named by the EO that would serve as reliable third-party assessors. (See Table 1.) Together, these institutions have conducted classified cyber and machine learning work in areas the EO identified as major priorities of frontier AI risk in Sec. 2(d), such as detecting and patching software vulnerabilities. All four FFRDCs are sponsored by agencies the EO has tasked with implementation.

FFRDCWhat it doesFY24 funding
IDA’s Center for Communications and Computingconducts classified cryptography and computing research in partnership with DOD and NSAReceived $85.2 million in complete federal funding during FY 2024
MITRE’s National Security Engineering Center (NSEC)produced ATLAS as a global knowledge base of tactics against attacks by AI-enabled systemsApproximately $1.44 billion
MIT’s Lincoln Laboratorysponsored by DOD, have created an open-source Responsible AI Toolbox to design, evaluate, and monitor AI systemsApproximately $1.36 billion
CMU’s Software Engineering Institute (SEI)sponsored by DOoD houses the AI division which covers designing benchmark AI systems and securing ML vulnerabilities,Received $147.4 million

Delegating frontier AI assessment to FFRDCs increases government technical capacity, since these institutions already maintain secure infrastructure and dedicated personnel with relevant expertise. By directing major frontier AI projects to FFRDCs, the government also reinvests funding and time back into American scientific personnel, which in turn spurs further technological innovation and produces reusable capacity for AI tasks that come after EO 14409.

FFRDCs’ unique status as a non-profit with access to private-sector resources required to “operate in the public interest with objectivity and independence” allows them to sit outside of government while working with it. Because FFRDCs operate under sponsoring agreements that restrict the amount of outside work they can accept, and because frontier AI vendors cannot control FFRDCs’ revenue as their funding is directly sourced from their sponsoring agencies, FFRDCs do not hold a stake in the outcome of the benchmarking process. Furthermore, since FFRDCs are required to “fully disclose their activities to their sponsoring agency,” security agencies will still be able to maintain close monitoring of activities with sensitive data and preserve safeguards against dangerous actors that the EO seeks to establish. 

The effectiveness of differentiating the roles of standards-setter and assessors has been proven effective under previous federal practice with FedRAMP’s accredited third-party assessment organizations (3PAOs), or assessors, as the current FedRAMP Consolidated Rules for 2026 refer to them. Under this system, assessors conduct independent security assessments of commercial cloud service providers (CSPs) and document Security Assessment Reports (SARs) detailing how well services meet federal standards. However, authorizing officials (AO) from federal agencies get the final say in the risk decision. Therefore, the current administration can designate FFRDCs a similar role and separate verification structurally from the final authorization of frontier AI models. Doing so guarantees that FFRDCs provide additional technical verification against established benchmarks while retaining the final say in the hands of trusted federal agencies.

Additionally, as EO 14409’s Sec. 3(b)(ii)-(iii) currently lays out, industry and the federal government will collaborate to select “trusted partners” who gain early access to the most powerful frontier models. The EO explicitly states that selecting trusted partners serves to “promote secure innovation and strengthen the cybersecurity of critical infrastructure.” Early access is therefore treated both as a risk to be managed and a benefit to be allocated. The emphasis on “secure innovation” suggests that powerful models must be placed in the hands of entities trusted to use them responsibly, since each additional user of early-access models adds more risk of the model being leaked. At the same time, these tools must reach the frontline, critical infrastructure defenders who would otherwise lack access to tools of similar capability. 

Although the general purpose of selecting trusted partners is stated, there is no definition of who could serve as trusted partners or what specific qualifications make an entity eligible for this title. The absence of published, concrete criteria creates conditions that allow regulatory capture, whereas the very regulatory rules made to protect public interest may become a tool to push industry interests. The government will effectively be picking the private firms that “win” the market by selecting which firms receive the commercial advantage of the most powerful frontier models. Without explicit, transparent rules, the allocation of this title may become based on preexisting relationships with government or industry actors rather than real need and qualification, leaving smaller critical-infrastructure firms that could benefit from having early access at a disadvantage. 

It is imperative that change is enacted promptly to minimize the resource and time cost of amending the verification process once it has been embedded into agencies’ operations. NIST can publish a publicly accessible set of qualifications for “trusted partners” before the first round of assessments is made and before trusted partners are selected.

Plan of Action

Recommendation 1. Establish FFRDCs as the third-party assessors of AI models against classified benchmarks

DOD should task FFRDCs as third-party assessors who will check submitted AI models against classified frontier AI benchmarks through existing sponsoring agreements under the FAR 35.017. Because their sponsoring contracts restrict outside work and competition with private industry, FFRDCs cannot be funded by the private companies whose models they may be asked to assess. Additionally, existing contracts with the most central agencies identified in the EO mark the opportunity for a low-burden process to map tasks onto an existing structure between FFRDCs and DOD.

In reference to FedRAMP’s assessment model, FFRDCs would not be the final decision-makers on whether a frontier AI model exceeds the threshold set by the established benchmarks. Their role would be limited to providing strictly technical evaluation and reporting recommended actions back to agencies the EO already designates as assessors, such as the NSA and CISA. Ultimately, authority would remain where the order previously placed it with cleared national security actors, but the addition of FFRDCs will ensure that the technical work behind the most influential decisions in AI and cybersecurity today gains more credibility and rigor. While outsourcing the verification process would likely lead to additional time for interagency coordination, the benefits of increasing credibility, as well as the act of reinvesting in fostering the scientific talent around AI technology in America, far outweigh these considerations.

A complete pipeline connecting FFRDCs to the sponsoring agency and NSA could be established as follows:

  1. Benchmark development. The same group of actors identified in EO 14409 Sect. 3—including the Secretaries of the Treasury, War (through the Director of NSA), and Homeland Security (through the Director of CISA), as well as corresponding consultants—develops a set of classified benchmarks on what qualifies as a “covered frontier model.”
  2. Task Assignment. AI developers voluntarily engage the Federal Government in evaluation of their model. DOD issues a new task order to the appropriate FFRDC while developers provide model access directly to the FFRDC under confidentiality and cybersecurity protections.
  3. Evaluation Execution. The FFRDCs assess model capabilities against the classified benchmarks within secure computing environments. Projected areas of testing may include cyber-specific capabilities that frontier models present, such as the ability to discover and exploit cybersecurity weaknesses, and whether safeguards hold against repeated jailbreaking. However, the precise areas of interest would be tailored to the benchmarks being measured against. Evaluation is confined within isolated environments, and full evaluation logs and code versions are preserved to be combined into the final report.
  4. Reporting. FFRDCs provide a classified report to the sponsor agency, outlining whether the model “exceeds threshold,” “undetermined,” or “does not exceed threshold,” to each of the set standards. The report should be completed with methodology, results, as well as model limitations.
  5. Final Decision. NSA Director, referring to the FFRDC evaluation record and set benchmarks, makes the final decision as to whether the AI model should be designated as a “covered frontier model.”

To expand upon existing technical capacity and map new AI verification responsibilities to these FFRDCs, further funding should be secured. Given that this memo’s proposed FFRDC assessors are under DOD sponsorship, DOD would be the ideal actor to provide additional funding. Establishing a firm estimate would require a formal cost estimate by DOD and the corresponding FFRDCs. As a starting point, policymakers should plan for roughly $30-75 million in the first year for facilities, initial hiring, and compute, and $20-50 million per year onwards for ongoing costs such as salaries and maintenance. If full launch funding cannot be identified, DOD may instead opt for a pilot year to identify a smaller, one-year program. After the full run-time of the pilot year, agencies may then choose to renew the program and continue having FFRDCs’ technical expertise on board.

Ultimately, this recommendation aligns with the national priority to accelerate American AI innovation. By reinvesting in America’s scientific talent through delegating funding and more expansive AI projects to our federally funded laboratories, we can create a positive feedback loop that propels America further as a global technological leader. 

Recommendation 2. Develop and publish a set of publicly available function-based criteria to qualify as “trusted partners”

To ensure verifiable eligibility standards and minimize government control on the market for frontier AI, NIST, in consultation with NSA and CISA, should develop and publish a set of function-based criteria that outlines what will qualify as a “trusted partner” eligible for early access to covered frontier models. This set of standards should be published after a period for the public and private industry to submit comments to fully assess current industry needs and capabilities.

Undefined criteria for membership have been proven to be ineffective in previous government collaborations with the private industry. CISA’s Joint Cyber Defense Collaborative (JCDC) was established in 2021 to secure cyberspace by bringing collaborators from the government, industry, and international organizations, but it never published a set of clear qualifying criteria for membership. Its first cohort was drawn largely from the largest cloud and security vendors, including Amazon Web Services, AT&T, CrowdStrike, FireEye Mandiant, Google Cloud, Lumen, Microsoft, Palo Alto Networks and Verizon. But the ambiguity around what qualifies an entity for membership led to CISA’s own Cybersecurity Advisory Committee (CSAC) to advise CISA to clarify and provide more transparency on membership requirements and joining processes. Moving forward with the implementation of EO 14409, it’s imperative to learn from this past example that published requirements are necessary. 

NIST should be the leading actor for this effort. First, its Center for AI Standards and Innovation (CAISI) is already charged under the AI Action Plan with developing evaluation guidelines for federal agencies and has meaningfully coordinated efforts on pre-deployment standards with major companies such as OpenAI and Anthropic. NIST has also previously constructed and published the AI Risk Management Framework that established publicly accessible and uniform technical vocabulary as well as voluntary guidelines for AI systems. NSA and CISA should act as consulting agencies and contribute guidance on security and clearance requirements.

This set of function-based criteria might include:

  1. Demonstrated and Significant Mission. The entity has demonstrated significant and positive contributions to a critical infrastructure sector and can articulate a specific function that early access to frontier AI models would serve.
  2. Security Capacity. The entity is equipped with sufficient security capacity to protect access to models. 
  3. Matching Public Benefit to Commercial Advantage. The entity’s commercial advantage derived from being designated as a trusted partner should match the critical-infrastructure security benefit it would produce.
  4. Accountability and Auditability. The entity has accepted responsibility to oversee and log its usage of frontier AI models.

The drafted criteria should explicitly state that a commercial relationship with the developer, while not disqualifying, cannot serve as the basis for the designation as a trusted partner. Additionally, designated trusted partners should be re-evaluated periodically to ensure that the entity continues to align with published criteria. The estimated cost for this recommendation is a one-time $2.5 million to support a NIST drafting team (of five to ten staff), public comment processing, and consultation with NSA and CISA. Ongoing future monitoring and revision of the criteria may cost under $1 million per year. 

All critical infrastructure firms stand to benefit from this approach. A set of defined and functional standards, rather than an omitted list of names, will allow any existing firm to assess its own eligibility and work toward meeting it. As a result, this recommendation would contribute to building a collaborative frontier AI industry that benefits from its working relationship with the government.

Conclusion

While EO 14409 is an important first step toward creating a national response to frontier AI development, further measures are needed to reinforce the credibility of the AI evaluation process and define the currently unspecified “trusted partner” criteria.

Under the recommendations of this policy memo:

Implementing these changes would strengthen national technological capacity, reinvest in American scientific talent, and foster a collaborative frontier AI field. 

Frequently Asked Questions
Who might oppose this proposal, and how can concerns be addressed?

This proposal orients itself around improving accountability and transparency. These recommendations address two pressing American AI concerns: one, reinvesting in American scientific talent by equipping FFRDCs with relevant AI experience and funding, and two, promoting an equitable frontier AI market without additional regulations that may increase burden and slow innovation.


Private industry may express concern over the concentration of authority and the absence of external technical review. They may believe this leads to an ad-hoc process of AI evaluation and designation that is biased. However, having clear and published trusted partner criteria and an evaluation process promotes fair partnership and spurs healthy competition. That opens participation to smaller and newer firms (i.e. startups and middle tech).

How will implementation and accountability be measured?

Implementation would be built upon an existing contract and relationships between DOD and its sponsored FFRDCs rather than requiring a new group of actors to execute. Accountability would be measured by DOD, which acts as the sponsoring agency, receives all direct reports, and oversees task order. The Director of NSA then receives the final report and uses it to make a documented and well-supported designation decision. The publication of trusted partner criteria will be verifiable via the draft released during the public comment period by the projected timeline and the final, publicly accessible document. .

If the Director of NSA is one of the actors setting the classified AI benchmarks, and DOD (the broader department overseeing NSA) is sponsoring the FFRDC third-party verifier, is that a conflict of interest?

Although both the Director of NSA and the broader DOD will remain central actors in the policy memo’s recommendations, the conflict of interest is reduced because the evaluation records are produced outside of the office making the final decision. Since FFRDCs are not the final decision makers and only serve to provide technical evaluation reports, their role remains as an independent actor rather than an entity that could potentially gain from swaying frontier AI delegation decisions.

How much will this cost? Where would this money come from?

DOD budgets FFRDC work through assigning staff years of technical effort (STE) (see: GAO-20-31, FEDERAL RESEARCH: DOD’s Use of Study and Analysis) that represents the amount of resources required by one employee for one year. Congress caps the total number of STEs that DOD may allocate each fiscal year, and the current ceiling is set at 6,053 STEs, and Defense STE during the FY may be reallocated as needed. There is no publicly accessible record of the amount of STEs assigned to each DOD FFRDCs as well as the precise amount of STEs required to support respective research projects. A very general and hypothetical estimate of how many STEs can support a large-scale remapping of AI verification tasks onto the recommended FFRDCs would likely take an estimated 38-58 STEs.


Per the policy memo recommendations, the larger FFRDC evaluation process may be broken down to three areas that will require personnel and resource support. First, approx 20-30 STEs may be assigned to support conducting model evaluations, with approximately 15-30 evaluations per year starting off and expanded as needed. Second, a portion or approx 12-18 STEs may be needed to build secure evaluation infrastructure with appropriate GPUs and equipment, building isolated networks for red-team testing, etc. Finally, a portion of approx 6-10 STEs could be used to support efforts of writing and delivering evaluation reports to sponsoring agencies who will hold the power to make final audit decisions.


This amount of STE will likely support the number of needed technical evaluators at FFRDC rates, secure infrastructure, and increasing computation dedicated solely to supporting verifying frontierAI. However, as is not disclosed to the public, further input from involved FFRDC and DOD personnel is needed to precisely identify the amount of needed resources.

FFRDCs have received criticism from GAO and other sponsoring agencies. Why outsource to FFRDCs?

The verification role requires an assessor who is independent from developers and holds no stakes in the outcome. FFRDCs are the best fit given their access to government clearance, status as a non-profit, and secure facilities which ordinary private contractors and academic experts do not have. GAO’s concerns are based on oversight failures, noncompetitive contracts, mission creep, as well as contract management. These are governance and cost problems that do not significantly reduce FFRDCs’ technical capacity for such work or the quality of outputs.


FFRDCs are not perfectly insulated, but against the current layout of the EO 14409 where the verifier is also acting as the standards-setter, outsourcing to FFRDCs is still a significant improvement.

NIST CAISI already conducts AI evaluations, why not CAISI?

First, NIST CAISI operates on a very limited budget. There has been a lot of effort to advocate for an expanded budget, but it has not yet been granted. Due to constrained staffing resources, NIST CAISI may not be able to meaningfully produce the volume of model evaluations at the speed (30 days) required by the EO’s process.


Second, all major actors named in the EO are located from federal agencies operating behind very classified standards and processes. Therefore, it is unlikely that any responsibilities can be re-directed to public-facing organizations. Ultimately, it seeks to keep the authority within national security actors, which the NIST CAISI does not qualify for. Therefore, to find a meaningful compromise between national security and the need for transparency, FFRDCs are the most viable actors that allow ultimate authority to reside with central actors while adding technical capacity and credibility.

What’s Progress and What’s Not in the Trump Administration’s AI Action Plan

Artificial intelligence is already shaping how Americans work, learn, and receive vital services—and its influence is only accelerating. To steer this technology toward the public good, the United States needs a coherent, government-wide AI agenda that encourages innovation and trustworthiness, grounded in the best scientific evidence. 

In February 2025, the Trump Administration sought public comment on its development of an AI Action Plan. The Federation of American Scientists saw this as an opportunity to contribute expert, nonpartisan guidance, combining insights from our policy team with ideas from the broader science and technology community, developed as part of our Day One Project. In our comments to the White House Office of Science and Technology Policy we recommended incorporating responsible policies to unleash AI innovation, accelerate AI adoption, ensure secure and trustworthy AI, and strengthen our existing world-class government institutions. 

Last week, the Trump Administration released their AI Action Plan. The document contains many promising aspects related to AI research and development, interpretability and control, managing national security risks, and new models for accelerating scientific research. However, there are also concerning provisions, such as those inhibiting state regulations and removing mentions of diversity, equity, and inclusion and climate change from the NIST AI Risk Management Framework. These omissions weaken the United States’ ability to lead on some of the most pressing societal challenges associated with AI technologies.

Despite the AI Action Plan’s ambitious proposals, it will remain aspirational without funding, proper staffing, and clear timelines. The deep cuts to budgets and personnel across the government present an incongruous picture of the Administration’s priorities and policy agenda for emerging technologies, and places pressure on Congress to ensure this plan is properly supported.

Promising Advances & Opportunities

AI Interpretability  

As an organization, we’ve developed and shared concrete ideas for advancing AI interpretability—the science of understanding how AI works under the hood. The Administration’s elevation of AI interpretability in the plan is a promising step. Improving interpretability is not only critical for technical progress but also essential to fostering public trust and confidence in AI systems. 

We have provided a roadmap for the government to deliver on the promise of interpretable AI in both our AI Action Plan comments and a more detailed memo. In these documents we’ve advocated for advancing AI explainability through open-access resources, standardized benchmarks, common tasks, user-centered research, and a robust repository of techniques to ensure consistent, meaningful, and widely applicable progress across the field. We’ve also argued for the federal government to prioritize interpretable AI in procurement—especially for high-stakes applications—and to establish research and development agreements with AI companies and interpretability research organizations to red team critical systems and conduct targeted interpretability research.

AI Research and Development 

Beyond interpretability, the AI Action Plan lays out an ambitious and far-reaching agenda for AI research and development, including robustness and control, advancing the science of AI, and building an AI evaluation ecosystem. We recognize that the Administration has incorporated forward-looking proposals that echo those from our Day One Project—such as building world-class scientific datasets and using AI to accelerate materials discovery. These policy proposals showcase our perspective that the federal government has a critical role to play in supporting groundbreaking scientific and technical research. 

A Toolbox for AI Procurement

The Administration’s focus on strengthening the federal workforce’s capacity to use and manage AI is an essential step toward responsible deployment, cross-agency coordination, and reliability in government AI use. The proposed GSA-led AI procurement toolbox closely mirrors our recommendation for a resource to guide agencies through the AI acquisition process. Proper implementation of this policy could further support government efficiency and agility to respond to the needs of constituents. 

Managing National Security Risks

The Administration also clearly recognizes the emerging national security risks posed by AI. While the exact nature of many of these risks remains uncertain, the plan contains prudent recommendations on key areas like biosecurity and cybersecurity, and highlights the important role that the Center for AI Standards and Innovation can play in responding to these risks. FAS has previously published policy ideas on how to prepare for emerging AI threats and create a system for reporting AI incidents, as well as outlining how CAISI can play a greater role in advancing AI reliability and security. These proposals can help the government implement the recommendations advanced in the Action Plan.

Focused Research Organizations

The Administration’s support of Focused Research Organizations (FROs) is a promising step. FROs are organizations that address well-defined challenges that require scale and coordination but that are not immediately profitable, and are an exciting model for accelerating scientific progress. FAS first published on FROs in 2020, and has since released a range of proposals from experts that are well-suited to the FRO model. Since 2020, various FROs have gained over $100 million in philanthropic funding, but we believe that this is the first time that the U.S. government has explicitly embraced the FRO model.

Where the AI Action Plan Falls Short

Restricting State-Level Guardrails

The Administration’s AI Action Plan proposes to restrict federal AI funding to states when state AI rules “hinder the effectiveness” of that funding. While avoiding unnecessary red tape is sensible, this unclear standard could offer the administration a wide latitude to block state rules at its discretion. FAS has recently opposed preemption of state-level AI regulation by Congress in the absence of federal action. Without national standards for AI, state rules provide an opportunity to develop best practices for responsible AI adoption. 

Failing to Address Bias in AI Systems 

We are also concerned by the recommended revision to the NIST AI Risk Management Framework (RMF) that would eliminate references to diversity, equity, and inclusion. AI bias is a proven, measurable phenomenon, as documented by a broad scientific consensus from leading researchers and practitioners across sectors. Failing to address such biases leaves the public vulnerable to the harms of discriminatory or unfair systems that can affect people in areas like healthcare, housing, hiring, and access to public services. This includes deeply consequential biases, such as those affecting rural communities. A lack of action to address AI bias will only inhibit beneficial adoption and further erode trust in the accuracy of algorithmic systems. 

The AI Action Plan contains a direction for the federal government to only procure AI models from developers who “ensure that their systems are objective and free from top-down ideological bias,” which is implemented via an associated executive order. Building modern AI systems involves a huge range of choices, including which data to use for training, how to “fine tune” the model for particular use-cases, and the “system prompt” which guides model behavior. Each of these stages can affect model outputs in ways that are not well understood and can be difficult to control. There is no standard definition for what constitutes a model that is “free from top-down ideological bias”, and this vague standard could easily be misused or improperly implemented at the agency level with unintended consequences for the public. We encourage the administration to instead focus on increasing transparency and explainability of systems as a mechanism to prevent unintended bias in outputs. 

Ignoring the Environmental Costs and Opportunities

The Administration’s direction to remove mention of climate change from the RMF overlooks the very real climate and environment impacts associated with the growing resource demands of large-scale AI systems. Measuring and managing environmental impacts is an important component of AI infrastructure buildout, and removing this policy lever will also restrict AI adoption. This is also a missed opportunity to push forward the ways that AI can help tackle climate change and other environmental issues. In our recent AI and Energy Policy Sprint, we developed policy memos which highlighted the benefits AI could bring to our energy system and environment, while also highlighting ways of responding to AI’s environmental and health impacts. 

The Importance of Public Trust

The current lack of public trust in AI risks inhibiting innovation and adoption of AI systems, meaning new methods will not be discovered and new benefits won’t be felt. A failure to uphold high standards in the technology we deploy will also place our nation at a strategic disadvantage compared to our competitors. Recognizing this issue, both the first and second Trump administrations have emphasized public trust as a key theme in their AI policy documents. Many of the research directions outlined in the administration’s AI Action Plan promise to steer AI technology in a more trustworthy direction and deliver widespread benefits to the public. However, several measures simultaneously threaten to undermine important guardrails, while cuts to important government programs also work against the goals the administration has set for itself. 

The Federation of American Scientists will continue to collaborate with the scientific community to place rigorous evidence-based policy at the heart of delivering AI that works for all Americans.

Moving Beyond Pilot Programs to Codify and Expand Continuous AI Benchmarking in Testing and Evaluation

Rapid and advanced AI integration and diffusion within the Department of Defense (DoD) and other government agencies has emerged as a critical national security priority. This convergence of rapid AI advancement and DoD prioritization creates an urgent need to ensure that AI models integrated into defense operations are reliable, safe, and mission-enhancing. For this purpose, the DoD must deploy and expand one of its most critical tools available within its Testing and Evaluation (T&E) process: benchmarking—the structured practice of applying shared tasks and metrics to compare models, track progress, and expose performance gaps.

A standardized AI benchmarking framework is critical for delivering uniform, mission-aligned evaluations across the DoD. Despite their importance, the DoD currently lacks standardized, enforceable AI safety benchmarks, especially for open-ended or adaptive use cases. A shift from ad hoc to structured assessments will support more informed, trusted, and effective procurement decisions. 

Particularly at the acquisition stage for AI models, rapid DoD acquisition platforms such as Tradewinds can serve as the policy vehicle for enabling more robust benchmarking efforts. This can be done with the establishment of a federally coordinated benchmarking hub, spearheaded by a coordinated effort between the Chief Data and Artificial Intelligence Officer (CDAO) and Defense Innovation Unit (DIU) in consultation with the newly established Chief AI Officer’s Council (CAIOC) of the White House Office of Management and Budget (OMB). 

Challenge and Opportunity

Experts at the intersection of both AI and defense, such as the retired Lieutenant General John (Jack) N.T. Shanahan, have emphasized the profound impact of AI on the way the United States will fight future wars – with the character of war continuously reshaped by AI’s diffusion across all domains. The DoD is committed to remaining at the forefront of these changes: between 2022-2023, the value of federal AI contracts increased by over 1200%, with the surge driven by increases in DoD spending. Secretary of Defense Pete Hegseth has pledged increased investment in AI specifically for military modernization efforts, and has tasked the Army to implement AI in command and control across the theater, corps, and division headquarters by 2027–further underscoring AI’s transformative impact on modern warfare.

Strategic competitors—especially the People’s Republic of China—are rapidly integrating AI into their military and technological systems. The Chinese Communist Party views AI-enabled science and technology as central to accelerating military modernization and achieving global leadership. At this pivotal moment, the DoD is pushing to adopt advanced AI across operations to preserve the U.S. edge in military and national security applications. Yet, accelerating too quickly without proper safeguards risks exposing vulnerabilities adversaries could exploit.

With the DoD at a unique inflection point, it must balance the rapid adoption and integration of AI into its operations with the need for oversight and safety. DoD needs AI systems that consistently meet clearly defined performance standards set by acquisition authorities, operate strictly within the scope of their intended use, and do not exhibit unanticipated or erratic behaviors under operational conditions. These systems can deliver measurable value to mission outcomes while fostering trust and confidence among human operators through predictability, transparency, and alignment with mission-specific requirements.

AI benchmarks are standardized tasks and metrics that systematically measure a model’s performance, reliability, and safety, and have increasingly been adopted as a key measurement tool by the AI industry. Currently, DoD lacks standardized, comprehensive AI safety benchmarks, especially for open-ended or adaptive use cases. Without these benchmarks, the DoD risks acquiring models that underperform, deviate from mission requirements, or introduce avoidable vulnerabilities, leading to increased operational risk, reduced mission effectiveness, and costly contract revisions.  

A recent report from the Center for a New American Security (CNAS) on best practices for AI T&E outlined that the rapid and unpredictable pace of AI advancement presents distinctive challenges for both policymakers and end-users. The accelerating pace of adoption and innovation heightens both the urgency and complexity of establishing effective AI benchmarks to ensure acquired models meet the mission-specific performance standards required by the DoD and the services. 

The DoD faces particularly outsized risk, as its unique operational demands can expose AI models to extreme conditions where performance may degrade. For example, under adversarial conditions, or when encountering data that is different from its training, an AI model may behave unpredictably, posing heightened risk to the mission. Robust evaluations, such as those offered through benchmarking, help to identify points of failure or harmful model capabilities before they become apparent during critical use cases. By measuring model performance in real-world applicable scenarios and environments, we increase understanding of attack surface vulnerabilities to adversarial inputs. We can identify inaccurate or over-confident measurements of outputs, and recognize potential failures in edge cases and extreme scenarios (including those beyond training parameters, Moreover, we improve human-AI performance and trust factors, and avoid unintended capabilities. Benchmarking helps to surface these issues early. 

Robust AI benchmarking frameworks can enhance U.S. leadership by shaping international norms for military AI safety, improving acquisition efficiency by screening out underperforming systems, and surfacing unintended or high-risk model behaviors before deployment. Furthermore, benchmarking enables AI performance to be quantified in alignment with mission needs, using guidance from the CDAO RAI Toolkit and clear acquisition parameters to support decision-making for both procurement officers and warfighters. Given the DoD’s high-risk use cases and unique mission requirements, robust benchmarking is even more essential than in the commercial sector.

The DoD now has an opportunity to formalize AI safety benchmark frameworks within its Testing and Evaluation (T&E) processes, tailored to both dual-use and defense-specific applications. T&E is already embedded in DoD culture, offering a strong foundation for expanding benchmarking. Public-private AI testing initiatives, such as the DoD collaboration with Scale AI to create effective T&E (including through benchmarking) for AI models show promise and existing motivation for such initiatives. Yet, critical policy gaps still exist. With pilot programs underway, the DoD can move beyond vendor-led or ad hoc evaluations to introduce DoD-led testing, assess mission-specific capabilities, launch post-acquisition benchmarking, and develop human-AI team metrics. The widely used Tradewinds platform offers an existing vehicle to integrate these enhanced benchmarks without reinventing the wheel. 

To implement robust benchmarking at DoD, this memo proposes the following policy recommendations, to be coordinated by DoD Chief Digital and Artificial Intelligence Office (CDAO):

Plan of Action 

The CDAO should launch a formalized AI Benchmarking Initiative, moving beyond current vendor-led pilot programs, while continuing to refine its private industry initiatives. This effort should be comprehensive and collaborative in nature, leveraging internal technical expertise. This includes the newly established coordinating bodies on AI such as the Chief AI Officer’s Council, which can help to ensure that DoD benchmarking practices are aligned with federal priorities, and the Defense Innovation Unit, which can be an excellent private industry-national defense sector bridge and coordinator in these efforts. Specifically, the CDAO should integrate benchmarking into the acquisition pipeline. This will  establish ongoing benchmarking practices that facilitate continuous model performance evaluation through the entirety of the model lifecycle. 

Policy Recommendations 

Recommendation 1. Establish a Standardized Defense AI Benchmarking Initiative and create a Centralized Repository of Benchmarks

The DoD should build on lessons learned from its partnership with Scale AI (and others) developing benchmarks specifically for defense use cases. This should expand  into a standardized, agency-wide framework. 

This recommendation is in line with findings outlined by RAND, which calls for developing a comprehensive framework for robust evaluation and emphasizes the need for collaborative practices, and measurable performance metrics for model performance. 

The DoD should incorporate the following recommendations and government entities to achieve this goal:

Develop a Whole-of-Government Approach to AI Benchmarking

If internal reallocations from the $500 million allocation proves insufficient or unviable, Congressional approval for additional funds can be another funding source. Given the strategic importance of AI in defense, such requests can readily find bipartisan support, particularly when tied to operational success and risk mitigation.

Recommendation 2. Formalize Pre-Deployment Benchmarking for AI Models at the Acquisition Stage 

The key to meaningful benchmarking lies in integrating it at the pre-award stage of procurement. The DoD should establish a formal process that:

Recommendation 3. Contextualize Benchmarking into Operational Environments

Current efforts to scale and integrate AI reflect the distinct operational realities of the DoD and military services. Scale AI, in partnership with the DoD, Anduril, Microsoft, and the CDAO, is  developing AI-powered solutions which are focused on the United States Indo-Pacific Command (INDOPACOM) and United States European Command (EUCOM). With these regional command focused AI solutions, it makes sense to create equally focused benchmarking standards to test AI model performance in specific environments and under unique and focused conditions. In fact, researchers have been identifying the limits of traditional AI benchmarking and making the case for bespoke, holistic, and use-case relevant benchmark development. This is vital because as AI models advance, they introduce entirely new capabilities which require more robust testing and evaluation. For example, large language models, which have introduced new functionalities including natural language querying or multimodal search interfaces, require entirely new benchmarks that measure: natural language understanding, modal integration accuracy, context retention, and result usefulness. In the same vein, DoD relevant benchmarks must be developed in an operationally-relevant context. This can be achieved by:

Frameworks such as Holistic Evaluation of Language Models (HELM) and Focused LLM Ability Skills and Knowledge (FLASK) can offer valuable guidance for developing LLM-focused benchmarks within the DoD, by enabling more comprehensive evaluations based on specific model skill sets, use-case scenarios, and tailored performance metrics.

Recommendation 4. Integration of Human-in-the-Loop Benchmarking 

An additional layer of AI benchmarking for safe and effective AI diffusion into the DoD ecosystem is evaluating AI-human team performance, and measuring user trust, perceptions and confidence in various AI models. “Human‑in‑the‑loop” systems require a person to approve or adjust the AI’s decision before action, while “human‑on‑the‑loop” systems allow autonomous operation but keep a person supervising and ready to intervene. Both  “Human in the loop”  and “Human on the loop” are critical components of the DoD and military approach to AI. Both  require continued human oversight of ethical and safety considerations over AI-enabled capabilities with national security implications. A recent study by MIT study found that there are surprising performance gaps between AI only, human only, and AI-human teams. For the DoD particularly, it is important to effectively measure these performance gaps across the various AI models it plans to integrate into its operations due to heavy reliance on user-AI teams. 

A CNAS report on effective T&E for AI spotlighted the DARPA Air Combat Evolution (ACE) program, which sought autonomous air‑combat agents needing minimal human intervention. Expert test pilots could override the system, yet often did so prematurely, distrusting its unfamiliar tactics. This case underscores the need for early, extensive benchmarks that test user capacity, surface trust gaps that can cripple human‑AI teams, and assure operators that models meet legal and ethical standards. Accordingly, this memo urges expanding benchmarking beyond pure model performance to AI‑human team evaluations in high‑risk national‑security, lethal, or error‑sensitive environments.

Conclusion

The Department of Defense is racing to integrate AI across every domain of warfare, yet speed without safety will  jeopardize mission success and national security. Standardized, acquisition‑integrated, continuous, and mission‑specific benchmarking is therefore not a luxury—it is the backbone of responsible AI deployment. Current pilot programs with private partners are encouraging starts, but they remain too ad hoc and narrow to match the scale and tempo of modern AI development.

Benchmarking must begin at the pre‑award acquisition stage and follow systems through their entire lifecycle, detecting risks, performance drift, and adversarial vulnerabilities before they threaten operations. As the DARPA ACE program showed, early testing of human‑AI teams and rigorous red‑teaming surface trust gaps and hidden failure modes that vendor‑led evaluations often miss. Because AI models—and enemy capabilities—evolve constantly, our evaluation methods must evolve just as quickly.

By institutionalizing robust benchmarks under CDAO leadership, in concert with the Defense Innovation Unit and the Chief AI Officers Council, the DoD can set world‑class standards for military AI safety while accelerating reliable procurement. Ultimately, AI benchmarking is not a hurdle to innovation and acquisition, but rather it is the infrastructure that can make rapid acquisition more reliable and innovation more viable. The DoD cannot afford the risk of deploying AI systems which are risky, unreliable, ineffective or misaligned with mission needs and standards in high-risk operational environments. At this inflection point, the choice is not between speed and safety but between ungoverned acceleration and a calculated momentum that allows our strategic AI advantage to be both sustained and secured.

This memo was written by an AI Safety Policy Entrepreneurship Fellow over the course of a six-month, part-time program that supports individuals in advancing their policy ideas into practice. You can read more policy memos and learn about Policy Entrepreneurship Fellows here.

What is the Scale AI benchmarking pilot program at DoD, and why and how does this policy proposal build on this initiative?

he Scale AI benchmarking initiative, launched in February 2024 in partnership with the DoD, is a pilot framework designed to evaluate the performance of AI models intended for defense and national security applications. It is part of the broader efforts to create a framework for T&E of AI models for the CDAO.

This memo builds on that foundation by:



  • Formalizing benchmarking as a standard requirement at the procurement stage across DoD acquisition processes.

  • Inserting benchmarking protocols into rapid acquisition platforms like Tradewinds.

  • Establishing a defense-specific benchmarking repository and enabling red-teaming led by the AI Rapid Capabilities Cell (AI RCC) within the CDAO.

  • Shifting the lead on benchmarking from vendor-enabled to internally developed, led, and implemented, creating bespoke evaluation criteria tailored to specific mission needs.

What types of AI systems will these benchmarks apply to, and how will they be tailored for national security use cases?

The proposed benchmarking framework will apply to a diverse range of AI systems, including:



  • Decision-making and command and control support tools (sensors, target recognition, process automation, and tools involved in natural language processing).

  • Generative models for planning, logistics, intelligence, or data generation.

  • Autonomous agents, such as drones and robotic systems.


Benchmarks will be theater and context-specific, reflecting real-world environments (e.g. contested INDOPACOM scenarios), end-user roles (human-AI teaming in combat), and mission-specific risk factors such as adversarial interference and model drift.

How will this benchmarking framework approach open-source or non-proprietary AI models intended for DoD use?

Open-source models present distinct challenges due to model ownership and origin, additional possible exposure to data poisoning, and downstream user manipulation. However, due to the nature of open-source models, it should be noted that the general increase in transparency and potential access to training data could make open-source models less challenging to put through rigorous T&E.


This memo recommends:



  • Applying standardized evaluation criteria across both open-source and proprietary models which can be developed by utilizing the AI benchmarking repository and applying model evaluations based on possible use cases of the model.

  • Incorporating benchmarking to test possible areas of vulnerability for downstream user manipulation.

  • Measuring the transparency of training data.

  • Performing adversarial testing to assess resilience against manipulated inputs via red-teaming.

  • Logging the open-source model performance in the proposed centralized repository, enabling ongoing monitoring for drift and other issues

Why is red-teaming a necessity in addition to AI benchmarking, and how will it be executed?

Red-teaming implements adversarial stress-testing (which can be more robust and operationally relevant if led by an internal team as this memo proposes), and can identify vulnerabilities and unintended capabilities before deployment. Internally led red-teaming, in particular, is critical for evaluating models intended for use in unpredictable or hostile environments.

How will red-teaming be executed?

To effectively employ the red-teaming efforts, this policy recommends that:



  • The AI Rapid Capabilities Cell within the CDAO should lead red-teaming operations, leveraging the team’s technical capabilities with its experience and mission set to integrate and rapidly scale AI at the speed of relevance — delivering usable capability fast enough to affect current operations and decision cycles.

  • Internal, technically skilled teams should be created who are capable of incorporating classified threat models and edge-case scenarios.

  • Red-teaming should focus on simulating realistic mission conditions, and searching for specific model capabilities, going beyond generic or vendor-supplied test cases.

How does this benchmarking framework improve acquisition decisions and reduce risks?

Integrating benchmarking at the acquisition stage enables procurement officers to:



  • Compare models on mission-relevant, standardized performance metrics and ensure that there is evidence of measurable performance metrics which align with their own “vision of success” procurement requirements for the models.

  • Identify and avoid models with unsafe, misaligned, unverified, or ineffective capabilities.

  • Prevent cost-overruns or contract revisions.


Benchmarking workshops for acquisition officers can further equip them with the skills to interpret benchmark results and apply them to their operational requirements.

Develop a Risk Assessment Framework for AI Integration into Nuclear Weapons Command, Control, and Communications Systems

As the United States overhauls nearly every element of its strategic nuclear forces, artificial intelligence is set to play a larger role—initially in early‑warning sensors and decision‑support tools, and likely in other mission areas. Improved detection could strengthen deterrence, but only if accompanying hazards—automation bias, model hallucinations, exploitable software vulnerabilities, and the risk of eroding assured second‑strike capability—are well managed. 

To ensure responsible AI integration, the Office of the Assistant Secretary of Defense for Nuclear Deterrence, Chemical, and Biological Defense Policy and Programs (OASD (ND-CBD)), the U.S. Strategic Command (STRATCOM), the Defense Advanced Research Projects Agency (DARPA), the Office of the Undersecretary of Defense for Policy (OUSD(P)), and the National Nuclear Security Administration (NNSA), should jointly develop a standardized AI risk-assessment framework guidance document, with implementation led by the Department of Defense’s Chief Digital and Artificial Intelligence Office (CDAO) and STRATCOM. Furthermore, DARPA and CDAO should join the Nuclear Weapons Council to ensure AI-related risks are systematically evaluated alongside traditional nuclear modernization decisions. 

Challenge and Opportunity 

The United States is replacing or modernizing nearly every component of its strategic nuclear forces, estimated to cost at least $1.7 trillion over the next 30 years. This includes its:

Simultaneously, artificial intelligence (AI) capabilities are rapidly advancing and being applied across the national security enterprise, including nuclear weapons stockpile stewardship and some components of command, control, and communications (NC3) systems, which encompass early warning, decision-making, and force deployment components.

The NNSA, responsible for stockpile stewardship, is increasingly integrating AI into its work. This includes using AI for advanced modeling and simulation of nuclear warheads. For example, by creating a digital twin of existing weapons systems to analyze aging and performance issues, as well as using AI to accelerate the lifecycle of nuclear weapons development. Furthermore, NNSA is leading some aspects of the safety testing and systematic evaluations of frontier AI models on behalf of the U.S. government, with a specific focus on assessing nuclear and radiological risk.

Within the NC3 architecture, a complex “system of systems” with over 200 components, simpler forms of AI are already being used in areas including early‑warning sensors, and may be applied to  decision‑support tools and other subsystems as confidence and capability grow. General Anthony J. Cotton—who leads STRATCOM, the combatant command that directs America’s global nuclear forces and their command‑and‑control network—told a 2024 conference that STRATCOM is “exploring all possible technologies, techniques, and methods” to modernize NC3. Advanced AI and data‑analytics tools, he said, can sharpen decision‑making, fuse nuclear and conventional operations, speed data‑sharing with allies, and thus strengthen deterrence. General Cotton added that research must also map the cascading risks, emergent behaviors, and unintended pathways that AI could introduce into nuclear decision processes.

Thus, from stockpile stewardship to NC3 systems, AI is likely to be integrated across multiple nuclear capabilities, some potentially stabilizing, others potentially highly destabilizing. For example, on the stabilizing effects, AI could enhance early warning systems by processing large volumes of satellite, radar, and other signals intelligence, thus providing more time to decision-makers. On the destabilizing side, the ability for AI to detect or track other countries’ nuclear forces could be destabilizing, triggering an expansionary arms race if countries doubt the credibility of their second-strike capability. Furthermore, countries may misinterpret each other’s nuclear deterrence doctrines or have no means of verification of human control of their nuclear weapons.

While several public research reports have been conducted on how AI integration into NC3 could upset the balance of strategic stability, less research has focused on the fundamental challenges with AI systems themselves that must be accounted for in any risk framework. Per the National Institute of Standards and Technology’s (NIST) AI Risk Management Framework, several fundamental AI challenges at a technical level must be accounted for in the integration of AI into stockpile stewardship and NC3.

Not all AI applications within the nuclear enterprise carry the same level of risk. For example, using AI to model warhead aging in stockpile stewardship is largely internal to the Department of Energy (DOE) and involves less operational risk. Despite lower risk, there is still potential for an insufficiently secure model to lead to leaked technical data about nuclear weapons. 

However, integrating AI into decision support systems or early warning functions within NC3 introduces significantly higher stakes. These systems require time-sensitive, high-consequence judgments, and AI integration in this context raises serious concerns about issues including confabulations, human-AI interactions, and information security: 

This is not an exhaustive list of issues with AI systems, however it highlights several key areas that must be managed. A risk framework must account for these distinctions and apply stricter oversight where system failure could have direct consequences for escalation or deterrence credibility. Without such a framework, it will be challenging to harness the benefits AI has to offer.

Plan of Action 

Recommendation 1. OASD (ND-CBD),  STRATCOM, DARPA, OUSD(P), and NNSA, should develop a standardized risk assessment framework guidance document to evaluate the integration of artificial intelligence into nuclear stockpile stewardship and NC3 systems. 

This framework would enable systematic evaluation of risks, including confabulations, human-AI configuration, and information security, across modernization efforts. The framework could assess the extent to which an AI model is prone to confabulations, involving performance evaluations (or “benchmarking”) under a wide range of realistic conditions. While there are public measurements for confabulations, it is essential to evaluate AI systems on data relevant to the deployment circumstances, which could involve highly sensitive military information.  

Additionally, the framework could assess human-AI configuration with specific focus on risks from automation bias and the degree of human oversight. For these tests, it is important to put the AI systems in contact with human operators in situations that are as close to real deployment as possible, for example when operators are tired, distracted, or under pressure. 

Finally, the framework could include assessments of information security under extreme conditions. This should include simulating comprehensive adversarial attacks (or “red-teaming”) to understand how the AI system and its human operators behave when subject to a range of known attacks on AI systems.

NNSA should be included in this development due to their mission ownership of stockpile stewardship and nuclear safety, and leadership in advanced modeling and simulation capabilities. DARPA should be included due to its role as the cutting edge research and development agency, extensive experience in AI red-teaming, and understanding of the AI vulnerabilities landscape. STRATCOM must be included as the operational commander of NC3 systems, to ensure the framework accounts for real-word needs and escalation risks. OASD (ND-CBD) should be involved given the office’s responsibilities to oversee nuclear modernization and coordinate across the interagency. The OUSD (P) should be included to provide strategic oversight and ensure the risk assessment aligns with broader defense policy objectives and international commitments.

Recommendation 2. CDAO should implement the Risk Assessment Framework with STRATCOM

While NNSA, DARPA, OASD (ND-CBD) and STRATCOM can jointly create the risk assessment framework, CDAO and STRATCOM should serve as the implementation leads for utilizing the framework. Given that the CDAO is already responsible for AI assurance, testing and evaluation, and algorithmic oversight, they would be well-positioned to work with relevant stakeholders to support implementation of the technical assessment. STRATCOM would have the strongest understanding of operational contexts with which to apply the framework. NNSA and DARPA therefore could advise on technical underpinnings with regards to AI of the framework, while the CDAO would prioritize operational governance and compliance, ensuring that there are clear risk assessments completed and understood when considering integration of AI into nuclear-related defense systems.

Recommendation 3. DARPA and CDAO should join the Nuclear Weapons Council 

Given their roles in the creation and implementation of the AI risk assessment framework, stakeholders from both DARPA and the CDAO should be incorporated into the Nuclear Weapons Council (NWC), either as full members or attendees to a subcommittee. As the NWC is the interagency body the DOE and the DoD responsible for sustaining and modernizing the U.S. nuclear deterrent, the NWC is responsible for endorsing military requirements, approving trade-offs, and ensuring alignment between DoD delivery systems and NNSA weapons. 

As AI capabilities become increasingly embedded in nuclear weapons stewardship, NC3 systems, and broader force modernization, the NWC must be equipped to evaluate associated risks and technological implications. Currently, the NWC is composed of senior officials from the Department of Defense, the Joint Chiefs of Staff, and the Department of Energy, including the NNSA. While these entities bring deep domain expertise in nuclear policy, military operations, and weapons production, the Council lacks additional representation focused on AI.

DARPA’s inclusion would ensure that early-stage technology developments and red-teaming insights are considered upstream in decision-making. Likewise, CDAO’s presence would provide continuity in AI assurance, testing, and digital system governance across operational defense components. Their participation would enhance the Council’s ability to address new categories of risk, such as model confabulation, automation bias, and adversarial manipulation of AI systems, that are not traditionally covered by existing nuclear stakeholders. By incorporating DARPA and CDAO, the NWC would be better positioned to make informed decisions that reflect both traditional nuclear considerations and the rapidly evolving technological landscape that increasingly shapes them.

Conclusion 

While AI is likely to be integrated into components of the U.S. nuclear enterprise, without a standardized initial approach to assessing and managing AI-specific risk, including confabulations, automation bias, and novel cybersecurity threats, this integration could undermine an effective deterrent. A risk assessment framework coordinated by OASD (ND-CBD), with STRATCOM, NNSA and DARPA, and implemented with support of the CDAO, could provide a starting point for NWC decisions and assessments of the alignment between DoD delivery system needs, the NNSA stockpile, and NC3 systems.

This memo was written by an AI Safety Policy Entrepreneurship Fellow over the course of a six-month, part-time program that supports individuals in advancing their policy ideas into practice. You can read more policy memos and learn about Policy Entrepreneurship Fellows here.

Frequently Asked Questions
Does the NWC have the authority to create a new subcommittee including DARPA and the CDAO?

Yes, NWC subordinate organizations or subcommittees are not codified in Title 10 USC §179, so the NWC has the flexibility to create, merge, or abolish organizations and subcommittees as needed.

Are there existing regulations that the United States has declared with respect to AI integration into NC3?

Section 1638 of the FY2025 National Defense Authorization Act established a Statement of Policy emphasizing that any use of AI in support of strategic deterrence should not compromise, “the principle of requiring positive human actions in execution of decisions by the President with respect to the employment of nuclear weapons.” However, as this memo describes, AI presents further challenges outside of solely keeping a human in the loop in terms of decision-making.

A National Center for Advanced AI Reliability and Security

While AI’s transformative advances have enormous positive potential, leading scientists and industry executives are also sounding the alarm about catastrophic risks on a global scale. If left unmanaged, these risks could undermine our ability to reap the benefits of AI progress. While the U.S. government has made some progress, including by establishing the Center for AI Standards and Innovation (CAISI)—formerly the US AI Safety Institute—current government capacity is insufficient to respond to these extreme frontier AI threats. To address this problem, this memo proposes scaling up a significantly enhanced “CAISI+” within the Department of Commerce. CAISI+ would require dedicated high-security compute facilities, specialized talent, and an estimated annual operating budget of $67-155 million, with a setup cost of $155-275 million. CAISI+ would have expanded capacity for conducting advanced model evaluations for catastrophic risks, provide direct emergency assessments to the President and National Security Council (NSC), and drive critical AI reliability and security research, ensuring America is prepared to lead on AI and safeguard its national interests.

Challenge and Opportunity 

Frontier AI is advancing rapidly toward powerful general-purpose capabilities. While this progress has produced widely useful products, it is also generating significant security risks. Recent evaluations on Anthropic’s Claude Opus 4 model were unable to rule out the risk that the model could be used to advise novice actors to produce bioweapons, triggering additional safeguards. Meanwhile, the FBI warns that AI “increases cyber-attack speed, scale, and automation”, with a 442% increase in AI-enhanced voice phishing attacks in 2024, and recent evaluations showing AI models rapidly gaining offensive cyber capabilities. 

AI company CEOs and leading researchers have predicted that this progress will continue, with potentially transformative AI capabilities arriving in the next few years–and fast progress in AI capabilities will continue to generate novel threats greater than those from existing models. As AI systems are predicted to become increasingly capable of performing complex tasks and taking extended autonomous actions, researchers warn of these additional risks, such as loss of human control, AI-enabled WMD proliferation, and strategic surprise with severe national security implications. While timelines to AI systems surpassing dangerous capability thresholds are uncertain, this proposal attempts to lay out a US government response that is robust to a range of possible timelines, while taking the above trends seriously.

Current U.S. Government capabilities, including the existing Center for AI Standards and Innovation (CAISI), are not adequately resourced or empowered to independently evaluate, monitor, or respond to the most advanced AI threats. For example, current CAISI funding is precarious, its home institution (NIST)’s offices are reportedly “crumbling”, and its budget is roughly one-tenth of its counterpart in the UK. Despite previous underinvestment, CAISI has consistently produced rigorous model evaluations, and in doing so, has earned strong credibility with industry and government stakeholders. This also includes support from legislators: bipartisan legislation has been introduced in both chambers of Congress to authorize CAISI in statute, while just last month, the House China Committee released a letter noting that CAISI has a role to play in “understanding, predicting, and preparing for” national security risks from AI development in the PRC.

A dedicated and properly resourced national entity is essential for supporting the development of safe, secure, and trustworthy AI to drive widespread adoption, by providing sustained, independent technical assessments and emergency coordination—roles that ad-hoc industry consultations or self-reporting cannot fulfill for paramount matters of national security and public safety.

Establishing CAISI+ now is a critical opportunity to proactively manage these profound risks, ensure American leadership in AI, and prevent strategic disadvantage as global AI capabilities advance. While full operational capacity may not be needed immediately, certain infrastructure, such as highly secure computing, has significant lead times, demanding foresight and preparatory action. This blueprint offers a scalable framework to build these essential national capabilities, safeguarding our future against AI-related catastrophic events and enabling the U.S. to shape the trajectory of this transformative technology.

Plan of Action 

To effectively address extreme AI risks, develop more trustworthy AI systems, and secure U.S. interests, the Administration and Congress should collaborate to establish and resource a world-class national entity to inform the federal response to the above trendlines. 

Recommendation 1. Establish CAISI+ to Lead National AI Safety and Coordinate Crisis Response. 

CAISI+, evolving from the current CAISI within the National Institute of Standards and Technology, under the Department of Commerce, must have a clear mandate focused on large-scale AI risks. Core functions include: 

Governance will feature clear interagency coordination (e.g., with the Department of Defense, Department of Energy, Department of Homeland Security, and other relevant bodies in the intelligence community) and an internal structure with distinct directorates for evaluations, emergency response, and research, coordinated by CAISI+ leadership.

Recommendation 2. Equip CAISI+ with Elite American Talent and Sustained Funding

CAISI+’s efficacy hinges on world-class personnel and reliable funding to execute its mission. This necessitates: 

  1. Exceptional American Talent: Special hiring authorities (e.g., direct hire, excepted service) and competitive compensation are paramount to attract and retain leading U.S. AI researchers, evaluators, and security experts, ensuring our AI standards reflect American values.
  2. Significant, Sustained Funding: Initial mainline estimates (see “Funding estimates for CAISI+” below) suggest $155-$275 million for setup and an annual operating budget of $67-$155 million for the recommended implementation level, sourced via new appropriations, to ensure America develops strong domestic capacity for defending against AI-powered threats. If funding is not appropriated, or if appropriations fall short, additional support may be able to be sourced via a NIST Foundation.

Recommendation 3. Equip CAISI+ with Essential Secure Compute Infrastructure. 

CAISI+ must be able to access secure compute in order to run certain evaluations involving proprietary models and national security data. This cluster can remain relatively modest in scale. Other researchers have hypothesized that a “Trusted AI Verification and Evaluation Cluster” for verifying and evaluating frontier AI development would need only 128 to 512 state-of-the-art graphical processing units (GPU)s–orders of magnitude smaller than the scale of training compute, such as the recent Llama 3.1 405 B model’s training run use of a 16,000 H100 GPU cluster, or xAI’s 200,000 GPU Colossus cluster. 

However, the cluster will need to be highly secure–in other words, able to defend against attacks from nation-state adversaries. Certain evaluations will require full access to the internal “weights” of AI models, which requires hosting the model. Model hosting introduces the risk of model theft and proliferation of dangerous capabilities. Some evaluations will also involve the use of very sensitive data, such as nuclear weapons design evals–introducing additional incentive for cyberattacks. Researchers at Gladstone AI, a national security-focused AI policy consulting firm, write that in several years, powerful AI systems may confer significant strategic advantages to nation-states, and will therefore be top-priority targets for theft or sabotage by adversary nation-states. They also note that neither existing datacenters nor AI labs are secure enough to prevent this theft–thereby necessitating novel research and buildout to reach the necessary security level, outlined as “Security Level-5” (SL-5) in RAND’s Playbook for Securing AI Model Weights.

Therefore, we suggest a hybrid strategy for specialized secure compute, featuring a highly secure SL-5 air-gapped core facility for sensitive model analysis (a long-lead item requiring immediate planning), with access to a secondary pool of compute for additional capacity to run less sensitive evaluations via a formal partnership with DOE to access national lab resources. CAISI+ may also want to coordinate with the NITRD National Strategic Computing Reserve Pilot Program to explore needs for AI-crisis-related surge computing capability.

If a sufficiently secure compute cluster is infeasible or not developed in time, CAISI+ will ultimately be unable to host model internals without introducing unacceptable risks of model theft, severely limiting its ability to evaluate frontier AI systems.

Recommendation 4. Explore Granting Critical Authorities 

While current legal authorities may suffice for CAISI+’s core missions, evolving AI threats could require additional tools. The White House (specifically the Office of Science and Technology Policy [OSTP], in collaboration with the Office of Management and Budget [OMB]) should analyze existing federal powers (such as the Defense Production Act or the International Emergency Economic Powers Act) to identify gaps in AI threat response capabilities–including potential needs for an incident reporting system and related subpoena authorities (similar to the function of the National Transportation Safety Board), or for model access for safety evaluations, or compute oversight authorities. Based on this analysis, the executive branch should report to Congress where new statutory authorities may be necessary, with defined risk criteria and appropriate safeguards.

Recommendation 5. Implement CAISI+ Enhancements Through Urgent, Phased Approach

Building on CAISI’s existing foundation within NIST/DoC, the Administration should enhance its capabilities to address AI risks that extend beyond current voluntary evaluation frameworks. Given expert warnings that transformative AI could emerge within the current Administration’s term, immediate action is essential to augment CAISI’s capacity to handle extreme scenarios. To achieve full operational capacity by early 2027, initial-phase activities must begin now  due to long infrastructure lead times:

Immediate Enhancements (0-6 months):

  1. Leverage NIST’s existing relationships with DOE labs to secure interim access to classified computing facilities for sensitive evaluations
  2. Initiate the security research and procurement process for the SL-5 compute facility outlined in Recommendation 3
  3. Work with OMB and Department of Commerce leadership to secure initial funding through reprogramming or supplemental appropriations
  4. Build on CAISI’s current voluntary agreements to develop protocols for emergency model access and crisis response
  5. Begin the OSTP-led analysis of existing federal authorities (per Recommendation 4) to identify potential gaps in AI threat response capabilities

Subsequent phases will extend CAISI’s current work through:

Conclusion

Enhancing and empowering CAISI+ is a strategic investment in U.S. national security, far outweighed by the potential costs of inaction on this front. With an estimated annual operating budget of $67-155 million, CAISI+ will provide essential technical capabilities to evaluate and respond to the most serious AI risks, ensuring the U.S. leads in developing and governing AI safely and securely, irrespective of where advanced capabilities emerge. While timelines to AI systems surpassing dangerous capability thresholds are uncertain, by acting now to establish the necessary infrastructure, expertise, and authorities, the Administration can safeguard American interests and our technological future through a broad range of possible scenarios.

This memo was written by an AI Safety Policy Entrepreneurship Fellow over the course of a six-month, part-time program that supports individuals in advancing their policy ideas into practice. You can read more policy memos and learn about Policy Entrepreneurship Fellows here.

A Grant Program to Enhance State and Local Government AI Capacity and Address Emerging Threats

States and localities are eager to leverage artificial intelligence (AI) to optimize service delivery and infrastructure management, but they face significant resource gaps. Without sufficient personnel and capital, these jurisdictions cannot properly identify and mitigate the risks associated with AI adoption, including cyber threats, surging power demands, and data privacy issues. Congress should establish a new grant program, coordinated by the Cybersecurity and Infrastructure Security Agency (CISA), to assist state and local governments in addressing these challenges. Such funding will allow the federal government to instill best security and operating practices nationwide, while identifying effective strategies from the grassroots that can inform federal rulemaking. Ultimately, federal, state, and local capacity are interrelated; federal investments in state and local government will help the entire country harness AI’s potential and reduce the risk of catastrophic events such as a large, AI-powered cyberattack.

Challenge and Opportunity 

In 2025, 45 state legislatures have introduced more than 550 bills focused on the regulation of artificial intelligence, covering everything from procurement guidelines to acceptable AI uses in K-12 education to liability standards for AI misuse and error. Major cities have followed suit with sweeping guidance of their own, identifying specific AI risks related to bias and hallucination and directives to reduce their impact on government functions. The influx of regulatory action reflects burgeoning enthusiasm about AI’s ability to streamline public services and increase government efficiency.

Yet two key roadblocks stand in the way: inconsistent rules and uneven capacity. AI regulations vary widely across jurisdictions — sometimes offering contradictory guidance — and public agencies often lack the staff and skills needed to implement them. In a 2024 survey, six in ten public sector professionals cited the AI skills gap as their biggest obstacle in implementing AI tools. This reflects a broader IT staffing crisis, with over 450,000 unfilled cybersecurity roles nationwide, which is particularly acute in the public sector given lower salaries and smaller budgets.

These roadblocks at the state and local level pose a major risk to the entire country. In the cyber space, ransomware attacks on state and local targets have demonstrated that hackers can exploit small vulnerabilities in legacy systems to gain broad access and cause major disruption, extending far beyond their initial targets. The same threat trajectory is conceivable with AI. States and cities, lacking the necessary workforce and adhering to a patchwork of different regulations, will find themselves unable to safely adopt AI tools and mount a uniform response in an AI-related crisis. 

In 2021, Congress established the State and Local Cybersecurity Grant Program (SLCGP) at CISA, which focused on resourcing states, localities, and tribal territories to better respond to cyber threats. States have received almost $1 billion in funding to implement CISA’s security best practices like multifactor authentication and establish cybersecurity planning committees, which effectively coordinate strategic planning and cyber governance among state, municipal, and private sector information technology leaders. 

Federal investment in state and local AI capacity-building can help standardize the existing, disparate guidance and bridge resource gaps, just as it has in the cybersecurity space. AI coordination is less mature today than the cybersecurity space was when the SLCGP was established in 2021. The updated Federal Information Security Modernization Act, which enabled the Department of Homeland Security to set information security standards across government, had been in effect for seven years by 2021, and some of its best practices had already trickled down to states and localities. 

Thus, the need for clear AI state capacity, guardrails, and information-sharing across all levels of government is even greater. A small federal investment now can unlock large returns by enabling safe, effective AI adoption and avoiding costly failures. Local governments are eager to deploy AI but lack the resources to do so securely. Modest funding can align fragmented rules, train high-impact personnel, and surface replicable models—lowering the cost of responsible AI use nationwide. Each successful pilot creates a multiplier effect, accelerating progress while reducing risk.

Plan of Action 

Recommendation 1. Congress should authorize a three-year pilot grant program focused on state and local AI capacity-building.

SLCGP’s authorization expires on August 31, 2025, which provides two unique pathways for a pilot grant program. The Homeland Security Committees in the House and Senate could amend and renew the existing SLCGP provision to make room for an AI-focused pilot. Alternatively, Congress could pass a new authorization, which would likely set the stage for a sustained grant program, upon successful completion of the pilot. A separate authorization would also allow Congress to consider other federal agencies as program facilitators or co-facilitators, in case they want to cover AI integrations that do not directly touch critical infrastructure, which is CISA’s primary focus. 

Alternatively, the House Energy and Commerce and Senate Commerce, Science, and Transportation Committees could authorize a program coordinated by the National Institute of Standards and Technology, which produced the AI Risk Management Framework and has strong expertise in a range of vulnerabilities embedded within AI models. Congress might also consider mandating an interagency advisory committee to oversee the program, including, for example, experts from the Department of Energy to provide technical assistance and guidance on projects related to energy infrastructure.

In either case, the authorization should be coupled with a starting appropriation of $55 million over three years, which would fund ten statewide pilot projects totaling up to $5 million plus administrative costs. The structure of the program will broadly parallel SLCGP’s goals. First, it would align state and local AI approaches with existing federal guidance, such as the NIST AI Risk Management Framework and the Trump Administration’s OMB guidance on the regulation and procurement of artificial intelligence applications. Second, the program would establish better coordination between local and state authorities on AI rules. A new authorization for AI, however, allows Congress and the agency tasked with managing the program the opportunity to improve upon SLCGP’s existing provisions. This new program should permit states to coordinate their AI activities through existing leadership structures rather than setting up a new planning committee. The legislative language should also prioritize skills training and allocate a portion of grant funding to be spent on recruiting and retaining AI professionals within state and local government who can oversee projects.

Recommendation 2. Pilot projects should be implementation-focused and rooted in one of three significant risks: cybersecurity, energy usage, or data privacy.

Similar to SLCGP, this pilot grant program should be focused on implementation. The target product for a grant is a functional local or state AI application that has undergone risk mitigation, rather than a report that identifies issues in the abstract. For example, under this program, a state would receive federal funding to integrate AI into the maintenance of its cities’ wastewater treatment plants without compromising cybersecurity. Funding would support AI skills training for the relevant municipal employees and scaling of certain cybersecurity best practices like data encryption that minimize the project’s risk. States will submit reports to the federal government at each phase of their project: first documenting the risks they identified, then explaining their prioritization of risks to mitigate, then walking through their specific mitigation actions, and later, retrospectively reporting on the outcomes of those mitigations after the project has gone into operational use.

This approach would maximize the pilot’s return on investment. States will be able to complete high-impact AI projects without taking on the associated security costs. The frameworks generated from the project can be reused many times over for later projects, as can the staff who are hired or trained with federal support. 

Given the inconsistency of priorities surfaced in state and local AI directives, the federal government should set the agenda of risks to focus on. The clearest set of risks for the pilot are cybersecurity, energy usage, and data privacy, all of which are highlighted in NIST’s Risk Management Framework. 

If successful, the pilot could expand to address additional risks or support broader, multi-risk, multi-state interventions.

Recommendation 3. The pilot program must include opportunities for grantees to share their ideas with other states and localities.

Arguably the most important facet of this new AI program will be forums where grantees share their learnings. Administrative costs for this program should go toward funding a twice-yearly (bi-annual) in-person forum, where grantees can publicly share updates on their projects. An in-person forum would also provide states with the space to coordinate further projects on the margins. CISA is particularly well positioned to host a forum like this given its track record of convening critical infrastructure operators. Grantees should be required to publish guidance, tools, and templates in a public, digital repository. Ideally, states that did not secure grants can adopt successful strategies from their peers and save taxpayers the cost of duplicate planning work. 

Conclusion 

Congress should establish a new grant program to assist state and local governments in addressing AI risks, including cybersecurity, energy usage, and data privacy. Such federal investments will give structure to the dynamic yet disparate national AI regulatory conversation. The grant program, which will cost $55 million to pilot over three years, will yield a high return on investment for both the ten grantee states and the peers that learn from its findings. By making these investments now, Congress can keep states moving fast toward AI without opening the door to critical, costly vulnerabilities.

This memo was written by an AI Safety Policy Entrepreneurship Fellow over the course of a six-month, part-time program that supports individuals in advancing their policy ideas into practice. You can read more policy memos and learn about Policy Entrepreneurship Fellows here.

Frequently Asked Questions
Does Congress have to authorize a new grant program to operate this pilot?

No, Congress could leverage SLCGP’s existing authorization to focus on projects that look at the intersection of AI and cybersecurity. They could offer an amendment to the next Homeland Security Appropriations package that directs modest SLCGP funding (e.g. $10-20 million) to AI projects. Alternatively, Congress could insert language on AI into SLCGP’s reauthorization, which is due on August 31, 2025.


Although leveraging the existing authorization would be easier, Congress would be better served by authorizing a new program, which can focus on multiple priorities including energy usage and data privacy. To stay agile, the language in the statute could allow CISA to direct funds toward new emerging risks, as they are identified by NIST and other agencies. Finally, a specific authorization would pave the way for an expansion of this program assuming the initial 10 state pilot goes well.

Why focus on individual state and local projects rather than an across-the-board effort to improve capacity in all states across all vectors?

This pilot is right-sized for efficiency, impact, and cost savings. A program to bring all 50 states into compliance with certain AI risk mitigation guidelines would cost hundreds of millions, which is not feasible in the current budgetary environment. States are starting from very different baselines, especially with their energy infrastructure, which makes it difficult to bring them all to a single end-point. Moreover, because AI is evolving so rapidly, guidance is likely to age poorly. The energy needs of AI might change before states finish their plan to build data centers. Similarly, federal data privacy laws might go in place that undercut or contradict the best practices established by this program.

What are the benefits to this deployment approach?

This pilot will allow 10 states and/or localities to quickly deploy AI implementations that produce real value: for example, quicker emergency response times and savings on infrastructure maintenance. CISA can learn from the grantees’ experiences to iterate on federal guidance. They might identify a stumbling block on one project and refine their guidance to prevent 49 other states from encountering the same obstacle. If grantees effectively share their learnings, they can cut massive amounts of time off other states’ planning processes and help the federal government build guidance that is more rooted in the realities of AI deployment.

Some have expressed concerns that planning-focused grants create additional layers of bureaucracy. Will this pilot just add more red tape to AI integration?

No. If done correctly, this pilot will cut red tape and allow the entire country to harness AI’s positive potential. States and localities are developing AI regulations in a vacuum. Some of the laws proposed are contradictory or duplicative precisely because many state legislatures are not coordinating effectively with state and local government technical experts. When bills do pass, guidance is often poorly implemented because there is no overarching figure, beyond a state chief information officer, to bring departments and cities into compliance. In essence, 50 states are producing 50 sets of regulations because there is scant federal guidance and few mechanisms for them to learn from other states and coordinate within their state on best practices.

How will this program streamline and optimize state and local AI planning processes?

This program aims to cut down on bureaucratic redundancy by leveraging states’ existing cyber planning bodies to take a comprehensive approach to AI. By convening the appropriate stakeholders from the public sector, private sector, and academia to work on a funded AI project, states will develop more efficient coordination processes and identify regulations that stand in the way of effective technological implementation. States and localities across the country will build their guidelines based on successful grantee projects, absorbing best practices and casting aside inefficient rules. It is impossible to mount a coordinated response to significant challenges like AI-enabled cyberattacks without some centralized government planning, but this pilot is designed to foster efficient and effective coordination across federal, state, and local governments.

Accelerating AI Interpretability To Promote U.S. Technological Leadership

The most advanced AI systems remain ‘black boxes’ whose inner workings even their developers cannot fully understand, leading to issues with reliability and trustworthiness. However, as AI systems become more capable, there is a growing desire to deploy them in high-stakes scenarios. The bipartisan National Security Commission on AI cautioned that AI systems perceived as unreliable or unpredictable will ‘stall out’: leaders will not adopt them, operators will mistrust them, Congress will not fund them, and the public will not support them (NSCAI, Final Report, 2021). AI interpretability research—the science of opening these black boxes and attempting to comprehend why they do what they do—could turn opacity into understanding and enable wider AI adoption.

With AI capabilities racing ahead, the United States should accelerate interpretability research now to keep its technological edge and field high-stakes AI deployment with justified confidence. This memorandum describes three policy recommendations that could help the United States seize the moment and maintain a lead on AI interpretability: (1) creatively investing in interpretability research, (2) entering into research and development agreements between interpretability experts and government agencies and laboratories, and (3) prioritizing interpretable AI in federal procurement. 

Challenge and Opportunity

AI capabilities are progressing rapidly. According to many frontier AI companies’ CEOs and independent researchers, AI systems could reach general-purpose capabilities that equal or even surpass humans within the next decade. As capabilities progress, there is a growing desire to incorporate these systems into high-stakes use cases, from military and intelligence uses (DARPA, 2025; Ewbank, 2024) to key sectors of the economy (AI for American Industry, 2025). 

However, the most advanced AI systems are still ‘black boxes’ (Sharkey et al., 2024) that we observe from the outside and that we ‘grow,’ more than we ‘build’ (Olah, 2024). Our limited comprehension of the inner workings of neural networks means that we still really do not understand what happens within these black boxes, leaving uncertainty regarding their safety and reliability. This could have resounding consequences. As the 2021 final report of the National Security Commission on AI (NSCAI) highlighted, “[i]f AI systems routinely do not work as designed or are unpredictable in ways that can have significant negative consequences, then leaders will not adopt them, operators will not use them, Congress will not fund them, and the American people will not support them” (NSCAI, Final Report, 2021). In other words, if AI systems are not always reliable and secure, this could inhibit or limit their adoption, especially in high-stakes scenarios, potentially compromising the AI leadership and national security goals outlined in the Trump administration’s agenda (Executive Order, 2025). 

AI interpretability is a subfield of AI safety that is specifically concerned with opening and peeking inside the black box to comprehend “why AI systems do what they do, and … put this into human-understandable terms” (Nanda, 2024; Sharkey et al., 2025). In other words, interpretability is the AI equivalent of an MRI (Amodei, 2025) because it attempts to provide observers with an understandable image of the hidden internal processes of AI systems.

The Challenge of Understanding AI Systems Before They Reach or Even Surpass Human-Level Capabilities

Recent years have brought breakthroughs across several research areas focused on making AI more trustworthy and reliable, including in AI interpretability. Among other efforts, the same companies developing the most advanced AI systems have designed systems that are easier to understand and have reached new research milestones (Marks et al., 2025; Lindsey et al., 2025; Lieberum et al. 2024; Kramar et al., 2024; Gao et al., 2024; Tillman & Mossing, 2025). 

AI interpretability, however, is still trailing behind raw AI capabilities. AI companies project that it could take 5–10 years to reliably understand model internals (Amodei, 2025), while experts expect systems exhibiting human‑level general-purpose capabilities by as early as 2027 (Kokotajlo et al., 2025). That gap will force policymakers into a difficult corner once AI systems reach similar capabilities: deploy unprecedentedly powerful yet opaque systems, or slow deployment and fall behind. Unless interpretability accelerates, the United States could risk both competitive and security advantages. 

The Challenge of Trusting Today’s Systems for High-Stakes Applications

We must understand the inner workings of highly advanced AI systems before they reach human or above-human general-purpose capabilities, especially if we want to trust them in high-stakes scenarios. There are several reasons why current AI systems might not always be reliable and secure. For instance, AI systems could exhibit the following vulnerabilities. First, AI systems inherit the blind spots of their training data. When the world changes—alliances shift, governments fall, regulations update—systems still reason from outdated facts, undermining reliability in high-stakes diplomatic or military settings (Jensen et al., 2025).

Second, AI systems are unusually easy to strip‑mine for memorized secrets, especially if these secrets come as uncommon word combinations (e.g., proprietary blueprints). Data‑extraction attacks are now “practical and highly realistic” and will grow even more effective as system size increases (Carlini et al., 2021; Nasr et al., 2023; Li et al., 2025). The result could be wholesale leakage of classified or proprietary information (DON, 2023).

Third, cleverly crafted prompts can still jailbreak cutting‑edge systems, bypassing safety rails and exposing embedded hazardous knowledge (Hughes et al., 2024; Ramesh et al., 2024). With attack success rates remaining uncomfortably high across even the leading systems, adversaries could manipulate AI systems with these vulnerabilities in real‑time national security scenarios (Caballero & Jenkins, 2024).

This is not a comprehensive list. Systems could exhibit vulnerabilities in high-stakes applications for many other reasons. For instance, AI systems could be misaligned and engage in scheming behavior (Meinke et al., 2024; Phuong et al., 2025) or have baked-in backdoors that an attacker could exploit (Hubinger et al., 2024; Davidson et al., 2025). 

The Opportunity to Promote AI Leadership Through Interpretability

Interpretability offers an opportunity to address these described challenges and reduce barriers to the safe adoption of the most advanced AI systems, thereby further promoting innovation and increasing the existing advantages those systems present over adversaries’ systems. In this sense, accelerating interpretability could help promote and secure U.S. AI leadership (Bau et al., 2025; IFP, 2025). For example, by helping ensure that highly advanced AI systems are deployed safely in high-stakes scenarios, interpretability could improve national security and help mitigate the risk of state and non-state adversaries using AI capabilities against the United States (NSCAI, Final Report, 2021). Interpretability could therefore serve as a front‑line defense against vulnerabilities in today’s most advanced AI systems.

Making future AI systems safe and trustworthy could become easier the more we understand how they work (Shah et al., 2025). Anthropic’s CEO recently endorsed the importance and urgency of interpretability, noting that “every advance in interpretability quantitatively increases our ability to look inside models and diagnose their problems” (Amodei, 2025). This means that interpretability not only enhances reliability in the deployment of today’s AI systems, but understanding AI systems could also lead to breakthroughs in designing more targeted systems or attaining more robust monitoring of deployed systems. This could then enable the United States to deploy tomorrow’s human-level or above-human general-purpose AI systems with increased confidence, thus securing strategic advantages when engaging  geopolitically. The following uses the vulnerabilities discussed above to demonstrate three ways in which interpretability could improve the reliability of today’s AI systems when deployed in high-stakes scenarios.

First, interpretability could help systems selectively update outdated information through model editing, without risking a reduction in performance. Model editing allows us to selectively inject new facts or fix mistakes (Cohen et al., 2023; Hase et al., 2024) by editing activations without updating the entire model. However, this ‘surgical tool’ has shown ‘side effects’ causing performance degradation (Gu et al., 2024; Gupta et al., 2024). Interpretability could help us understand how stored knowledge alters parameters as well as develop stronger memorization measures (Yao et al., 2023; Carlini et al., 2019), enabling us to ‘incise and excise’ AI models with fewer side effects.

Second, interpretability could help systems selectively forget training data through machine unlearning, once again without losing performance. Machine unlearning allows systems to forget specific data classes (such as memorized secrets or hazardous knowledge) while remembering the rest (Tarun et al., 2023). Like model editing, this ‘surgical tool’ suffers from performance degradation. Interpretability could help develop new unlearning techniques that preserve performance (Guo et al., 2024; Belrose et al., 2023; Zou et al., 2024).

Third, interpretability could help effectively block jailbreak attempts, which can only currently be discovered empirically (Amodei, 2025). Interpretability could lead to a breakthrough in understanding models’ persistent vulnerability to jailbreaking by allowing us to characterize dangerous knowledge. Existing interpretability research has already analyzed how AI models process harmful prompts (He et al., 2024; Ball et al., 2024; Lin et al., 2024; Zhou et al., 2024), and additional research could build on these initial findings

The conditions are ripe to promote technological leadership and national security through interpretability. Many of the same problems that were highlighted in the 2019 National AI R&D Strategic Plan remained the same in its 2023 update, echoing those included in NSCAI’s 2021 final report. We have made relatively little progress addressing these challenges. AI systems are still vulnerable to attacks (NSCAI, Final Report, 2021) and can still “be made do the wrong thing, reveal the wrong thing” and “be easily fooled, evaded, and misled in ways that can have profound security implications” (National AI R&D Strategic Plan, 2019). The field of interpretability is gaining some momentum among AI companies (Amodei, 2025; Shah et al., 2025; Goodfire, 2025) and AI researchers (IFP, 2025; Bau et al., 2025; FAS, 2025). 

To be sure, despite recent progress, interpretability remains challenging and has attracted some skepticism (Hendrycks & Hiscott, 2025). Accordingly, a strong AI safety strategy must include many components beyond interpretability, including robust AI evaluations (Apollo Research, 2025) and control measures (Redwood Research, 2025).

Plan of Action 

The United States has an opportunity to seize the moment and lead an acceleration of AI interpretability. The following three recommendations establish a strategy for how the United States could promptly incentivize AI interpretability research. 

Recommendation 1. The federal government should prioritize and invest in foundational AI interpretability research, which would include identifying interpretability as a ‘strategic priority’ in the 2025 update of the National AI R&D Strategic Plan. 

The National Science and Technology Council (NSTC) should identify AI interpretability as a ‘strategic priority’ in the upcoming National AI R&D Strategic Plan. Congress should then appropriate federal R&D funding for federal agencies (including DARPA and the NSF) to catalyze and support AI interpretability acceleration through various mechanisms, including grants and prizes, R&D credits, tax credits, advanced market commitments, and buyer-of-first-resort mechanisms. 

This first recommendation echoes not only the 2019 update of the National AI R&D Strategic Plan and NSCAI’s 2021 final report––which recommended allocating more federal R&D investments to advance the interpretability of Al systems (NSCAI, Final Report, 2021; National AI R&D Strategic Plan, 2019),, but also the more recent remarks by the Director of the Office of Science and Technology Policy (OSTP), according to whom we need creative R&D funding approaches to enable scientists and engineers to create new theories and put them into practice (OSTP Director’s Remarks, 2025). This recommendation is also in line with calls from AI companies, asserting that “we still need significant investment in ‘basic science’” (Shah et al., 2025).

The United States could incentivize and support AI interpretability work through various approaches. In addition to prize competitions, advanced market commitments, fast and flexible grants (OSTP Director’s Remarks, 2025; Institute for Progress, 2025), and challenge-based acquisition programs (Institute for Progress, 2025), funding mechanisms could include R&D tax credits for AI companies undertaking or investing in interpretability research, and tax credits to adopters of interpretable AI, such as downstream deployers. If the federal government acts as “an early adopter and avid promoter of American technology” (OSTP Director’s Remarks, 2025), federal agencies could also rely on buyer-of-first-resort mechanisms for interpretability platforms. 

These strategies may require developing a clearer understanding of which frontier AI companies undertake sufficient interpretability efforts when developing their most advanced systems, and which companies currently do not. Requiring AI companies to disclose how they use interpretability to test models before release (Amodei, 2025) could be helpful, but might not be enough to devise a ‘ranking’ of interpretability efforts. While potentially premature given the state of the art in interpretability, an option could be to start developing standardized metrics and benchmarks to evaluate interpretability (Mueller et al., 2025; Stephenson et al., 2025). This task could be carried out by the National Institute of Standards and Technology (NIST), within which some AI researchers have recommended creating an AI Interpretability and Control Standards Working Group (Bau et al., 2025). 

A great way to operationalize this first recommendation would be for the National Science and Technology Council (NSTC) to include interpretability as a “strategic priority” in the 2025 update of the National AI R&D Strategic Plan (RFI, 2025). These “strategic priorities” seek to target and focus AI innovation for the next 3–5 years, paying particular attention to areas of “high-risk, high-reward AI research” that the industry is unlikely to address because it may not provide immediate commercial returns (RFI, 2025). If interpretability were included as a “strategic priority,” then the Office of Management and Budget (OMB) could instruct agencies to align their budgets with the 2025 National AI R&D Strategic Plan priorities in its memorandum addressed to executive department heads. Relevant agencies, including DARPA and the National Science Foundation (NSF), would then develop their budget requests for Congress, aligning them with the 2025 National AI R&D Strategic Plan and the OMB memorandum. After Congress reviews these proposals and appropriates funding, agencies could launch initiatives that incentivize interpretability work, including grants and prizes, R&D credits, tax credits, advanced market commitments, and buyer-of-first-resort mechanisms. 

Recommendation 2. The federal government should enter into research and development agreements with AI companies and interpretability research organizations to red team AI systems applied in high-stakes scenarios and conduct targeted interpretability research. 

AI companies, interpretability organizations, and federal agencies and laboratories (such as DARPA, the NSF, and the U.S. Center for AI Standards and Innovation) should enter into research and development agreements to pursue targeted AI interpretability research to solve national security vulnerabilities identified through security-focused red teaming. 

This second recommendation takes into account the fact that the federal government possesses unique expertise and knowledge in national security issues to support national security testing and evaluation (FMF, 2025). Federal agencies and laboratories (such as DARPA, the NSF, and the U.S. Center for AI Standards and Innovation), frontier AI companies, and interpretability organizations could enter into research and development agreements to undertake red teaming of national security vulnerabilities (as, for instance, SABER which aims to assess AI-enabled battlefield systems for the DoD; SABER, 2025) and provide state-of-the-art interpretability platforms to patch the revealed vulnerabilities. In the future, AI companies could also apply the most advanced AI systems to support interpretability research. 

Recommendation 3. The federal government should prioritize interpretable AI in federal procurement, especially for high-stakes applications. 

If federal agencies are procuring highly advanced AI for high-stakes scenarios and national security missions, they should preferentially procure interpretable AI systems. This preference could be accounted for by weighing the lack of understanding of an AI system’s inner workings when calculating cost. 

This third and final recommendation provides for the interim and assumes interpretable AI systems will coexist in a ‘gradient of interpretability’ with other AI systems that are less interpretable. In that scenario, agencies procuring AI systems should give preference to AI systems that are more interpretable. One way to account for this preference would be by weighing the potential vulnerabilities of uninterpretable AI systems within calculating costs during federal acquisition analyses. This recommendation also requires establishing a defined ‘ranking’ of interpretability efforts. While defining this ranking is currently challenging, the research outlined in recommendations 1 and 2 could better position the government to measure and rank the interpretability of different AI systems. 

Conclusion 

Now is the time for the United States to take action and lead the charge on AI interpretability research. While research is never guaranteed to lead to desired outcomes or to solve persistent problems, the potential high reward—understanding and trusting future AI systems and making today’s systems more robust to adversarial attacks—justifies this investment. Not only could AI interpretability make AI safer and more secure, but it could also establish justified confidence in the prompt adoption of future systems that are as capable as or even more capable than humans, and enable the deployment of today’s most advanced AI systems to high-stakes scenarios, thus promoting AI leadership and national security. With this goal in mind, this policy memorandum recommends that the United States, through the relevant federal agencies and laboratories (including DARPA, the NSF, and the U.S. Center for AI Standards and Innovation), invest in interpretability research, form research and development agreements to red team high-stakes AI systems and undertake targeted interpretability research, and prioritize interpretable AI systems in federal acquisitions.

Acknowledgments 

I wish to thank Oliver Stephenson, Dan Braun, Lee Sharkey, and Lucius Bushnaq for their ideas, comments, and feedback on this memorandum.

This memo was written by an AI Safety Policy Entrepreneurship Fellow over the course of a six-month, part-time program that supports individuals in advancing their policy ideas into practice. You can read more policy memos and learn about Policy Entrepreneurship Fellows here.

Accelerating R&D for Critical AI Assurance and Security Technologies

The opportunities presented by advanced artificial intelligence are immense, from accelerating cutting-edge scientific research to improving key government services. However, for these benefits to be realized, both the private and public sectors need confidence that AI tools are reliable and secure. This will require R&D effort to solve urgent technical challenges related to understanding and evaluating emergent AI behaviors and capabilities, securing AI hardware and infrastructure, and preparing for a world with many advanced AI agents.

To secure global adoption of U.S. AI technology and ensure America’s workforce can fully leverage advanced AI, the federal government should take a strategic and coordinated approach to support AI assurance and security R&D by: clearly defining AI assurance and security R&D priorities; establishing an AI R&D consortium and deploying agile funding mechanisms for critical R&D areas; and establishing an AI Frontier Science Fellowship to ensure a pipeline of technical AI talent. 

Challenge and Opportunity

AI systems have progressed rapidly in the past few years, demonstrating human-level and even superhuman performance across diverse tasks. Yet, they remain plagued by flaws that produce unpredictable and potentially dangerous failures. Frontier systems are vulnerable to attacks that can manipulate them into executing unintended actions, hallucinate convincing but incorrect information, and exhibit other behaviors that researchers struggle to predict or control.

As AI capabilities rapidly advance toward more consequential applications—from medical diagnosis to financial decision-making to military systems—these reliability issues could pose increasingly severe risks to public safety and national security, while reducing beneficial uses. Recent polling shows that just 32% of Americans trust AI, and this limited trust will slow the uptake of impactful AI use-cases that could drive economic growth and enhance national competitiveness.

The federal government has an opportunity to secure America’s technological lead and promote global adoption of U.S. AI by catalyzing research to address urgent AI reliability and security challenges—challenges that align with broader policy consensus reflected in the National Security Commission on AI’s recommendations and bipartisan legislative efforts like the VET AI Act. Recent research has surfaced substantial expert consensus around priority research areas that address the following three challenges.

The first challenge involves understanding emergent AI capabilities and behaviors. As AI systems get larger, also referred to as “scaling”, they develop unexpected capabilities and reasoning patterns that researchers cannot predict, making it difficult to anticipate risks or ensure reliable performance. Addressing this means advancing the science of AI scaling and evaluations.

Scientific methods to understand emergent behaviors and capabilities

This research aims to build a scientific understanding of how AI systems learn, reason, and exhibit diverse capabilities. This involves not only studying specific phenomena like emergence and scaling but, more broadly, employing and refining evaluations as the core empirical methodology to characterize all facets of AI behavior. This includes evaluations in areas such as CBRN weapons, cybersecurity, and deception, and broader research on AI evaluations to ensure that AI systems can be accurately assessed and understood. Example work includes Wijk et al. (2024) and McKenzie et al. (2023)

The second challenge is securing AI hardware and infrastructure. AI systems require robust protection of model weights, secure deployment environments, and resilient supply chains to prevent theft, manipulation, or compromise by malicious actors seeking to exploit these powerful technologies. Addressing this means advancing hardware and infrastructure security for AI.

Hardware and infrastructure security for AI

Ensuring the security of AI systems at the hardware and infrastructure level involves protecting model weights, securing deployment environments, maintaining supply chain integrity, and implementing robust monitoring and threat detection mechanisms. Methods include the use of confidential computing, rigorous access controls, specialized hardware protections, and continuous security oversight. Example work includes Nevo et al. (2024) and Hepworth et al. (2024)

The third challenge involves preparing for a world with many AI agents—AI models that can act autonomously. Alongside their potentially immense benefits, the increasing deployment of AI agents creates critical blind spots, as agents could coordinate covertly beyond human oversight, amplify failures into system-wide cascades, and combine capabilities in ways that circumvent existing safeguards. Addressing this means advancing agent metrology, infrastructure, and security.

Agent metrology, infrastructure, and security

Developing a deeper understanding of agentic behavior in LLM-based systems, including clarifying how LLM agents learn over time, respond to underspecified goals, and engage with their environments. This also includes research that ensures safe multi-agent interactions, such as detecting and preventing malicious collective behaviors, studying how transparency can affect agent interactions, and developing evaluations for agent behavior and interaction. Example work includes Lee and Tiwari (2024) and Chan et al. (2024)

While academic and industry researchers have made progress on these problems, this progress is not keeping pace with AI development and deployment. The market is likely to underinvest in research that is more experimental or with no immediate commercial applications. The U.S. government, as the R&D lab of the world, has an opportunity to unlock AI’s transformative potential through accelerating assurance and security research.

Plan of Action

The rapid pace of AI advancement demands a new strategic, coordinated approach to federal R&D for AI assurance and security.  Given financial constraints, it is more important than ever to make sure that the impact of every dollar invested in R&D is maximized.

Much of the critical technical expertise now resides in universities, startups, and leading AI companies rather than traditional government labs. To harness this distributed talent, we need R&D mechanisms that move at the pace of innovation, leverage academic research excellence, engage early-career scientists who drive breakthroughs, and partner with industry leaders who can share access to essential compute resources and frontier models. Traditional bureaucratic processes risk leaving federal efforts perpetually behind the curve.

The U.S. government should implement a three-pronged plan to advance the above R&D priorities.

Recommendation 1. Clearly define AI assurance and security R&D priorities

The Office of Science and Technology Policy (OSTP) and the National Science Foundation (NSF) should highlight critical areas of AI assurance and security as R&D priorities by including these in the 2025 update of the National AI R&D Strategic Plan and the forthcoming AI Action Plan. All federal agencies conducting AI R&D should engage with the construction of these plans to explain how their expertise could best contribute to these goals. For example, the Defense Advanced Research Projects Agency (DARPA)’s Information Innovation Office could leverage its expertise in AI security to investigate ways to design secure interaction protocols and environments for AI agents that eliminate risks from rogue agents.

The priorities would help coordinate government R&D activities by providing funding agencies with a common set of priorities, public research institutes such as the National Labs to conduct fundamental R&D activities, Congress with information to support relevant legislative decisions, and industry to serve as a guide to R&D.

Additionally, given the dynamic nature of frontier AI research, OSTP and NSF should publish an annual survey of progress in critical AI assurance and security areas and identify which challenges are the highest priority.

Recommendation 2. Establish an AI R&D consortium and deploy agile funding mechanisms for critical R&D

As noted by OSTP Director Michael Kratsios, “prizes, challenges, public-private partnerships, and other novel funding mechanisms, can multiply the impact of targeted federal dollars. We must tie grants to clear strategic targets, while still allowing for the openness of scientific exploration.” Federal funding agencies should develop and implement agile funding mechanisms for AI assurance and security R&D in line with established priorities. Congress should include reporting language in its Commerce, Justice, Science (CJS) appropriations bill that supports accelerated R&D disbursements for investment into prioritized areas.

A central mechanism should be the creation of an AI Assurance and Security R&D Consortium, jointly led by DARPA and NSF, bringing together government, AI companies, and universities. In this model:

This consortium structure would enable rapid resource sharing, collaborative research projects, and accelerated translation of research into practice. It would operate under flexible contracting mechanisms using Other Transaction Authority (OTA) to reduce administrative barriers.

Beyond the consortium, funding agencies should leverage Other Transaction Authority (OTA) and Prize Competition Authority to flexibly contract and fund research projects related to priority areas. New public-private grant vehicles focused on funding fundamental research in priority areas should be set up via existing foundations linked to funding agencies such as the NSF Foundation, DOE’s Foundation for Energy Security and Innovation, or the proposed NIST Foundation.

Specific funding mechanisms should be chosen based on the target technology’s maturity level. For example, the NSF can support more fundamental research through fast grants via its EAGER and RAPID programs. Previous fast-grant programs, such as SGER, were found to be wildly effective, with “transformative research results tied to more than 10% of projects.”

For research areas where clear, well-defined technical milestones are achievable, such as developing secure cluster-scale environments for large AI training workloads, the government can support the creation of focused research organizations (FROs) and implement advanced market commitments (AMCs) to take technologies across the ‘valley of death’. DARPA and IARPA can administer higher-risk, more ambitious R&D programs with national security applications.

Recommendation 3. Establish an AI Frontier Science Fellowship to ensure a pipeline of technical AI talent that can contribute directly to R&D and support fast-grant program management

It is critical to ensure that America has a growing pool of talented researchers entering the field of AI assurance and security, given its strategic importance to American competitiveness and national security. 

The NSF should launch an AI Frontier Science Fellowship targeting early-career researchers in critical AI assurance and security R&D. Drawing from proven models like CyberCorp Scholarship for Service, COVID-19 Fast Grants, and  proposals such as for “micro-ARPAs”, this program operates on two tracks:

  1. Frontier Scholars: This track would provide comprehensive research support for PhD students and post-docs conducting relevant research on priority AI security and reliability topics. This includes computational resources, research rotations at government labs and agencies, and financial support.
  2. Rapid Grant Program Managers (PM): This track recruits researchers to serve fixed terms as Rapid Grant PMs, responsible for administering EAGER/RAPID grants focused on AI assurance and security. 

This fellowship solves multiple problems at once. It builds the researcher pipeline while creating a nimble, decentralized approach to science funding that is more in line with the dynamic nature of the field. This should improve administrative efficiency and increase the surface area for innovation by allowing for more early-stage high-risk projects to be funded. Also, PMs who perform well in administering these small, fast grants can then become full-fledged program officers and PMs at agencies like the NSF and DARPA. This program (including grant budget) would cost around $40 million per year.

Conclusion

To unlock AI’s immense potential, from research to defense, we must ensure these tools are reliable and secure. This demands R&D breakthroughs to better understand emergent AI capabilities and behaviors, secure AI hardware and infrastructure, and prepare for a multi-agent world. The federal government must lead by setting clear R&D priorities, building foundational research talent, and injecting targeted funding to fast-track innovation.  This unified push is key to securing America’s AI leadership and ensuring that American AI is the global gold standard.

This memo was written by an AI Safety Policy Entrepreneurship Fellow over the course of a six-month, part-time program that supports individuals in advancing their policy ideas into practice. You can read more policy memos and learn about Policy Entrepreneurship Fellows here.

Frequently Asked Questions
Can this be done with reallocating existing budget and under existing authority?

Yes, the recommendations are achievable by reallocating the existing budget and using existing authorities, but this would likely mean accepting a smaller initial scale.


In terms of authorities, OSTP and NSF can already update the National AI R&D Strategic Plan and establish AI assurance and security priorities through normal processes. To implement agile funding mechanisms, agencies can use OTA and Prize Competition Authority. Fast grants require no special statute and can be done under existing grant authorities.


In terms of budget, agencies can reallocate 5-10% of existing AI research funds towards security and assurance R&D. The Frontier Science Fellowship could start as a $5-10 million pilot under NSF’s existing education authorities, e.g. drawing from NSF’s Graduate Research Fellowship Program.


While agencies have flexibility to begin this work, achieving the memo’s core objective – ensuring AI systems are trustworthy and reliable for workforce and military adoption – requires dedicated funding. Congress could provide authorization and appropriation for a named fellowship, which would make the program more stable and allow it to survive personnel turnover.

Why does the federal government need to step in – won’t the private sector fix AI reliability and security on its own?

Market incentives drive companies to fix AI failures that directly impact their bottom line, e.g., chatbots giving bad customer service or autonomous vehicles crashing. More visible, immediate problems are likely to be prioritized because customers demand it or because of liability concerns. This memo focuses on R&D areas that the private sector is less likely to tackle adequately.


The private will address some security and reliability issues, but there are likely to be significant gaps. Understanding emergent model capabilities demands costly fundamental research that generates little immediate commercial return. Likewise, securing AI infrastructure against nation-state attacks will likely require multi-year R&D processes, and companies can fail to coordinate to develop these technologies without a clear demand signal. Finally, systemic dangers arising from multi-agent interactions might be left unmanaged because these failures emerge from complex dynamics with unclear liability attribution.


The government can step in to fund the foundational research that the market is likely to undersupply by default and help coordinate the key stakeholders in the process.

How do we encourage industry cooperation, given that researchers will require access to frontier AI models and systems from private companies to conduct this research?

Companies need security solutions to access regulated industries and enterprise customers. Collaboration on government-funded research provides these solutions while sharing costs and risks.


The proposed AI Assurance and Security R&D Consortium in Recommendation 2 create a structured framework for cooperation. Companies contribute model access and compute credits while receiving:



  • Government-funded researchers working on their deployment challenges

  • Shared IP rights under consortium agreements

  • Early access to security and reliability innovations

  • Risk mitigation through collaborative cost-sharing


Under the consortia’s IP framework, companies retain full commercial exploitation rights while the government gets unlimited rights for government purposes. In the absence of a consortium agreement, an alternative arrangement could be a patent pool, where companies can access patented technologies in the pool through a single agreement. These structures, combined with the fellowship program providing government-funded researchers, creates strong incentives for private sector participation while advancing critical public research objectives.

The Federation of American Scientists Calls on OMB to Maintain the Agency AI Use Case Inventories at Their Current Level of Detail

The federal government’s approach to deploying AI systems is a defining force in shaping industry standards, academic research, and public perception of these technologies. Public sentiment toward AI remains mixed, with many Americans expressing a lack of trust in AI systems. To fully harness the benefits of AI, the public must have confidence that these systems are deployed responsibly and enhance their lives and livelihoods.

The first Trump Administration’s AI policies clearly recognized the opportunity to promote AI adoption through transparency and public trust. President Trump’s Executive Order 13859 explicitly stated that agencies must design, develop, acquire, and use “AI in a manner that fosters public trust and confidence while protecting privacy, civil rights, civil liberties, and American values.” This commitment laid the foundation for increasing government accountability in AI use.

A major step in this direction was the AI Use Case Inventory, established under President Trump’s Executive Order 13960 and later codified in the 2023 Advancing American AI Act. The agency inventories have since become a crucial tool in fostering public trust and innovation in government AI use. Recent OMB guidance (M-24-10) has expanded its scope, standardizing AI definitions, and collecting information on potential adverse impacts. The detailed inventory enhances accountability by ensuring transparency in AI deployments, tracks AI successes and risks to improve government services, and supports AI vendors by providing visibility into public-sector AI needs, thereby driving industry innovation.

The end of 2024 marked a major leap in government transparency regarding AI use. Agency reporting on AI systems saw dramatic improvements, with federal AI inventories capturing more than 1,700 AI use cases —a 200% increase in reported use cases from the previous year. The Department of Homeland Security (DHS) alone reported 158 active AI use cases. Of these, 29 were identified as high-risk, with detailed documentation on how 24 of those use cases are mitigating potential risks. This level of disclosure is essential for maintaining public trust and ensuring responsible AI deployment.

OMB is set to release revisions to its AI guidance (M-24-10) in mid-March, presenting an opportunity to ensure that transparency remains a top priority.

To support continued transparency and accountability in government AI use, the Federation of American Scientists has written a letter urging OMB to maintain its detailed guidance on AI inventories. We believe that sustained transparency is crucial to ensuring responsible AI governance, fostering public trust, and enabling industry innovation.

Federation of American Scientists and 16 Tech Organizations Call on OMB and OSTP to Maintain Agency AI Use Case Inventories

The first Trump Administration’s E.O. 13859 commitment laid the foundation for increasing government accountability in AI use; this should continue

Washington, D.C. – March 6, 2025 – The Federation of American Scientists (FAS), a non-partisan, nonprofit science think tank dedicated to developing evidence-based policies to address national challenges, today released a letter to the White House Office of Management and Budget (OMB) and the Office of Science and Technology Policy (OSTP), signed by 16 additional scientific and technical organizations, urging the current Trump administration to maintain the federal agency AI use cases inventories at the current level of detail.

“The federal government has immense power to shape industry standards, academic research, and public perception of artificial intelligence,” says Daniel Correa, CEO of the Federation of American Scientists. “By continuing the work set forth by the first Trump administration in Executive Order 13960  and continued by the bipartisan 2023 Advancing American AI Act, OMB’s detailed use cases help us understand the depth and scope of AI systems used for government services.”

“FAS and our fellow organizations urge the administration to maintain these use case standards because these inventories provide a critical check on government AI use,” says Dr. Jedidah Isler, Chief Science Officer at FAS.

AI Guidance Update Mid-March

“Transparency is essential for public trust, which in turn is critical to maximizing the benefits of government AI use. That’s why FAS is leading a letter urging the administration to uphold the current level of agency AI use case detail—ensuring transparency remains a top priority,” says Oliver Stephenson, Associate Director of AI and Emerging Tech Policy at FAS.

“Americans want reassurances that the development and use of artificial intelligence within the federal government is safe;  and that we have the ability to mitigate any adverse impacts. By maintaining guidance that federal agencies have to collect and publish information on risks, development status, oversight, data use and so many other elements, OMB will continue strengthening Americans’ trust in the development and use of artificial intelligence,” says Clara Langevin, AI Policy Specialist at FAS.

Surging Use of AI in Government 

This letter follows the dramatic rise in the use of artificial intelligence across government, with anticipated growth coming at a rapid rate. For example, at the end of 2024 the Department of Homeland Security (DHS) alone reported 158 active AI use cases. Of these, 29 were identified as high-risk, with detailed documentation on how 24 of those use cases are mitigating potential risks. OMB and OSTP have the ability and authority to set the guidelines that can address the growing pace of government innovation. 

FAS and our signers believe that sustained transparency is crucial to ensuring responsible AI governance, fostering public trust, and enabling responsible industry innovation.

Signatories Urging AI Use Case Inventories at Current Level of Detail

Federation of American Scientists
Beeck Center for Social Impact + Innovation at Georgetown University
Bonner Enterprises, LLC
Center for AI and Digital Policy
Center for Democracy & Technology
Center for Inclusive Change
CUNY Public Interest Tech Lab
Electronic Frontier Foundation
Environmental Policy Innovation Center
Mozilla
National Fair Housing Alliance
NETWORK Lobby for Catholic Social Justice
New America’s Open Technology Institute
POPVOX Foundation
Public Citizen
SeedAI
The Governance Lab



###

ABOUT FAS

The Federation of American Scientists (FAS) works to advance progress on a broad suite of contemporary issues where science, technology, and innovation policy can deliver dramatic progress, and seeks to ensure that scientific and technical expertise have a seat at the policymaking table. Established in 1945 by scientists in response to the atomic bomb, FAS continues to work on behalf of a safer, more equitable, and more peaceful world. More information about FAS work at fas.org.


ABOUT THIS COALITION

Organizations signed on to this letter represent a range of technology stakeholders in industry, academia, and nonprofit realms. We share a commitment to AI transparency.  We urge the current administration, OMB, and OSTP to retain the policies set forth in Trump’s Executive Order 13960 and continued in the bipartisan 2023 Advancing American AI Act.


Three Artificial Intelligence Bills Endorsed by Federation of American Scientists Advance from the House Committee

Proposed bills advance research ecosystems, economic development, and education access and move now to the U.S. House of Representatives for a vote

Washington, D.C. – September 12, 2024 – Three proposed artificial intelligence bills endorsed by the Federation of American Scientists (FAS), a nonpartisan science think tank, advance forward from a House Science, Space, and Technology Committee markup held on September 11th, 2024. These bills received bipartisan support and will now be reported to the full chamber. The three bills are: H.R. 9403, the Expanding AI Voices Act, co-sponsored by Rep. Vince Fong (CA-20) and Rep. Andrea Salinas (OR-06); H.R. 9197, the Small Business AI Act, co-sponsored by Rep. Mike Collins (GA-10) and Rep. Haley Stevens (MI-11), and H.R. 9403, the Expand AI Act, co-sponsored by Rep. Valerie Foushee (NC-04) and Rep. Frank Lucas (OK-03).

“FAS endorsed these bills based on the evaluation of their strengths. Among these are the development of infrastructure to develop AI safely and responsibly; the deployment of resources to ensure development benefits more equitably across our economy; and investment in the talent pool necessary for this consequential, emerging technology,” says Dan Correa, CEO of FAS.

“These three bills pave a vision for the equitable and safe use of AI in the U.S. Both the Expanding AI Voices Act and the NSF AI Education Act will create opportunities for underrepresented voices to have a say in how AI is developed and deployed. Additionally, the Small Business AI Act will ensure that an important sector of our society feels empowered to use AI safely and securely,” says Clara Langevin, FAS AI Policy Specialist. 

Expanding AI Voices Act

The Expanding AI Voices Act will support a broad and diverse interdisciplinary research community for the advancement of artificial intelligence and AI-powered innovation through partnerships and capacity building at certain institutions of higher education to expand AI capacity in populations historically underrepresented in STEM.

Specifically, the Expanding AI Voices Act of 2024 will:

Small Business AI Act

Emerging science is central to new and established small businesses, across industries and around the country. This bill will require the Director of the National Institute of Standards and Technology (NIST) to develop resources for small businesses in utilizing artificial intelligence, and for other purposes. 

National Science Foundation Artificial Intelligence Education Act of 2024 (NSF AI Education Act).


The National Artificial Intelligence Initiative Act of 2020 (15 U.S.C. 9451) will  bolster educational skills in AI through new learning initiatives and workforce training programs. Specifically, the bill will: 

###

ABOUT FAS

The Federation of American Scientists (FAS) works to advance progress on a broad suite of contemporary issues where science, technology, and innovation policy can deliver dramatic progress, and seeks to ensure that scientific and technical expertise have a seat at the policymaking table. Established in 1945 by scientists in response to the atomic bomb, FAS continues to work on behalf of a safer, more equitable, and more peaceful world. More information at fas.org.