Rapid and advanced AI integration and diffusion within the Department of Defense (DoD) and other government agencies has emerged as a critical national security priority. This convergence of rapid AI advancement and DoD prioritization creates an urgent need to ensure that AI models integrated into defense operations are reliable, safe, and mission-enhancing. For this purpose, the DoD must deploy and expand one of its most critical tools available within its Testing and Evaluation (T&E) process: benchmarking—the structured practice of applying shared tasks and metrics to compare models, track progress, and expose performance gaps.

A standardized AI benchmarking framework is critical for delivering uniform, mission-aligned evaluations across the DoD. Despite their importance, the DoD currently lacks standardized, enforceable AI safety benchmarks, especially for open-ended or adaptive use cases. A shift from ad hoc to structured assessments will support more informed, trusted, and effective procurement decisions. 

Particularly at the acquisition stage for AI models, rapid DoD acquisition platforms such as Tradewinds can serve as the policy vehicle for enabling more robust benchmarking efforts. This can be done with the establishment of a federally coordinated benchmarking hub, spearheaded by a coordinated effort between the Chief Data and Artificial Intelligence Officer (CDAO) and Defense Innovation Unit (DIU) in consultation with the newly established Chief AI Officer’s Council (CAIOC) of the White House Office of Management and Budget (OMB). 

Challenge and Opportunity

Experts at the intersection of both AI and defense, such as the retired Lieutenant General John (Jack) N.T. Shanahan, have emphasized the profound impact of AI on the way the United States will fight future wars – with the character of war continuously reshaped by AI’s diffusion across all domains. The DoD is committed to remaining at the forefront of these changes: between 2022-2023, the value of federal AI contracts increased by over 1200%, with the surge driven by increases in DoD spending. Secretary of Defense Pete Hegseth has pledged increased investment in AI specifically for military modernization efforts, and has tasked the Army to implement AI in command and control across the theater, corps, and division headquarters by 2027–further underscoring AI’s transformative impact on modern warfare.

Strategic competitors—especially the People’s Republic of China—are rapidly integrating AI into their military and technological systems. The Chinese Communist Party views AI-enabled science and technology as central to accelerating military modernization and achieving global leadership. At this pivotal moment, the DoD is pushing to adopt advanced AI across operations to preserve the U.S. edge in military and national security applications. Yet, accelerating too quickly without proper safeguards risks exposing vulnerabilities adversaries could exploit.

With the DoD at a unique inflection point, it must balance the rapid adoption and integration of AI into its operations with the need for oversight and safety. DoD needs AI systems that consistently meet clearly defined performance standards set by acquisition authorities, operate strictly within the scope of their intended use, and do not exhibit unanticipated or erratic behaviors under operational conditions. These systems can deliver measurable value to mission outcomes while fostering trust and confidence among human operators through predictability, transparency, and alignment with mission-specific requirements.

AI benchmarks are standardized tasks and metrics that systematically measure a model’s performance, reliability, and safety, and have increasingly been adopted as a key measurement tool by the AI industry. Currently, DoD lacks standardized, comprehensive AI safety benchmarks, especially for open-ended or adaptive use cases. Without these benchmarks, the DoD risks acquiring models that underperform, deviate from mission requirements, or introduce avoidable vulnerabilities, leading to increased operational risk, reduced mission effectiveness, and costly contract revisions.  

A recent report from the Center for a New American Security (CNAS) on best practices for AI T&E outlined that the rapid and unpredictable pace of AI advancement presents distinctive challenges for both policymakers and end-users. The accelerating pace of adoption and innovation heightens both the urgency and complexity of establishing effective AI benchmarks to ensure acquired models meet the mission-specific performance standards required by the DoD and the services. 

The DoD faces particularly outsized risk, as its unique operational demands can expose AI models to extreme conditions where performance may degrade. For example, under adversarial conditions, or when encountering data that is different from its training, an AI model may behave unpredictably, posing heightened risk to the mission. Robust evaluations, such as those offered through benchmarking, help to identify points of failure or harmful model capabilities before they become apparent during critical use cases. By measuring model performance in real-world applicable scenarios and environments, we increase understanding of attack surface vulnerabilities to adversarial inputs. We can identify inaccurate or over-confident measurements of outputs, and recognize potential failures in edge cases and extreme scenarios (including those beyond training parameters, Moreover, we improve human-AI performance and trust factors, and avoid unintended capabilities. Benchmarking helps to surface these issues early. 

Robust AI benchmarking frameworks can enhance U.S. leadership by shaping international norms for military AI safety, improving acquisition efficiency by screening out underperforming systems, and surfacing unintended or high-risk model behaviors before deployment. Furthermore, benchmarking enables AI performance to be quantified in alignment with mission needs, using guidance from the CDAO RAI Toolkit and clear acquisition parameters to support decision-making for both procurement officers and warfighters. Given the DoD’s high-risk use cases and unique mission requirements, robust benchmarking is even more essential than in the commercial sector.

The DoD now has an opportunity to formalize AI safety benchmark frameworks within its Testing and Evaluation (T&E) processes, tailored to both dual-use and defense-specific applications. T&E is already embedded in DoD culture, offering a strong foundation for expanding benchmarking. Public-private AI testing initiatives, such as the DoD collaboration with Scale AI to create effective T&E (including through benchmarking) for AI models show promise and existing motivation for such initiatives. Yet, critical policy gaps still exist. With pilot programs underway, the DoD can move beyond vendor-led or ad hoc evaluations to introduce DoD-led testing, assess mission-specific capabilities, launch post-acquisition benchmarking, and develop human-AI team metrics. The widely used Tradewinds platform offers an existing vehicle to integrate these enhanced benchmarks without reinventing the wheel. 

To implement robust benchmarking at DoD, this memo proposes the following policy recommendations, to be coordinated by DoD Chief Digital and Artificial Intelligence Office (CDAO):

• Expanding on existing benchmarking efforts
• Standardizing AI safety thresholds during the procurement cycle
• Implementing benchmarking during the lifecycle of the model
• Establishing a benchmarking repository
• Enabling adversarial stress testing, or “red-teaming”, prior to deployment to enhance current benchmarking gaps for DoD AI use cases

Plan of Action 

The CDAO should launch a formalized AI Benchmarking Initiative, moving beyond current vendor-led pilot programs, while continuing to refine its private industry initiatives. This effort should be comprehensive and collaborative in nature, leveraging internal technical expertise. This includes the newly established coordinating bodies on AI such as the Chief AI Officer’s Council, which can help to ensure that DoD benchmarking practices are aligned with federal priorities, and the Defense Innovation Unit, which can be an excellent private industry-national defense sector bridge and coordinator in these efforts. Specifically, the CDAO should integrate benchmarking into the acquisition pipeline. This will  establish ongoing benchmarking practices that facilitate continuous model performance evaluation through the entirety of the model lifecycle. 

Policy Recommendations 

Recommendation 1. Establish a Standardized Defense AI Benchmarking Initiative and create a Centralized Repository of Benchmarks

The DoD should build on lessons learned from its partnership with Scale AI (and others) developing benchmarks specifically for defense use cases. This should expand  into a standardized, agency-wide framework. 

This recommendation is in line with findings outlined by RAND, which calls for developing a comprehensive framework for robust evaluation and emphasizes the need for collaborative practices, and measurable performance metrics for model performance. 

The DoD should incorporate the following recommendations and government entities to achieve this goal:

Develop a Whole-of-Government Approach to AI Benchmarking

• Develop and expand on existing pilot benchmarking frameworks, similar to Massive Multitask Language Understanding (MMLU) but tailored to military-relevant tasks and DoD-specific use cases. 
• Expand the $10 million T&E and research budget by $10 million, with allocations specifically for bolstering internal benchmarking capabilities. One crucial piece is identifying and recruiting technically capable talent to aid in developing internal benchmarking guidelines. As AI models advance, new “reasoning” models with advanced capabilities become far costlier to benchmark, and the DoD must plan for these future demands now. Part of this allocation can come from the $500 million allocated for the combatant command AI budgets. This monetary allocation is critical to successfully implementing this policy because model benchmarking for more advanced models – such as OpenAI’s GPT-3 – can cost millions. This modest budgetary increase is  a starting point for moving beyond piecemeal and ad hoc benchmarking, to a comprehensive and standardized process. This funding increases would facilitate:
  • Development of and expansion of internal and customized benchmarking capabilities 
  • Recruitment and retention of technical talent 
  • Development of simulation environment for more mission-relevant benchmarks 

If internal reallocations from the $500 million allocation proves insufficient or unviable, Congressional approval for additional funds can be another funding source. Given the strategic importance of AI in defense, such requests can readily find bipartisan support, particularly when tied to operational success and risk mitigation.

• Create a centralized AI benchmarking repository under the CDAO. This will standardize categories, performance metrics, mission alignment, and lessons learned across defense-specific use cases. This repository will enable consistent tracking of model performance over time, support analysis across model iterations, and allow for benchmarking transferability across similar operational scenarios. By compiling performance data at scale, the repository will  also help identify interoperability risks and system-level vulnerabilities—particularly how different AI models may behave when integrated—thereby enhancing the DoD’s ability to assess, document, and mitigate potential performance and safety failures.
• Convene a partnership, organized by OMB, between the CDAO,  the DIU and the CAIOC, to jointly establish and maintain a centralized benchmarking repository. While many CAIOC members represent civilian agencies, their involvement is crucial: numerous departments (such as the Department of Homeland Security, the Department of Energy, and the National Institute of Standards and Technology) are already employing AI in high-stakes contexts and bring relevant technical expertise, safety frameworks, and risk management policies. Incorporating these perspectives ensures that DoD benchmarking practices are not developed in isolation but reflect best practices across the federal government. This partnership will leverage the DIU’s insights on emerging private-sector technologies, the CDAO’s acquisition and policy authorities, and CAIOC’s alignment with broader executive branch priorities, thereby ensuring that benchmarking practices are technically sound, risk-informed, and consistent with government-wide standards and priorities for trustworthy, safe, and reliable AI.

Recommendation 2. Formalize Pre-Deployment Benchmarking for AI Models at the Acquisition Stage 

The key to meaningful benchmarking lies in integrating it at the pre-award stage of procurement. The DoD should establish a formal process that:

• Integrates benchmarking into existing AI acquisition platforms, such as Tradewinds, and embeds it within the T&E process.
• Requires participation from third-party vendors in benchmarking the products they propose for DoD acquisition and use.
• Embeds internal adversarial stress testing, or “red-teaming”, into AI benchmarking ensures more realistic, mission-aligned evaluations that account for adversarial threats and the unique, high-risk operating environments the military faces. By leveraging its internal expertise in mission context, classified threat models, and domain-specific edge cases that external vendors are unlikely to fully replicate, the DoD can produce a more comprehensive and defense-relevant assessment of AI system safety, efficacy, and suitability for deployment. Specifically, this policy memo recommends that the AI Rapid Capabilities Cell (AI RCC) be tasked with carrying out the red-teaming, as a technically qualified element of the CDAO.
• Assures procurement officers understand the value of incorporating benchmarking performance metrics into their contract award decision-making. This can be done by hosting benchmarking workshops for procurement officers, which outline the benchmarking results for model performance for various models in the acquisition pipeline and to guide them on how to apply these metrics to their own performance requirements and guidelines. 

Recommendation 3. Contextualize Benchmarking into Operational Environments

Current efforts to scale and integrate AI reflect the distinct operational realities of the DoD and military services. Scale AI, in partnership with the DoD, Anduril, Microsoft, and the CDAO, is  developing AI-powered solutions which are focused on the United States Indo-Pacific Command (INDOPACOM) and United States European Command (EUCOM). With these regional command focused AI solutions, it makes sense to create equally focused benchmarking standards to test AI model performance in specific environments and under unique and focused conditions. In fact, researchers have been identifying the limits of traditional AI benchmarking and making the case for bespoke, holistic, and use-case relevant benchmark development. This is vital because as AI models advance, they introduce entirely new capabilities which require more robust testing and evaluation. For example, large language models, which have introduced new functionalities including natural language querying or multimodal search interfaces, require entirely new benchmarks that measure: natural language understanding, modal integration accuracy, context retention, and result usefulness. In the same vein, DoD relevant benchmarks must be developed in an operationally-relevant context. This can be achieved by:

• Developing simulation environments for benchmarking that are mission-specific across a broader set of domains, including technical and regional commands, to test AI models under specific conditions which are likely to be encountered by users in unique, contested, and/or adversarial environments. The Bipartisan House Task Force on Artificial Intelligence report provides useful guidance on AI model functionality, reliability, and safety in operating in contested, denied, and degraded environments.
• Prioritizing use-case-specific benchmarks over broad commercial metrics by incorporating user feedback and identifying tailored risk scenarios that more accurately measure model performance.
• Introducing context relevant benchmarks to measure performance in specific, DoD-relevant scenarios, such as:
  • Task-specific accuracy (i.e. correct ID in satellite imagery cases)
  • Alignment with context-specific rules of engagement
  • Instances of degraded performance under high-stress conditions
  • Susceptibility to adversarial manipulation (i.e. data poisoning)
  • Latency in high-risk, fast-paced decision-making scenarios
• Creating post-deployment benchmarking to ensure ongoing performance and risk compliance, and to detect and address issues like model drift, a phenomenon where model performance degrades over time. As there is no established consensus on how often continuous model benchmarking should be performed, the DoD should study the appropriate practical, risk-informed timelines for re-evaluating deployed systems.

Frameworks such as Holistic Evaluation of Language Models (HELM) and Focused LLM Ability Skills and Knowledge (FLASK) can offer valuable guidance for developing LLM-focused benchmarks within the DoD, by enabling more comprehensive evaluations based on specific model skill sets, use-case scenarios, and tailored performance metrics.

Recommendation 4. Integration of Human-in-the-Loop Benchmarking 

An additional layer of AI benchmarking for safe and effective AI diffusion into the DoD ecosystem is evaluating AI-human team performance, and measuring user trust, perceptions and confidence in various AI models. “Human‑in‑the‑loop” systems require a person to approve or adjust the AI’s decision before action, while “human‑on‑the‑loop” systems allow autonomous operation but keep a person supervising and ready to intervene. Both  “Human in the loop”  and “Human on the loop” are critical components of the DoD and military approach to AI. Both  require continued human oversight of ethical and safety considerations over AI-enabled capabilities with national security implications. A recent study by MIT study found that there are surprising performance gaps between AI only, human only, and AI-human teams. For the DoD particularly, it is important to effectively measure these performance gaps across the various AI models it plans to integrate into its operations due to heavy reliance on user-AI teams. 

A CNAS report on effective T&E for AI spotlighted the DARPA Air Combat Evolution (ACE) program, which sought autonomous air‑combat agents needing minimal human intervention. Expert test pilots could override the system, yet often did so prematurely, distrusting its unfamiliar tactics. This case underscores the need for early, extensive benchmarks that test user capacity, surface trust gaps that can cripple human‑AI teams, and assure operators that models meet legal and ethical standards. Accordingly, this memo urges expanding benchmarking beyond pure model performance to AI‑human team evaluations in high‑risk national‑security, lethal, or error‑sensitive environments.

Conclusion

The Department of Defense is racing to integrate AI across every domain of warfare, yet speed without safety will  jeopardize mission success and national security. Standardized, acquisition‑integrated, continuous, and mission‑specific benchmarking is therefore not a luxury—it is the backbone of responsible AI deployment. Current pilot programs with private partners are encouraging starts, but they remain too ad hoc and narrow to match the scale and tempo of modern AI development.

Benchmarking must begin at the pre‑award acquisition stage and follow systems through their entire lifecycle, detecting risks, performance drift, and adversarial vulnerabilities before they threaten operations. As the DARPA ACE program showed, early testing of human‑AI teams and rigorous red‑teaming surface trust gaps and hidden failure modes that vendor‑led evaluations often miss. Because AI models—and enemy capabilities—evolve constantly, our evaluation methods must evolve just as quickly.

By institutionalizing robust benchmarks under CDAO leadership, in concert with the Defense Innovation Unit and the Chief AI Officers Council, the DoD can set world‑class standards for military AI safety while accelerating reliable procurement. Ultimately, AI benchmarking is not a hurdle to innovation and acquisition, but rather it is the infrastructure that can make rapid acquisition more reliable and innovation more viable. The DoD cannot afford the risk of deploying AI systems which are risky, unreliable, ineffective or misaligned with mission needs and standards in high-risk operational environments. At this inflection point, the choice is not between speed and safety but between ungoverned acceleration and a calculated momentum that allows our strategic AI advantage to be both sustained and secured.

This memo was written by an AI Safety Policy Entrepreneurship Fellow over the course of a six-month, part-time program that supports individuals in advancing their policy ideas into practice. You can read more policy memos and learn about Policy Entrepreneurship Fellows here.

What is the Scale AI benchmarking pilot program at DoD, and why and how does this policy proposal build on this initiative?

he Scale AI benchmarking initiative, launched in February 2024 in partnership with the DoD, is a pilot framework designed to evaluate the performance of AI models intended for defense and national security applications. It is part of the broader efforts to create a framework for T&E of AI models for the CDAO.

This memo builds on that foundation by:

• Formalizing benchmarking as a standard requirement at the procurement stage across DoD acquisition processes.


• Inserting benchmarking protocols into rapid acquisition platforms like Tradewinds.


• Establishing a defense-specific benchmarking repository and enabling red-teaming led by the AI Rapid Capabilities Cell (AI RCC) within the CDAO.


• Shifting the lead on benchmarking from vendor-enabled to internally developed, led, and implemented, creating bespoke evaluation criteria tailored to specific mission needs.

read more

What types of AI systems will these benchmarks apply to, and how will they be tailored for national security use cases?

The proposed benchmarking framework will apply to a diverse range of AI systems, including:

• Decision-making and command and control support tools (sensors, target recognition, process automation, and tools involved in natural language processing).


• Generative models for planning, logistics, intelligence, or data generation.


• Autonomous agents, such as drones and robotic systems.

Benchmarks will be theater and context-specific, reflecting real-world environments (e.g. contested INDOPACOM scenarios), end-user roles (human-AI teaming in combat), and mission-specific risk factors such as adversarial interference and model drift.

read more

How will this benchmarking framework approach open-source or non-proprietary AI models intended for DoD use?

Open-source models present distinct challenges due to model ownership and origin, additional possible exposure to data poisoning, and downstream user manipulation. However, due to the nature of open-source models, it should be noted that the general increase in transparency and potential access to training data could make open-source models less challenging to put through rigorous T&E.

This memo recommends:

• Applying standardized evaluation criteria across both open-source and proprietary models which can be developed by utilizing the AI benchmarking repository and applying model evaluations based on possible use cases of the model.


• Incorporating benchmarking to test possible areas of vulnerability for downstream user manipulation.


• Measuring the transparency of training data.


• Performing adversarial testing to assess resilience against manipulated inputs via red-teaming.


• Logging the open-source model performance in the proposed centralized repository, enabling ongoing monitoring for drift and other issues

read more

Why is red-teaming a necessity in addition to AI benchmarking, and how will it be executed?

Red-teaming implements adversarial stress-testing (which can be more robust and operationally relevant if led by an internal team as this memo proposes), and can identify vulnerabilities and unintended capabilities before deployment. Internally led red-teaming, in particular, is critical for evaluating models intended for use in unpredictable or hostile environments.

read more

How will red-teaming be executed?

To effectively employ the red-teaming efforts, this policy recommends that:

• The AI Rapid Capabilities Cell within the CDAO should lead red-teaming operations, leveraging the team’s technical capabilities with its experience and mission set to integrate and rapidly scale AI at the speed of relevance — delivering usable capability fast enough to affect current operations and decision cycles.


• Internal, technically skilled teams should be created who are capable of incorporating classified threat models and edge-case scenarios.


• Red-teaming should focus on simulating realistic mission conditions, and searching for specific model capabilities, going beyond generic or vendor-supplied test cases.

read more

How does this benchmarking framework improve acquisition decisions and reduce risks?

Integrating benchmarking at the acquisition stage enables procurement officers to:

• Compare models on mission-relevant, standardized performance metrics and ensure that there is evidence of measurable performance metrics which align with their own “vision of success” procurement requirements for the models.


• Identify and avoid models with unsafe, misaligned, unverified, or ineffective capabilities.


• Prevent cost-overruns or contract revisions.

Benchmarking workshops for acquisition officers can further equip them with the skills to interpret benchmark results and apply them to their operational requirements.

read more

As the United States overhauls nearly every element of its strategic nuclear forces, artificial intelligence is set to play a larger role—initially in early‑warning sensors and decision‑support tools, and likely in other mission areas. Improved detection could strengthen deterrence, but only if accompanying hazards—automation bias, model hallucinations, exploitable software vulnerabilities, and the risk of eroding assured second‑strike capability—are well managed. 

To ensure responsible AI integration, the Office of the Assistant Secretary of Defense for Nuclear Deterrence, Chemical, and Biological Defense Policy and Programs (OASD (ND-CBD)), the U.S. Strategic Command (STRATCOM), the Defense Advanced Research Projects Agency (DARPA), the Office of the Undersecretary of Defense for Policy (OUSD(P)), and the National Nuclear Security Administration (NNSA), should jointly develop a standardized AI risk-assessment framework guidance document, with implementation led by the Department of Defense’s Chief Digital and Artificial Intelligence Office (CDAO) and STRATCOM. Furthermore, DARPA and CDAO should join the Nuclear Weapons Council to ensure AI-related risks are systematically evaluated alongside traditional nuclear modernization decisions. 

Challenge and Opportunity 

The United States is replacing or modernizing nearly every component of its strategic nuclear forces, estimated to cost at least $1.7 trillion over the next 30 years. This includes its:

• Intercontinental ballistic missiles (ICBMs)
• Ballistic missile submarines and their submarine-launched ballistic missiles (SLBMs) 
• Strategic bombers, cruise missiles, and gravity bombs
• Nuclear warhead production and plutonium pit fabrication facilities 

Simultaneously, artificial intelligence (AI) capabilities are rapidly advancing and being applied across the national security enterprise, including nuclear weapons stockpile stewardship and some components of command, control, and communications (NC3) systems, which encompass early warning, decision-making, and force deployment components.

The NNSA, responsible for stockpile stewardship, is increasingly integrating AI into its work. This includes using AI for advanced modeling and simulation of nuclear warheads. For example, by creating a digital twin of existing weapons systems to analyze aging and performance issues, as well as using AI to accelerate the lifecycle of nuclear weapons development. Furthermore, NNSA is leading some aspects of the safety testing and systematic evaluations of frontier AI models on behalf of the U.S. government, with a specific focus on assessing nuclear and radiological risk.

Within the NC3 architecture, a complex “system of systems” with over 200 components, simpler forms of AI are already being used in areas including early‑warning sensors, and may be applied to  decision‑support tools and other subsystems as confidence and capability grow. General Anthony J. Cotton—who leads STRATCOM, the combatant command that directs America’s global nuclear forces and their command‑and‑control network—told a 2024 conference that STRATCOM is “exploring all possible technologies, techniques, and methods” to modernize NC3. Advanced AI and data‑analytics tools, he said, can sharpen decision‑making, fuse nuclear and conventional operations, speed data‑sharing with allies, and thus strengthen deterrence. General Cotton added that research must also map the cascading risks, emergent behaviors, and unintended pathways that AI could introduce into nuclear decision processes.

Thus, from stockpile stewardship to NC3 systems, AI is likely to be integrated across multiple nuclear capabilities, some potentially stabilizing, others potentially highly destabilizing. For example, on the stabilizing effects, AI could enhance early warning systems by processing large volumes of satellite, radar, and other signals intelligence, thus providing more time to decision-makers. On the destabilizing side, the ability for AI to detect or track other countries’ nuclear forces could be destabilizing, triggering an expansionary arms race if countries doubt the credibility of their second-strike capability. Furthermore, countries may misinterpret each other’s nuclear deterrence doctrines or have no means of verification of human control of their nuclear weapons.

While several public research reports have been conducted on how AI integration into NC3 could upset the balance of strategic stability, less research has focused on the fundamental challenges with AI systems themselves that must be accounted for in any risk framework. Per the National Institute of Standards and Technology’s (NIST) AI Risk Management Framework, several fundamental AI challenges at a technical level must be accounted for in the integration of AI into stockpile stewardship and NC3.

Not all AI applications within the nuclear enterprise carry the same level of risk. For example, using AI to model warhead aging in stockpile stewardship is largely internal to the Department of Energy (DOE) and involves less operational risk. Despite lower risk, there is still potential for an insufficiently secure model to lead to leaked technical data about nuclear weapons. 

However, integrating AI into decision support systems or early warning functions within NC3 introduces significantly higher stakes. These systems require time-sensitive, high-consequence judgments, and AI integration in this context raises serious concerns about issues including confabulations, human-AI interactions, and information security: 

• Confabulations: A phenomenon in which generative AI systems (GAI) systems generate and confidently present erroneous or false content in response to user inputs, or
  prompts. These phenomena are colloquially also referred to as “hallucinations” or “fabrications”, and could have particularly dangerous consequences in high-stakes settings. 

• Human-AI Interactions: Due to the complexity and human-like nature of GAI technology, humans may over-rely on GAI systems or may unjustifiably perceive GAI content to be of higher quality than that produced by other sources. This phenomenon is an example of automation bias or excessive deference to automated systems. This deference can lead to a shift from a human making the final decision (“human in the loop”), to a human merely observing AI generated decisions (“human on the loop”). Automation bias therefore risks exacerbating other risks of GAI systems as it can lead to humans maintaining insufficient oversight.

• Information Security: AI expands the cyberattack surface of NC3. Poisoned AI training data and tampered code can embed backdoors, and, once deployed, prompt‑injection or adversarial examples can hijack AI decision tools, distort early‑warning analytics, or leak secret data. The opacity of large AI models can let these exploits spread unnoticed, and as models become more complex, they will be harder to debug.

This is not an exhaustive list of issues with AI systems, however it highlights several key areas that must be managed. A risk framework must account for these distinctions and apply stricter oversight where system failure could have direct consequences for escalation or deterrence credibility. Without such a framework, it will be challenging to harness the benefits AI has to offer.

Plan of Action 

Recommendation 1. OASD (ND-CBD),  STRATCOM, DARPA, OUSD(P), and NNSA, should develop a standardized risk assessment framework guidance document to evaluate the integration of artificial intelligence into nuclear stockpile stewardship and NC3 systems. 

This framework would enable systematic evaluation of risks, including confabulations, human-AI configuration, and information security, across modernization efforts. The framework could assess the extent to which an AI model is prone to confabulations, involving performance evaluations (or “benchmarking”) under a wide range of realistic conditions. While there are public measurements for confabulations, it is essential to evaluate AI systems on data relevant to the deployment circumstances, which could involve highly sensitive military information.  

Additionally, the framework could assess human-AI configuration with specific focus on risks from automation bias and the degree of human oversight. For these tests, it is important to put the AI systems in contact with human operators in situations that are as close to real deployment as possible, for example when operators are tired, distracted, or under pressure. 

Finally, the framework could include assessments of information security under extreme conditions. This should include simulating comprehensive adversarial attacks (or “red-teaming”) to understand how the AI system and its human operators behave when subject to a range of known attacks on AI systems.

NNSA should be included in this development due to their mission ownership of stockpile stewardship and nuclear safety, and leadership in advanced modeling and simulation capabilities. DARPA should be included due to its role as the cutting edge research and development agency, extensive experience in AI red-teaming, and understanding of the AI vulnerabilities landscape. STRATCOM must be included as the operational commander of NC3 systems, to ensure the framework accounts for real-word needs and escalation risks. OASD (ND-CBD) should be involved given the office’s responsibilities to oversee nuclear modernization and coordinate across the interagency. The OUSD (P) should be included to provide strategic oversight and ensure the risk assessment aligns with broader defense policy objectives and international commitments.

Recommendation 2. CDAO should implement the Risk Assessment Framework with STRATCOM

While NNSA, DARPA, OASD (ND-CBD) and STRATCOM can jointly create the risk assessment framework, CDAO and STRATCOM should serve as the implementation leads for utilizing the framework. Given that the CDAO is already responsible for AI assurance, testing and evaluation, and algorithmic oversight, they would be well-positioned to work with relevant stakeholders to support implementation of the technical assessment. STRATCOM would have the strongest understanding of operational contexts with which to apply the framework. NNSA and DARPA therefore could advise on technical underpinnings with regards to AI of the framework, while the CDAO would prioritize operational governance and compliance, ensuring that there are clear risk assessments completed and understood when considering integration of AI into nuclear-related defense systems.

Recommendation 3. DARPA and CDAO should join the Nuclear Weapons Council 

Given their roles in the creation and implementation of the AI risk assessment framework, stakeholders from both DARPA and the CDAO should be incorporated into the Nuclear Weapons Council (NWC), either as full members or attendees to a subcommittee. As the NWC is the interagency body the DOE and the DoD responsible for sustaining and modernizing the U.S. nuclear deterrent, the NWC is responsible for endorsing military requirements, approving trade-offs, and ensuring alignment between DoD delivery systems and NNSA weapons. 

As AI capabilities become increasingly embedded in nuclear weapons stewardship, NC3 systems, and broader force modernization, the NWC must be equipped to evaluate associated risks and technological implications. Currently, the NWC is composed of senior officials from the Department of Defense, the Joint Chiefs of Staff, and the Department of Energy, including the NNSA. While these entities bring deep domain expertise in nuclear policy, military operations, and weapons production, the Council lacks additional representation focused on AI.

DARPA’s inclusion would ensure that early-stage technology developments and red-teaming insights are considered upstream in decision-making. Likewise, CDAO’s presence would provide continuity in AI assurance, testing, and digital system governance across operational defense components. Their participation would enhance the Council’s ability to address new categories of risk, such as model confabulation, automation bias, and adversarial manipulation of AI systems, that are not traditionally covered by existing nuclear stakeholders. By incorporating DARPA and CDAO, the NWC would be better positioned to make informed decisions that reflect both traditional nuclear considerations and the rapidly evolving technological landscape that increasingly shapes them.

Conclusion 

While AI is likely to be integrated into components of the U.S. nuclear enterprise, without a standardized initial approach to assessing and managing AI-specific risk, including confabulations, automation bias, and novel cybersecurity threats, this integration could undermine an effective deterrent. A risk assessment framework coordinated by OASD (ND-CBD), with STRATCOM, NNSA and DARPA, and implemented with support of the CDAO, could provide a starting point for NWC decisions and assessments of the alignment between DoD delivery system needs, the NNSA stockpile, and NC3 systems.

This memo was written by an AI Safety Policy Entrepreneurship Fellow over the course of a six-month, part-time program that supports individuals in advancing their policy ideas into practice. You can read more policy memos and learn about Policy Entrepreneurship Fellows here.

Frequently Asked Questions

Does the NWC have the authority to create a new subcommittee including DARPA and the CDAO?

Yes, NWC subordinate organizations or subcommittees are not codified in Title 10 USC §179, so the NWC has the flexibility to create, merge, or abolish organizations and subcommittees as needed.

read more

Are there existing regulations that the United States has declared with respect to AI integration into NC3?

Section 1638 of the FY2025 National Defense Authorization Act established a Statement of Policy emphasizing that any use of AI in support of strategic deterrence should not compromise, “the principle of requiring positive human actions in execution of decisions by the President with respect to the employment of nuclear weapons.” However, as this memo describes, AI presents further challenges outside of solely keeping a human in the loop in terms of decision-making.

read more

While AI’s transformative advances have enormous positive potential, leading scientists and industry executives are also sounding the alarm about catastrophic risks on a global scale. If left unmanaged, these risks could undermine our ability to reap the benefits of AI progress. While the U.S. government has made some progress, including by establishing the Center for AI Standards and Innovation (CAISI)—formerly the US AI Safety Institute—current government capacity is insufficient to respond to these extreme frontier AI threats. To address this problem, this memo proposes scaling up a significantly enhanced “CAISI+” within the Department of Commerce. CAISI+ would require dedicated high-security compute facilities, specialized talent, and an estimated annual operating budget of $67-155 million, with a setup cost of $155-275 million. CAISI+ would have expanded capacity for conducting advanced model evaluations for catastrophic risks, provide direct emergency assessments to the President and National Security Council (NSC), and drive critical AI reliability and security research, ensuring America is prepared to lead on AI and safeguard its national interests.

Challenge and Opportunity 

Frontier AI is advancing rapidly toward powerful general-purpose capabilities. While this progress has produced widely useful products, it is also generating significant security risks. Recent evaluations on Anthropic’s Claude Opus 4 model were unable to rule out the risk that the model could be used to advise novice actors to produce bioweapons, triggering additional safeguards. Meanwhile, the FBI warns that AI “increases cyber-attack speed, scale, and automation”, with a 442% increase in AI-enhanced voice phishing attacks in 2024, and recent evaluations showing AI models rapidly gaining offensive cyber capabilities. 

AI company CEOs and leading researchers have predicted that this progress will continue, with potentially transformative AI capabilities arriving in the next few years–and fast progress in AI capabilities will continue to generate novel threats greater than those from existing models. As AI systems are predicted to become increasingly capable of performing complex tasks and taking extended autonomous actions, researchers warn of these additional risks, such as loss of human control, AI-enabled WMD proliferation, and strategic surprise with severe national security implications. While timelines to AI systems surpassing dangerous capability thresholds are uncertain, this proposal attempts to lay out a US government response that is robust to a range of possible timelines, while taking the above trends seriously.

Current U.S. Government capabilities, including the existing Center for AI Standards and Innovation (CAISI), are not adequately resourced or empowered to independently evaluate, monitor, or respond to the most advanced AI threats. For example, current CAISI funding is precarious, its home institution (NIST)’s offices are reportedly “crumbling”, and its budget is roughly one-tenth of its counterpart in the UK. Despite previous underinvestment, CAISI has consistently produced rigorous model evaluations, and in doing so, has earned strong credibility with industry and government stakeholders. This also includes support from legislators: bipartisan legislation has been introduced in both chambers of Congress to authorize CAISI in statute, while just last month, the House China Committee released a letter noting that CAISI has a role to play in “understanding, predicting, and preparing for” national security risks from AI development in the PRC.

A dedicated and properly resourced national entity is essential for supporting the development of safe, secure, and trustworthy AI to drive widespread adoption, by providing sustained, independent technical assessments and emergency coordination—roles that ad-hoc industry consultations or self-reporting cannot fulfill for paramount matters of national security and public safety.

Establishing CAISI+ now is a critical opportunity to proactively manage these profound risks, ensure American leadership in AI, and prevent strategic disadvantage as global AI capabilities advance. While full operational capacity may not be needed immediately, certain infrastructure, such as highly secure computing, has significant lead times, demanding foresight and preparatory action. This blueprint offers a scalable framework to build these essential national capabilities, safeguarding our future against AI-related catastrophic events and enabling the U.S. to shape the trajectory of this transformative technology.

Plan of Action 

To effectively address extreme AI risks, develop more trustworthy AI systems, and secure U.S. interests, the Administration and Congress should collaborate to establish and resource a world-class national entity to inform the federal response to the above trendlines. 

Recommendation 1. Establish CAISI+ to Lead National AI Safety and Coordinate Crisis Response. 

CAISI+, evolving from the current CAISI within the National Institute of Standards and Technology, under the Department of Commerce, must have a clear mandate focused on large-scale AI risks. Core functions include: 

1. Advanced Model Evaluation: Developing and operating state-of-the-art platforms to test frontier AI models for dangerous capabilities, adversarial behavior or goals (such as deception or power-seeking), and potential weaponization. While the level of risk presented by current models is very uncertain, even those who are skeptical of particular risk models are often supportive of developing better evaluations. 
2. Emergency Assessment & Response: Providing rapid, expert risk assessments and warnings directly to the President and the National Security Council (NSC) in the event of severe AI-driven national security threats. The CAISI+ Director should be statutorily designated as the Principal Advisor on AI Risks to the President and NSC, with authority to:
   1. Submit AI threat assessments to the President’s Daily Brief (PDB) when intelligence indicates imminent or critical risks
   2. Convene emergency sessions of the NSC Deputies Committee or Principals Committee for time-sensitive AI security threats
   3. Maintain direct communication channels to the National Security Advisor for immediate threat notification
   4. Issue “Critical AI Threat Warnings” through established NSC emergency communication protocols, similar to those used for terrorism or WMD threats 
3. Foundational AI Reliability and Security Research: Driving and funding research into core AI alignment, control, and security challenges to maintain U.S. technological leadership while developing trustworthy AI systems. This research will yield dual benefits to both the public and industry, by enabling broader adoption of reliable AI tools and preventing catastrophic incidents that could devastate the AI sector, similar to how the Three Mile Island disaster impacted nuclear energy development. Following the model of NIST’s successful encryption standards, establishing rigorous AI safety benchmarks and protocols will create industry-wide confidence while ensuring American competitiveness.

Governance will feature clear interagency coordination (e.g., with the Department of Defense, Department of Energy, Department of Homeland Security, and other relevant bodies in the intelligence community) and an internal structure with distinct directorates for evaluations, emergency response, and research, coordinated by CAISI+ leadership.

Recommendation 2. Equip CAISI+ with Elite American Talent and Sustained Funding

CAISI+’s efficacy hinges on world-class personnel and reliable funding to execute its mission. This necessitates: 

1. Exceptional American Talent: Special hiring authorities (e.g., direct hire, excepted service) and competitive compensation are paramount to attract and retain leading U.S. AI researchers, evaluators, and security experts, ensuring our AI standards reflect American values.
2. Significant, Sustained Funding: Initial mainline estimates (see “Funding estimates for CAISI+” below) suggest $155-$275 million for setup and an annual operating budget of $67-$155 million for the recommended implementation level, sourced via new appropriations, to ensure America develops strong domestic capacity for defending against AI-powered threats. If funding is not appropriated, or if appropriations fall short, additional support may be able to be sourced via a NIST Foundation.

Funding estimates for CAISI+

Initial Setup (First 2 Years)

Category	Details	Cost of Recommended Implementation
Facility Construction	RAND Safety Level-5 (SL-5) Secure Computing Facility: 15,000+ sq ft at $1,500-3,000/sq ft for specialized construction; TEMPEST shielding and electromagnetic security; Multi-factor physical access controls and surveillance; Dedicated power/cooling infrastructure; Facility certification and accreditation	$50-70M*
Computing Infrastructure	Core SL-5 Cluster: ~128-512 Blackwell B200 GPUs (~$4-18M) + supporting infrastructure ($12M); Storage Systems: Secure, tiered storage architecture ($8-15M); Networking: Air-gapped internal networks + secure external interfaces ($3-5M); External Access: DOE lab arrangements ($8-15M)	$35-65M
Personnel (2-year)	Technical Staff: 60-120 AI researchers, evaluators, security experts ($18-60M); Leadership: 10-15 senior positions with SL/ST pay scales ($3.6-6.2M); Support: 30-60 admin, legal, procurement staff ($4-14.4M); Hiring Incentives: Recruitment bonuses, loan repayment, training ($10-20M)	$40-80M
Initial Operations	Platform Development: Evaluation frameworks, red team capabilities ($14-30M); Partnerships & Grants: Academic/industry collaboration setup ($8-15M); Model Access: Licensing, legal frameworks ($5-10M); Security Operations: Initial clearance processing, protocols ($3-5M)	$30-60M
Total Initial Investment		$155-275M

*Cost estimates for SL-5 facilities are highly uncertain due to limited precedent (see RAND SL study); actual costs could vary significantly.

Annual Operating Budget (Steady State, ~1.2x current UK AISI level)

Category	Details	Cost of Recommended Implementation
Personnel	Core Technical: 80-150 staff ($12-45M); Leadership: Senior positions ($1.8-3.1M annually); Specialized Contractors: High-end AI talent at competitive rates ($6-12M); Retention/Training: Ongoing development programs ($2-4M)	$22-64M
Compute Resources	Facility Power: 1.2MW consumption ($750K annually); DOE Lab Access: Formal computing allocations ($5-10M); Cloud/External: FedRAMP services for non-sensitive workloads ($3-7M); Hardware Refresh: Ongoing upgrades and replacements ($8-15M)	$20-40M
Operations & Maintenance	Facility Ops: Security personnel, maintenance, utilities ($5-10M); Security Operations: Continuous monitoring, threat assessment ($3-5M); Infrastructure Management: System administration, cybersecurity ($2-5M)	$10-20M
Research & Partnerships	External Grants: Academic partnerships ($8-15M for ~20 three-year academic projects at $400k/yr); International Coordination: Allied CAISI collaboration ($2-5M); Model Acquisition: Ongoing access agreements ($3-5M); Emergency Response: Crisis surge capacity maintenance ($2-5M)	$15-30M
Total Annual Budget		$67-155M

Implementation Considerations

• Phased approach: The facility could be developed in stages, prioritizing core evaluation capabilities before expanding to full emergency response capacity.
• Leverage existing assets: Initial operations could utilize existing DOE relationships rather than immediately building dedicated infrastructure.
• Partnership model: Some costs could be offset through public-private partnerships with technology companies and research institutions.
• Talent acquisition strategy: Use of special hiring authorities (direct hire, excepted service) and competitive compensation (SL/ST pay scales, retention bonuses) may help compete with private sector AI companies.
• Sustainable funding: For stability, a multi-year Congressional appropriation with dedicated line-item funding would be crucial.

Staffing Breakdown by Function

• Technical Research (40-60% of staff): AI evaluations, safety research, alignment, interpretability research
• Security Operations (25-35% of staff): Red-teaming, misuse assessment, weaponization evaluation, security management
• Policy & Strategy (10-15% of staff): Leadership, risk assessment, interagency coordination, international liaisons
• Support Functions (15-20% of staff): Legal, procurement, compute infrastructure management, administration

For context, current funding levels include:

• Current CAISI funding (mid-2025): $10 million annually
• UK AISI (CAISI counterpart) initial funding: £100 million (~$125 million)
• Oak Ridge Leadership Computing Facility operations: ~$200-300 million annually
• Standard DOE supercomputing facility construction: $400-600 million

Even the minimal implementation would require substantially greater resources than the current CAISI, but remains well within the scale of other national-priority technology initiatives. The recommended implementation level would position CAISI+ to effectively fulfill its expanded mission of frontier AI evaluation, monitoring, and emergency response.

Funding Longevity

• Initial authorization: 5-year authorization with specific milestones and metrics
• Review mechanism: Independent assessment by the Government Accountability Office at 3-year mark to evaluate effectiveness and adjust scope/resources, supplemented by a National Academies study specifically tasked with evaluating the scientific and technical rigor of the CAISI+.
• Long-term vision: Transition to permanent authorization for core functions with periodic reauthorization of specific initiatives
• Accountability: Annual reporting to Congress on key performance metrics and risk assessments

Recommendation 3. Equip CAISI+ with Essential Secure Compute Infrastructure. 

CAISI+ must be able to access secure compute in order to run certain evaluations involving proprietary models and national security data. This cluster can remain relatively modest in scale. Other researchers have hypothesized that a “Trusted AI Verification and Evaluation Cluster” for verifying and evaluating frontier AI development would need only 128 to 512 state-of-the-art graphical processing units (GPU)s–orders of magnitude smaller than the scale of training compute, such as the recent Llama 3.1 405 B model’s training run use of a 16,000 H100 GPU cluster, or xAI’s 200,000 GPU Colossus cluster. 

However, the cluster will need to be highly secure–in other words, able to defend against attacks from nation-state adversaries. Certain evaluations will require full access to the internal “weights” of AI models, which requires hosting the model. Model hosting introduces the risk of model theft and proliferation of dangerous capabilities. Some evaluations will also involve the use of very sensitive data, such as nuclear weapons design evals–introducing additional incentive for cyberattacks. Researchers at Gladstone AI, a national security-focused AI policy consulting firm, write that in several years, powerful AI systems may confer significant strategic advantages to nation-states, and will therefore be top-priority targets for theft or sabotage by adversary nation-states. They also note that neither existing datacenters nor AI labs are secure enough to prevent this theft–thereby necessitating novel research and buildout to reach the necessary security level, outlined as “Security Level-5” (SL-5) in RAND’s Playbook for Securing AI Model Weights.

Therefore, we suggest a hybrid strategy for specialized secure compute, featuring a highly secure SL-5 air-gapped core facility for sensitive model analysis (a long-lead item requiring immediate planning), with access to a secondary pool of compute for additional capacity to run less sensitive evaluations via a formal partnership with DOE to access national lab resources. CAISI+ may also want to coordinate with the NITRD National Strategic Computing Reserve Pilot Program to explore needs for AI-crisis-related surge computing capability.

If a sufficiently secure compute cluster is infeasible or not developed in time, CAISI+ will ultimately be unable to host model internals without introducing unacceptable risks of model theft, severely limiting its ability to evaluate frontier AI systems.

Recommendation 4. Explore Granting Critical Authorities 

While current legal authorities may suffice for CAISI+’s core missions, evolving AI threats could require additional tools. The White House (specifically the Office of Science and Technology Policy [OSTP], in collaboration with the Office of Management and Budget [OMB]) should analyze existing federal powers (such as the Defense Production Act or the International Emergency Economic Powers Act) to identify gaps in AI threat response capabilities–including potential needs for an incident reporting system and related subpoena authorities (similar to the function of the National Transportation Safety Board), or for model access for safety evaluations, or compute oversight authorities. Based on this analysis, the executive branch should report to Congress where new statutory authorities may be necessary, with defined risk criteria and appropriate safeguards.

Recommendation 5. Implement CAISI+ Enhancements Through Urgent, Phased Approach

Building on CAISI’s existing foundation within NIST/DoC, the Administration should enhance its capabilities to address AI risks that extend beyond current voluntary evaluation frameworks. Given expert warnings that transformative AI could emerge within the current Administration’s term, immediate action is essential to augment CAISI’s capacity to handle extreme scenarios. To achieve full operational capacity by early 2027, initial-phase activities must begin now  due to long infrastructure lead times:

Immediate Enhancements (0-6 months):

1. Leverage NIST’s existing relationships with DOE labs to secure interim access to classified computing facilities for sensitive evaluations
2. Initiate the security research and procurement process for the SL-5 compute facility outlined in Recommendation 3
3. Work with OMB and Department of Commerce leadership to secure initial funding through reprogramming or supplemental appropriations
4. Build on CAISI’s current voluntary agreements to develop protocols for emergency model access and crisis response
5. Begin the OSTP-led analysis of existing federal authorities (per Recommendation 4) to identify potential gaps in AI threat response capabilities

Subsequent phases will extend CAISI’s current work through:

• Foundation-building activities (6-12 months): Implementing the special hiring authorities described in Recommendation 2, formalizing enhanced interagency MOUs to support coordination described in Recommendation 1, and establishing the direct NSC reporting channels for the CAISI+ Director as Principal Advisor on AI Risks.
• Capability expansion (12-18 months): Beginning construction of the SL-5 facility, operationalizing the three core functions (Advanced Model Evaluation, Emergency Assessment & Response, and Foundational AI Reliability Research), and recruiting the 80-150 technical staff outlined in the funding breakdown.
• Full enhanced capacity (18+ months): Achieving the operational capabilities described in Recommendation 1, including mature evaluation platforms, direct Presidential/NSC threat warning protocols, and comprehensive research programs.

Conclusion

Enhancing and empowering CAISI+ is a strategic investment in U.S. national security, far outweighed by the potential costs of inaction on this front. With an estimated annual operating budget of $67-155 million, CAISI+ will provide essential technical capabilities to evaluate and respond to the most serious AI risks, ensuring the U.S. leads in developing and governing AI safely and securely, irrespective of where advanced capabilities emerge. While timelines to AI systems surpassing dangerous capability thresholds are uncertain, by acting now to establish the necessary infrastructure, expertise, and authorities, the Administration can safeguard American interests and our technological future through a broad range of possible scenarios.

This memo was written by an AI Safety Policy Entrepreneurship Fellow over the course of a six-month, part-time program that supports individuals in advancing their policy ideas into practice. You can read more policy memos and learn about Policy Entrepreneurship Fellows here.

States and localities are eager to leverage artificial intelligence (AI) to optimize service delivery and infrastructure management, but they face significant resource gaps. Without sufficient personnel and capital, these jurisdictions cannot properly identify and mitigate the risks associated with AI adoption, including cyber threats, surging power demands, and data privacy issues. Congress should establish a new grant program, coordinated by the Cybersecurity and Infrastructure Security Agency (CISA), to assist state and local governments in addressing these challenges. Such funding will allow the federal government to instill best security and operating practices nationwide, while identifying effective strategies from the grassroots that can inform federal rulemaking. Ultimately, federal, state, and local capacity are interrelated; federal investments in state and local government will help the entire country harness AI’s potential and reduce the risk of catastrophic events such as a large, AI-powered cyberattack.

Challenge and Opportunity 

In 2025, 45 state legislatures have introduced more than 550 bills focused on the regulation of artificial intelligence, covering everything from procurement guidelines to acceptable AI uses in K-12 education to liability standards for AI misuse and error. Major cities have followed suit with sweeping guidance of their own, identifying specific AI risks related to bias and hallucination and directives to reduce their impact on government functions. The influx of regulatory action reflects burgeoning enthusiasm about AI’s ability to streamline public services and increase government efficiency.

Yet two key roadblocks stand in the way: inconsistent rules and uneven capacity. AI regulations vary widely across jurisdictions — sometimes offering contradictory guidance — and public agencies often lack the staff and skills needed to implement them. In a 2024 survey, six in ten public sector professionals cited the AI skills gap as their biggest obstacle in implementing AI tools. This reflects a broader IT staffing crisis, with over 450,000 unfilled cybersecurity roles nationwide, which is particularly acute in the public sector given lower salaries and smaller budgets.

These roadblocks at the state and local level pose a major risk to the entire country. In the cyber space, ransomware attacks on state and local targets have demonstrated that hackers can exploit small vulnerabilities in legacy systems to gain broad access and cause major disruption, extending far beyond their initial targets. The same threat trajectory is conceivable with AI. States and cities, lacking the necessary workforce and adhering to a patchwork of different regulations, will find themselves unable to safely adopt AI tools and mount a uniform response in an AI-related crisis. 

In 2021, Congress established the State and Local Cybersecurity Grant Program (SLCGP) at CISA, which focused on resourcing states, localities, and tribal territories to better respond to cyber threats. States have received almost $1 billion in funding to implement CISA’s security best practices like multifactor authentication and establish cybersecurity planning committees, which effectively coordinate strategic planning and cyber governance among state, municipal, and private sector information technology leaders. 

Federal investment in state and local AI capacity-building can help standardize the existing, disparate guidance and bridge resource gaps, just as it has in the cybersecurity space. AI coordination is less mature today than the cybersecurity space was when the SLCGP was established in 2021. The updated Federal Information Security Modernization Act, which enabled the Department of Homeland Security to set information security standards across government, had been in effect for seven years by 2021, and some of its best practices had already trickled down to states and localities. 

Thus, the need for clear AI state capacity, guardrails, and information-sharing across all levels of government is even greater. A small federal investment now can unlock large returns by enabling safe, effective AI adoption and avoiding costly failures. Local governments are eager to deploy AI but lack the resources to do so securely. Modest funding can align fragmented rules, train high-impact personnel, and surface replicable models—lowering the cost of responsible AI use nationwide. Each successful pilot creates a multiplier effect, accelerating progress while reducing risk.

Plan of Action 

Recommendation 1. Congress should authorize a three-year pilot grant program focused on state and local AI capacity-building.

SLCGP’s authorization expires on August 31, 2025, which provides two unique pathways for a pilot grant program. The Homeland Security Committees in the House and Senate could amend and renew the existing SLCGP provision to make room for an AI-focused pilot. Alternatively, Congress could pass a new authorization, which would likely set the stage for a sustained grant program, upon successful completion of the pilot. A separate authorization would also allow Congress to consider other federal agencies as program facilitators or co-facilitators, in case they want to cover AI integrations that do not directly touch critical infrastructure, which is CISA’s primary focus. 

Alternatively, the House Energy and Commerce and Senate Commerce, Science, and Transportation Committees could authorize a program coordinated by the National Institute of Standards and Technology, which produced the AI Risk Management Framework and has strong expertise in a range of vulnerabilities embedded within AI models. Congress might also consider mandating an interagency advisory committee to oversee the program, including, for example, experts from the Department of Energy to provide technical assistance and guidance on projects related to energy infrastructure.

In either case, the authorization should be coupled with a starting appropriation of $55 million over three years, which would fund ten statewide pilot projects totaling up to $5 million plus administrative costs. The structure of the program will broadly parallel SLCGP’s goals. First, it would align state and local AI approaches with existing federal guidance, such as the NIST AI Risk Management Framework and the Trump Administration’s OMB guidance on the regulation and procurement of artificial intelligence applications. Second, the program would establish better coordination between local and state authorities on AI rules. A new authorization for AI, however, allows Congress and the agency tasked with managing the program the opportunity to improve upon SLCGP’s existing provisions. This new program should permit states to coordinate their AI activities through existing leadership structures rather than setting up a new planning committee. The legislative language should also prioritize skills training and allocate a portion of grant funding to be spent on recruiting and retaining AI professionals within state and local government who can oversee projects.

Recommendation 2. Pilot projects should be implementation-focused and rooted in one of three significant risks: cybersecurity, energy usage, or data privacy.

Similar to SLCGP, this pilot grant program should be focused on implementation. The target product for a grant is a functional local or state AI application that has undergone risk mitigation, rather than a report that identifies issues in the abstract. For example, under this program, a state would receive federal funding to integrate AI into the maintenance of its cities’ wastewater treatment plants without compromising cybersecurity. Funding would support AI skills training for the relevant municipal employees and scaling of certain cybersecurity best practices like data encryption that minimize the project’s risk. States will submit reports to the federal government at each phase of their project: first documenting the risks they identified, then explaining their prioritization of risks to mitigate, then walking through their specific mitigation actions, and later, retrospectively reporting on the outcomes of those mitigations after the project has gone into operational use.

This approach would maximize the pilot’s return on investment. States will be able to complete high-impact AI projects without taking on the associated security costs. The frameworks generated from the project can be reused many times over for later projects, as can the staff who are hired or trained with federal support. 

Given the inconsistency of priorities surfaced in state and local AI directives, the federal government should set the agenda of risks to focus on. The clearest set of risks for the pilot are cybersecurity, energy usage, and data privacy, all of which are highlighted in NIST’s Risk Management Framework. 

• Cybersecurity. Cybersecurity projects should focus on detecting AI-assisted social engineering tactics, used to gain access into secure systems, and adversarial attacks like “poisoning” or “jailbreaking”, which manipulate AI models to produce undesirable outputs. Consider emergency response systems: the transition to IP-based, interconnected 911 systems increases the cyberattack surface, making it easier for an attack targeting one response center to spread across other jurisdictions. A municipality  could seek funding to trial an AI dispatcher with necessary guardrails. As part of their project, they could ensure they have the appropriate cyber hygiene protocols in place to prevent cyberattacks from rendering the dispatcher useless or exploiting vulnerabilities in the dispatcher to gain access to underlying 911 systems that multiple localities rely on. 

• Energy Usage. Energy usage projects should calculate power needs associated with AI development and implementation and the additional energy resources available to prevent outages. Much of the country faces a heightened risk of power outages due to antiquated grids, under-resourced providers, and a dearth of new electricity generation. AI integrations and supportive infrastructure that require significant power will place a heavy burden on states and potentially impact the operation of other critical infrastructure. A sample project might examine the energy demands of a new data center, powering an AI integration into traffic monitoring, and determine where that data center can best be constructed to accommodate available grid capacity.

• Data Privacy. Finally, data privacy projects should focus on bringing AI systems into compliance with existing data laws like the Health Insurance Portability and Accountability Act (HIPAA) and the Children’s Online Privacy Protection Act (COPPA) for AI interventions in healthcare and education, respectively. Because the U.S. lacks a comprehensive data privacy law, states might also experiment with additional best practices, such as training models to detect and reject prompts that contain personally identifiable information. A sample project in this domain might integrate a chatbot into the state Medicaid system to more efficiently triage patients and identify the steps the state can take to prevent the chatbot from handling PII in a manner that does not comply with HIPAA.

If successful, the pilot could expand to address additional risks or support broader, multi-risk, multi-state interventions.

Recommendation 3. The pilot program must include opportunities for grantees to share their ideas with other states and localities.

Arguably the most important facet of this new AI program will be forums where grantees share their learnings. Administrative costs for this program should go toward funding a twice-yearly (bi-annual) in-person forum, where grantees can publicly share updates on their projects. An in-person forum would also provide states with the space to coordinate further projects on the margins. CISA is particularly well positioned to host a forum like this given its track record of convening critical infrastructure operators. Grantees should be required to publish guidance, tools, and templates in a public, digital repository. Ideally, states that did not secure grants can adopt successful strategies from their peers and save taxpayers the cost of duplicate planning work. 

Conclusion 

Congress should establish a new grant program to assist state and local governments in addressing AI risks, including cybersecurity, energy usage, and data privacy. Such federal investments will give structure to the dynamic yet disparate national AI regulatory conversation. The grant program, which will cost $55 million to pilot over three years, will yield a high return on investment for both the ten grantee states and the peers that learn from its findings. By making these investments now, Congress can keep states moving fast toward AI without opening the door to critical, costly vulnerabilities.

This memo was written by an AI Safety Policy Entrepreneurship Fellow over the course of a six-month, part-time program that supports individuals in advancing their policy ideas into practice. You can read more policy memos and learn about Policy Entrepreneurship Fellows here.

Frequently Asked Questions

Does Congress have to authorize a new grant program to operate this pilot?

No, Congress could leverage SLCGP’s existing authorization to focus on projects that look at the intersection of AI and cybersecurity. They could offer an amendment to the next Homeland Security Appropriations package that directs modest SLCGP funding (e.g. $10-20 million) to AI projects. Alternatively, Congress could insert language on AI into SLCGP’s reauthorization, which is due on August 31, 2025.

Although leveraging the existing authorization would be easier, Congress would be better served by authorizing a new program, which can focus on multiple priorities including energy usage and data privacy. To stay agile, the language in the statute could allow CISA to direct funds toward new emerging risks, as they are identified by NIST and other agencies. Finally, a specific authorization would pave the way for an expansion of this program assuming the initial 10 state pilot goes well.

read more

Why focus on individual state and local projects rather than an across-the-board effort to improve capacity in all states across all vectors?

This pilot is right-sized for efficiency, impact, and cost savings. A program to bring all 50 states into compliance with certain AI risk mitigation guidelines would cost hundreds of millions, which is not feasible in the current budgetary environment. States are starting from very different baselines, especially with their energy infrastructure, which makes it difficult to bring them all to a single end-point. Moreover, because AI is evolving so rapidly, guidance is likely to age poorly. The energy needs of AI might change before states finish their plan to build data centers. Similarly, federal data privacy laws might go in place that undercut or contradict the best practices established by this program.

read more

What are the benefits to this deployment approach?

This pilot will allow 10 states and/or localities to quickly deploy AI implementations that produce real value: for example, quicker emergency response times and savings on infrastructure maintenance. CISA can learn from the grantees’ experiences to iterate on federal guidance. They might identify a stumbling block on one project and refine their guidance to prevent 49 other states from encountering the same obstacle. If grantees effectively share their learnings, they can cut massive amounts of time off other states’ planning processes and help the federal government build guidance that is more rooted in the realities of AI deployment.

read more

Some have expressed concerns that planning-focused grants create additional layers of bureaucracy. Will this pilot just add more red tape to AI integration?

No. If done correctly, this pilot will cut red tape and allow the entire country to harness AI’s positive potential. States and localities are developing AI regulations in a vacuum. Some of the laws proposed are contradictory or duplicative precisely because many state legislatures are not coordinating effectively with state and local government technical experts. When bills do pass, guidance is often poorly implemented because there is no overarching figure, beyond a state chief information officer, to bring departments and cities into compliance. In essence, 50 states are producing 50 sets of regulations because there is scant federal guidance and few mechanisms for them to learn from other states and coordinate within their state on best practices.

read more

How will this program streamline and optimize state and local AI planning processes?

This program aims to cut down on bureaucratic redundancy by leveraging states’ existing cyber planning bodies to take a comprehensive approach to AI. By convening the appropriate stakeholders from the public sector, private sector, and academia to work on a funded AI project, states will develop more efficient coordination processes and identify regulations that stand in the way of effective technological implementation. States and localities across the country will build their guidelines based on successful grantee projects, absorbing best practices and casting aside inefficient rules. It is impossible to mount a coordinated response to significant challenges like AI-enabled cyberattacks without some centralized government planning, but this pilot is designed to foster efficient and effective coordination across federal, state, and local governments.

read more

The most advanced AI systems remain ‘black boxes’ whose inner workings even their developers cannot fully understand, leading to issues with reliability and trustworthiness. However, as AI systems become more capable, there is a growing desire to deploy them in high-stakes scenarios. The bipartisan National Security Commission on AI cautioned that AI systems perceived as unreliable or unpredictable will ‘stall out’: leaders will not adopt them, operators will mistrust them, Congress will not fund them, and the public will not support them (NSCAI, Final Report, 2021). AI interpretability research—the science of opening these black boxes and attempting to comprehend why they do what they do—could turn opacity into understanding and enable wider AI adoption.

With AI capabilities racing ahead, the United States should accelerate interpretability research now to keep its technological edge and field high-stakes AI deployment with justified confidence. This memorandum describes three policy recommendations that could help the United States seize the moment and maintain a lead on AI interpretability: (1) creatively investing in interpretability research, (2) entering into research and development agreements between interpretability experts and government agencies and laboratories, and (3) prioritizing interpretable AI in federal procurement. 

Challenge and Opportunity

AI capabilities are progressing rapidly. According to many frontier AI companies’ CEOs and independent researchers, AI systems could reach general-purpose capabilities that equal or even surpass humans within the next decade. As capabilities progress, there is a growing desire to incorporate these systems into high-stakes use cases, from military and intelligence uses (DARPA, 2025; Ewbank, 2024) to key sectors of the economy (AI for American Industry, 2025). 

However, the most advanced AI systems are still ‘black boxes’ (Sharkey et al., 2024) that we observe from the outside and that we ‘grow,’ more than we ‘build’ (Olah, 2024). Our limited comprehension of the inner workings of neural networks means that we still really do not understand what happens within these black boxes, leaving uncertainty regarding their safety and reliability. This could have resounding consequences. As the 2021 final report of the National Security Commission on AI (NSCAI) highlighted, “[i]f AI systems routinely do not work as designed or are unpredictable in ways that can have significant negative consequences, then leaders will not adopt them, operators will not use them, Congress will not fund them, and the American people will not support them” (NSCAI, Final Report, 2021). In other words, if AI systems are not always reliable and secure, this could inhibit or limit their adoption, especially in high-stakes scenarios, potentially compromising the AI leadership and national security goals outlined in the Trump administration’s agenda (Executive Order, 2025). 

AI interpretability is a subfield of AI safety that is specifically concerned with opening and peeking inside the black box to comprehend “why AI systems do what they do, and … put this into human-understandable terms” (Nanda, 2024; Sharkey et al., 2025). In other words, interpretability is the AI equivalent of an MRI (Amodei, 2025) because it attempts to provide observers with an understandable image of the hidden internal processes of AI systems.

The Challenge of Understanding AI Systems Before They Reach or Even Surpass Human-Level Capabilities

Recent years have brought breakthroughs across several research areas focused on making AI more trustworthy and reliable, including in AI interpretability. Among other efforts, the same companies developing the most advanced AI systems have designed systems that are easier to understand and have reached new research milestones (Marks et al., 2025; Lindsey et al., 2025; Lieberum et al. 2024; Kramar et al., 2024; Gao et al., 2024; Tillman & Mossing, 2025). 

AI interpretability, however, is still trailing behind raw AI capabilities. AI companies project that it could take 5–10 years to reliably understand model internals (Amodei, 2025), while experts expect systems exhibiting human‑level general-purpose capabilities by as early as 2027 (Kokotajlo et al., 2025). That gap will force policymakers into a difficult corner once AI systems reach similar capabilities: deploy unprecedentedly powerful yet opaque systems, or slow deployment and fall behind. Unless interpretability accelerates, the United States could risk both competitive and security advantages. 

The Challenge of Trusting Today’s Systems for High-Stakes Applications

We must understand the inner workings of highly advanced AI systems before they reach human or above-human general-purpose capabilities, especially if we want to trust them in high-stakes scenarios. There are several reasons why current AI systems might not always be reliable and secure. For instance, AI systems could exhibit the following vulnerabilities. First, AI systems inherit the blind spots of their training data. When the world changes—alliances shift, governments fall, regulations update—systems still reason from outdated facts, undermining reliability in high-stakes diplomatic or military settings (Jensen et al., 2025).

Second, AI systems are unusually easy to strip‑mine for memorized secrets, especially if these secrets come as uncommon word combinations (e.g., proprietary blueprints). Data‑extraction attacks are now “practical and highly realistic” and will grow even more effective as system size increases (Carlini et al., 2021; Nasr et al., 2023; Li et al., 2025). The result could be wholesale leakage of classified or proprietary information (DON, 2023).

Third, cleverly crafted prompts can still jailbreak cutting‑edge systems, bypassing safety rails and exposing embedded hazardous knowledge (Hughes et al., 2024; Ramesh et al., 2024). With attack success rates remaining uncomfortably high across even the leading systems, adversaries could manipulate AI systems with these vulnerabilities in real‑time national security scenarios (Caballero & Jenkins, 2024).

This is not a comprehensive list. Systems could exhibit vulnerabilities in high-stakes applications for many other reasons. For instance, AI systems could be misaligned and engage in scheming behavior (Meinke et al., 2024; Phuong et al., 2025) or have baked-in backdoors that an attacker could exploit (Hubinger et al., 2024; Davidson et al., 2025). 

The Opportunity to Promote AI Leadership Through Interpretability

Interpretability offers an opportunity to address these described challenges and reduce barriers to the safe adoption of the most advanced AI systems, thereby further promoting innovation and increasing the existing advantages those systems present over adversaries’ systems. In this sense, accelerating interpretability could help promote and secure U.S. AI leadership (Bau et al., 2025; IFP, 2025). For example, by helping ensure that highly advanced AI systems are deployed safely in high-stakes scenarios, interpretability could improve national security and help mitigate the risk of state and non-state adversaries using AI capabilities against the United States (NSCAI, Final Report, 2021). Interpretability could therefore serve as a front‑line defense against vulnerabilities in today’s most advanced AI systems.

Making future AI systems safe and trustworthy could become easier the more we understand how they work (Shah et al., 2025). Anthropic’s CEO recently endorsed the importance and urgency of interpretability, noting that “every advance in interpretability quantitatively increases our ability to look inside models and diagnose their problems” (Amodei, 2025). This means that interpretability not only enhances reliability in the deployment of today’s AI systems, but understanding AI systems could also lead to breakthroughs in designing more targeted systems or attaining more robust monitoring of deployed systems. This could then enable the United States to deploy tomorrow’s human-level or above-human general-purpose AI systems with increased confidence, thus securing strategic advantages when engaging  geopolitically. The following uses the vulnerabilities discussed above to demonstrate three ways in which interpretability could improve the reliability of today’s AI systems when deployed in high-stakes scenarios.

First, interpretability could help systems selectively update outdated information through model editing, without risking a reduction in performance. Model editing allows us to selectively inject new facts or fix mistakes (Cohen et al., 2023; Hase et al., 2024) by editing activations without updating the entire model. However, this ‘surgical tool’ has shown ‘side effects’ causing performance degradation (Gu et al., 2024; Gupta et al., 2024). Interpretability could help us understand how stored knowledge alters parameters as well as develop stronger memorization measures (Yao et al., 2023; Carlini et al., 2019), enabling us to ‘incise and excise’ AI models with fewer side effects.

Second, interpretability could help systems selectively forget training data through machine unlearning, once again without losing performance. Machine unlearning allows systems to forget specific data classes (such as memorized secrets or hazardous knowledge) while remembering the rest (Tarun et al., 2023). Like model editing, this ‘surgical tool’ suffers from performance degradation. Interpretability could help develop new unlearning techniques that preserve performance (Guo et al., 2024; Belrose et al., 2023; Zou et al., 2024).

Third, interpretability could help effectively block jailbreak attempts, which can only currently be discovered empirically (Amodei, 2025). Interpretability could lead to a breakthrough in understanding models’ persistent vulnerability to jailbreaking by allowing us to characterize dangerous knowledge. Existing interpretability research has already analyzed how AI models process harmful prompts (He et al., 2024; Ball et al., 2024; Lin et al., 2024; Zhou et al., 2024), and additional research could build on these initial findings

The conditions are ripe to promote technological leadership and national security through interpretability. Many of the same problems that were highlighted in the 2019 National AI R&D Strategic Plan remained the same in its 2023 update, echoing those included in NSCAI’s 2021 final report. We have made relatively little progress addressing these challenges. AI systems are still vulnerable to attacks (NSCAI, Final Report, 2021) and can still “be made do the wrong thing, reveal the wrong thing” and “be easily fooled, evaded, and misled in ways that can have profound security implications” (National AI R&D Strategic Plan, 2019). The field of interpretability is gaining some momentum among AI companies (Amodei, 2025; Shah et al., 2025; Goodfire, 2025) and AI researchers (IFP, 2025; Bau et al., 2025; FAS, 2025). 

To be sure, despite recent progress, interpretability remains challenging and has attracted some skepticism (Hendrycks & Hiscott, 2025). Accordingly, a strong AI safety strategy must include many components beyond interpretability, including robust AI evaluations (Apollo Research, 2025) and control measures (Redwood Research, 2025).

Plan of Action 

The United States has an opportunity to seize the moment and lead an acceleration of AI interpretability. The following three recommendations establish a strategy for how the United States could promptly incentivize AI interpretability research. 

Recommendation 1. The federal government should prioritize and invest in foundational AI interpretability research, which would include identifying interpretability as a ‘strategic priority’ in the 2025 update of the National AI R&D Strategic Plan. 

The National Science and Technology Council (NSTC) should identify AI interpretability as a ‘strategic priority’ in the upcoming National AI R&D Strategic Plan. Congress should then appropriate federal R&D funding for federal agencies (including DARPA and the NSF) to catalyze and support AI interpretability acceleration through various mechanisms, including grants and prizes, R&D credits, tax credits, advanced market commitments, and buyer-of-first-resort mechanisms. 

This first recommendation echoes not only the 2019 update of the National AI R&D Strategic Plan and NSCAI’s 2021 final report––which recommended allocating more federal R&D investments to advance the interpretability of Al systems (NSCAI, Final Report, 2021; National AI R&D Strategic Plan, 2019),, but also the more recent remarks by the Director of the Office of Science and Technology Policy (OSTP), according to whom we need creative R&D funding approaches to enable scientists and engineers to create new theories and put them into practice (OSTP Director’s Remarks, 2025). This recommendation is also in line with calls from AI companies, asserting that “we still need significant investment in ‘basic science’” (Shah et al., 2025).

The United States could incentivize and support AI interpretability work through various approaches. In addition to prize competitions, advanced market commitments, fast and flexible grants (OSTP Director’s Remarks, 2025; Institute for Progress, 2025), and challenge-based acquisition programs (Institute for Progress, 2025), funding mechanisms could include R&D tax credits for AI companies undertaking or investing in interpretability research, and tax credits to adopters of interpretable AI, such as downstream deployers. If the federal government acts as “an early adopter and avid promoter of American technology” (OSTP Director’s Remarks, 2025), federal agencies could also rely on buyer-of-first-resort mechanisms for interpretability platforms. 

These strategies may require developing a clearer understanding of which frontier AI companies undertake sufficient interpretability efforts when developing their most advanced systems, and which companies currently do not. Requiring AI companies to disclose how they use interpretability to test models before release (Amodei, 2025) could be helpful, but might not be enough to devise a ‘ranking’ of interpretability efforts. While potentially premature given the state of the art in interpretability, an option could be to start developing standardized metrics and benchmarks to evaluate interpretability (Mueller et al., 2025; Stephenson et al., 2025). This task could be carried out by the National Institute of Standards and Technology (NIST), within which some AI researchers have recommended creating an AI Interpretability and Control Standards Working Group (Bau et al., 2025). 

A great way to operationalize this first recommendation would be for the National Science and Technology Council (NSTC) to include interpretability as a “strategic priority” in the 2025 update of the National AI R&D Strategic Plan (RFI, 2025). These “strategic priorities” seek to target and focus AI innovation for the next 3–5 years, paying particular attention to areas of “high-risk, high-reward AI research” that the industry is unlikely to address because it may not provide immediate commercial returns (RFI, 2025). If interpretability were included as a “strategic priority,” then the Office of Management and Budget (OMB) could instruct agencies to align their budgets with the 2025 National AI R&D Strategic Plan priorities in its memorandum addressed to executive department heads. Relevant agencies, including DARPA and the National Science Foundation (NSF), would then develop their budget requests for Congress, aligning them with the 2025 National AI R&D Strategic Plan and the OMB memorandum. After Congress reviews these proposals and appropriates funding, agencies could launch initiatives that incentivize interpretability work, including grants and prizes, R&D credits, tax credits, advanced market commitments, and buyer-of-first-resort mechanisms. 

Recommendation 2. The federal government should enter into research and development agreements with AI companies and interpretability research organizations to red team AI systems applied in high-stakes scenarios and conduct targeted interpretability research. 

AI companies, interpretability organizations, and federal agencies and laboratories (such as DARPA, the NSF, and the U.S. Center for AI Standards and Innovation) should enter into research and development agreements to pursue targeted AI interpretability research to solve national security vulnerabilities identified through security-focused red teaming. 

This second recommendation takes into account the fact that the federal government possesses unique expertise and knowledge in national security issues to support national security testing and evaluation (FMF, 2025). Federal agencies and laboratories (such as DARPA, the NSF, and the U.S. Center for AI Standards and Innovation), frontier AI companies, and interpretability organizations could enter into research and development agreements to undertake red teaming of national security vulnerabilities (as, for instance, SABER which aims to assess AI-enabled battlefield systems for the DoD; SABER, 2025) and provide state-of-the-art interpretability platforms to patch the revealed vulnerabilities. In the future, AI companies could also apply the most advanced AI systems to support interpretability research. 

Recommendation 3. The federal government should prioritize interpretable AI in federal procurement, especially for high-stakes applications. 

If federal agencies are procuring highly advanced AI for high-stakes scenarios and national security missions, they should preferentially procure interpretable AI systems. This preference could be accounted for by weighing the lack of understanding of an AI system’s inner workings when calculating cost. 

This third and final recommendation provides for the interim and assumes interpretable AI systems will coexist in a ‘gradient of interpretability’ with other AI systems that are less interpretable. In that scenario, agencies procuring AI systems should give preference to AI systems that are more interpretable. One way to account for this preference would be by weighing the potential vulnerabilities of uninterpretable AI systems within calculating costs during federal acquisition analyses. This recommendation also requires establishing a defined ‘ranking’ of interpretability efforts. While defining this ranking is currently challenging, the research outlined in recommendations 1 and 2 could better position the government to measure and rank the interpretability of different AI systems. 

Conclusion 

Now is the time for the United States to take action and lead the charge on AI interpretability research. While research is never guaranteed to lead to desired outcomes or to solve persistent problems, the potential high reward—understanding and trusting future AI systems and making today’s systems more robust to adversarial attacks—justifies this investment. Not only could AI interpretability make AI safer and more secure, but it could also establish justified confidence in the prompt adoption of future systems that are as capable as or even more capable than humans, and enable the deployment of today’s most advanced AI systems to high-stakes scenarios, thus promoting AI leadership and national security. With this goal in mind, this policy memorandum recommends that the United States, through the relevant federal agencies and laboratories (including DARPA, the NSF, and the U.S. Center for AI Standards and Innovation), invest in interpretability research, form research and development agreements to red team high-stakes AI systems and undertake targeted interpretability research, and prioritize interpretable AI systems in federal acquisitions.

Acknowledgments 

I wish to thank Oliver Stephenson, Dan Braun, Lee Sharkey, and Lucius Bushnaq for their ideas, comments, and feedback on this memorandum.

This memo was written by an AI Safety Policy Entrepreneurship Fellow over the course of a six-month, part-time program that supports individuals in advancing their policy ideas into practice. You can read more policy memos and learn about Policy Entrepreneurship Fellows here.

The opportunities presented by advanced artificial intelligence are immense, from accelerating cutting-edge scientific research to improving key government services. However, for these benefits to be realized, both the private and public sectors need confidence that AI tools are reliable and secure. This will require R&D effort to solve urgent technical challenges related to understanding and evaluating emergent AI behaviors and capabilities, securing AI hardware and infrastructure, and preparing for a world with many advanced AI agents.

To secure global adoption of U.S. AI technology and ensure America’s workforce can fully leverage advanced AI, the federal government should take a strategic and coordinated approach to support AI assurance and security R&D by: clearly defining AI assurance and security R&D priorities; establishing an AI R&D consortium and deploying agile funding mechanisms for critical R&D areas; and establishing an AI Frontier Science Fellowship to ensure a pipeline of technical AI talent. 

Challenge and Opportunity

AI systems have progressed rapidly in the past few years, demonstrating human-level and even superhuman performance across diverse tasks. Yet, they remain plagued by flaws that produce unpredictable and potentially dangerous failures. Frontier systems are vulnerable to attacks that can manipulate them into executing unintended actions, hallucinate convincing but incorrect information, and exhibit other behaviors that researchers struggle to predict or control.

As AI capabilities rapidly advance toward more consequential applications—from medical diagnosis to financial decision-making to military systems—these reliability issues could pose increasingly severe risks to public safety and national security, while reducing beneficial uses. Recent polling shows that just 32% of Americans trust AI, and this limited trust will slow the uptake of impactful AI use-cases that could drive economic growth and enhance national competitiveness.

The federal government has an opportunity to secure America’s technological lead and promote global adoption of U.S. AI by catalyzing research to address urgent AI reliability and security challenges—challenges that align with broader policy consensus reflected in the National Security Commission on AI’s recommendations and bipartisan legislative efforts like the VET AI Act. Recent research has surfaced substantial expert consensus around priority research areas that address the following three challenges.

The first challenge involves understanding emergent AI capabilities and behaviors. As AI systems get larger, also referred to as “scaling”, they develop unexpected capabilities and reasoning patterns that researchers cannot predict, making it difficult to anticipate risks or ensure reliable performance. Addressing this means advancing the science of AI scaling and evaluations.

Scientific methods to understand emergent behaviors and capabilities

This research aims to build a scientific understanding of how AI systems learn, reason, and exhibit diverse capabilities. This involves not only studying specific phenomena like emergence and scaling but, more broadly, employing and refining evaluations as the core empirical methodology to characterize all facets of AI behavior. This includes evaluations in areas such as CBRN weapons, cybersecurity, and deception, and broader research on AI evaluations to ensure that AI systems can be accurately assessed and understood. Example work includes Wijk et al. (2024) and McKenzie et al. (2023)

The second challenge is securing AI hardware and infrastructure. AI systems require robust protection of model weights, secure deployment environments, and resilient supply chains to prevent theft, manipulation, or compromise by malicious actors seeking to exploit these powerful technologies. Addressing this means advancing hardware and infrastructure security for AI.

Hardware and infrastructure security for AI

Ensuring the security of AI systems at the hardware and infrastructure level involves protecting model weights, securing deployment environments, maintaining supply chain integrity, and implementing robust monitoring and threat detection mechanisms. Methods include the use of confidential computing, rigorous access controls, specialized hardware protections, and continuous security oversight. Example work includes Nevo et al. (2024) and Hepworth et al. (2024)

The third challenge involves preparing for a world with many AI agents—AI models that can act autonomously. Alongside their potentially immense benefits, the increasing deployment of AI agents creates critical blind spots, as agents could coordinate covertly beyond human oversight, amplify failures into system-wide cascades, and combine capabilities in ways that circumvent existing safeguards. Addressing this means advancing agent metrology, infrastructure, and security.

Agent metrology, infrastructure, and security

Developing a deeper understanding of agentic behavior in LLM-based systems, including clarifying how LLM agents learn over time, respond to underspecified goals, and engage with their environments. This also includes research that ensures safe multi-agent interactions, such as detecting and preventing malicious collective behaviors, studying how transparency can affect agent interactions, and developing evaluations for agent behavior and interaction. Example work includes Lee and Tiwari (2024) and Chan et al. (2024)

While academic and industry researchers have made progress on these problems, this progress is not keeping pace with AI development and deployment. The market is likely to underinvest in research that is more experimental or with no immediate commercial applications. The U.S. government, as the R&D lab of the world, has an opportunity to unlock AI’s transformative potential through accelerating assurance and security research.

Plan of Action

The rapid pace of AI advancement demands a new strategic, coordinated approach to federal R&D for AI assurance and security.  Given financial constraints, it is more important than ever to make sure that the impact of every dollar invested in R&D is maximized.

Much of the critical technical expertise now resides in universities, startups, and leading AI companies rather than traditional government labs. To harness this distributed talent, we need R&D mechanisms that move at the pace of innovation, leverage academic research excellence, engage early-career scientists who drive breakthroughs, and partner with industry leaders who can share access to essential compute resources and frontier models. Traditional bureaucratic processes risk leaving federal efforts perpetually behind the curve.

The U.S. government should implement a three-pronged plan to advance the above R&D priorities.

Recommendation 1. Clearly define AI assurance and security R&D priorities

The Office of Science and Technology Policy (OSTP) and the National Science Foundation (NSF) should highlight critical areas of AI assurance and security as R&D priorities by including these in the 2025 update of the National AI R&D Strategic Plan and the forthcoming AI Action Plan. All federal agencies conducting AI R&D should engage with the construction of these plans to explain how their expertise could best contribute to these goals. For example, the Defense Advanced Research Projects Agency (DARPA)’s Information Innovation Office could leverage its expertise in AI security to investigate ways to design secure interaction protocols and environments for AI agents that eliminate risks from rogue agents.

The priorities would help coordinate government R&D activities by providing funding agencies with a common set of priorities, public research institutes such as the National Labs to conduct fundamental R&D activities, Congress with information to support relevant legislative decisions, and industry to serve as a guide to R&D.

Additionally, given the dynamic nature of frontier AI research, OSTP and NSF should publish an annual survey of progress in critical AI assurance and security areas and identify which challenges are the highest priority.

Recommendation 2. Establish an AI R&D consortium and deploy agile funding mechanisms for critical R&D

As noted by OSTP Director Michael Kratsios, “prizes, challenges, public-private partnerships, and other novel funding mechanisms, can multiply the impact of targeted federal dollars. We must tie grants to clear strategic targets, while still allowing for the openness of scientific exploration.” Federal funding agencies should develop and implement agile funding mechanisms for AI assurance and security R&D in line with established priorities. Congress should include reporting language in its Commerce, Justice, Science (CJS) appropriations bill that supports accelerated R&D disbursements for investment into prioritized areas.

A central mechanism should be the creation of an AI Assurance and Security R&D Consortium, jointly led by DARPA and NSF, bringing together government, AI companies, and universities. In this model:

• Government provides funding for personnel, administrative support, and manages the consortium’s strategic direction
• AI companies contribute model access, compute credits, and engineering expertise
• Universities provide researchers and facilities for conducting fundamental research

This consortium structure would enable rapid resource sharing, collaborative research projects, and accelerated translation of research into practice. It would operate under flexible contracting mechanisms using Other Transaction Authority (OTA) to reduce administrative barriers.

Beyond the consortium, funding agencies should leverage Other Transaction Authority (OTA) and Prize Competition Authority to flexibly contract and fund research projects related to priority areas. New public-private grant vehicles focused on funding fundamental research in priority areas should be set up via existing foundations linked to funding agencies such as the NSF Foundation, DOE’s Foundation for Energy Security and Innovation, or the proposed NIST Foundation.

Specific funding mechanisms should be chosen based on the target technology’s maturity level. For example, the NSF can support more fundamental research through fast grants via its EAGER and RAPID programs. Previous fast-grant programs, such as SGER, were found to be wildly effective, with “transformative research results tied to more than 10% of projects.”

For research areas where clear, well-defined technical milestones are achievable, such as developing secure cluster-scale environments for large AI training workloads, the government can support the creation of focused research organizations (FROs) and implement advanced market commitments (AMCs) to take technologies across the ‘valley of death’. DARPA and IARPA can administer higher-risk, more ambitious R&D programs with national security applications.

Recommendation 3. Establish an AI Frontier Science Fellowship to ensure a pipeline of technical AI talent that can contribute directly to R&D and support fast-grant program management

It is critical to ensure that America has a growing pool of talented researchers entering the field of AI assurance and security, given its strategic importance to American competitiveness and national security. 

The NSF should launch an AI Frontier Science Fellowship targeting early-career researchers in critical AI assurance and security R&D. Drawing from proven models like CyberCorp Scholarship for Service, COVID-19 Fast Grants, and  proposals such as for “micro-ARPAs”, this program operates on two tracks:

1. Frontier Scholars: This track would provide comprehensive research support for PhD students and post-docs conducting relevant research on priority AI security and reliability topics. This includes computational resources, research rotations at government labs and agencies, and financial support.
2. Rapid Grant Program Managers (PM): This track recruits researchers to serve fixed terms as Rapid Grant PMs, responsible for administering EAGER/RAPID grants focused on AI assurance and security. 

This fellowship solves multiple problems at once. It builds the researcher pipeline while creating a nimble, decentralized approach to science funding that is more in line with the dynamic nature of the field. This should improve administrative efficiency and increase the surface area for innovation by allowing for more early-stage high-risk projects to be funded. Also, PMs who perform well in administering these small, fast grants can then become full-fledged program officers and PMs at agencies like the NSF and DARPA. This program (including grant budget) would cost around $40 million per year.

Conclusion

To unlock AI’s immense potential, from research to defense, we must ensure these tools are reliable and secure. This demands R&D breakthroughs to better understand emergent AI capabilities and behaviors, secure AI hardware and infrastructure, and prepare for a multi-agent world. The federal government must lead by setting clear R&D priorities, building foundational research talent, and injecting targeted funding to fast-track innovation.  This unified push is key to securing America’s AI leadership and ensuring that American AI is the global gold standard.

This memo was written by an AI Safety Policy Entrepreneurship Fellow over the course of a six-month, part-time program that supports individuals in advancing their policy ideas into practice. You can read more policy memos and learn about Policy Entrepreneurship Fellows here.

Frequently Asked Questions

Can this be done with reallocating existing budget and under existing authority?

Yes, the recommendations are achievable by reallocating the existing budget and using existing authorities, but this would likely mean accepting a smaller initial scale.

In terms of authorities, OSTP and NSF can already update the National AI R&D Strategic Plan and establish AI assurance and security priorities through normal processes. To implement agile funding mechanisms, agencies can use OTA and Prize Competition Authority. Fast grants require no special statute and can be done under existing grant authorities.

In terms of budget, agencies can reallocate 5-10% of existing AI research funds towards security and assurance R&D. The Frontier Science Fellowship could start as a $5-10 million pilot under NSF’s existing education authorities, e.g. drawing from NSF’s Graduate Research Fellowship Program.

While agencies have flexibility to begin this work, achieving the memo’s core objective – ensuring AI systems are trustworthy and reliable for workforce and military adoption – requires dedicated funding. Congress could provide authorization and appropriation for a named fellowship, which would make the program more stable and allow it to survive personnel turnover.

read more

Why does the federal government need to step in – won’t the private sector fix AI reliability and security on its own?

Market incentives drive companies to fix AI failures that directly impact their bottom line, e.g., chatbots giving bad customer service or autonomous vehicles crashing. More visible, immediate problems are likely to be prioritized because customers demand it or because of liability concerns. This memo focuses on R&D areas that the private sector is less likely to tackle adequately.

The private will address some security and reliability issues, but there are likely to be significant gaps. Understanding emergent model capabilities demands costly fundamental research that generates little immediate commercial return. Likewise, securing AI infrastructure against nation-state attacks will likely require multi-year R&D processes, and companies can fail to coordinate to develop these technologies without a clear demand signal. Finally, systemic dangers arising from multi-agent interactions might be left unmanaged because these failures emerge from complex dynamics with unclear liability attribution.

The government can step in to fund the foundational research that the market is likely to undersupply by default and help coordinate the key stakeholders in the process.

read more

How do we encourage industry cooperation, given that researchers will require access to frontier AI models and systems from private companies to conduct this research?

Companies need security solutions to access regulated industries and enterprise customers. Collaboration on government-funded research provides these solutions while sharing costs and risks.

The proposed AI Assurance and Security R&D Consortium in Recommendation 2 create a structured framework for cooperation. Companies contribute model access and compute credits while receiving:

• Government-funded researchers working on their deployment challenges


• Shared IP rights under consortium agreements


• Early access to security and reliability innovations


• Risk mitigation through collaborative cost-sharing

Under the consortia’s IP framework, companies retain full commercial exploitation rights while the government gets unlimited rights for government purposes. In the absence of a consortium agreement, an alternative arrangement could be a patent pool, where companies can access patented technologies in the pool through a single agreement. These structures, combined with the fellowship program providing government-funded researchers, creates strong incentives for private sector participation while advancing critical public research objectives.

read more

DC’s first-ever Climate Week brought with it many chances to discuss the hottest-button topics in climate innovation and policy. FAS took the opportunity to do just that, by hosting a panel to explore the intersection of artificial intelligence (AI), energy, and climate issues with leading experts. Dr. Oliver Stephenson, FAS’ Associate Director of Artificial Intelligence and Emerging Technology Policy, sat down with Dr. Tanya Das, Dr. Costa Samaras, and Charles Hua to discuss what’s at stake at this critical crossroads moment. 

Missed the panel? Don’t fret. Read on to learn the need-to-knows. Here’s how these experts think we can maximize the “good” and minimize the “bad” of AI and data centers, leverage research and development (R&D) to make AI tools more successful and efficient, and how to better align incentives for AI growth with the public good.

First, Some Level Setting 

The panelists took their time to make sure the audience understood two key facts regarding this space. First, not all data centers are utilized for AI. The Electric Power Research Institute (EPRI) estimates that AI applications are only used in about 10-20% of data centers. The rest? Data storage, web hosting capabilities, other cloud computing, and more.

Second, load growth due to the energy demand of data centers is happening, but the exact degree still remains unknown. Lawrence Berkeley National Lab (LBNL) models project that data centers in the US will consume anywhere between 6.7% and 12% of US electricity generation by 2028. For a country that consumes roughly 4 trillion kilowatt hours (kWh) of electricity each year, this estimation spans a couple hundred billion kWh/year from the low end to the high. Also, these projections are calculated based on different assumptions that factor in AI energy efficiency improvements, hardware availability, regulatory decisions, modeling advancements, and just how much demand there will be for AI. When each of these conditions are evolving daily, even the most credible projections come with a good amount of uncertainty.

There is also ambiguity in the numbers and in the projections at the local and state levels, as many data center companies shop around to multiple utilities to get the best deal. This can sometimes lead to projects getting counted twice in local projections. Researchers at LBNL have recently said they can confidently make data center energy projections out to 2028. Beyond that, they can’t make reasonable assumptions about data center load growth amid growing load from other sectors working to electrify—like decarbonizing buildings and electric vehicle (EV) adoption.

Maximizing the Good, Minimizing the Bad

As data center clusters continue to proliferate across the United States, their impacts—on energy systems and load growth, water resources, housing markets, and electricity rates—will be most acutely felt at the state and local levels. DC’s nearby neighbor Northern Virginia has become a “data center alley” with more than 200 data centers in Loudoun County alone, and another 117 in the planning stages. 

States ultimately hold the power to shape the future of the industry through utility regulation, zoning laws, tax incentives, and grid planning – with specific emphasis on state Public Utility Commissions (PUCs). PUCs have a large influence on where data centers can be connected to the grid and the accompanying rate structure for how each data center pays for its power—whether through tariffs, increasing consumer rates, or other cost agreements. It is imperative that vulnerable ratepayers are not left to shoulder the costs and risks associated with the rapid expansion of data centers, including higher electricity bills, increased grid strain, and environmental degradation.

Panelists emphasized that despite the potential negative impacts of AI and data centers expansion, leaders have a real opportunity to leverage AI to maximize positive outcomes—like improving grid efficiency, accelerating clean energy deployment, and optimizing public services—while minimizing harms like overconsumption of energy and water, or reinforcing environmental injustice. Doing so, however, will require new economic and political incentives that align private investment with public benefit.

Research & Development at the Department of Energy

The U.S. Department of Energy (DOE) is uniquely positioned to help solve the challenges AI and data centers pose, as the agency sits at the critical intersection of AI development, high-performance computing, and energy systems. DOE’s national laboratories have been central to advancing AI capabilities: Oak Ridge National Laboratory (ORNL) was indeed the first to integrate graphics processing units (GPUs) into supercomputers, pioneering a new era of AI training and modeling capacity. DOE also runs two of the world’s most powerful supercomputers – Aurora at Argonne National Lab and Frontier at ORNL – cementing the U.S.’ leadership in high-performance computing.  

Beyond computing, DOE plays a key role in modernizing grid infrastructure, advancing clean energy technologies, and setting efficiency standards for energy-intensive operations like data centers. The agency has also launched programs like the Frontiers in Artificial Intelligence for Science, Security and Technology (FASST), overseen by the Office of Critical and Emerging Tech (CET), to coordinate AI-related activities across its programs.

As the intersection of AI and energy deepens—with AI driving data center expansion and offering tools to manage its impact—DOE must remain at the center of this conversation, and it must continue to deliver. The stakes are high: how we manage this convergence will influence not only the pace of technological innovation but also the equity and sustainability of our energy future.

Incentivizing Incentives: Aligning AI Growth with the Public Good 

The U.S. is poised to spend a massive amount of carbon to power the next wave of artificial intelligence. From training LLMs to supporting real-time AI applications, the energy intensity of this sector is undeniable—and growing. That means we’re not just investing financially in AI; we’re investing environmentally. To ensure that this investment delivers public value, we must align political and economic incentives with societal outcomes like grid stability, decarbonization, and real benefits for American communities.

One of the clearest opportunities lies in making data centers more responsive to the needs of the electric grid. While these facilities consume enormous amounts of power, they also hold untapped potential to act as flexible loads—adjusting their demand based on grid conditions to support reliability and integrate clean energy. The challenge? There’s currently little economic incentive for them to do so. One panelist noted skepticism that market structures alone will drive this shift without targeted policy support or regulatory nudges.

Instead, many data centers continue to benefit from “sweetheart deals”—generous tax abatements and economic development incentives offered by states and municipalities eager to attract investment. These agreements often lack transparency and rarely require companies to contribute to local energy resilience or emissions goals. For example, in several states, local governments have offered multi-decade property tax exemptions or reduced electricity rates without any accountability for climate impact or grid contributions.

New AI x Energy Policy Ideas Underway

If we’re going to spend gigatons of carbon in pursuit of AI-driven innovation, we must be strategic about where and how we direct incentives. That means:

• Conditioning public subsidies on data center flexibility and efficiency performance.
• Requiring visibility into private energy agreements and emissions footprints.
• Designing market signals—like time-of-use pricing or demand response incentives—that reward facilities for operating in sync with clean energy resources.

We don’t just need more incentives—we need better ones. And we need to ensure they serve public priorities, not just private profit. Through our AI x Energy Policy Sprint, FAS is working with leading experts to develop promising policy solutions for the Trump administration, Congress, and state and local governments. These policy memos will address how to: mitigate the energy and environmental impacts of AI systems and data centers, enhance the reliability and efficiency of energy systems using AI applications, and unlock transformative technological solutions with AI and energy R&D. 

Right now, we have a rare opportunity to shape U.S. policy at the critical intersection of AI and energy. Acting decisively today ensures we can harness AI to drive innovation, revolutionize energy solutions, and sustainably integrate transformative technologies into our infrastructure.

The Federation of American Scientists (FAS) submitted the following response to the Request for Information (RFI) issued by the Office of Science and Technology Policy (OSTP) in February 2025 regarding the development of an Artificial Intelligence (AI) Action Plan.

At a time when AI is poised to transform every sector of the economy, the Trump administration has a critical opportunity to solidify America’s leadership in this pivotal technology. Building on the foundations laid during the first Trump administration, bold and targeted policies can unleash innovation, unlocking AI’s vast potential to stimulate economic growth, revolutionize industries, and strengthen national security. However, innovation alone is insufficient; without public trust, AI adoption will stall. Ensuring AI systems are transparent, reliable, and aligned with American values will accelerate responsible adoption and solidify AI as a cornerstone of America’s economic and technological leadership.

To sustain America’s leadership in AI innovation, accelerate adoption across the economy, and guarantee that AI systems remain secure and trustworthy, we offer a set of actionable policy recommendations. Developed by FAS in partnership with prominent AI experts, industry leaders, and research institutions—including contributors to the recent FAS Day One 2025 Project and the 2024 AI Legislative Sprint—these proposals are structured around four strategic pillars: 1) unleashing AI innovation, 2) accelerating AI adoption, 3) ensuring secure and trustworthy AI, and 4) strengthening existing world-class U.S. government institutions and programs. 

1) Unleashing AI Innovation. American AI leadership has been driven by bold private-sector investments and world-class academic research. However, critical high-impact areas remain underfunded. The federal government can catalyze investment and innovation by expanding access to essential data, investing strategically in overlooked areas of AI R&D, defining priority research challenges, promoting public-private partnerships, and attracting and retaining global talent.

2) Accelerating AI Adoption Across the Economy. The United States leads in AI breakthroughs, but these breakthroughs must translate into widespread adoption to maximize their economic and societal benefits. Accelerating adoption—a critical yet often overlooked driver of national competitiveness—requires addressing workforce readiness, expanding government capacity, and managing rising energy demands.

3) Ensuring Secure and Trustworthy AI. Ensuring AI systems are secure and trustworthy is essential not only for fostering public confidence and accelerating widespread adoption, but also for improving government efficiency and ensuring the responsible use of taxpayer resources when AI is deployed by public agencies. While the previous Trump administration recognized the necessity of public trust when promoting AI adoption, concerns persist about AI’s rapid evolution, unpredictable capabilities, and potential for misuse. Future AI accidents could further erode this trust, stalling AI progress. To address these risks and fully harness AI’s potential, the U.S. government must proactively monitor emerging threats, rigorously evaluate AI technologies, and encourage innovation that upholds fundamental American values such as privacy. 

4) Strengthening Existing World-Class U.S. Government AI Institutions and Programs. Realizing the Trump Administration’s goals will require building on leading government AI capabilities. Key initiatives—including the NIST AI Safety Institute (AISI), the National AI Research Resource (NAIRR) Pilot, the AI Use Case Inventory, and the Department of Energy’s Office of Critical and Emerging Technologies (CET)—advance AI innovation, security, and transparency. The AISI evaluates AI models with broad industry support, while the NAIRR Pilot expands access to AI resources beyond Big Tech. Federal AI use case inventories enhance government transparency and industry engagement, building public trust. DOE’s CET drives AI-powered advancements in science and national security. Integrating these proven initiatives into the AI Action Plan will solidify America’s AI leadership.

By acting decisively, the administration can ensure American AI remains the gold standard, drive economic competitiveness, and accelerate science and innovation.

Overview of Policy Proposals

Policy Proposals to Unleash AI Innovation

• Recommendation 1: Promote innovation in trustworthy AI through a Public-Private National Initiative for AI Explainability.
• Recommendation 2: Direct the Department of Energy (DOE) to use AI to accelerate the discovery of new materials.
• Recommendation 3: Create AI-ready collaborative datasets to accelerate progress in the life sciences.
• Recommendation 4: Establish a NIST Foundation to amplify public-private collaboration, secure private investment, and accelerate innovation.
• Recommendation 5: Attract top global talent by creating a National Security AI Entrepreneur Visa for elite dual-use technology founders.

Policy Proposals to Accelerate AI Adoption Across the Economy

• Recommendation 1: Streamline procurement processes for government use of AI.
• Recommendation 2: Establish a Federal Center of Excellence to expand state and local government capacity for AI procurement and use.
• Recommendation 3: Pilot an AI Corps at HHS to drive government-wide AI adoption.
• Recommendation 4: Make America’s teacher workforce competitive for the AI era.
• Recommendation 5: Prepare U.S. energy infrastructure for AI growth through standardized measurement and forecasting.

Policy Proposals to Ensure Secure and Trustworthy AI

• Privacy:
  • Recommendation 1: Secure third party commercial data for AI through FedRAMP authorization.
  • Recommendation 2: Catalyze federal data sharing through privacy enhancing technologies.
  • Recommendation 3: Establish data-sharing standards to support AI development in healthcare.
• Security, Safety, and Trustworthiness:
  • Recommendation 1: Establish an early warning system for AI-powered threats to national security and public safety.
  • Recommendation 2: Create a voluntary AI incident reporting hub to monitor security incidents from AI.
  • Recommendation 3: Promote AI trustworthiness by providing a safe harbor for AI researchers.
  • Recommendation 4: Build a national digital content authentication technologies research ecosystem.
  • Recommendation 5: Strengthen national security by evaluating AI-driven biological threats.

Policy Proposals to Strengthen Existing World-Class U.S. Government AI Institutions and Programs that are Key to the Trump Administration’s AI Agenda

• Recommendation 1: Support the NIST AI Safety Institute as a key pillar of American AI excellence.
• Recommendation 2: Expand the National Artificial Intelligence Research Resource from pilot to full program.
• Recommendation 3: Enhance transparency, accountability, and industry engagement by preserving the AI use case inventory.
• Recommendation 4: Propel U.S. Scientific and Security AI Leadership by Supporting AI and Computing at DOE.

Policy Proposals to Unleash AI Innovation

As artificial intelligence continues transforming industries and reshaping global competition, the United States must take bold, coordinated action to maintain its technological leadership. A multi-agency approach could include launching a National Initiative for AI Explainability, accelerating materials science discovery through AI-powered autonomous laboratories, creating AI-ready datasets for the life sciences, establishing a NIST Foundation to enhance public-private collaboration in AI research, and creating a National Security AI Entrepreneur Visa to attract and retain top global talent. Together, these initiatives would strengthen America’s AI ecosystem by addressing critical challenges in transparency, scientific research, standards development, and talent acquisition—while ensuring the U.S. remains at the forefront of responsible AI innovation.

Recommendation 1. Promote Innovation in Trustworthy AI through a Public-Private National Initiative for AI Explainability 

Understanding the inner workings of AI systems is critical not only for reliability and risk mitigation in high-stakes areas such as defense, healthcare, and finance, but also for bolstering American technological leadership and maximizing government accountability and efficiency. However, despite promising progress in fields such as “mechanistic interpretability”, the study of explainability in AI systems is still nascent. A lack of explainability risks undermining trust and inhibiting AI adoption, particularly in safety-critical sectors.

To address the challenge of understanding and improving AI systems, we propose the launch of a Public-Private National Initiative for AI Explainability. Following in the footsteps of government-coordinated research projects like the Human Genome Project, this initiative would unite researchers, industry leaders, standards bodies, and government agencies to map the inner workings of advanced AI systems in a public-private partnership. 

Federal precedent for such work already exists: DARPA’s 2017-2021 Explainable AI (XAI) program sought to create machine learning systems capable of explaining their decisions in a way humans could understand. While the program advanced techniques for explainable models and human-friendly translations of complex AI reasoning, the rapid development and scaling of AI technologies in the past five years demand a renewed, more ambitious effort.

The objectives of the initiative would include:

• Creating Open-Access Resources: Developing AI models, datasets, and tools accessible to researchers and practitioners, allowing a larger number of actors to contribute to progress.
• Developing Standardized Metrics and Benchmarks: Establishing clear standards to evaluate the explainability of AI systems in different circumstances, ensuring consistency and reliability across applications.
• Defining Common Tasks: Establishing standardized metrics and open datasets to create “common tasks” in explainability—well-defined challenges that drive innovation and encourage widespread progress as the broader ecosystem competes to improve performance.
• Investigating User-Centric Explanation Needs: Conducting research to identify which types of AI explanations are most effective and meaningful, and which provide appropriate degrees of control, to users across various contexts and applications.  
• Developing a Repository of Explainability Techniques: Researching and disseminating explainability methods applicable across various AI domains, including an analysis of the strengths and weaknesses of different approaches and where they can be properly applied.

Implementation Strategy:

To launch this effort, the President should issue an executive order to signal national commitment and assign leadership to key federal agencies, including:

• Office of Science and Technology Policy: Playing a coordinating role across the government.
• Defense Advanced Research Projects Agency (DARPA): Building upon its prior experience with the XAI program to spearhead research efforts.
• National Institute of Standards and Technology (NIST): Developing standards and benchmarks for AI explainability, building on previous work in this area.
• National Science Foundation (NSF): Funding academic research through its National AI Research Institutes.
• Department of Energy (DOE): Leveraging its computational resources and expertise in large-scale research projects.
• Other government agencies with relevant expertise: For example, the National Institutes of Health (NIH) could focus on explainability in AI applications within the healthcare sector.

The White House should leverage its convening power to unite leading AI companies, top academic institutions, and government agencies in formal collaborations. These partnerships could encompass co-funded research, shared datasets and computing resources, collaborative access to advanced AI models, and joint development of open-source tools. Establishing a structured public-private partnership will facilitate coordinated funding, align strategic priorities, and streamline resource sharing, ensuring that advancements in AI explainability directly support both national interests and economic competitiveness. To sustain this initiative, the administration should also secure consistent, multi-year federal funding through appropriations requests to Congress. 

DARPA’s XAI program showed that AI explainability requires interdisciplinary collaboration to align technical development with human understanding. Building on these insights, this initiative should include experts from computer science, cognitive science, ethics, law, and domain-specific fields to ensure explanations are clear, useful, and actionable for decision-makers across critical sectors. 

By implementing this National Initiative for AI Explainability, the Trump administration can significantly enhance public confidence in AI technologies, accelerate responsible adoption by both the public and private sectors, and solidify America’s global leadership in AI innovation. Critically, a modest investment of government resources in this initiative could unlock substantial private-sector investment, spurring innovation and driving economic growth. This strategic approach will also enhance government accountability, optimize the responsible use of taxpayer resources, and ensure that American industry continues to lead in AI development and deployment.

Recommendation 2. Direct the Department of Energy (DOE) to use AI to Accelerate the Discovery of New Materials (link to full memo >>>)

Innovations in AI and robotics could revolutionize materials science by automating experimental processes and dramatically accelerating the discovery of new materials. Currently, materials science research involves manually testing different combinations of elements to identify promising materials, which limits the pace of discovery. Using AI foundation models for physics and chemistry, scientists could simulate new materials, while robotic “self-driving labs” could run 24/7 to synthesize and evaluate them autonomously. This approach would enable continuous data generation, refining AI models in a feedback loop that speeds up research and lowers costs. Given its expertise in supercomputing, AI, and a vast network of national labs, the Department of Energy (DOE) could lead this transformative initiative, potentially unlocking advancements in critical materials, such as improved battery components, that could have immense economic and technological impacts.

Recommendation 3. Create AI-ready Collaborative Datasets to Accelerate Progress in the Life Sciences (link to full memo >>>)

Large, high-quality datasets could revolutionize life science research by powering AI models that unlock new discoveries in areas like drug development and diagnostics. Currently, researchers often work in silos with limited incentives to collaborate and share meticulously curated data, slowing progress. By launching a government-funded, end-to-end initiative—from identifying critical dataset needs to certifying automated collection methods and hosting robust open repositories—scientists could continuously generate and refine data, fueling AI models in a feedback loop that boosts accuracy and lowers costs. Even a relatively modest government investment could produce vital resources for researchers and startups to spark new industries. This model could also be extended to a range of other scientific fields to accelerate U.S.science and innovation.

Recommendation 4. Create a NIST Foundation to Support the Agency’s AI Mandate (link to full memo >>>)

To maintain America’s competitive edge in AI, NIST needs greater funding, specialized talent, and the flexibility to work effectively with private-sector partners. One solution is creating a “NIST Foundation,” modeled on the DOE’s Foundation for Energy Security and Innovation (FESI), which combines federal and private resources to expand capacity, streamline operations, and spur innovation. Legislation enabling such a foundation was introduced with bipartisan support in the 118th Congress, signaling broad consensus on its value. The Trump administration can direct NIST to study how a nonprofit foundation might boost its AI initiatives and broader mission—just as a similar report helped pave the way for FESI—giving Congress the evidence it needs to formally authorize a NIST Foundation. The administration can also support passage of authorizing legislation through Congress.

Recommendation 5. Attract Top Global Talent by Creating a National Security AI Entrepreneur Visa for Elite Dual-use Technology Founders (link to full memo >>>)

America’s leadership in AI has been driven by the contributions of immigrant entrepreneurs, with companies like NVIDIA, Anthropic, OpenAI, X, and HuggingFace—all of which have at least one immigrant co-founder—leading the charge. To maintain this competitive edge as global competition intensifies, the administration should champion a National Security Startup Visa specifically targeted at high-skilled founders of AI firms. These entrepreneurs are at the forefront of developing dual-use technologies critical for both America’s economic leadership and national security. Although the linked proposal above is targeted at legislative action, the administration can take immediate steps to advance this priority by publicly supporting legislation to establish such a visa, engaging with Congressional allies to underscore its strategic importance, and directing agencies like the Department of Homeland Security and the Department of Commerce to explore ways to streamline pathways for these innovators. This decisive action would send a clear signal that America remains the destination of choice for world-class talent, ensuring that the nation stays ahead in the race for AI dominance.

Policy Proposals to Accelerate AI Adoption Across the Economy

AI has transformative potential to boost economic growth and unlock new levels of prosperity for all. The Trump administration should take bold action to encourage greater adoption of AI technologies and AI expertise by leveraging government procurement, hiring, and standards-setting processes, alongside coordinated support for America’s teachers to prepare students to join the future AI workforce. In government, a coordinated set of federal initiatives is needed to modernize and streamline effective AI adoption in the public sector. These proposals include developing a national digital platform through GSA to streamline AI procurement processes, establishing a federal center of excellence to support state and local governments in AI implementation, and pursuing innovative hiring models to expand AI expertise at HHS. Additionally, NIST should develop voluntary standards for measuring AI energy and resource usage to inform infrastructure planning efforts. Finally, the President should announce a national teacher talent surge and set AI as a competitive priority in American education. 

Recommendation 1. Streamline Procurement Processes for Government Use of AI (link to full memo >>>)

The federal government has a critical role in establishing standards for AI systems to enhance public services while ensuring they are implemented ethically and transparently. To streamline this effort and support federal agencies, the administration should direct the General Services Administration (GSA) to create a user-friendly, digital platform for AI procurement. This platform would simplify the acquisition process by providing agencies with clear, up-to-date guidelines, resources, and best practices, all tailored to align with existing procurement frameworks. The platform would empower agencies to make informed decisions that prioritize safety, fairness, and effective use of AI technologies, while demonstrating the administration’s commitment to modernizing government operations and ensuring America leads the way in adopting cutting-edge AI solutions.

Recommendation 2. Establish a Federal Center of Excellence to Expand State and Local Government Capacity for AI Procurement and Use (link to full memo >>>)

State and local governments often face challenges in effectively leveraging AI to enhance their efficiency and service capabilities. To support responsible AI adoption at the state, local, tribal, and territorial (SLTT) levels, the administration should establish a federal AI Center of Excellence. This center would provide hands-on guidance from experts in government, academia, and civil society, helping SLTT agencies navigate complex challenges such as limited technical expertise, budget constraints, privacy concerns, and evolving regulations. It would also translate existing federal AI standards—including Executive Order 13960 and the NIST Risk Management Framework—into practical, actionable advice. By developing in-house procurement and deployment expertise, SLTT governments could independently and confidently implement AI solutions, promoting innovation while ensuring responsible, effective, and efficient use of taxpayer resources.

Recommendation 3. Pilot an AI Corps at HHS to Drive Government-Wide AI Adoption (link to full memo >>>) 

Federal agencies often struggle to leverage AI effectively, due to limited technical expertise and complex oversight requirements. Modeled after the Department of Homeland Security’s successful AI Corps, which has improved disaster response and cybersecurity, this pilot would embed AI and machine learning experts within the Department of Health and Human Services’s (HHS) 10 agencies, accelerating responsible AI implementation in healthcare, driving greater efficiency, and demonstrating a scalable model that could be replicated across other federal departments. HHS is uniquely suited for piloting an AI Corps because it oversees critical health infrastructure and massive, sensitive datasets—presenting significant opportunities for AI-driven improvements but also requiring careful management. If successful, this pilot could serve as a strategic blueprint to enhance AI adoption, improve government performance, and maximize the responsible use of taxpayer resources across the federal government.

Recommendation 4. Make America’s Teacher Workforce Competitive for the AI Era (link to full memo >>>) 

With America facing a significant shortage of teachers and a growing need for AI and digital skills in the workforce, the Trump administration can rebuild America’s teaching profession by launching a coordinated strategy led by the Office of Science and Technology Policy (OSTP). This initiative should begin with a national teacher talent surge to expand annual teacher graduates by 100,000, addressing both the urgent workforce gap and the imperative to equip students for an AI-driven future. The plan includes a Challenge.gov competition to attract innovative recruitment and retention models, updating Department of Education scholarship programs (like the Graduate Assistance in Areas of National Need) to include AI, data science, and machine learning, convening colleges of education to modernize training, and directing agencies to prioritize AI-focused teacher development. By leveraging existing grants (e.g., Teacher Quality Partnerships, SEED, the STEM Corps, and Robert Noyce Scholarships), the administration can ensure a robust pipeline of educators ready to guide the next generation.

Recommendation 5. Prepare U.S. Energy Infrastructure for AI Growth Through Standardized Measurement and Forecasting

As AI adoption accelerates, America’s energy infrastructure faces a critical challenge: next-generation AI systems could place unprecedented demands on the power grid, yet the lack of standardized measurements, and wide variations in forecasted demand, leaves utilities and policymakers unprepared. Without proactive planning, energy constraints could slow AI innovation and undermine U.S. competitiveness.

To address this, the Administration should direct the National Institute of Standards and Technology (NIST) and the Department of Energy (DOE) to develop a standardized framework for measuring and forecasting AI’s energy and resource demands. This framework should be paired with a voluntary reporting program for AI developers—potentially collected by the Energy Information Administration (EIA)—to provide a clearer picture of AI’s impact on energy consumption. The EIA should also be tasked with forecasting AI-driven energy demand, ensuring that utilities, public utility commissions, and state energy planners have the data needed to modernize the grid efficiently.

Greater transparency will enable both government and industry to anticipate energy needs, drive investment in grid modernization, and prevent AI-related power shortages that could hinder economic growth. The proactive integration of AI and energy planning will strengthen America’s leadership in AI innovation while safeguarding the reliability of its infrastructure. FAS is actively developing policy proposals with the science and technology community at the intersection of AI and energy. We plan to share additional recommendations on this topic in the coming months.

Policy Proposals to Ensure Secure and Trustworthy AI

Privacy

Protecting Americans’ privacy while harnessing the potential of AI requires decisive federal action that prioritizes both individual rights and technological advancement. Strengthening privacy protections while enabling responsible data sharing is crucial for ensuring that AI-driven innovations improve public services without compromising sensitive information. Key initiatives include establishing NIST-led guidelines for secure data sharing and maintaining data integrity, implementing a FedRAMP authorization framework for third-party data sources used by government agencies, and promoting the use of Privacy Enhancing Technologies (PETs). Additionally, the administration should create a “Responsible Data Sharing Corps” to provide agencies with expert guidance and build capacity in responsible data practices.

Recommendation 1. Secure Third Party Commercial Data for AI through FedRAMP Authorization (link to full memo >>>)

The U.S. government is a major customer of commercial data brokers and should require a pre-evaluation process before agencies acquire large datasets, ensuring privacy and security from the outset. Thoroughly vetting data brokers and verifying compliance standards can help avert national security risks posed by compromised or unregulated third-party vendors. To formalize these safeguards, OMB and FedRAMP should create an authorization framework for data brokers that provide commercially available information, especially with personally identifiable information. Building on its established role in securing cloud providers FedRAMP is well positioned to guide these protocols, ensuring agencies work only with trusted vendors and strengthening overall data protection.

Recommendation 2. Catalyze Federal Data Sharing through Privacy Enhancing Technologies (link to full memo >>>)

To maintain America’s leadership in AI and digital innovation, the administration must ensure that government agencies can securely leverage data while protecting privacy and maintaining public trust. The federal government can lead by example through the adoption of Privacy Enhancing Technologies (PETs)—tools that enable data analysis while minimizing exposure of sensitive information. Agencies should be encouraged to adopt PETs with support from a Responsible Data Sharing Corps, while NIST develops a decision-making framework to guide their use. OMB should require agencies to apply this framework in data-sharing initiatives and report on PET adoption, with a PET Use Case Inventory and annual reports enhancing transparency. A federal fellowship program could also bring in experts from academia and industry to drive PET innovation. These measures would strengthen privacy, security, and public trust while positioning the U.S. as a global leader in responsible data use.

Recommendation 3. Establish Data-Sharing Standards to Support AI Development in Healthcare (link to full memo >>>) 

The U.S. healthcare system generates vast amounts of data daily, yet fragmentation, privacy concerns, and lack of interoperability severely limit its use in AI development, hindering medical innovation. To address this, the AI Action Plan should direct NIST to lead an interagency coalition in developing standardized protocols for health data anonymization, secure sharing, and third-party access. By establishing clear technical and governance standards—similar to NIST’s Cryptographic and Biometric Standards Programs—this initiative would enable responsible research while ensuring compliance with privacy and security requirements. These standards would unlock AI-driven advancements in diagnostics, treatment planning, and health system efficiency. Other nations, including the U.K., Australia, and Finland, are already implementing centralized data-sharing frameworks; without federal leadership, the U.S. risks falling behind. By taking decisive action, the administration can position the U.S. as a global leader in medical AI, accelerating innovation while maintaining strong privacy protections.

Security, Safety, and Trustworthiness

AI holds immense promise for job growth, national security, and innovation, but accidents or misuse risk undermining public trust and slowing adoption—threatening the U.S.’s leadership in this critical field. The following proposals use limited, targeted government action alongside private-sector collaboration to strengthen America’s AI capabilities while upholding public confidence and protecting our national interests.

Recommendation 1. Establish an Early Warning System for AI-Powered Threats to National Security and Public Safety (link to full memo >>>) 

Emerging AI capabilities could also pose severe threats to public safety and national security. AI companies are already evaluating their most advanced models to identify dual-use capabilities, such as the capacity to conduct offensive cyber operations, enable the development of biological or chemical weapons, and autonomously replicate and spread. These capabilities can arise unpredictably and undetected during development and after deployment. To prepare for these emerging risks, the federal government should establish a coordinated “early-warning system” for novel dual-use AI capabilities to gain awareness of emerging risks before models are deployed. A government agency could serve as a central information clearinghouse—an approach adapted from the original congressional proposal linked above. Advanced AI model developers could confidentially report newly discovered or assessed dual-use capabilities, and the White House could direct relevant government agencies to form specialized working groups that engage with private sector and other non-governmental partners to rapidly mitigate risks and leverage defensive applications. This initiative would ensure that the federal government and its stakeholders have maximum lead time to prepare for emerging AI-powered threats, positioning the U.S. as a leader in safe and responsible AI innovation.

Recommendation 2. Create a Voluntary AI Incident Reporting Hub to Monitor Security Incidents from AI (link to full memo >>>)

The federal government should establish a voluntary national Artificial Intelligence Incident Reporting Hub to better track, analyze, and address incidents from increasingly complex and capable AI systems that are deployed in the real world. Such an initiative could be modeled after successful incident reporting and info-sharing systems operated by the National Cybersecurity FFRDC, the Federal Aviation Administration, and the Food and Drug Administration. By providing comprehensive yet confidential data collection under the umbrella of an agency (e.g. NIST) this initiative would bolster public trust, facilitate the sharing of critical risk information, and enable prompt government action on emerging threats, from cybersecurity vulnerabilities to potential misuse of AI in sensitive areas like chemical, biological, radiological, or nuclear contexts. This proposal builds on bipartisan legislation introduced in the last Congress, as well as the memo linked above, which was originally targeted at Congressional action.

Recommendation 3. Promote AI Trustworthiness by Providing a Safe Harbor for AI Researchers (link to full memo >>>)

Independent AI research plays a key role in ensuring safe and reliable AI systems. In 2024, over 350 researchers signed an open letter calling for “a safe harbor for independent AI evaluation”, noting that generative AI companies offer no legal protections for independent safety researchers. This situation is unlike established voluntary protections from companies for traditional software, and Department of Justice (DOJ) guidance not to prosecute good faith security research. The proposal linked above was targeted at Congressional action, however the executive branch could adapt these ideas in several ways, by, for example: 1) instructing the Office of Management and Budget (OMB) to issue guidance to all federal agencies requiring that contracting documents for generative AI systems include safe-harbor provisions for good-faith external research, consistent with longstanding federal policies that promote responsible vulnerability disclosure. 2) Coordinating with DOJ and relevant agencies to clarify that good-faith AI security and safety testing—such as red-teaming and adversarial evaluation—does not violate the Computer Fraud and Abuse Act (CFAA) or other laws when conducted according to established guidelines.

Recommendation 4. Build a National Digital Content Authentication Technologies Research Ecosystem (link to full memo >>>) 

AI generated synthetic content (such as fake videos, images, and audio) is increasingly used by malicious actors to defraud elderly Americans, spread child sexual abuse material, and impersonate political figures. To counter these threats, the United States must invest in developing technical solutions for reliable synthetic content detection. Through the National Institute of Standards and Technology (NIST), the Trump Administration can: 1) establish dedicated university-led national research centers, 2) develop a national synthetic content database, and 3) run and coordinate prize competitions to strengthen technical countermeasures.These initiatives will help build a robust research ecosystem to keep pace with the rapidly evolving synthetic content threat landscape, maintaining America’s role as a global leader in responsible and secure AI.

Recommendation 5. Strengthen National Security by Evaluating AI-Driven Biological Threats (link to full memo >>>)

Over the past two years, the rapid advance of AI in biology and large language models has highlighted an urgent need for a targeted U.S. Government program to assess and mitigate biosecurity risks. While AI-enabled tools hold immense promise for drug discovery, vaccine research, and other beneficial applications, their dual-use potential (e.g., identifying viral mutations that enhance vaccine evasion) makes them a national security priority. Building on the Department of Homeland Security’s (DHS) previous work on AI and CBRN threats, the Department of Energy (DOE),  DHS, and other relevant agencies, should now jointly launch a “Bio Capability Evaluations” program, backed by sustained funding, to develop specialized benchmarks and standards for evaluating dangerous biological capabilities in AI-based research tools. By forming public-private partnerships, creating a DOE “sandbox” for ongoing testing, and integrating results into intelligence assessments, such a program would enable more nuanced, evidence-based regulations and help the United States stay ahead of potential adversaries seeking to exploit AI’s biological capabilities.

Policy Proposals to Strengthen Existing World-Class U.S. Government AI Institutions and Programs that are Key to the Trump Administration’s AI Agenda

A robust institutional framework is essential for ensuring that the government fulfills its role in AI research, industry coordination, and ecosystem development. The previous Trump administration laid the groundwork for American AI leadership, and the institutions established since then can be leveraged to further assert U.S. dominance in this critical technological space.

Recommendation 1. Support the NIST AI Safety Institute as a Key Pillar of American AI Excellence

The NIST AI Safety Institute (AISI) has assembled a world-leading team to ensure that the U.S. leads in safe, reliable, and trustworthy AI development. As AI integrates into critical sectors like national security, healthcare, and finance, strong safety standards are essential. AISI develops rigorous benchmarks, tests model security, and collaborates with industry to set standards, mitigating risks from unreliable AI. Strengthening AISI protects U.S. consumers, businesses, and national security while boosting global trust in the U.S. AI ecosystem—enhancing international adoption of American AI models. AISI has broad support, with bipartisan legislation to codify the AISI advanced in Congress and backing from organizations across industry and academia. The AI Action Plan should prioritize AISI as a pillar of AI policy.

Recommendation 2. Expand the National Artificial Intelligence Research Resource from Pilot to Full Program

For decades, academic researchers have driven AI breakthroughs, laying the foundation for the technologies that now shape global competition. However, as AI development becomes increasingly concentrated within large technology companies, the U.S. risks losing the ecosystem that made these advances possible. The National AI Research Resource (NAIRR) Pilot is a critical initiative to keep American AI innovation competitive and accessible. By providing researchers and educators across the country access to cutting-edge AI tools, datasets, and computing power, NAIRR ensures that innovation is not confined to a handful of dominant firms but widely distributed. To keep America at the forefront of AI, the Trump Administration should expand NAIRR into a full-fledged program. Allowing the program to lapse would erode America’s leadership in AI research, forcing top talent to seek resources elsewhere. To secure its future, the White House should support bipartisan legislation to fully authorize NAIRR and include it in the President’s Budget Request, ensuring sustained investment in this vital initiative.

Recommendation 3. Enhance Transparency, Accountability, and Industry Engagement by Preserving the AI Use Case Inventory (link to letter of support >>>)

The AI Use Case Inventory, established under President Trump’s Executive Order 13960 and later codified in section 7225 of the FY23 National Defense Authorization Act, plays a crucial role in fostering public trust and innovation in government AI use. Recent OMB guidance (M-24-10) has expanded its scope, refining AI classifications and standardizing AI definitions. The inventory enhances public trust and accountability by ensuring transparency in AI deployments, tracks AI successes and risks to improve government services, and supports AI vendors by providing visibility into public-sector AI needs, thereby driving industry innovation. As the federal government considers revisions to M-24-10 and its plan for AI adoption within federal agencies, OMB should uphold the 2024 guidance on federal agency AI Use Case Inventories and ensure agencies have the necessary resources to complete it effectively.

Recommendation 4. Propel U.S. Scientific and Security AI Leadership by Supporting AI and Computing at DOE 

The Department of Energy (DOE) hosts leading research and innovation centers, particularly under the Undersecretary for Science and Innovation. The Office of Critical and Emerging Technologies (CET), for example, plays a key role in coordinating AI initiatives, including the proposed Frontiers in Artificial Intelligence for Science, Security, and Technology (FASST) program. To fully harness AI’s potential, DOE should establish a dedicated AI and Computing Laboratory under the Undersecretary, ensuring a strategic, mission-driven approach to AI development. This initiative would accelerate scientific discovery, strengthen national security, and tackle energy challenges by leveraging DOE’s advanced computational infrastructure and expertise. To ensure success, it should be supported by a multi-year funding commitment and flexible operational authorities, modeled after ARPA-E, to streamline hiring, procurement, and industry-academic partnerships.

Conclusion

These recommendations offer a roadmap for securing America’s leadership in artificial intelligence while upholding the fundamental values of innovation, competitiveness, and trustworthiness. By investing in cutting-edge research, equipping government and educators with the tools to navigate the AI era, and ensuring safety, the new administration can position America as a global standard-bearer for trustworthy and effective AI development.

Deriving insights from data is essential for effective governance. However, collecting and sharing data—if not managed properly—can pose privacy risks for individuals. Current scientific understanding shows that so-called “anonymization” methods that have been widely used in the past are inadequate for protecting privacy in the era of big data and artificial intelligence. The evolving field of Privacy-Enhancing Technologies (PETs), including differential privacy and secure multiparty computation, offers a way forward for sharing data safely and responsibly.

The administration should prioritize the use of PETs by integrating them into data-sharing processes and strengthening the executive branch’s capacity to deploy PET solutions.

Challenge and Opportunity

A key function of modern government is the collection and dissemination of data. This role of government is enshrined in Article 1, Section 2 of the U.S. Constitution in the form of the decennial census—and has only increased with recent initiatives to modernize the federal statistical system and expand evidence-based policymaking. The number of datasets itself has also grown; there are now over 300,000 datasets on data.gov, covering everything from border crossings to healthcare. The release of these datasets not only accomplishes important transparency goals, but also represents an important step toward advancing American society fairer, as data are a key ingredient in identifying policies that benefit the public. 

Unfortunately, the collection and dissemination of data comes with significant privacy risks. Even with access to aggregated information, motivated attackers can extract information specific to individual data subjects and cause concrete harm. A famous illustration of this risk occurred in 1997 when Latanya Sweeney was able to identify the medical record of then-Governor of Massachusetts, William Weld, from a public, anonymized dataset. Since then, the power of data re-identification techniques—and incentives for third parties to learn sensitive information about individuals—have only increased, compounding this risk. As a democratic, civil-rights respecting nation, it is irresponsible for our government agencies to continue to collect and disseminate datasets without careful consideration of the privacy implications of data sharing.

While there may appear to be an irreconcilable tension between facilitating data-driven insight and protecting the privacy of individual’s data, an emerging scientific consensus shows that Privacy-Enhancing Technologies (PETs) offer a path forward. PETs are a collection of techniques that enable data to be used while tightly controlling the risk incurred by individual data subjects. One particular PET, differential privacy (DP), was recently used by the U.S. Census Bureau within their disclosure avoidance system for the 2020 decennial census in order to meet their dual mandates of data release and confidentiality. Other PETs, including variations of secure multiparty computation, have been used experimentally by other agencies, including to link long-term income data to college records and understand mental health outcomes for individuals who have earned doctorates. The National Institute of Standards and Technology (NIST) has produced frameworks and reports on data and information privacy, including PETs topics such as DP (see Q&A section). However, these reports still lack a comprehensive and actionable framework on how organizations should consider, use and deploy PETs in organizations. 

As artificial intelligence becomes more prevalent inside and outside government and relies on increasingly large datasets, the need for responsible data sharing is growing more urgent. The federal government is uniquely positioned to foster responsible innovation and set a strong example by promoting the use of PETs. The use of DP in the 2020 decennial census was an extraordinary example of the government’s capacity to lead global innovation in responsible data sharing practices. While the promise of continuing this trend is immense, expanding the use of PETs within government poses twin challenges: (1) sharing data within government comes with unique challenges—both technical and legal—that are only starting to be fully understood and (2) expertise on using PETs within government is limited. In this proposal, we outline a concrete plan to overcome these challenges and unlock the potential of PETs within government.

Plan of Action

Using PETs when sharing data should be a key priority for the executive branch. The new administration should encourage agencies to consider the use of PETs when sharing data and build a United States DOGE Service (USDS) “Responsible Data Sharing Corps” of professionals who can provide in-house guidance around responsible data sharing.

We believe that enabling data sharing with PETs requires (1) gradual, iterative refinement of norms and (2) increased capacity in government. With these in mind, we propose the following recommendations for the executive branch.

Strategy Component 1. Build consideration of PETs into the process of data sharing

Recommendation 1. NIST should produce a decision-making framework for organizations to rely on when evaluating the use of PETs.

NIST should provide a step-by-step decision-making framework for determining the appropriate use of PETs within organizations, including whether PETs should be used, and if so, which PET and how it should be deployed. Specifically, this guidance should be at the same level of granularity as NIST Risk Management Framework for Cybersecurity. NIST should consult with a range of stakeholders from the broad data sharing ecosystem to create this framework. This includes data curators (i.e., organizations that collect and share data, within and outside the government); data users (i.e., organizations that consume, use and rely on shared data, including government agencies, special interest groups and researchers); data subjects; experts across fields such as information studies, computer science, and statistics; and decision makers within public and private organizations who have prior experience using PETs for data sharing. The report may build on NIST’s existing related publications and other guides for policymakers considering the use of specific PETs, and should provide actionable guidance on factors to consider when using PETs. The output of this process should be not only a decision, but also a report documenting the execution of decision-making framework (which will be instrumental for Recommendation 3).

Recommendation 2. The Office of Management and Budget (OMB) should mandate government agencies interested in data sharing to use the NIST’s decision-making framework developed in Recommendation 1 to determine the appropriateness of PETs to protect their data pipelines.

The risks to data subjects associated with data releases can be significantly mitigated with the use of PETs, such as differential privacy. Along with considering other mechanisms of disclosure control (e.g., tiered access, limiting data availability), agencies should investigate the feasibility and tradeoffs around using PETs to protect data subjects while sharing data for policymaking and public use. To that end, OMB should require government agencies to use the decision-making framework produced by NIST (in Recommendation 1) for each instance of data sharing. We emphasize that this decision-making process may lead to a decision not to use PETs, as appropriate. Agencies should compile the produced reports such that they can be accessed by OMB as part of Recommendation 3.

Recommendation 3. OMB should produce a PET Use Case Inventory and annual reports that provide insights on the use of PETs in government data-sharing contexts.

To promote transparency and shared learning, agencies should share the reports produced as part of their PET deployments and associated decision-making processes with OMB. Using these reports, OMB should (1) publish a federal government PET Use Case Inventory (similar to the recently established Federal AI Use Case Inventory) and (2) synthesize these findings into an annual report. These findings should provide high-level insights into the decisions that are being made across agencies regarding responsible data sharing, and highlight the barriers to adoption of PETs within various government data pipelines. These reports can then be used to update the decision-making frameworks we propose that NIST should produce (Recommendation 1) and inspire further technical innovation in academia and the private sector.

Strategy Component 2. Build capacity around responsible data sharing expertise 

Increasing in-depth decision-making around responsible data sharing—including the use of PETs—will require specialized expertise. While there are some government agencies with teams well-trained in these topics (e.g., the Census Bureau and its team of DP experts), expertise across government is still lacking. Hence, we propose a capacity-building initiative that increases the number of experts in responsible data sharing across government.

Recommendation 4. Announce the creation of a “Responsible Data Sharing Corps.”

We propose that the USDS create a “Responsible Data Sharing Corps” (RDSC). This team will be composed of experts in responsible data sharing practices and PETs. RDSC experts can be deployed into other government agencies as needed to support decision-making about data sharing. They may also be available for as-needed consultations with agencies to answer questions or provide guidance around PETs or other relevant areas of expertise.

Recommendation 5. Build opportunities for continuing education and training for RDSC members.

Given the evolving nature of responsible data practices, including the rapid development of PETs and other privacy and security best practices, members of the RDSC should have 20% effort reserved for continuing education and training. This may involve taking online courses or attending workshops and conferences that describe state-of-the-art PETs and other relevant technologies and methodologies.

Recommendation 6. Launch a fellowship program to maintain the RDSC’s cutting-edge expertise in deploying PETS.

Finally, to ensure that the RDSC stays at the cutting edge of relevant technologies, we propose an RDSC fellowship program similar to or part of the Presidential Innovation Fellows. Fellows may be selected from academia or industry, but should have expertise in PETs and propose a novel use of PETs in a government data-sharing context. During their one-year terms, fellows will perform their proposed work and bring new knowledge to the RDSC.

Conclusion

Data sharing has become a key priority for the government in recent years, but privacy concerns make it critical to modernize technology for responsible data use to leverage data for policymaking and transparency. PETs such as differential privacy, secure multiparty computation, and others offer a promising way forward. However, deploying PETs at a broad scale requires changing norms and increasing capacity in government. The executive branch should lead these efforts by encouraging agencies to consider PETs when making data-sharing decisions and building a “Responsible Data Sharing Corps” who can provide expertise and support for agencies in this effort. By encouraging the deployment of PETs, the government can increase fairness, utility and transparency of data while protecting itself—and its data subjects—from privacy harms.

This action-ready policy memo is part of Day One 2025 — our effort to bring forward bold policy ideas, grounded in science and evidence, that can tackle the country’s biggest challenges and bring us closer to the prosperous, equitable and safe future that we all hope for whoever takes office in 2025 and beyond.

PLEASE NOTE (February 2025): Since publication several government websites have been taken offline. We apologize for any broken links to once accessible public data.

Frequently Asked Questions

What are the concrete risks associated with data sharing?

Data sharing requires a careful balance of multiple factors, with privacy and utility being particularly important.

• Data products released without appropriate and modern privacy protection measures could facilitate abuse, as attackers can weaponize information contained in these data products against individuals, e.g., blackmail, stalking, or publicly harassing those individuals.


• On the other hand, the lack of accessible data can also cause harm due to reduced utility: various actors, such as state and local government entities, may have limited access to accurate or granular data, resulting in the inefficient allocation of resources to small or marginalized communities.

read more

What are some examples of PETs to consider?

Privacy-Enhancing Technologies is a broad umbrella category that includes many different technical tools. Leading examples of these tools include differential privacy, secure multiparty computation, trusted execution environments, and federated learning. Each one of these technologies is designed to address different privacy threats. For additional information, we suggest the UN Guide on Privacy-Enhancing Technologies for Official Statistics and the ICO’s resources on Privacy-Enhancing Technologies.

read more

What NIST publications are relevant to PETs?

NIST has multiple publications related to data privacy, such as the Risk Management Framework for Cybersecurity and the Privacy Framework. The report De-Identifying Government Datasets: Techniques and Governance focuses on responsible data sharing by government organizations, while the Guidelines for Evaluating Differential Privacy Guarantees provides a framework to assess the privacy protection level provided by differential privacy for any organization.

read more

What is differential privacy (DP)?

Differential privacy is a framework for controlling the amount of information leaked about individuals during a statistical analysis. Typically, random noise is injected into the results of the analysis to hide individual people’s specific information while maintaining overall statistical patterns in the data. For additional information, we suggest Differential Privacy: A Primer for a Non-technical Audience.

read more

What is secure multiparty computation (MPC)?

Secure multiparty computation is a technique that allows several actors to jointly aggregate information while protecting each actor’s data from disclosure. In other words, it allows parties to jointly perform computations on their data while ensuring that each party learns only the result of the computation. For additional information, we suggest Secure Multiparty Computation FAQ for Non-Experts.

read more

How have privacy-enhancing technologies been used in government before, domestically and internationally?

There are multiple examples of PET deployments at both the federal and local levels both domestically and internationally. We list several examples below, and refer interested readers to the in-depth reports by Advisory Committee on Data for Evidence Building (report 1 and report 2):

• The Census Bureau used differential privacy in their disclosure avoidance system to release results from the 2020 decennial census data. Using differential privacy allowed the bureau to provide formal disclosure avoidance guarantees as well as precise information about the impact of this system on the accuracy of the data.


• The Boston Women’s Workforce Council (BWWC) measures wage disparities among employers in the greater Boston area using secure multiparty computation (MPC).


• The Israeli Ministry of Health publicly released its National Life Birth Registry using differential privacy.


• Privacy-preserving record linkage, a variant of secure multiparty computation, has been used experimentally by both the U.S. Department of Education and the National Center for Health Statistics. Additionally, it has been used at the county level in Allegheny County, PA.

Additional examples can also be found in the UN’s case-study repository of PET deployments.

read more

What type of expertise is required to deploy PETs solutions?

Data-sharing projects are not new to the government, and pockets of relevant expertise—particularly in statistics, software engineering, subject matter areas, and law—already exist. Deploying PET solutions requires technical computer science expertise for building and integrating PETs into larger systems, as well as sociotechnical expertise in communicating the use of PETs to relevant parties and facilitating decision-making around critical choices.

read more

Washington, D.C. – March 7, 2025 – The Federation of American Scientists (FAS) is pleased to welcome Dr. Yong-Bee Lim as the new Associate Director of Global Risk. In this role, Dr. Lim will help develop, organize, and implement FAS’s growing contribution in the area of catastrophic risk prevention, including on core areas of nuclear weapons, AI and national security, space and other emerging technologies.  

“The role of informed, credible and engaging organizations in support of sound public policy is more important than ever” said Jon Wolfsthal, FAS Director of Global Risk. “Yong-Bee embodies what it means to be an effective policy entrepreneur and to make meaningful contributions to US and global security. We are really excited that he is now part of the FAS team.”

Dr. Lim is a recognized expert in biosecurity, emerging technologies, and converging risks through his former roles as Deputy Director of both the the Converging Risks Lab and the Janne E. Nolan Center at the Council on Strategic Risks, his research and leadership roles in academia, and through his work at key agencies (DoD, HHS/ASPR, and DoE) in the United States. He completed his Ph.D. in Biodefense from George Mason University’s Biodefense program, where he conducted critical work on understanding the safety, security, and cultural dimensions of the U.S.-based Do-It-Yourself Biology (DIYBio) community. His recent accolades include being in the inaugural fellowship class of the Editorial Fellows program at the Bulletin of the Atomic Scientists and his selection and involvement in the Emerging Leaders in Biosecurity Initiative hosted by the Johns Hopkins Center for Health Security. 

“As emerging capabilities change the very contours of safety, security, and innovation, FAS has positioned itself to both highlight the global opportunities we must seize and address the global risks we must mitigate,” Lim said. “Founded in 1945, FAS continues to display thought leadership and impact because it has not forgotten its core mission: to ensure that scientific and technical expertise continue to have a seat at the policymaking table. I am honored to be part of an organization with a legacy and mission like FAS.”

ABOUT FAS

The Federation of American Scientists (FAS) works to advance progress on a broad suite of issues where science, technology, and innovation policy can deliver transformative impact, and seeks to ensure that scientific and technical expertise have a seat at the policymaking table. Established in 1945 by scientists in response to the atomic bomb, FAS continues to bring scientific rigor and analysis to address contemporary challenges. More information about FAS work at fas.org and Global Risk, here.

The federal government’s approach to deploying AI systems is a defining force in shaping industry standards, academic research, and public perception of these technologies. Public sentiment toward AI remains mixed, with many Americans expressing a lack of trust in AI systems. To fully harness the benefits of AI, the public must have confidence that these systems are deployed responsibly and enhance their lives and livelihoods.

The first Trump Administration’s AI policies clearly recognized the opportunity to promote AI adoption through transparency and public trust. President Trump’s Executive Order 13859 explicitly stated that agencies must design, develop, acquire, and use “AI in a manner that fosters public trust and confidence while protecting privacy, civil rights, civil liberties, and American values.” This commitment laid the foundation for increasing government accountability in AI use.

A major step in this direction was the AI Use Case Inventory, established under President Trump’s Executive Order 13960 and later codified in the 2023 Advancing American AI Act. The agency inventories have since become a crucial tool in fostering public trust and innovation in government AI use. Recent OMB guidance (M-24-10) has expanded its scope, standardizing AI definitions, and collecting information on potential adverse impacts. The detailed inventory enhances accountability by ensuring transparency in AI deployments, tracks AI successes and risks to improve government services, and supports AI vendors by providing visibility into public-sector AI needs, thereby driving industry innovation.

The end of 2024 marked a major leap in government transparency regarding AI use. Agency reporting on AI systems saw dramatic improvements, with federal AI inventories capturing more than 1,700 AI use cases —a 200% increase in reported use cases from the previous year. The Department of Homeland Security (DHS) alone reported 158 active AI use cases. Of these, 29 were identified as high-risk, with detailed documentation on how 24 of those use cases are mitigating potential risks. This level of disclosure is essential for maintaining public trust and ensuring responsible AI deployment.

OMB is set to release revisions to its AI guidance (M-24-10) in mid-March, presenting an opportunity to ensure that transparency remains a top priority.

To support continued transparency and accountability in government AI use, the Federation of American Scientists has written a letter urging OMB to maintain its detailed guidance on AI inventories. We believe that sustained transparency is crucial to ensuring responsible AI governance, fostering public trust, and enabling industry innovation.

The administration should create a federal center of excellence for state and local artificial intelligence (AI) procurement and use—a hub for expertise and resources on public sector AI procurement and use at the state, local, tribal, and territorial (SLTT) government levels. The center could be created by expanding the General Services Administration’s (GSA) existing Artificial Intelligence Center of Excellence (AI CoE). As new waves of AI technologies enter the market, shifting both practice and policy, such a center of excellence would help bridge the gap between existing federal resources on responsible AI and the specific, grounded challenges that individual agencies face. In the decades ahead, new AI technologies will touch an expanding breadth of government services—including public health, child welfare, and housing—vital to the wellbeing of the American people. An AI CoE federal center would equip public sector agencies with sustainable expertise and set a consistent standard for practicing responsible AI procurement and use. This resource ensures that AI truly enhances services, protects the public interest, and builds public trust in AI-integrated state and local government services. 

Challenge and Opportunity 

State, local, tribal, and territorial (SLTT) governments provide services that are critical to the welfare of our society. Among these: providing housing, child support, healthcare, credit lending, and teaching. SLTT governments are increasingly interested in using AI to assist with providing these services. However, they face immense challenges in responsibly procuring and using new AI technologies. While grappling with limited technical expertise and budget constraints, SLTT government agencies considering or deploying AI must navigate data privacy concerns, anticipate and mitigate biased model outputs, ensure model outputs are interpretable to workers, and comply with sector-specific regulatory requirements, among other responsibilities. 

The emergence of foundation models (large AI systems adaptable to many different tasks) for public sector use exacerbates these existing challenges. Technology companies are now rapidly developing new generative AI services tailored towards public sector organizations. For example, earlier this year, Microsoft announced that Azure OpenAI Service would be newly added to Azure Government—a set of AI services that target government customers. These types of services are not specifically created for public sector applications and use contexts, but instead are meant to serve as a foundation for developing specific applications. 

For SLTT government agencies, these generative AI services blur the line between procurement and development: Beyond procuring specific AI services, we anticipate that agencies will increasingly be tasked with the responsible use of general AI services to develop specific AI applications. Moreover, recent AI regulations suggest that responsibility and liability for the use and impacts of procured AI technologies will be shared by the public sector agency that deploys them, rather than just resting with the vendor supplying them.

SLTT agencies must be well-equipped with responsible procurement practices and accountability mechanisms pivotal to moving forward given the shifts across products, practice, and policy. Federal agencies have started to provide guidelines for responsible AI procurement (e.g., Executive Order 13960, OMB-M-21-06, NIST RMF). But research shows that SLTT governments need additional support to apply these resources.: Whereas existing federal resources provide high-level, general guidance, SLTT government agencies must navigate a host of challenges that are context-specific (e.g., specific to regional laws, agency practices, etc.). SLTT government agency leaders have voiced a need for individualized support in accounting for these context-specific considerations when navigating procurement decisions. 

Today, private companies are promising state and local government agencies that using their AI services can transform the public sector. They describe diverse potential applications, from supporting complex decision-making to automating administrative tasks. However, there is minimal evidence that these new AI technologies can improve the quality and efficiency of public services. There is evidence, on the other hand, that AI in public services can have unintended consequences, and when these technologies go wrong, they often worsen the problems they are aimed at solving. For example, by increasing disparities in decision-making when attempting to reduce them. 

Challenges to responsible technology procurement follow a historical trend: Government technology has frequently been critiqued for failures in the past decades. Because public services such as healthcare, social work, and credit lending have such high stakes, failures in these areas can have far-reaching consequences. They also entail significant financial costs, with millions of dollars wasted on technologies that ultimately get abandoned. Even when subpar solutions remain in use, agency staff may be forced to work with them for extended periods despite their poor performance.

The new administration is presented with a critical opportunity to redirect these trends. Training each relevant individual within SLTT government agencies, or hiring new experts within each agency, is not cost- or resource-effective. Without appropriate training and support from the federal government, AI adoption is likely to be concentrated in well-resourced SLTT agencies, leaving others with fewer resources (and potentially more low income communities) behind. This could lead to disparate AI adoption and practices among SLTT agencies, further exacerbating existing inequalities. The administration urgently needs a plan that supports SLTT agencies in learning how to handle responsible AI procurement and use–to develop sustainable knowledge about how to navigate these processes over time—without requiring that each relevant individual in the public sector is trained. This plan also needs to ensure that, over time, the public sector workforce is transformed in their ability to navigate complicated AI procurement processes and relationships—without requiring constant retraining of new waves of workforces. 

In the context of federal and SLTT governments, a federal center of excellence for state and local AI procurement would accomplish these goals through a “hub and spoke” model. This center of excellence would serve as the “hub” that houses a small number of selected experts from academia, non-profit organizations, and government. These experts would then train “spokes”—existing state and local public sector agency workers—in navigating responsible procurement practices. To support public sector agencies in learning from each others’ practices and challenges, this federal center of excellence could additionally create communication channels for information- and resource-sharing across the state and local agencies. 

Procured AI technologies in government will serve as the backbone of local public services for decades to come. Upskilling government agencies to make smart decisions about which AI technologies to procure (and which are best avoided) would not only protect the public from harmful AI systems but would also save the government money by decreasing the likelihood of adopting expensive AI technologies that end up getting dropped. 

Plan of Action 

A federal center of excellence for state and local AI procurement would ensure that procured AI technologies are responsibly selected and used to serve as a strong and reliable backbone for public sector services. This federal center of excellence can support both intra-agency and inter-agency capacity-building and learning about AI procurement and use—that is, mechanisms to support expertise development within a given public sector agency and between multiple public sector agencies. This federal center of excellence would not be deliberative (i.e., SLTT governments would receive guidance and support but would not have to seek approval on their practices). Rather, the goal would be to upskill SLTT agencies so they are better equipped to navigate their own AI procurement and use endeavors. 

To upskill SLTT agencies through inter-agency capacity-building, the federal center of excellence would house experts in relevant domain areas (e.g., responsible AI, public interest technology, and related topics). Fellows would work with cohorts of public sector agencies to provide training and consultation services. These fellows, who would come from government, academia, and civil society, would build on their existing expertise and experiences with responsible AI procurement, integrating new considerations proposed by federal standards for responsible AI (e.g., Executive Order 13960, OMB-M-21-06, NIST RMF). The fellows would serve as advisors to help operationalize these guidelines into practical steps and strategies, helping to set a consistent bar for responsible AI procurement and use practices along the way. 

Cohorts of SLTT government agency workers, including existing agency leaders, data officers, and procurement experts, would work together with an assigned advisor to receive consultation and training support on specific tasks that their agency is currently facing. For example, for agencies or programs with low AI maturity or familiarity (e.g., departments that are beginning to explore the adoption of new AI tools), the center of excellence can help navigate the procurement decision-making process, help them understand their agency-specific technology needs, draft procurement contracts, select amongst proposals, and negotiate plans for maintenance. For agencies and programs with high AI maturity or familiarity, the advisor can train the programs about unexpected AI behaviors and mitigation strategies, when this arises. These communication pathways would allow federal agencies to better understand the challenges state and local governments face in AI procurement and maintenance, which can help seed ideas for improving existing resources and create new resources for AI procurement support.

To scaffold intra-agency capacity-building, the center of excellence can build the foundations for cross-agency knowledge-sharing. In particular, it would include a communication platform and an online hub of procurement resources, both shared amongst agencies. The communication platform would allow state and local government agency leaders who are navigating AI procurement to share challenges, learned lessons, and tacit knowledge to support each other. The online hub of resources can be collected by the center of excellence and SLTT government agencies. Through the online hub, agencies can upload and learn about new responsible AI resources and toolkits (e.g., such as those created by government and the research community), as well as examples of procurement contracts that agencies themselves used. 

To implement this vision, the new administration should expand the U.S. General Services Administration’s (GSA) existing Artificial Intelligence Center of Excellence (AI CoE), which provides resources and infrastructural support for AI adoption across the federal government. We propose expanding this existing AI CoE to include the components of our proposed center of excellence for state and local AI procurement and use. This would direct support towards SLTT government agencies—which are currently unaccounted for in the existing AI CoE—specifically via our proposed capacity-building model.

Over the next 12 months, the goals of expanding the AI CoE would be three-fold:

1. Develop the core components of our proposed center of excellence within the AI CoE. 

• Recruit a core set of fellows with expertise in responsible AI, public interest technology, and related topics from government, academia, and civil society for a 1-2 year placement; 
• Develop a centralized onboarding and training program for the fellows to set standards for responsible AI procurement and use guidelines and goals; 
• Create a research strategy to streamline documentation of SLTT agencies’ on-the-ground practices and challenges for procuring new AI technologies, which could help prepare future fellows. 

2. Launch collaborations for the first sample of SLTT government agencies. Focus on building a path for successful collaborations: 

• Identify a small set of state and local government agencies who desire federal support in navigating AI procurement and use (e.g., deciding which AI use cases to adopt, how to effectively evaluate AI deployments through time, what organizational policies to create to help govern AI use); 
• Ensure there is a clear communication pathway between the agency and their assigned fellow; 
• Have each fellow and agency pair create a customized plan of action to ensure the agency is upskilled in their ability to independently navigate AI procurement and use with time. 

3. Build a path for our proposed center of excellence to grow and gain experience. If the first few collaborations show strong reviews, design a scaling strategy that will: 

• Incorporate the center of excellence’s core budget into future budget planning;
• Identify additional fellows for the program; 
• Roll out the program to additional state and local government agencies. 

Conclusion

Expanding the existing AI CoE to include our proposed federal center of excellence for AI procurement and use can help ensure that SLTT governments are equipped to make informed, responsible decisions about integrating AI technologies into public services. This body would provide necessary guidance and training, helping to bridge the gap between high-level federal resources and the context-specific needs of SLTT agencies. By fostering both intra-agency and inter-agency capacity-building for responsible AI procurement and use, this approach builds sustainable expertise, promotes equitable AI adoption, and protects public interest. This ensures that AI enhances—rather than harms—the efficiency and quality of public services. As new waves of AI technologies continue to enter the public sector, touching a breadth of services critical to the welfare of the American people, this center of excellence will help maintain high standards for responsible public sector AI for decades to come.

This action-ready policy memo is part of Day One 2025 — our effort to bring forward bold policy ideas, grounded in science and evidence, that can tackle the country’s biggest challenges and bring us closer to the prosperous, equitable and safe future that we all hope for whoever takes office in 2025 and beyond.

PLEASE NOTE (February 2025): Since publication several government websites have been taken offline. We apologize for any broken links to once accessible public data.

Frequently Asked Questions

What is existing guidance for responsible SLTT procurement and use of AI technologies?

Federal agencies have published numerous resources to support responsible AI procurement, including the Executive Order 13960, OMB-M-21-06, NIST RMF. Some of these resources provide guidance on responsible AI development in organizations broadly, across the public, private, and non-profit sectors. For example, the NIST RMF provides organizations with guidelines to identify, assess, and manage risks in AI systems to promote the deployment of more trustworthy and fair AI systems. Others focus on public sector AI applications. For instance, the OMB Memorandum published by the Office of Management and Budget describes strategies for federal agencies to follow responsible AI procurement and use practices.

read more

Why a federal center? Can’t SLTT governments do this on their own?

Research describes how these forms of resources often require additional skills and knowledge that make it challenging for agencies to effectively use on their own. A federal center of excellence for state and local AI procurement could help agencies learn to use these resources. Adapting these guidelines to specific SLTT agency contexts necessitates a careful task of interpretation which may, in turn, require specialized expertise or resources. The creation of this federal center of excellence to guide responsible SLTT procurement on-the-ground can help bridge this critical gap. Fellows in the center of excellence and SLTT procurement agencies can build on this existing pool of guidance to build a strong theoretical foundation to guide their practices.

read more

How has this “hub and spoke” model been used before?

The hub and spoke model has been used across a range of applications to support efficient management of resources and services. For instance, in healthcare, providers have used the hub and spoke model to organize their network of services; specialized, intensive services would be located in “hub” healthcare establishments whereas secondary services would be provided in “spoke” establishments, allowing for more efficient and accessible healthcare services. Similar organizational networks have been followed in transportation, retail, and cybersecurity. Microsoft follows a hub and spoke model to govern responsible AI practices and disseminate relevant resources. Microsoft has a single centralized “hub” within the company that houses responsible AI experts—those with expertise on the implementation of the company’s responsible AI goals. These responsible AI experts then train “spokes”—workers residing in product and sales teams across the company, who learn about best practices and support their team in implementing them.

read more

Who would be the experts selected as fellows by the center of excellence? What kind of training would they receive?

During the training, experts would form a stronger foundation for (1) on-the-ground challenges and practices that public sector agencies grapple with when developing, procuring, and using AI technologies and (2) existing AI procurement and use guidelines provided by federal agencies. The content of the training would be taken from syntheses of prior research on public sector AI procurement and use challenges, as well as existing federal resources available to guide responsible AI development. For example, prior research has explored public sector challenges to supporting algorithmic fairness and accountability and responsible AI design and adoption decisions, amongst other topics.

The experts who would serve as fellows for the federal center of excellence would be individuals with expertise and experience studying the impacts of AI technologies and designing interventions to support more responsible AI development, procurement, and use. Given the interdisciplinary nature of the expertise required for the role, individuals should have an applied, socio-technical background on responsible AI practices, ideally (but not necessarily) for the public sector. The individual would be expected to have the skills needed to share emerging responsible AI practices, strategies, and tacit knowledge with public sector employees developing or procuring AI technologies. This covers a broad range of potential backgrounds.

read more

What are some examples of the skills or competencies fellows might bring to the Center?

For example, a professor in academia who studies how to develop public sector AI systems that are more fair and aligned with community needs may be a good fit. A socio-technical researcher in civil society with direct experience studying or developing new tools to support more responsible AI development, who has intuition over which tools and practices may be more or less effective, may also be a good candidate. A data officer in a state government agency who has direct experience procuring and governing AI technologies in their department, with an ability to readily anticipate AI-related challenges other agencies may face, may also be a good fit. The cohort of fellows should include a balanced mix of individuals coming from government, academia, and civil society.

read more