FAS Position on “Schedule PC” and Impact on Federal Scientists

FAS shares the following formal comment in the Federal Register and asks that the scientific community, and the people across the nation who benefit from their research, to do the same.


The Federation of American Scientists opposes the proposed “Schedule Policy/Career” (“Schedule PC”) in present form because it rescinds civil servant employment protections, placing unnecessary and undesirable political pressure on highly specialized scientific and technical career professionals serving in government.


FAS encourages the Office of Personnel Management to rescind or substantially overhaul the Proposed Rule on Improving Performance, Accountability and Responsiveness in the Civil Service. We ask that OPM respond to the following comments and reflect how it will revise the Proposed Rule or abandon it.

New Employment Category is Unnecessary

Instead of creating a new employment category – the Schedule P/C for federal civil servants – the same goals can be accomplished by requiring agencies to regularly review and update critical elements in the performance appraisal system and their rating factors. Changing performance elements will have the impact of ensuring attention to accountability and responsiveness to policy without the ambiguity or determining assignment to the Schedule or the taxpayer expense of defending it.

The Administration is already taking this action by changing the performance appraisal system for the Senior Executive Service to make senior executives more responsive to Executive-branch priorities and policies. FAS advocates for updates to performance standards and rating factors appropriate for non-executives–based on the best available evidence–to achieve the intended accountability and responsiveness goals in this Proposed Rule. 

Proposed Rule Conflates Accountability with Administration

The Proposed Rule makes several errors in interpretation of the Civil Service Act of 1978, including the one potentially most detrimental to scientific enquiry, innovation, and exploration:

Securing American AI Leadership: A Strategic Action Plan for Innovation, Adoption, and Trust

The Federation of American Scientists (FAS) submitted the following response to the Request for Information (RFI) issued by the Office of Science and Technology Policy (OSTP) in February 2025 regarding the development of an Artificial Intelligence (AI) Action Plan.

At a time when AI is poised to transform every sector of the economy, the Trump administration has a critical opportunity to solidify America’s leadership in this pivotal technology. Building on the foundations laid during the first Trump administration, bold and targeted policies can unleash innovation, unlocking AI’s vast potential to stimulate economic growth, revolutionize industries, and strengthen national security. However, innovation alone is insufficient; without public trust, AI adoption will stall. Ensuring AI systems are transparent, reliable, and aligned with American values will accelerate responsible adoption and solidify AI as a cornerstone of America’s economic and technological leadership.

To sustain America’s leadership in AI innovation, accelerate adoption across the economy, and guarantee that AI systems remain secure and trustworthy, we offer a set of actionable policy recommendations. Developed by FAS in partnership with prominent AI experts, industry leaders, and research institutions—including contributors to the recent FAS Day One 2025 Project and the 2024 AI Legislative Sprint—these proposals are structured around four strategic pillars: 1) unleashing AI innovation, 2) accelerating AI adoption, 3) ensuring secure and trustworthy AI, and 4) strengthening existing world-class U.S. government institutions and programs. 

1) Unleashing AI Innovation. American AI leadership has been driven by bold private-sector investments and world-class academic research. However, critical high-impact areas remain underfunded. The federal government can catalyze investment and innovation by expanding access to essential data, investing strategically in overlooked areas of AI R&D, defining priority research challenges, promoting public-private partnerships, and attracting and retaining global talent.

2) Accelerating AI Adoption Across the Economy. The United States leads in AI breakthroughs, but these breakthroughs must translate into widespread adoption to maximize their economic and societal benefits. Accelerating adoption—a critical yet often overlooked driver of national competitiveness—requires addressing workforce readiness, expanding government capacity, and managing rising energy demands.

3) Ensuring Secure and Trustworthy AI. Ensuring AI systems are secure and trustworthy is essential not only for fostering public confidence and accelerating widespread adoption, but also for improving government efficiency and ensuring the responsible use of taxpayer resources when AI is deployed by public agencies. While the previous Trump administration recognized the necessity of public trust when promoting AI adoption, concerns persist about AI’s rapid evolution, unpredictable capabilities, and potential for misuse. Future AI accidents could further erode this trust, stalling AI progress. To address these risks and fully harness AI’s potential, the U.S. government must proactively monitor emerging threats, rigorously evaluate AI technologies, and encourage innovation that upholds fundamental American values such as privacy. 

4) Strengthening Existing World-Class U.S. Government AI Institutions and Programs. Realizing the Trump Administration’s goals will require building on leading government AI capabilities. Key initiatives—including the NIST AI Safety Institute (AISI), the National AI Research Resource (NAIRR) Pilot, the AI Use Case Inventory, and the Department of Energy’s Office of Critical and Emerging Technologies (CET)—advance AI innovation, security, and transparency. The AISI evaluates AI models with broad industry support, while the NAIRR Pilot expands access to AI resources beyond Big Tech. Federal AI use case inventories enhance government transparency and industry engagement, building public trust. DOE’s CET drives AI-powered advancements in science and national security. Integrating these proven initiatives into the AI Action Plan will solidify America’s AI leadership.

By acting decisively, the administration can ensure American AI remains the gold standard, drive economic competitiveness, and accelerate science and innovation.

Overview of Policy Proposals

Policy Proposals to Unleash AI Innovation

Policy Proposals to Accelerate AI Adoption Across the Economy

Policy Proposals to Ensure Secure and Trustworthy AI

Policy Proposals to Strengthen Existing World-Class U.S. Government AI Institutions and Programs that are Key to the Trump Administration’s AI Agenda

Policy Proposals to Unleash AI Innovation

As artificial intelligence continues transforming industries and reshaping global competition, the United States must take bold, coordinated action to maintain its technological leadership. A multi-agency approach could include launching a National Initiative for AI Explainability, accelerating materials science discovery through AI-powered autonomous laboratories, creating AI-ready datasets for the life sciences, establishing a NIST Foundation to enhance public-private collaboration in AI research, and creating a National Security AI Entrepreneur Visa to attract and retain top global talent. Together, these initiatives would strengthen America’s AI ecosystem by addressing critical challenges in transparency, scientific research, standards development, and talent acquisition—while ensuring the U.S. remains at the forefront of responsible AI innovation.

Recommendation 1. Promote Innovation in Trustworthy AI through a Public-Private National Initiative for AI Explainability 

Understanding the inner workings of AI systems is critical not only for reliability and risk mitigation in high-stakes areas such as defense, healthcare, and finance, but also for bolstering American technological leadership and maximizing government accountability and efficiency. However, despite promising progress in fields such as “mechanistic interpretability”, the study of explainability in AI systems is still nascent. A lack of explainability risks undermining trust and inhibiting AI adoption, particularly in safety-critical sectors.

To address the challenge of understanding and improving AI systems, we propose the launch of a Public-Private National Initiative for AI Explainability. Following in the footsteps of government-coordinated research projects like the Human Genome Project, this initiative would unite researchers, industry leaders, standards bodies, and government agencies to map the inner workings of advanced AI systems in a public-private partnership. 

Federal precedent for such work already exists: DARPA’s 2017-2021 Explainable AI (XAI) program sought to create machine learning systems capable of explaining their decisions in a way humans could understand. While the program advanced techniques for explainable models and human-friendly translations of complex AI reasoning, the rapid development and scaling of AI technologies in the past five years demand a renewed, more ambitious effort.

The objectives of the initiative would include:

Implementation Strategy:

To launch this effort, the President should issue an executive order to signal national commitment and assign leadership to key federal agencies, including:

The White House should leverage its convening power to unite leading AI companies, top academic institutions, and government agencies in formal collaborations. These partnerships could encompass co-funded research, shared datasets and computing resources, collaborative access to advanced AI models, and joint development of open-source tools. Establishing a structured public-private partnership will facilitate coordinated funding, align strategic priorities, and streamline resource sharing, ensuring that advancements in AI explainability directly support both national interests and economic competitiveness. To sustain this initiative, the administration should also secure consistent, multi-year federal funding through appropriations requests to Congress. 

DARPA’s XAI program showed that AI explainability requires interdisciplinary collaboration to align technical development with human understanding. Building on these insights, this initiative should include experts from computer science, cognitive science, ethics, law, and domain-specific fields to ensure explanations are clear, useful, and actionable for decision-makers across critical sectors. 

By implementing this National Initiative for AI Explainability, the Trump administration can significantly enhance public confidence in AI technologies, accelerate responsible adoption by both the public and private sectors, and solidify America’s global leadership in AI innovation. Critically, a modest investment of government resources in this initiative could unlock substantial private-sector investment, spurring innovation and driving economic growth. This strategic approach will also enhance government accountability, optimize the responsible use of taxpayer resources, and ensure that American industry continues to lead in AI development and deployment.

Recommendation 2. Direct the Department of Energy (DOE) to use AI to Accelerate the Discovery of New Materials (link to full memo >>>)

Innovations in AI and robotics could revolutionize materials science by automating experimental processes and dramatically accelerating the discovery of new materials. Currently, materials science research involves manually testing different combinations of elements to identify promising materials, which limits the pace of discovery. Using AI foundation models for physics and chemistry, scientists could simulate new materials, while robotic “self-driving labs” could run 24/7 to synthesize and evaluate them autonomously. This approach would enable continuous data generation, refining AI models in a feedback loop that speeds up research and lowers costs. Given its expertise in supercomputing, AI, and a vast network of national labs, the Department of Energy (DOE) could lead this transformative initiative, potentially unlocking advancements in critical materials, such as improved battery components, that could have immense economic and technological impacts.

Recommendation 3. Create AI-ready Collaborative Datasets to Accelerate Progress in the Life Sciences (link to full memo >>>)

Large, high-quality datasets could revolutionize life science research by powering AI models that unlock new discoveries in areas like drug development and diagnostics. Currently, researchers often work in silos with limited incentives to collaborate and share meticulously curated data, slowing progress. By launching a government-funded, end-to-end initiative—from identifying critical dataset needs to certifying automated collection methods and hosting robust open repositories—scientists could continuously generate and refine data, fueling AI models in a feedback loop that boosts accuracy and lowers costs. Even a relatively modest government investment could produce vital resources for researchers and startups to spark new industries. This model could also be extended to a range of other scientific fields to accelerate U.S.science and innovation.

Recommendation 4. Create a NIST Foundation to Support the Agency’s AI Mandate (link to full memo >>>)

To maintain America’s competitive edge in AI, NIST needs greater funding, specialized talent, and the flexibility to work effectively with private-sector partners. One solution is creating a “NIST Foundation,” modeled on the DOE’s Foundation for Energy Security and Innovation (FESI), which combines federal and private resources to expand capacity, streamline operations, and spur innovation. Legislation enabling such a foundation was introduced with bipartisan support in the 118th Congress, signaling broad consensus on its value. The Trump administration can direct NIST to study how a nonprofit foundation might boost its AI initiatives and broader mission—just as a similar report helped pave the way for FESI—giving Congress the evidence it needs to formally authorize a NIST Foundation. The administration can also support passage of authorizing legislation through Congress.

Recommendation 5. Attract Top Global Talent by Creating a National Security AI Entrepreneur Visa for Elite Dual-use Technology Founders (link to full memo >>>)

America’s leadership in AI has been driven by the contributions of immigrant entrepreneurs, with companies like NVIDIA, Anthropic, OpenAI, X, and HuggingFace—all of which have at least one immigrant co-founder—leading the charge. To maintain this competitive edge as global competition intensifies, the administration should champion a National Security Startup Visa specifically targeted at high-skilled founders of AI firms. These entrepreneurs are at the forefront of developing dual-use technologies critical for both America’s economic leadership and national security. Although the linked proposal above is targeted at legislative action, the administration can take immediate steps to advance this priority by publicly supporting legislation to establish such a visa, engaging with Congressional allies to underscore its strategic importance, and directing agencies like the Department of Homeland Security and the Department of Commerce to explore ways to streamline pathways for these innovators. This decisive action would send a clear signal that America remains the destination of choice for world-class talent, ensuring that the nation stays ahead in the race for AI dominance.

Policy Proposals to Accelerate AI Adoption Across the Economy

AI has transformative potential to boost economic growth and unlock new levels of prosperity for all. The Trump administration should take bold action to encourage greater adoption of AI technologies and AI expertise by leveraging government procurement, hiring, and standards-setting processes, alongside coordinated support for America’s teachers to prepare students to join the future AI workforce. In government, a coordinated set of federal initiatives is needed to modernize and streamline effective AI adoption in the public sector. These proposals include developing a national digital platform through GSA to streamline AI procurement processes, establishing a federal center of excellence to support state and local governments in AI implementation, and pursuing innovative hiring models to expand AI expertise at HHS. Additionally, NIST should develop voluntary standards for measuring AI energy and resource usage to inform infrastructure planning efforts. Finally, the President should announce a national teacher talent surge and set AI as a competitive priority in American education. 

Recommendation 1. Streamline Procurement Processes for Government Use of AI (link to full memo >>>)

The federal government has a critical role in establishing standards for AI systems to enhance public services while ensuring they are implemented ethically and transparently. To streamline this effort and support federal agencies, the administration should direct the General Services Administration (GSA) to create a user-friendly, digital platform for AI procurement. This platform would simplify the acquisition process by providing agencies with clear, up-to-date guidelines, resources, and best practices, all tailored to align with existing procurement frameworks. The platform would empower agencies to make informed decisions that prioritize safety, fairness, and effective use of AI technologies, while demonstrating the administration’s commitment to modernizing government operations and ensuring America leads the way in adopting cutting-edge AI solutions.

Recommendation 2. Establish a Federal Center of Excellence to Expand State and Local Government Capacity for AI Procurement and Use (link to full memo >>>)

State and local governments often face challenges in effectively leveraging AI to enhance their efficiency and service capabilities. To support responsible AI adoption at the state, local, tribal, and territorial (SLTT) levels, the administration should establish a federal AI Center of Excellence. This center would provide hands-on guidance from experts in government, academia, and civil society, helping SLTT agencies navigate complex challenges such as limited technical expertise, budget constraints, privacy concerns, and evolving regulations. It would also translate existing federal AI standards—including Executive Order 13960 and the NIST Risk Management Framework—into practical, actionable advice. By developing in-house procurement and deployment expertise, SLTT governments could independently and confidently implement AI solutions, promoting innovation while ensuring responsible, effective, and efficient use of taxpayer resources.

Recommendation 3. Pilot an AI Corps at HHS to Drive Government-Wide AI Adoption (link to full memo >>>) 

Federal agencies often struggle to leverage AI effectively, due to limited technical expertise and complex oversight requirements. Modeled after the Department of Homeland Security’s successful AI Corps, which has improved disaster response and cybersecurity, this pilot would embed AI and machine learning experts within the Department of Health and Human Services’s (HHS) 10 agencies, accelerating responsible AI implementation in healthcare, driving greater efficiency, and demonstrating a scalable model that could be replicated across other federal departments. HHS is uniquely suited for piloting an AI Corps because it oversees critical health infrastructure and massive, sensitive datasets—presenting significant opportunities for AI-driven improvements but also requiring careful management. If successful, this pilot could serve as a strategic blueprint to enhance AI adoption, improve government performance, and maximize the responsible use of taxpayer resources across the federal government.

Recommendation 4. Make America’s Teacher Workforce Competitive for the AI Era (link to full memo >>>) 

With America facing a significant shortage of teachers and a growing need for AI and digital skills in the workforce, the Trump administration can rebuild America’s teaching profession by launching a coordinated strategy led by the Office of Science and Technology Policy (OSTP). This initiative should begin with a national teacher talent surge to expand annual teacher graduates by 100,000, addressing both the urgent workforce gap and the imperative to equip students for an AI-driven future. The plan includes a Challenge.gov competition to attract innovative recruitment and retention models, updating Department of Education scholarship programs (like the Graduate Assistance in Areas of National Need) to include AI, data science, and machine learning, convening colleges of education to modernize training, and directing agencies to prioritize AI-focused teacher development. By leveraging existing grants (e.g., Teacher Quality Partnerships, SEED, the STEM Corps, and Robert Noyce Scholarships), the administration can ensure a robust pipeline of educators ready to guide the next generation.

Recommendation 5. Prepare U.S. Energy Infrastructure for AI Growth Through Standardized Measurement and Forecasting

As AI adoption accelerates, America’s energy infrastructure faces a critical challenge: next-generation AI systems could place unprecedented demands on the power grid, yet the lack of standardized measurements, and wide variations in forecasted demand, leaves utilities and policymakers unprepared. Without proactive planning, energy constraints could slow AI innovation and undermine U.S. competitiveness.

To address this, the Administration should direct the National Institute of Standards and Technology (NIST) and the Department of Energy (DOE) to develop a standardized framework for measuring and forecasting AI’s energy and resource demands. This framework should be paired with a voluntary reporting program for AI developers—potentially collected by the Energy Information Administration (EIA)—to provide a clearer picture of AI’s impact on energy consumption. The EIA should also be tasked with forecasting AI-driven energy demand, ensuring that utilities, public utility commissions, and state energy planners have the data needed to modernize the grid efficiently.

Greater transparency will enable both government and industry to anticipate energy needs, drive investment in grid modernization, and prevent AI-related power shortages that could hinder economic growth. The proactive integration of AI and energy planning will strengthen America’s leadership in AI innovation while safeguarding the reliability of its infrastructure. FAS is actively developing policy proposals with the science and technology community at the intersection of AI and energy. We plan to share additional recommendations on this topic in the coming months.

Policy Proposals to Ensure Secure and Trustworthy AI

Privacy

Protecting Americans’ privacy while harnessing the potential of AI requires decisive federal action that prioritizes both individual rights and technological advancement. Strengthening privacy protections while enabling responsible data sharing is crucial for ensuring that AI-driven innovations improve public services without compromising sensitive information. Key initiatives include establishing NIST-led guidelines for secure data sharing and maintaining data integrity, implementing a FedRAMP authorization framework for third-party data sources used by government agencies, and promoting the use of Privacy Enhancing Technologies (PETs). Additionally, the administration should create a “Responsible Data Sharing Corps” to provide agencies with expert guidance and build capacity in responsible data practices.

Recommendation 1. Secure Third Party Commercial Data for AI through FedRAMP Authorization (link to full memo >>>)

The U.S. government is a major customer of commercial data brokers and should require a pre-evaluation process before agencies acquire large datasets, ensuring privacy and security from the outset. Thoroughly vetting data brokers and verifying compliance standards can help avert national security risks posed by compromised or unregulated third-party vendors. To formalize these safeguards, OMB and FedRAMP should create an authorization framework for data brokers that provide commercially available information, especially with personally identifiable information. Building on its established role in securing cloud providers FedRAMP is well positioned to guide these protocols, ensuring agencies work only with trusted vendors and strengthening overall data protection.

Recommendation 2. Catalyze Federal Data Sharing through Privacy Enhancing Technologies (link to full memo >>>)

To maintain America’s leadership in AI and digital innovation, the administration must ensure that government agencies can securely leverage data while protecting privacy and maintaining public trust. The federal government can lead by example through the adoption of Privacy Enhancing Technologies (PETs)—tools that enable data analysis while minimizing exposure of sensitive information. Agencies should be encouraged to adopt PETs with support from a Responsible Data Sharing Corps, while NIST develops a decision-making framework to guide their use. OMB should require agencies to apply this framework in data-sharing initiatives and report on PET adoption, with a PET Use Case Inventory and annual reports enhancing transparency. A federal fellowship program could also bring in experts from academia and industry to drive PET innovation. These measures would strengthen privacy, security, and public trust while positioning the U.S. as a global leader in responsible data use.

Recommendation 3. Establish Data-Sharing Standards to Support AI Development in Healthcare (link to full memo >>>) 

The U.S. healthcare system generates vast amounts of data daily, yet fragmentation, privacy concerns, and lack of interoperability severely limit its use in AI development, hindering medical innovation. To address this, the AI Action Plan should direct NIST to lead an interagency coalition in developing standardized protocols for health data anonymization, secure sharing, and third-party access. By establishing clear technical and governance standards—similar to NIST’s Cryptographic and Biometric Standards Programs—this initiative would enable responsible research while ensuring compliance with privacy and security requirements. These standards would unlock AI-driven advancements in diagnostics, treatment planning, and health system efficiency. Other nations, including the U.K., Australia, and Finland, are already implementing centralized data-sharing frameworks; without federal leadership, the U.S. risks falling behind. By taking decisive action, the administration can position the U.S. as a global leader in medical AI, accelerating innovation while maintaining strong privacy protections.

Security, Safety, and Trustworthiness

AI holds immense promise for job growth, national security, and innovation, but accidents or misuse risk undermining public trust and slowing adoption—threatening the U.S.’s leadership in this critical field. The following proposals use limited, targeted government action alongside private-sector collaboration to strengthen America’s AI capabilities while upholding public confidence and protecting our national interests.

Recommendation 1. Establish an Early Warning System for AI-Powered Threats to National Security and Public Safety (link to full memo >>>) 

Emerging AI capabilities could also pose severe threats to public safety and national security. AI companies are already evaluating their most advanced models to identify dual-use capabilities, such as the capacity to conduct offensive cyber operations, enable the development of biological or chemical weapons, and autonomously replicate and spread. These capabilities can arise unpredictably and undetected during development and after deployment. To prepare for these emerging risks, the federal government should establish a coordinated “early-warning system” for novel dual-use AI capabilities to gain awareness of emerging risks before models are deployed. A government agency could serve as a central information clearinghouse—an approach adapted from the original congressional proposal linked above. Advanced AI model developers could confidentially report newly discovered or assessed dual-use capabilities, and the White House could direct relevant government agencies to form specialized working groups that engage with private sector and other non-governmental partners to rapidly mitigate risks and leverage defensive applications. This initiative would ensure that the federal government and its stakeholders have maximum lead time to prepare for emerging AI-powered threats, positioning the U.S. as a leader in safe and responsible AI innovation.

Recommendation 2. Create a Voluntary AI Incident Reporting Hub to Monitor Security Incidents from AI (link to full memo >>>)

The federal government should establish a voluntary national Artificial Intelligence Incident Reporting Hub to better track, analyze, and address incidents from increasingly complex and capable AI systems that are deployed in the real world. Such an initiative could be modeled after successful incident reporting and info-sharing systems operated by the National Cybersecurity FFRDC, the Federal Aviation Administration, and the Food and Drug Administration. By providing comprehensive yet confidential data collection under the umbrella of an agency (e.g. NIST) this initiative would bolster public trust, facilitate the sharing of critical risk information, and enable prompt government action on emerging threats, from cybersecurity vulnerabilities to potential misuse of AI in sensitive areas like chemical, biological, radiological, or nuclear contexts. This proposal builds on bipartisan legislation introduced in the last Congress, as well as the memo linked above, which was originally targeted at Congressional action.

Recommendation 3. Promote AI Trustworthiness by Providing a Safe Harbor for AI Researchers (link to full memo >>>)

Independent AI research plays a key role in ensuring safe and reliable AI systems. In 2024, over 350 researchers signed an open letter calling for “a safe harbor for independent AI evaluation”, noting that generative AI companies offer no legal protections for independent safety researchers. This situation is unlike established voluntary protections from companies for traditional software, and Department of Justice (DOJ) guidance not to prosecute good faith security research. The proposal linked above was targeted at Congressional action, however the executive branch could adapt these ideas in several ways, by, for example: 1) instructing the Office of Management and Budget (OMB) to issue guidance to all federal agencies requiring that contracting documents for generative AI systems include safe-harbor provisions for good-faith external research, consistent with longstanding federal policies that promote responsible vulnerability disclosure. 2) Coordinating with DOJ and relevant agencies to clarify that good-faith AI security and safety testing—such as red-teaming and adversarial evaluation—does not violate the Computer Fraud and Abuse Act (CFAA) or other laws when conducted according to established guidelines.

Recommendation 4. Build a National Digital Content Authentication Technologies Research Ecosystem (link to full memo >>>) 

AI generated synthetic content (such as fake videos, images, and audio) is increasingly used by malicious actors to defraud elderly Americans, spread child sexual abuse material, and impersonate political figures. To counter these threats, the United States must invest in developing technical solutions for reliable synthetic content detection. Through the National Institute of Standards and Technology (NIST), the Trump Administration can: 1) establish dedicated university-led national research centers, 2) develop a national synthetic content database, and 3) run and coordinate prize competitions to strengthen technical countermeasures.These initiatives will help build a robust research ecosystem to keep pace with the rapidly evolving synthetic content threat landscape, maintaining America’s role as a global leader in responsible and secure AI.

Recommendation 5. Strengthen National Security by Evaluating AI-Driven Biological Threats (link to full memo >>>)

Over the past two years, the rapid advance of AI in biology and large language models has highlighted an urgent need for a targeted U.S. Government program to assess and mitigate biosecurity risks. While AI-enabled tools hold immense promise for drug discovery, vaccine research, and other beneficial applications, their dual-use potential (e.g., identifying viral mutations that enhance vaccine evasion) makes them a national security priority. Building on the Department of Homeland Security’s (DHS) previous work on AI and CBRN threats, the Department of Energy (DOE),  DHS, and other relevant agencies, should now jointly launch a “Bio Capability Evaluations” program, backed by sustained funding, to develop specialized benchmarks and standards for evaluating dangerous biological capabilities in AI-based research tools. By forming public-private partnerships, creating a DOE “sandbox” for ongoing testing, and integrating results into intelligence assessments, such a program would enable more nuanced, evidence-based regulations and help the United States stay ahead of potential adversaries seeking to exploit AI’s biological capabilities.

Policy Proposals to Strengthen Existing World-Class U.S. Government AI Institutions and Programs that are Key to the Trump Administration’s AI Agenda

A robust institutional framework is essential for ensuring that the government fulfills its role in AI research, industry coordination, and ecosystem development. The previous Trump administration laid the groundwork for American AI leadership, and the institutions established since then can be leveraged to further assert U.S. dominance in this critical technological space.

Recommendation 1. Support the NIST AI Safety Institute as a Key Pillar of American AI Excellence

The NIST AI Safety Institute (AISI) has assembled a world-leading team to ensure that the U.S. leads in safe, reliable, and trustworthy AI development. As AI integrates into critical sectors like national security, healthcare, and finance, strong safety standards are essential. AISI develops rigorous benchmarks, tests model security, and collaborates with industry to set standards, mitigating risks from unreliable AI. Strengthening AISI protects U.S. consumers, businesses, and national security while boosting global trust in the U.S. AI ecosystem—enhancing international adoption of American AI models. AISI has broad support, with bipartisan legislation to codify the AISI advanced in Congress and backing from organizations across industry and academia. The AI Action Plan should prioritize AISI as a pillar of AI policy.

Recommendation 2. Expand the National Artificial Intelligence Research Resource from Pilot to Full Program

For decades, academic researchers have driven AI breakthroughs, laying the foundation for the technologies that now shape global competition. However, as AI development becomes increasingly concentrated within large technology companies, the U.S. risks losing the ecosystem that made these advances possible. The National AI Research Resource (NAIRR) Pilot is a critical initiative to keep American AI innovation competitive and accessible. By providing researchers and educators across the country access to cutting-edge AI tools, datasets, and computing power, NAIRR ensures that innovation is not confined to a handful of dominant firms but widely distributed. To keep America at the forefront of AI, the Trump Administration should expand NAIRR into a full-fledged program. Allowing the program to lapse would erode America’s leadership in AI research, forcing top talent to seek resources elsewhere. To secure its future, the White House should support bipartisan legislation to fully authorize NAIRR and include it in the President’s Budget Request, ensuring sustained investment in this vital initiative.

Recommendation 3. Enhance Transparency, Accountability, and Industry Engagement by Preserving the AI Use Case Inventory (link to letter of support >>>)

The AI Use Case Inventory, established under President Trump’s Executive Order 13960 and later codified in section 7225 of the FY23 National Defense Authorization Act, plays a crucial role in fostering public trust and innovation in government AI use. Recent OMB guidance (M-24-10) has expanded its scope, refining AI classifications and standardizing AI definitions. The inventory enhances public trust and accountability by ensuring transparency in AI deployments, tracks AI successes and risks to improve government services, and supports AI vendors by providing visibility into public-sector AI needs, thereby driving industry innovation. As the federal government considers revisions to M-24-10 and its plan for AI adoption within federal agencies, OMB should uphold the 2024 guidance on federal agency AI Use Case Inventories and ensure agencies have the necessary resources to complete it effectively.

Recommendation 4. Propel U.S. Scientific and Security AI Leadership by Supporting AI and Computing at DOE 

The Department of Energy (DOE) hosts leading research and innovation centers, particularly under the Undersecretary for Science and Innovation. The Office of Critical and Emerging Technologies (CET), for example, plays a key role in coordinating AI initiatives, including the proposed Frontiers in Artificial Intelligence for Science, Security, and Technology (FASST) program. To fully harness AI’s potential, DOE should establish a dedicated AI and Computing Laboratory under the Undersecretary, ensuring a strategic, mission-driven approach to AI development. This initiative would accelerate scientific discovery, strengthen national security, and tackle energy challenges by leveraging DOE’s advanced computational infrastructure and expertise. To ensure success, it should be supported by a multi-year funding commitment and flexible operational authorities, modeled after ARPA-E, to streamline hiring, procurement, and industry-academic partnerships.

Conclusion

These recommendations offer a roadmap for securing America’s leadership in artificial intelligence while upholding the fundamental values of innovation, competitiveness, and trustworthiness. By investing in cutting-edge research, equipping government and educators with the tools to navigate the AI era, and ensuring safety, the new administration can position America as a global standard-bearer for trustworthy and effective AI development.

Public Comment on Executive Branch Agency Handling of CAI containing PII

Public comments serve the executive branch by informing more effective, efficient program design and regulation. As part of our commitment to evidence-based, science-backed policy, FAS staff leverage public comment opportunities to embed science, technology, and innovation into policy decision-making.

The Federation of American Scientists (FAS) is a non-partisan, nonprofit organization committed to using science and technology to benefit humanity by delivering on the promise of equitable and impactful policy. FAS believes that society benefits from a federal government that harnesses science, technology, and innovation to meet ambitious policy goals and deliver impactful results to the public. 

We are writing in response to your Request for Information on the Executive Branch Agency Handling of Commercially Available Information (CAI) Containing Personally Identifiable Information (PII). Specifically, we will be answering questions 2 and 5 in your request for information: 

2. What frameworks, models, or best practices should [the White House Office of Management and Budget] consider as it evaluates agency standards and procedures associated with the handling of CAI containing PII and considers potential guidance to agencies on ways to mitigate privacy risks from agencies’ handling of CAI containing PII?

5.  Agencies provide transparency into the handling of PII through various means (e.g., policies and directives, Privacy Act statements and other privacy notices at the point of collection, Privacy Act system of records notices, and privacy impact assessments). What, if any, improvements would enhance the public’s understanding of how agencies handle CAI containing PII?

Background

In the digital landscape, commercially available information (CAI) represents a vast ecosystem of personal data that can be easily obtained, sold, or licensed to various entities. The Executive Order on Safe, Secure, and Trustworthy Development and Use of Artificial Intelligence (EO 14110) defines CAI comprehensively as information about individuals or groups that is publicly accessible, encompassing details like device information and location data.

A 2017 report by the Georgetown Law Review found that 63% of Americans can be uniquely identified using just three basic attributes—gender, birth date, and ZIP code—with an astonishing 99.98% of individuals potentially re-identifiable from a dataset containing only 15 fundamental characteristics. This vulnerability underscores the critical challenges of data privacy in an increasingly interconnected world. 

CAI takes on heightened significance in the context of artificial intelligence (AI) deployment, as these systems enable both data collection and the use of advanced inference models to analyze datasets and produce predictions, insights, and assumptions that reveal patterns or relationships not directly evident in the data. Some AI systems can allow the intentional or unintentional reidentification of supposedly anonymized private data. These capabilities raise questions about privacy, consent, and the potential for unprecedented levels of personal information aggregation and analysis, challenging existing data protection frameworks and individual rights.

The United States federal government is one of the largest customers of commercial data brokers. Government entities increasingly use CAI to empower public programs, enabling federal agencies to augment decision-making, policy development, and resource allocation and enrich research and innovation goals with large yet granular datasets. For example, the National Institutes of Health have discussed within their data strategies how to incorporate commercially available data into research projects. The use of commercially available electronic health records is essential for understanding social inequalities within the healthcare system but includes sensitive personal data that must be protected. 

However, government agencies face significant public scrutiny over their use of CAI in areas including law enforcement, homeland security, immigration, and tax administration. This scrutiny stems from concerns about privacy violations, algorithmic bias, and the risks of invasive surveillance, profiling, and discriminatory enforcement practices that could disproportionately harm vulnerable populations.  For example, federal agencies like Immigration and Customs Enforcement (ICE) and Customs and Border Protection (CBP) have used broker-purchased location data to track individuals without warrants, raising constitutional concerns. 

In 2020, the American Civil Liberties Union filed a Freedom of Information Act lawsuit against several Department of Homeland Security (DHS) agencies, arguing that the DHS’s use of cellphone data and data from smartphone apps constitutes unreasonable searches without a warrant and violates the Fourth Amendment. A report by the Electronic Frontier Foundation found that CAI was used for mass surveillance practices, including geofence warrants that query all phones in specific locations, further challenging constitutional protections. 

While the Privacy Act of 1974 covers the use of federally collected personal information by agencies, there is no explicit guidance governing federal use of third-party data. The bipartisan Fourth Amendment is Not for Sale Act (H.R.4639) would bar certain technology providers—such as remote computing service and electronic communication service providers—from sharing the contents of stored electronic communications with anyone (including government actors) and from sharing customer records with government agencies. The bill has passed the House of Representatives in the 118th Congress but has yet to pass the Senate as of December 2024. Without protections in statute, it is imperative that the federal government crafts clear guidance on the use of CAI containing PII in AI systems. In this response to the Office of Management and Budget’s (OMB) request for information, FAS will outline three policy ideas that can improve how federal agencies navigate the use of CAI containing PII, including in AI use. 

Summary of Recommendations

The federal government is responsible for ensuring the safety and privacy of the processing of personally identifiable information within commercially available information used for the development and deployment of artificial intelligence systems. For this RFI, FAS brings three proposals to increase government capacity in ensuring transparency and risk mitigation in how CAI containing PII is used, including in agency use of AI: 

  1. Enable FedRAMP to Create an Authorization System for Third-Party Data Sources: An authorization framework for CAI containing PII would ensure a standardized approach for data collection, management, and contracting, mitigating risks, and ensuring ethical data use.
  2. Expand Existing Privacy Impact Assessments (PIA) to Incorporate Additional Requirements and Periodic Evaluations: Regular public reports on CAI sources and usage will enable stakeholders to monitor federal data practices effectively.
  3. Build Government Capacity for the Use of Privacy Enhancing Technologies to Bolster Anonymization Techniques by harnessing existing resources such as the United States Digital Service (USDS). 

Recommendation 1. Enable FedRAMP to Create an Authorization System for Third-Party Data Sources

Government agencies utilizing CAI should implement a pre-evaluation process before acquiring large datasets to ensure privacy and security. OMB, along with other agencies that are a part of the governing board of the Federal Risk and Authorization Management Program (FedRAMP), should direct FedRAMP to create an authorization framework for third-party data sources that contract with government agencies, especially data brokers that provide CAI with PII, to ensure that these vendors comply with privacy and security requirements. FedRAMP is uniquely positioned for this task because of its previous mandate to ensure the safety of cloud service providers used by the federal government and its recent expansion of this mandate to standardize AI technologies. The program could additionally harmonize its new CAI requirements with its forthcoming AI authorization framework.

When designing the content of the CAI authorization, a useful benchmark in terms of evaluation criteria is the Ag Data Transparent (ADT) certification process. Companies applying for this certification must submit contracts and respond to 11 data collection, usage, and sharing questions. Like the FedRAMP authorization process, a third-party administrator reviews these materials for consistency, granting the ADT seal only if the company’s practices align with its contracts. Any discrepancies must be corrected, promoting transparency and protecting farmers’ data rights. The ADT is a voluntary certification, and therefore does not provide a good model for enforcement. However, it does provide a framework for the kind of documentation that should be required. The CAI authorization should thus include the following information required by the ADT certification process:

Unlike the ADT, a FedRAMP authorization process can be strictly enforced. FedRAMP is mandatory for all cloud service providers working with the executive branch and follows a detailed authorization process with evaluations and third-party auditors. It would be valuable to bring that assessment rigor to federal agency use of CAI, and would help provide clarity to commercial vendors. 

The authorization framework should also document the following specific protocols for the use of CAI within AI systems:

By setting these standards, this authorization could help agencies understand privacy risks and ensure the reliability of CAI data vendors before deploying purchased datasets within AI systems or other information systems, therefore setting them up to create appropriate mitigation strategies. 

By encouraging data brokers to follow best practices, this recommendation would allow agencies to focus on authorized datasets that meet privacy and security standards. Public availability of this information could drive market-wide improvements in data governance and elevate trust in responsible data usage. This approach would support ethical data governance in AI projects and create a more transparent, publicly accountable framework for CAI use in government.  

Recommendation 2. Expand Privacy Impact Assessments (PIA) to Incorporate Additional Requirements and Periodic Evaluations 

Public transparency regarding the origins and details of government-acquired CAI containing PII is critical, especially given the largely unregulated nature of the data broker industry at the federal level. Privacy Impact Assessments (PIAs) are mandated under Section 208 of the 2002 E-Government Act and OMB Memo M-03-22, and can serve as a vital policy tool for ensuring such transparency. Agencies must complete PIAs at the outset of any new electronic information collection process that includes “information in identifiable form for ten or more persons.” Under direction from Executive Order 14110 on Safe, Secure, and Trustworthy Development and Use of Artificial Intelligence, OMB issued a request for information in April 2024 to explore updating PIA guidance for AI-era privacy concerns, although new guidance has not yet been issued. 

To ensure that PIAs can effectively provide transparency into government practices on CAI that contains PII, we recommend that OMB provide updated guidance requiring agencies to regularly review and update their PIAs at least every three years, and also require agencies to report more comprehensive information in PIAs. We provide more details on these recommendations below.

First, OMB should guide agencies to periodically update their PIAs to ensure evolutions in agency data practices are publicly captured, which is increasingly important as data-driven AI systems are adopted by government actors and create novel privacy concerns. Under OMB Memo M-03-22, agencies must initiate or update PIAs when new privacy risks or factors emerge that affect the collection and handling of PII, including when agencies incorporate PII obtained from commercial or public sources into existing information systems. However, a public comment submitted by the Electronic Privacy Information Center (EPIC) pointed out that many agencies fail to publish and update required PIAs in a timely manner, indicating that a stricter schedule is needed to maintain accountability for PIA reporting requirements. As data privacy risks evolve through the advancement of AI systems, increased cybersecurity risks, and new legislation, it is essential that a minimum standard schedule for updating PIAs is created to ensure agencies provide the public with an up-to-date understanding of the potential risks resulting from using CAI that includes PII. For example, the European Union’s ​​General Data Protection Regulation (Art. 35) requires PIAs to be reconducted every three years. 

Second, agency PIAs should report more detailed information on the CAI’s source, vendor information, contract agreements, and licensing arrangements. A frequent critique of existing PIAs is that they contain too little information to inform the public of relevant privacy harms. Such a lack of transparency risks damaging public trust in government. One model for expanded reporting frameworks for CAI containing PII is the May 2024 Policy Framework for CAI, established for the Intelligence Community (IC) by the Office of the Director of National Intelligence (ODNI). This framework requires the IC to document and report “the source of the Sensitive CAI and from whom the Sensitive CAI was accessed or collected” and “any licensing agreements and/or contract restrictions applicable to the Sensitive CAI”. OMB should incorporate these reporting practices into agency PIA requirements and explicitly require agencies to identify the CAI data vendor in order to provide insight into the source and quality of purchased data.

Many of these elements are also present in Recommendation 1, for a new FedRAMP authorization framework. However, that recommendation does not include existing agency projects using CAI or agencies that could contract CAI datasets outside of the FedRAMP authorization. Including this information within the PIA framework also allows for an iterative understanding of privacy risks throughout the lifecycle of a project using CAI. 

By obligating agencies to provide more frequent PIA updates and include additional details on the source, vendor, contract and licensing arrangements for CAI containing PII, the public gains valuable insight into how government agencies acquire, use, and manage sensitive data. These updates to PIAs would allow civil society groups, journalists, and other external stakeholders to track government data management practices over time during this critical juncture where federal uptake of AI systems is rapidly increasing.

Recommendation 3. Build Government Capacity for the Use of Privacy Enhancing Technologies to Bolster Anonymization Techniques

Privacy Enhancing Technologies (PETs) are a diverse set of tools that can be used throughout the data lifecycle to ensure privacy by design. They can also be powerful tools in ensuring that PII within CAI) is adequately anonymized and secure. OMB should collect information on current agency PET usage, gather best practices, and identify deployment gaps. To address these gaps, OMB should collaborate with agencies like the USDS to establish capacity-building programs, leveraging initiatives like the proposed “Responsible Data Sharing Core” to provide expert consultations and enhance responsible data-sharing practices.

Meta’s Open Loop project identified eight types of PETs that are ripe to be deployed in AI systems, categorizing them into maturity levels, context of deployment, and limitations. One type of PET is differential privacy, a mathematical framework designed to protect individuals’ privacy in datasets by introducing controlled noise to the data. This ensures that the output of data analysis or AI models does not reveal whether a specific individual’s information is included in the dataset. The noise is calibrated to balance privacy with data utility, allowing meaningful insights to be derived without compromising personal information. Differential privacy is particularly useful in AI models that rely on large-scale data for training, as it prevents the inadvertent exposure of PII during the learning process. Within the federal government, the U.S. Census Bureau is using differential privacy to anonymize data while preserving its aggregate utility, ensuring compliance with privacy regulations and reducing re-identification within datasets.

Scaling the use of PETs in other agencies has been referenced in several U.S. government strategy documents, such as the National Strategy to Advance Privacy-Preserving Data Sharing and Analytics, which encourages federal agencies to adopt and invest in the development of PETs, and the Executive Order (EO) on the Safe, Secure, and Trustworthy Development and Use of Artificial Intelligence, which calls for federal agencies to identify where they could use PETs. As a continuation of this EO, the National Science Foundation and the Department of Energy established a Research Coordination Network on PETs that will “address the barriers to widespread adoption of PETs, including regulatory considerations.”  

Although the ongoing research and development of PETS is vital to this growing field, there is an increasing need to ensure these technologies are implemented across the federal government. To kick this off, OMB should collect detailed information on how agencies currently use PETs, especially in projects that use CAI containing PII. This effort should include gathering best practices from agencies with successful PET implementations, such as the previous U.S. Census Bureau’s use of differential privacy. Additionally, OMB should identify gaps in PET deployment, assessing barriers such as technical capacity, funding, and awareness of relevant PETs. To address these gaps, OMB should collaborate with other federal agencies to design and implement capacity-building programs, equipping personnel with the knowledge and tools needed to integrate PETs effectively. For example, a forthcoming FAS’ Day One Project publication, “Increasing Responsible Data Sharing Capacity throughout Government,” seeks to harness existing government capabilities to build government capacity in deploying PETs. This proposal aims to enhance responsible data sharing in government by creating a capacity-building initiative called the  “Responsible Data Sharing Core” (RDSC). Managed by the USDS, the RDSC would deploy fellows and industry experts to agencies to consult on data use and sharing decisions and offer consultations on which PETs are appropriate for different contexts.   

Conclusion

The federal government’s increasing reliance on CAI containing PII presents significant privacy challenges. The current landscape of data procurement and AI deployment by agencies like ICE, CBP, and others raises critical concerns about potential Fourth Amendment violations, discriminatory profiling, and lack of transparency.

The ideas proposed in this memo—implementing FedRAMPamp authorization for data brokers, expanding privacy impact assessment requirements, and developing capacity-building programs for privacy-enhancing technologies—represent crucial first steps in addressing these systemic risks. As AI systems become increasingly integrated into government processes, maintaining a delicate balance between technological advancement and fundamental constitutional protections will be paramount to preserving individual privacy, promoting responsible adoption, and maintaining public trust.

We appreciate the opportunity to contribute to this Request for Information on Executive Branch Agency Handling of Commercially Available Information Containing Personally Identifiable Information. Please contact clangevin@fas.org if you have any questions or need additional information.

Public Comment on the U.S. Artificial Intelligence Safety Institute’s Draft Document: NIST AI 800-1, Managing Misuse Risk for Dual-Use Foundation Models

Public comments serve the executive branch by informing more effective, efficient program design and regulation. As part of our commitment to evidence-based, science-backed policy, FAS staff leverage public comment opportunities to embed science, technology, and innovation into policy decision-making.

The Federation of American Scientists (FAS) is a non-partisan organization dedicated to using science and technology to benefit humanity through equitable and impactful policy. With a strong track record in AI governance, FAS has actively contributed to the development of AI standards and frameworks, including providing feedback on NIST AI 600-1, the Generative AI Profile. Our work spans advocating for federal AI testbeds, recommending policy measures for frontier AI developers, and evaluating industry adoption of the NIST AI Risk Management Framework. We are members of the U.S. AI Safety Institute Research Consortium, and we responded to NIST’s request for information earlier this year concerning its responsibilities under sections 4.1, 4.5, and 11 of the AI Executive Order.

We commend NIST’s U.S. Artificial Intelligence Safety Institute for developing the draft guidance on “Managing Misuse Risk for Dual-Use Foundation Models.” This document represents a significant step toward establishing robust practices for mitigating catastrophic risks associated with advanced AI systems. The guidance’s emphasis on comprehensive risk assessment, transparent decision-making, and proactive safeguards aligns with FAS’s vision for responsible AI development.

In our response, we highlight several strengths of the guidance, including its focus on anticipatory risk assessment and the importance of clear documentation. We also identify areas for improvement, such as the need for harmonized language and more detailed guidance on model development safeguards. Our key suggestions include recommending a more holistic socio-technical approach to risk evaluation, strengthening language around halting development for unmanageable risks, and expanding the range of considered safeguards. We believe these adjustments will further strengthen NIST’s crucial role in shaping responsible AI development practices.

Background and Context

The rapid advancement of AI foundation models has spurred novel industry-led risk mitigation strategies. Leading AI companies have voluntarily adopted frameworks like Responsible Scaling Policies and Preparedness Frameworks, outlining risk thresholds and mitigation strategies for increasingly capable AI systems. (Our response to NIST’s February RFI was largely an exploration of these policies, their benefits and drawbacks, and how they could be strengthened.)

Managing misuse risks in foundation models is of paramount importance given their broad applicability and potential for dual use. As these models become more powerful, they may inadvertently enable malicious actors to cause significant harm, including facilitating the development of weapons, enabling sophisticated cyber attacks, or generating harmful content. The challenge lies not only in identifying current risks but also in anticipating future threats that may emerge as AI capabilities expand.

NIST’s new guidance on “Managing Misuse Risk for Dual-Use Foundation Models” builds upon these industry initiatives, providing a more standardized and comprehensive approach to risk management. By focusing on objectives such as anticipating potential misuse, establishing clear risk thresholds, and implementing robust evaluation procedures, the guidance creates a framework that can be applied across the AI development ecosystem. This approach is crucial for ensuring that as AI technology advances, appropriate safeguards are in place to protect against potential misuse while still fostering innovation.

Strengths of the guidance

1. Comprehensive Documentation and Transparency

The guidance’s emphasis on thorough documentation and transparency represents a significant advancement in AI risk management. For every practice under every objective, the guidance indicates appropriate documentation; this approach is more thorough in advancing transparency than any comparable guidance to date. The creation of a paper trail for decision-making and risk evaluation is crucial for both internal governance and potential external audits.

The push for transparency extends to collaboration with external stakeholders. For instance, practice 6.4 recommends providing “safe harbors for third-party safety research,” including publishing “a clear vulnerability disclosure policy for model safety issues.” This openness to external scrutiny and feedback is essential for building trust and fostering collaborative problem-solving in AI safety. (FAS has published a legislative proposal calling for enshrining “safe harbor” protections for AI researchers into law.)

2. Lifecycle Approach to Risk Management

The guidance excels in its holistic approach to risk management, covering the entire lifecycle of foundation models from pre-development assessment through to post-deployment monitoring. This comprehensive approach is evident in the structure of the document itself, which follows a logical progression from anticipating risks (Objective 1) through to responding to misuse after deployment (Objective 6).

The guidance demonstrates a proactive stance by recommending risk assessment before model development. Practice 1.3 suggests to “Estimate the model’s capabilities of concern before it is developed…”, which helps anticipate and mitigate potential harms before they materialize. The framework for red team evaluations (Practice 4.2) is particularly robust, recommending independent external experts and suggesting ways to compensate for gaps between red teams and real threat actors. The guidance also emphasizes the importance of ongoing risk assessment. Practice 3.2 recommends to “Periodically revisit estimates of misuse risk stemming from model theft…” This acknowledgment of the dynamic nature of AI risks encourages continuous vigilance.

3. Strong Stance on Model Security and Risk Tolerance

The guidance takes a firm stance on model security and risk tolerance, particularly in Objective 3. It unequivocally states that models relying on confidentiality for misuse risk management should only be developed when theft risk is sufficiently mitigated. This emphasizes the critical importance of security in AI development, including considerations for insider threats (Practice 3.1).

The guidance also demonstrates a realistic approach to the challenges posed by different deployment strategies. In Practice 5.1, it notes, “For example, allowing fine-tuning via API can significantly limit options to prevent jailbreaking and sharing the model’s weights can significantly limit options to monitor for misuse (Practice 6.1) and respond to instances of misuse (Practice 6.2).” This candid discussion of the limitations of safety interventions for open weight foundation models is crucial for fostering realistic risk assessments.

Additionally, the guidance promotes a conservative approach to risk management. Practice 5.3 recommends to “Consider leaving a margin of safety between the estimated level of risk at the point of deployment and the organization’s risk tolerance.” It further suggests considering “a larger margin of safety to manage risks that are more severe or less certain.” This approach provides an extra layer of protection against unforeseen risks or rapid capability advancements, which is crucial given the uncertainties inherent in AI development.

These elements collectively demonstrate NIST’s commitment to promoting realistic and robust risk management practices that prioritize safety and security in AI development and deployment. However, while the NIST guidance demonstrates several important strengths, there are areas where it could be further improved to enhance its effectiveness in managing misuse risks for dual-use foundation models.

Areas for improvement

1. Need for a More Comprehensive Socio-technical Approach to Measuring Misuse Risk

Objective 4 of the guidance demonstrates a commendable effort to incorporate elements of a socio-technical approach in measuring misuse risk. The guidance recognizes the importance of considering both technical and social factors, emphasizes the use of red teams to assess potential misuse scenarios, and acknowledges the need to consider different levels of access and various threat actors. Furthermore, it highlights the importance of avoiding harm during the measurement process, which is crucial in a socio-technical framework.

However, the guidance falls short in fully embracing a comprehensive socio-technical perspective. While it touches on the importance of external experts, it does not sufficiently emphasize the value of diverse perspectives, particularly from individuals with lived experiences relevant to specific risk scenarios. The guidance also lacks a structured approach to exploring the full range of potential misuse scenarios across different contexts and risk areas. Finally, the guidance does not mention measuring absolute versus marginal risks (ie., how much total misuse risk a model poses in a specific context versus how much marginal risk it poses compared to existing tools). These gaps limit the effectiveness of the proposed risk measurement approach in capturing the full complexity of AI system interactions with human users and broader societal contexts.

Specific recommendations for improving socio-technical approach

The NIST guidance in Practice 1.3 suggests estimating model capabilities by comparison to existing models, but provides little direction on how to conduct these comparisons effectively. To improve this, NIST could incorporate the concept of “available affordances.” This concept emphasizes that an AI system’s risk profile depends not just on its absolute capabilities, but also on the environmental resources and opportunities for affecting the world that are available to it.

Additionally, Kapoor et al. (2024) emphasize the importance of assessing the marginal risk of open foundation models compared to existing technologies or closed models. This approach aligns with a comprehensive socio-technical perspective by considering not just the absolute capabilities of AI systems, but also how they interact with existing technological and social contexts. For instance, when evaluating cybersecurity risks, they suggest considering both the potential for open models to automate vulnerability detection and the existing landscape of cybersecurity tools and practices. This marginal risk framework helps to contextualize the impact of open foundation models within broader socio-technical systems, providing a more nuanced understanding of their potential benefits and risks. 

NIST could recommend that organizations assess both the absolute capabilities of their AI systems and the affordances available to them in potential deployment contexts. This approach would provide a more comprehensive view of potential risks than simply comparing models in isolation. For instance, the guidance could suggest evaluating how a system’s capabilities might change when given access to different interfaces, actuators, or information sources.

Similarly, Weidinger et al. (2023) argue that while quantitative benchmarks are important, they are insufficient for comprehensive safety evaluation. They suggest complementing quantitative measures with qualitative assessments, particularly at the human interaction and systemic impact layers. NIST could enhance its guidance by providing more specific recommendations for integrating qualitative evaluation methods alongside quantitative benchmarks.

NIST should acknowledge potential implementation challenges with a comprehensive socio-technical approach. Organizations may struggle to create benchmarks that accurately reflect real-world misuse scenarios, particularly given the rapid evolution of AI capabilities and threat landscapes. Maintaining up-to-date benchmarks in a fast-paced field presents another ongoing challenge. Additionally, organizations may face difficulties in translating quantitative assessments into actionable risk management strategies, especially when dealing with novel or complex risks. NIST could enhance the guidance by providing strategies for navigating these challenges, such as suggesting collaborative industry efforts for benchmark development or offering frameworks for scalable testing approaches.

OpenAI‘s approach of using human participants to evaluate AI capabilities provides both a useful model for more comprehensive evaluation and an example of quantification challenges. While their evaluation attempted to quantify biological risk increase from AI access, they found that, as they put it, “Translating quantitative results into a meaningfully calibrated threshold for risk turns out to be difficult.” This underscores the need for more research on how to set meaningful thresholds and interpret quantitative results in the context of AI safety.

2. Inconsistencies in Risk Management Language

There are instances where the guidance uses varying levels of strength in its recommendations, particularly regarding when to halt or adjust development. For example, Practice 2.2 recommends to “Plan to adjust deployment or development strategies if misuse risks rise to unacceptable levels,” while Practice 3.2 uses stronger language, suggesting to “Adjust or halt further development until the risk of model theft is adequately managed.” This variation in language could lead to confusion and potentially weaker implementation of risk management strategies.

Furthermore, while the guidance emphasizes the importance of managing risks before deployment, it does not provide clear criteria for what constitutes “adequately managed” risk, particularly in the context of development rather than deployment. More consistent and specific language around these critical decision points would strengthen the guidance’s effectiveness in promoting responsible AI development.

Specific recommendations for strengthening language on halting development for unmanageable risks

To address the inconsistencies noted above, we suggest the following changes:

1. Standardize the language across the document to consistently use strong phrasing such as “Adjust or halt further development” when discussing responses to unacceptable levels of risk. 

The current guidance uses varying levels of strength in its recommendations regarding development adjustments. For instance, Recommendation 4 of Practice 2.2 uses the phrase “Plan to adjust deployment or development strategies,” while Recommendation 3 of Practice 3.2 more strongly suggests to “Adjust or halt further development.” Consistent language would emphasize the critical nature of these decisions and reduce potential confusion or weak implementation of risk management strategies. This could be accomplished by changing the language of Practice 2.2, Recommendation 4 to “Plan to adjust or halt further development or deployment if misuse risks rise to unacceptable levels before adequate security and safeguards are available to manage risk.”

The need for stronger language regarding halting development is reflected both in NIST’s other work and in commitments that many frontier AI developers have publicly agreed to. For instance, the NIST AI Risk Management Framework, section 1.2.3 (Risk Prioritization), suggests: “In some cases where an AI system presents the highest risk – where negative impacts are imminent, severe harms are actually occurring, or catastrophic risks are present – development and deployment should cease in a safe manner until risks can be sufficiently mitigated.” Further, the AI Seoul Summit frontier AI safety commitments explicitly state that organizations should “set out explicit processes they intend to follow if their model or system poses risks that meet or exceed the pre-defined thresholds.” Importantly, these commitments go on to specify that “In the extreme, organisations commit not to develop or deploy a model or system at all, if mitigations cannot be applied to keep risks below the thresholds.” 

2. Add to the list of transparency documentation for Practice 2.2 the following: “A decision-making framework for determining when risks have become truly unmanageable, considering factors like the severity of potential harm, the likelihood of the risk materializing, and the feasibility of mitigation strategies.”

While the current guidance emphasizes the importance of managing risks before deployment (e.g., in Practice 5.3), it does not provide clear criteria for what constitutes “adequately managed” risk, particularly in the context of development rather than deployment. A decision-making framework would provide clearer guidance on when to take the serious step of halting development. This addition would help prevent situations where development continues despite unacceptable risks due to a lack of clear stopping criteria. This recommendation aligns with the approach suggested by Alaga and Schuett (2023) in their paper on coordinated pausing, where they emphasize the need for clear thresholds and decision criteria to determine when AI development should be halted due to unacceptable risks. 

3. Gaps in Model Development Safeguards

The guidance’s treatment of safeguards, particularly those related to model development, lacks sufficient detail to be practically useful. This is most evident in Appendix B, which lists example safeguards. While this appendix is a valuable addition, the safeguards related to model training (“Improve the model’s training”) are notably lacking in detail compared to the safeguards around model security and detecting misuse.

While the guidance covers many aspects of risk management comprehensively, especially model security, it does not provide enough specific recommendations for technical approaches to building safer models during the development phase. This gap could limit the practical utility of the guidance for AI developers seeking to implement safety measures from the earliest stages of model creation.

Specific recommendations for additional safeguards for model development

For some safeguards, we recommend that the misuse risk guidance explicitly reference relevant sections of NIST 600-1, the Generative Artificial Intelligence Profile. Specifically, the GAI profile offers more comprehensive guidance on data-related and monitoring safeguards. For instance, the profile emphasizes documenting training data curation policies (MP-4.1-004) and establishing policies for data collection, retention, and quality (MP-4.1-005), which are crucial for managing misuse risk from the earliest stages of development. Additionally, the profile suggests implementing real-time monitoring processes for analyzing generated content performance and trustworthiness characteristics (MG-3.2-006), which could significantly enhance ongoing risk management during development. These references to the GAI Profile on model development safeguards could take the form of an additional item in Appendix B, or be incorporated into the relevant sections earlier in the guidance.

Beyond pointing to the model development safeguards included in the GAI Profile, we also recommend expanding Appendix B to include further safeguards for the model development phase. Both the GAI Profile and the current misuse risk guidance lack specific recommendations for two key model development safeguards: iterative safety testing throughout development and staged development/release processes. Below are two proposed additions to Appendix B:

SafeguardPossible Implementation Methods
Implement iterative safety testing throughout development.* Develop and continuously update a comprehensive suite of safety tests covering identified risk areas.

* Establish quantitative safety benchmarks and ensure the model meets predefined thresholds before progressing to next development stages.

* Conduct regular adversarial testing, updating the test suite based on discovered vulnerabilities or emerging threats.
Consider a staged development and release process.* Define clear safety criteria that must be met before advancing to each subsequent stage of model development or deployment.

* Implement a phased release strategy, incrementally increasing model capabilities or access only after thorough safety evaluations at each stage.

* If possible, maintain the capability to rapidly revert to previous versions or restrict access if safety issues are identified post-release.

The proposed safeguard “Implement iterative safety testing throughout development” addresses the current guidance’s limited detail on model training and development safeguards. This approach aligns with Barrett, et al.’s AI Risk-Management Standards Profile for General-Purpose AI Systems and Foundation Models (the “GPAIS Profile”)’s emphasis on proactive and ongoing risk assessment. Specifically, the Profile recommends identifying “GPAIS impacts…and risks (including potential uses, misuses, and abuses), starting from an early AI lifecycle stage and repeatedly through new lifecycle phases or as new information becomes available” (Barrett et al., 2023, p. 19). The GPAIS Profile further suggests that for larger models, developers should “analyze, customize, reanalyze, customize differently, etc., then deploy and monitor” (Barrett et al., 2023, p. 19), where “analyze” encompasses probing, stress testing, and red teaming. This iterative safety testing would integrate safety considerations throughout development, aligning with the guidance’s emphasis on proactive risk management and anticipating potential misuse risk.

Similarly, the proposed safeguard “Establish a staged development and release process” addresses a significant gap in the current guidance. While Practice 5.1 discusses pre-deployment risk assessment, it lacks a structured approach to incrementally increasing model capabilities or access. Solaiman et al. (2023) propose a “gradient of release” framework for generative AI, a phased approach to model deployment that allows for iterative risk assessment and mitigation. This aligns with the guidance’s emphasis on ongoing risk management and could enhance the ‘margin of safety’ concept in Practice 5.3. Implementing such a staged process would introduce multiple risk assessment checkpoints throughout development and deployment, potentially improving safety outcomes.

Conclusion

NIST’s guidance on “Managing Misuse Risk for Dual-Use Foundation Models” represents a significant step forward in establishing robust practices for mitigating catastrophic risks associated with advanced AI systems. The document’s emphasis on comprehensive risk assessment, transparent decision-making, and proactive safeguards demonstrates a commendable commitment to responsible AI development. However, to more robustly contribute to risk mitigation, the guidance must evolve to address key challenges, including a stronger approach to measuring misuse risk, consistent language on halting development, and more detailed model development safeguards.

As the science of AI risk assessment advances, this guidance should be recursively updated to address emerging risks and incorporate new best practices. While voluntary guidance is crucial, it is important to recognize that it cannot replace the need for robust policy and regulation. A combination of industry best practices, government oversight, and international cooperation will be necessary to ensure the responsible development of high-risk AI systems.

We appreciate the opportunity to provide input on this important document. FAS stands ready to continue assisting NIST in refining and implementing this guidance, as well as in developing further resources for responsible AI development. We believe that close collaboration between government agencies, industry leaders, and civil society organizations is key to realizing the benefits of AI while effectively mitigating its most serious risks.