Federation of American Scientists, Future of Life Institute Present Converging Risks Report, AI Impact Awards at Gala
FAS AI Impact Awards Presented to Advocates, Civil Society Entrepreneurs, Industry Experts, and Policymakers
Washington, D.C. – May 20, 2026 – Tonight at the International Spy Museum in downtown Washington, D.C., the Federation of American Scientists (FAS), a non-partisan, nonprofit science and technology policy organization, in partnership with the Future of Life Institute, the world’s oldest and largest AI think tank, conclude an 18 month project to investigate the implications of artificial intelligence on global risk.
FAS and FLI partnered to build a series of convenings and reports across the intersections of artificial intelligence (AI) with biosecurity, cybersecurity, nuclear command and control, military integration, and frontier AI governance. This project brought together leaders across these areas and created a space that was rigorous, transpartisan, and solutions-oriented to approach how we should think about how AI is rapidly changing global risks. Adapting to this reality will demand that policy entrepreneurs take action; scientific and technological expertise is a must for successful policymaking.
“FAS is dedicated to developing evidence-based policies to address national challenges, and the technical advances of artificial intelligence are already outpacing our expectations. We recognized an urgency in convening expertise across disciplines to better understand how we can reduce risk and increase societal rewards,” says FAS CEO Daniel Correa.
“AI is no longer a single-domain challenge. It is a force multiplier reshaping the risk landscape across nuclear, biological, cyber, and military systems simultaneously, and it is doing so faster than our institutions can adapt,” says Future of Life President and CEO, Anthony Aguirre. “That is precisely why this partnership with FAS has mattered so much. The report gives decision-makers a clear-eyed map of how these threats are compounding, and what we can do about it. The window to put sensible guardrails in place is open, but it is closing quickly. The leaders we are honoring show that rigorous, bipartisan action on the most consequential technology of our era is both necessary and possible.”
The AI x Global Risk Gala, moderated by Ashley Gold, Senior Technology Policy Reporter at Axios, will highlight a capstone report and present awards in recognition of AI policy leaders. Bloomberg‘s cyber and emerging tech reporter, Katrina Manson will host a discussion panel about the report. The panel will include FAS board member and former Acting Under Secretary for Science and Technology at the Department of Homeland Security, Dr. Daniel Gerstein.
‘Converging Risks’ Report
The primary report, Converging Risks: AI and the Future of Global Security, is the synthesis of sector-specific investigations into nuclear policy, cyber policy, biotechnology, defense, and critical infrastructure. Increasingly, AI cuts across all of them simultaneously.
The FAS team evaluated risks through the “Threat, Vulnerability, and Consequence” or “TVC” framework, a powerful acknowledgement of how stakes rise alongside introduction and interaction with multiple factors.
The report illustrates how AI is complicating the risk calculus, adding complexity to systems and events, changing the speed at which we need to respond, and often increasing the scale of the risk.
“Despite the very real risks artificial intelligence presents, our report is not fatalistic,” says Dr. Jedidah Isler, FAS’s Chief Science Officer. “We know that productive conversations and proactive policy cannot happen if we operate from a state of hype, fear or ignorance. As scientists, we must use all of the tools at our disposal to reckon with what is very likely to be one of the most consequential technologies of this era. It’s innately a sociotechnical problem: it’s not just the technology, but what we think about it and how we collaborate in the face of tremendous change. We must begin by building government capacity, coordination, and translation infrastructure now.”
FAS AI Impact Award Winners
FAS will also present four awards at the Gala: the AI Advocacy Award, AI Impact Award for Civil Society, AI Impact Award for Industry, and the AI Policy Award.
Joseph Gordon Levitt, AI Impact Award for Advocacy
Joseph Gordon Levitt, the UN’s first global advocate for “human-centric digital governance”, will receive the AI Impact Award for Advocacy for his work raising awareness of AI risks to non-technical audiences using his skills as a writer, director, communicator, and educator.
Mr. Levitt’s recent advocacy includes speaking out about Meta’s AI chatbots endangering children (September, 2025) and supporting an AI and child safety bill in Utah (January 2026).
Mr. Levitt and his organization, HITRECORD, explore the intersection of technology and society through both his creative work and advocacy around digital governance.
Sneha Revanur, AI Impact Award for Civil Society
Sneha Revanur will receive the AI Impact Award for Civil Society for her work founding a civil society organization, Encode, that works to influence federal AI policy that unifies pro-AI, pro-human perspectives.
Ms. Revanur began her activism work at age 15 when she learned that California was considering replacing its cash bail system with a risk-based algorithm and that the algorithm had serious racial bias baked into it. She organized a statewide coalition of high school students, fought the ballot measure, and helped defeat it by 13 percentage points.
Today, Ms. Revanur continues her activism work in AI regulation to ensure that trust and fairness are built into the often invisible systems that can have enormous impact on daily life.
Chris Meserole, AI Impact Award for Industry
Chris Meserole, Executive Director of the Frontier Model Forum, will receive the AI Impact Award for Industry for his work examining the security risks associated with artificial intelligence. He’s working to determine best practices to ensure strong interconnection between industry, research, and government.
Prior to the Frontier Model Forum, Chris served as Director of the AI and Emerging Technology Initiative at the Brookings Institution and a fellow in its Foreign Policy program.
Today, Mr. Meserole works extensively on safeguarding large-scale AI systems against the risks of accidental or malicious use.
Senator Blackburn (R-TN) and Senator Blumenthal (D-CT),
AI Impact Awards for Policy Leadership
How we govern AI’s impact on society is of utmost importance. Decisions made today will drive outcomes for years, and potentially decades, to come. FAS is presenting two AI Impact Award for Policy Leadership to honor work that anticipates and addresses future risks presented by artificial intelligence.
Senator Marsha Blackburn (R-TN), Senator Richard Blumenthal (D-CT) will be presented with the AI Impact Awards for Policy Leadership for their respective leadership navigating fast-moving technology and its implications.
Senator Blackburn of Tennessee has been a bold and consequential leader on AI policy. Last summer she successfully fought to remove a provision from federal legislation that would have blocked states from protecting their own citizens from AI harms for a decade. In December, she put forward a comprehensive national framework for AI governance that requires companies to conduct real risk assessments and establishes concrete rules on training data and deepfakes. Senator Blackburn also leads the Transparency and Responsibility for Artificial Intelligence Networks (TRAIN) Act, a bipartisan bill aimed at helping musicians, artists, writers, and other copyright holders determine whether their work has been used to train generative artificial intelligence models.
Senator Blackburn’s forward thinking on AI has driven leadership on quantum computing development. She is advancing bipartisan legislation like the National Quantum Initiative Reauthorization Act to provide necessary infrastructure for future AI capabilities.
Senator Blackburn serves on the Senate Committee on Commerce, Science, and Transportation, of which she is Chairman of the Consumer Protection, Technology, and Data Privacy Subcommittee, as well as on the Senate Judiciary Committee, of which she is Chairman of the Privacy, Technology, and the Law Subcommittee.
Senator Blumenthal of Connecticut has been one of the earliest and most consistent voices on Capitol Hill regarding technology and its implications for society. He has been using his voice to demand that Congress show up for this moment. He brought Sam Altman to Congress for the first time back in 2023 to help educate lawmakers and urge them to act. He has since pushed for his AI Accountability and Personal Data Protection Act, bipartisan legislation to hold AI companies accountable for how they use copyrighted material to train their models. He also introduced the bipartisan AI Risk Evaluation Act which would create a dedicated AI risk-evaluation program within the Department of Energy focused specifically on national security, civil liberties, and labor protections. Senator Blumenthal co-leads the bipartisan Guidelines for User Age-verification and Responsible Dialogue (GUARD) Act to protect children against harms from AI bots, and this legislation is advancing in the Senate.
Senator Blumenthal serves on Senate Committees on Armed Services, Judiciary, and Homeland Security and Government Affairs.
Two senators. Different parties. Different states. Different politics. Same conclusion: Congress cannot afford to sit this one out.
Policymakers in Attendance
Additional policymakers invited to the Gala have demonstrated leadership in advancing evidence-based artificial intelligence legislation, including:
Congressman Jim Himes (D-CT) serves as Ranking Member on the House Permanent Select Committee on Intelligence, has deep experience and unique insights into how U.S. intelligence agencies and the national security apparatus integrate artificial intelligence models, including how models could be used for hacking and cyberdefense. He will be a panelist at the gala.
Senator Elissa Slotkin (D-MI) serves on the Senate Armed Services Committee as Ranking Member of the Subcommittee on Emerging Threats and Capabilities, and introduced the AI Guardrails Act to address AI use around lethal force, spying on Americans and nuclear weapons. The bill seeks to codify two existing Defense Department guidelines into law: that AI cannot autonomously decide to kill a target and that the technology cannot be used to conduct mass surveillance on Americans. It would also ban the use of artificial intelligence for launching or detonating a nuclear weapon.
Congressman Don Bacon (R-NE) serves on the House Armed Services Committee as Chairman of the Subcommittee on Cyber, Information Technology and Innovation. Congressman Bacon has championed and overseen the passage of numerous provisions pertaining to AI and risk in the FY26 NDAA. Bacon joined the Congressional probe into Elon Musk’s Grok AI over allegations of antisemitism and ‘deeply alarming messages’ (July 2025).
Congressman Bill Foster (D-IL), Congress’s only member holding a PhD in physics, introduced the bipartisan Responsible and Ethical AI Labeling (REAL) Act, which would mandate a “clear, conspicuous, and prominently displayed” disclaimer notifying readers or viewers that content was created with or manipulated by AI.
Congressman Rich McCormick (R-GA) serves on the House Armed Services Committee and as the chairman of the Subcommittee on Oversight and Investigations. He also serves on the Armed Services Committee, Oversight and Government Reform Committee, and is a former member of the bipartisan Task Force on Artificial Intelligence.
###
About the Federation of American Scientists (FAS)
The Federation of American Scientists (FAS) works to advance progress on a broad suite of contemporary issues where science, technology, and innovation policy can deliver transformative impact, and seeks to ensure that scientific and technical expertise have a seat at the policymaking table. Established in 1945 by scientists in response to the atomic bomb, FAS continues to bring scientific rigor and analysis to address national challenges. More information about FAS’s work at fas.org.
About the Future of Life Institute
The Future of Life Institute (FLI) is the world’s oldest and largest AI think tank, with a team of 35+ full-time staff operating across the US and Europe. FLI has been working to steer the development of transformative technologies towards benefiting life and away from extreme large-scale risks since its founding in 2014. Find out more at futureoflife.org.
RESOURCES
AI x Global Risk Nexus Project
Converging Risks: AI and the Future of Global Security (and briefing booklet)
- AI and Nuclear Command, Control, and Communications
- AI and Military Integration
- AI and Biological Risks
- AGI and Global Risk
- AI, Cyber, and Global Risk
- AI and Nuclear Risks
FAS AI Impact Award Winners
More on AI Advocacy Award winner Joseph Gordon Levitt
More on AI Impact Award for Civil Society winner Sneha Revanur and Encode
More on AI Impact Award winner Chris Meserole and Frontier Model Forum
More on AI Impact Awards for Policy winners Senator Marsha Blackburn (R-TN) and Senator Richard Blumenthal (D-CT)
Public Comment on the U.S. Artificial Intelligence Safety Institute’s Draft Document: NIST AI 800-1, Managing Misuse Risk for Dual-Use Foundation Models
Public comments serve the executive branch by informing more effective, efficient program design and regulation. As part of our commitment to evidence-based, science-backed policy, FAS staff leverage public comment opportunities to embed science, technology, and innovation into policy decision-making.
The Federation of American Scientists (FAS) is a non-partisan organization dedicated to using science and technology to benefit humanity through equitable and impactful policy. With a strong track record in AI governance, FAS has actively contributed to the development of AI standards and frameworks, including providing feedback on NIST AI 600-1, the Generative AI Profile. Our work spans advocating for federal AI testbeds, recommending policy measures for frontier AI developers, and evaluating industry adoption of the NIST AI Risk Management Framework. We are members of the U.S. AI Safety Institute Research Consortium, and we responded to NIST’s request for information earlier this year concerning its responsibilities under sections 4.1, 4.5, and 11 of the AI Executive Order.
We commend NIST’s U.S. Artificial Intelligence Safety Institute for developing the draft guidance on “Managing Misuse Risk for Dual-Use Foundation Models.” This document represents a significant step toward establishing robust practices for mitigating catastrophic risks associated with advanced AI systems. The guidance’s emphasis on comprehensive risk assessment, transparent decision-making, and proactive safeguards aligns with FAS’s vision for responsible AI development.
In our response, we highlight several strengths of the guidance, including its focus on anticipatory risk assessment and the importance of clear documentation. We also identify areas for improvement, such as the need for harmonized language and more detailed guidance on model development safeguards. Our key suggestions include recommending a more holistic socio-technical approach to risk evaluation, strengthening language around halting development for unmanageable risks, and expanding the range of considered safeguards. We believe these adjustments will further strengthen NIST’s crucial role in shaping responsible AI development practices.
Background and Context
The rapid advancement of AI foundation models has spurred novel industry-led risk mitigation strategies. Leading AI companies have voluntarily adopted frameworks like Responsible Scaling Policies and Preparedness Frameworks, outlining risk thresholds and mitigation strategies for increasingly capable AI systems. (Our response to NIST’s February RFI was largely an exploration of these policies, their benefits and drawbacks, and how they could be strengthened.)
Managing misuse risks in foundation models is of paramount importance given their broad applicability and potential for dual use. As these models become more powerful, they may inadvertently enable malicious actors to cause significant harm, including facilitating the development of weapons, enabling sophisticated cyber attacks, or generating harmful content. The challenge lies not only in identifying current risks but also in anticipating future threats that may emerge as AI capabilities expand.
NIST’s new guidance on “Managing Misuse Risk for Dual-Use Foundation Models” builds upon these industry initiatives, providing a more standardized and comprehensive approach to risk management. By focusing on objectives such as anticipating potential misuse, establishing clear risk thresholds, and implementing robust evaluation procedures, the guidance creates a framework that can be applied across the AI development ecosystem. This approach is crucial for ensuring that as AI technology advances, appropriate safeguards are in place to protect against potential misuse while still fostering innovation.
Strengths of the guidance
1. Comprehensive Documentation and Transparency
The guidance’s emphasis on thorough documentation and transparency represents a significant advancement in AI risk management. For every practice under every objective, the guidance indicates appropriate documentation; this approach is more thorough in advancing transparency than any comparable guidance to date. The creation of a paper trail for decision-making and risk evaluation is crucial for both internal governance and potential external audits.
The push for transparency extends to collaboration with external stakeholders. For instance, practice 6.4 recommends providing “safe harbors for third-party safety research,” including publishing “a clear vulnerability disclosure policy for model safety issues.” This openness to external scrutiny and feedback is essential for building trust and fostering collaborative problem-solving in AI safety. (FAS has published a legislative proposal calling for enshrining “safe harbor” protections for AI researchers into law.)
2. Lifecycle Approach to Risk Management
The guidance excels in its holistic approach to risk management, covering the entire lifecycle of foundation models from pre-development assessment through to post-deployment monitoring. This comprehensive approach is evident in the structure of the document itself, which follows a logical progression from anticipating risks (Objective 1) through to responding to misuse after deployment (Objective 6).
The guidance demonstrates a proactive stance by recommending risk assessment before model development. Practice 1.3 suggests to “Estimate the model’s capabilities of concern before it is developed…”, which helps anticipate and mitigate potential harms before they materialize. The framework for red team evaluations (Practice 4.2) is particularly robust, recommending independent external experts and suggesting ways to compensate for gaps between red teams and real threat actors. The guidance also emphasizes the importance of ongoing risk assessment. Practice 3.2 recommends to “Periodically revisit estimates of misuse risk stemming from model theft…” This acknowledgment of the dynamic nature of AI risks encourages continuous vigilance.
3. Strong Stance on Model Security and Risk Tolerance
The guidance takes a firm stance on model security and risk tolerance, particularly in Objective 3. It unequivocally states that models relying on confidentiality for misuse risk management should only be developed when theft risk is sufficiently mitigated. This emphasizes the critical importance of security in AI development, including considerations for insider threats (Practice 3.1).
The guidance also demonstrates a realistic approach to the challenges posed by different deployment strategies. In Practice 5.1, it notes, “For example, allowing fine-tuning via API can significantly limit options to prevent jailbreaking and sharing the model’s weights can significantly limit options to monitor for misuse (Practice 6.1) and respond to instances of misuse (Practice 6.2).” This candid discussion of the limitations of safety interventions for open weight foundation models is crucial for fostering realistic risk assessments.
Additionally, the guidance promotes a conservative approach to risk management. Practice 5.3 recommends to “Consider leaving a margin of safety between the estimated level of risk at the point of deployment and the organization’s risk tolerance.” It further suggests considering “a larger margin of safety to manage risks that are more severe or less certain.” This approach provides an extra layer of protection against unforeseen risks or rapid capability advancements, which is crucial given the uncertainties inherent in AI development.
These elements collectively demonstrate NIST’s commitment to promoting realistic and robust risk management practices that prioritize safety and security in AI development and deployment. However, while the NIST guidance demonstrates several important strengths, there are areas where it could be further improved to enhance its effectiveness in managing misuse risks for dual-use foundation models.
Areas for improvement
1. Need for a More Comprehensive Socio-technical Approach to Measuring Misuse Risk
Objective 4 of the guidance demonstrates a commendable effort to incorporate elements of a socio-technical approach in measuring misuse risk. The guidance recognizes the importance of considering both technical and social factors, emphasizes the use of red teams to assess potential misuse scenarios, and acknowledges the need to consider different levels of access and various threat actors. Furthermore, it highlights the importance of avoiding harm during the measurement process, which is crucial in a socio-technical framework.
However, the guidance falls short in fully embracing a comprehensive socio-technical perspective. While it touches on the importance of external experts, it does not sufficiently emphasize the value of diverse perspectives, particularly from individuals with lived experiences relevant to specific risk scenarios. The guidance also lacks a structured approach to exploring the full range of potential misuse scenarios across different contexts and risk areas. Finally, the guidance does not mention measuring absolute versus marginal risks (ie., how much total misuse risk a model poses in a specific context versus how much marginal risk it poses compared to existing tools). These gaps limit the effectiveness of the proposed risk measurement approach in capturing the full complexity of AI system interactions with human users and broader societal contexts.
Specific recommendations for improving socio-technical approach
The NIST guidance in Practice 1.3 suggests estimating model capabilities by comparison to existing models, but provides little direction on how to conduct these comparisons effectively. To improve this, NIST could incorporate the concept of “available affordances.” This concept emphasizes that an AI system’s risk profile depends not just on its absolute capabilities, but also on the environmental resources and opportunities for affecting the world that are available to it.
Additionally, Kapoor et al. (2024) emphasize the importance of assessing the marginal risk of open foundation models compared to existing technologies or closed models. This approach aligns with a comprehensive socio-technical perspective by considering not just the absolute capabilities of AI systems, but also how they interact with existing technological and social contexts. For instance, when evaluating cybersecurity risks, they suggest considering both the potential for open models to automate vulnerability detection and the existing landscape of cybersecurity tools and practices. This marginal risk framework helps to contextualize the impact of open foundation models within broader socio-technical systems, providing a more nuanced understanding of their potential benefits and risks.
NIST could recommend that organizations assess both the absolute capabilities of their AI systems and the affordances available to them in potential deployment contexts. This approach would provide a more comprehensive view of potential risks than simply comparing models in isolation. For instance, the guidance could suggest evaluating how a system’s capabilities might change when given access to different interfaces, actuators, or information sources.
Similarly, Weidinger et al. (2023) argue that while quantitative benchmarks are important, they are insufficient for comprehensive safety evaluation. They suggest complementing quantitative measures with qualitative assessments, particularly at the human interaction and systemic impact layers. NIST could enhance its guidance by providing more specific recommendations for integrating qualitative evaluation methods alongside quantitative benchmarks.
NIST should acknowledge potential implementation challenges with a comprehensive socio-technical approach. Organizations may struggle to create benchmarks that accurately reflect real-world misuse scenarios, particularly given the rapid evolution of AI capabilities and threat landscapes. Maintaining up-to-date benchmarks in a fast-paced field presents another ongoing challenge. Additionally, organizations may face difficulties in translating quantitative assessments into actionable risk management strategies, especially when dealing with novel or complex risks. NIST could enhance the guidance by providing strategies for navigating these challenges, such as suggesting collaborative industry efforts for benchmark development or offering frameworks for scalable testing approaches.
OpenAI‘s approach of using human participants to evaluate AI capabilities provides both a useful model for more comprehensive evaluation and an example of quantification challenges. While their evaluation attempted to quantify biological risk increase from AI access, they found that, as they put it, “Translating quantitative results into a meaningfully calibrated threshold for risk turns out to be difficult.” This underscores the need for more research on how to set meaningful thresholds and interpret quantitative results in the context of AI safety.
2. Inconsistencies in Risk Management Language
There are instances where the guidance uses varying levels of strength in its recommendations, particularly regarding when to halt or adjust development. For example, Practice 2.2 recommends to “Plan to adjust deployment or development strategies if misuse risks rise to unacceptable levels,” while Practice 3.2 uses stronger language, suggesting to “Adjust or halt further development until the risk of model theft is adequately managed.” This variation in language could lead to confusion and potentially weaker implementation of risk management strategies.
Furthermore, while the guidance emphasizes the importance of managing risks before deployment, it does not provide clear criteria for what constitutes “adequately managed” risk, particularly in the context of development rather than deployment. More consistent and specific language around these critical decision points would strengthen the guidance’s effectiveness in promoting responsible AI development.
Specific recommendations for strengthening language on halting development for unmanageable risks
To address the inconsistencies noted above, we suggest the following changes:
1. Standardize the language across the document to consistently use strong phrasing such as “Adjust or halt further development” when discussing responses to unacceptable levels of risk.
The current guidance uses varying levels of strength in its recommendations regarding development adjustments. For instance, Recommendation 4 of Practice 2.2 uses the phrase “Plan to adjust deployment or development strategies,” while Recommendation 3 of Practice 3.2 more strongly suggests to “Adjust or halt further development.” Consistent language would emphasize the critical nature of these decisions and reduce potential confusion or weak implementation of risk management strategies. This could be accomplished by changing the language of Practice 2.2, Recommendation 4 to “Plan to adjust or halt further development or deployment if misuse risks rise to unacceptable levels before adequate security and safeguards are available to manage risk.”
The need for stronger language regarding halting development is reflected both in NIST’s other work and in commitments that many frontier AI developers have publicly agreed to. For instance, the NIST AI Risk Management Framework, section 1.2.3 (Risk Prioritization), suggests: “In some cases where an AI system presents the highest risk – where negative impacts are imminent, severe harms are actually occurring, or catastrophic risks are present – development and deployment should cease in a safe manner until risks can be sufficiently mitigated.” Further, the AI Seoul Summit frontier AI safety commitments explicitly state that organizations should “set out explicit processes they intend to follow if their model or system poses risks that meet or exceed the pre-defined thresholds.” Importantly, these commitments go on to specify that “In the extreme, organisations commit not to develop or deploy a model or system at all, if mitigations cannot be applied to keep risks below the thresholds.”
2. Add to the list of transparency documentation for Practice 2.2 the following: “A decision-making framework for determining when risks have become truly unmanageable, considering factors like the severity of potential harm, the likelihood of the risk materializing, and the feasibility of mitigation strategies.”
While the current guidance emphasizes the importance of managing risks before deployment (e.g., in Practice 5.3), it does not provide clear criteria for what constitutes “adequately managed” risk, particularly in the context of development rather than deployment. A decision-making framework would provide clearer guidance on when to take the serious step of halting development. This addition would help prevent situations where development continues despite unacceptable risks due to a lack of clear stopping criteria. This recommendation aligns with the approach suggested by Alaga and Schuett (2023) in their paper on coordinated pausing, where they emphasize the need for clear thresholds and decision criteria to determine when AI development should be halted due to unacceptable risks.
3. Gaps in Model Development Safeguards
The guidance’s treatment of safeguards, particularly those related to model development, lacks sufficient detail to be practically useful. This is most evident in Appendix B, which lists example safeguards. While this appendix is a valuable addition, the safeguards related to model training (“Improve the model’s training”) are notably lacking in detail compared to the safeguards around model security and detecting misuse.
While the guidance covers many aspects of risk management comprehensively, especially model security, it does not provide enough specific recommendations for technical approaches to building safer models during the development phase. This gap could limit the practical utility of the guidance for AI developers seeking to implement safety measures from the earliest stages of model creation.
Specific recommendations for additional safeguards for model development
For some safeguards, we recommend that the misuse risk guidance explicitly reference relevant sections of NIST 600-1, the Generative Artificial Intelligence Profile. Specifically, the GAI profile offers more comprehensive guidance on data-related and monitoring safeguards. For instance, the profile emphasizes documenting training data curation policies (MP-4.1-004) and establishing policies for data collection, retention, and quality (MP-4.1-005), which are crucial for managing misuse risk from the earliest stages of development. Additionally, the profile suggests implementing real-time monitoring processes for analyzing generated content performance and trustworthiness characteristics (MG-3.2-006), which could significantly enhance ongoing risk management during development. These references to the GAI Profile on model development safeguards could take the form of an additional item in Appendix B, or be incorporated into the relevant sections earlier in the guidance.
Beyond pointing to the model development safeguards included in the GAI Profile, we also recommend expanding Appendix B to include further safeguards for the model development phase. Both the GAI Profile and the current misuse risk guidance lack specific recommendations for two key model development safeguards: iterative safety testing throughout development and staged development/release processes. Below are two proposed additions to Appendix B:
The proposed safeguard “Implement iterative safety testing throughout development” addresses the current guidance’s limited detail on model training and development safeguards. This approach aligns with Barrett, et al.’s AI Risk-Management Standards Profile for General-Purpose AI Systems and Foundation Models (the “GPAIS Profile”)’s emphasis on proactive and ongoing risk assessment. Specifically, the Profile recommends identifying “GPAIS impacts…and risks (including potential uses, misuses, and abuses), starting from an early AI lifecycle stage and repeatedly through new lifecycle phases or as new information becomes available” (Barrett et al., 2023, p. 19). The GPAIS Profile further suggests that for larger models, developers should “analyze, customize, reanalyze, customize differently, etc., then deploy and monitor” (Barrett et al., 2023, p. 19), where “analyze” encompasses probing, stress testing, and red teaming. This iterative safety testing would integrate safety considerations throughout development, aligning with the guidance’s emphasis on proactive risk management and anticipating potential misuse risk.
Similarly, the proposed safeguard “Establish a staged development and release process” addresses a significant gap in the current guidance. While Practice 5.1 discusses pre-deployment risk assessment, it lacks a structured approach to incrementally increasing model capabilities or access. Solaiman et al. (2023) propose a “gradient of release” framework for generative AI, a phased approach to model deployment that allows for iterative risk assessment and mitigation. This aligns with the guidance’s emphasis on ongoing risk management and could enhance the ‘margin of safety’ concept in Practice 5.3. Implementing such a staged process would introduce multiple risk assessment checkpoints throughout development and deployment, potentially improving safety outcomes.
Conclusion
NIST’s guidance on “Managing Misuse Risk for Dual-Use Foundation Models” represents a significant step forward in establishing robust practices for mitigating catastrophic risks associated with advanced AI systems. The document’s emphasis on comprehensive risk assessment, transparent decision-making, and proactive safeguards demonstrates a commendable commitment to responsible AI development. However, to more robustly contribute to risk mitigation, the guidance must evolve to address key challenges, including a stronger approach to measuring misuse risk, consistent language on halting development, and more detailed model development safeguards.
As the science of AI risk assessment advances, this guidance should be recursively updated to address emerging risks and incorporate new best practices. While voluntary guidance is crucial, it is important to recognize that it cannot replace the need for robust policy and regulation. A combination of industry best practices, government oversight, and international cooperation will be necessary to ensure the responsible development of high-risk AI systems.
We appreciate the opportunity to provide input on this important document. FAS stands ready to continue assisting NIST in refining and implementing this guidance, as well as in developing further resources for responsible AI development. We believe that close collaboration between government agencies, industry leaders, and civil society organizations is key to realizing the benefits of AI while effectively mitigating its most serious risks.
Six Policy Ideas for the National AI Strategy
The White House Office of Science and Technology Policy (OSTP) has sought public input for the Biden administration’s National AI Strategy, acknowledging the potential benefits and risks of advanced AI. The Federation of American Scientists (FAS) was happy to recommend specific actions for federal agencies to safeguard Americans’ rights and safety. With U.S. companies creating powerful frontier AI models, the federal government must guide this technology’s growth toward public benefit and risk mitigation.
Recommendation 1: OSTP should work with a suitable agency to develop and implement a pre-deployment risk assessment protocol that applies to any frontier AI model.
Before launching a frontier AI system, developers must ensure safety, trustworthiness, and reliability through pre-deployment risk assessment. This protocol aims to thoroughly analyze potential risks and vulnerabilities in AI models before deployment.
We advocate for increased funding towards the National Institute of Standards and Technology (NIST) to enhance its risk measurement capacity and develop robust benchmarks for AI model risk assessment. Building upon NIST’s AI Risk Management Framework (RMF) will standardize metrics for evaluation incorporating various cases such as open-source models, academic research, and fine-tuning of models which differ from larger labs like OpenAI’s GPT-4.
We propose the Federal Trade Commission (FTC), under Section 5 of the FTC Act, implement and enforce this pre-deployment risk assessment strategy. The FTC’s role to prevent unfair or deceptive practices in commerce is aligned with mitigating potential risks from AI systems.
Recommendation 2: Adherence to the appropriate risk management framework should be compulsory for any AI-related project that receives federal funding.
The U.S. government, as a significant funder of AI through contracts and grants, has both a responsibility and opportunity. Responsibility: to ensure that its AI applications meet a high bar for risk management. Opportunity: to enhance a culture of safety in AI development more broadly. Adherence to a risk management framework should be a prerequisite for AI projects seeking federal funds.
Currently, voluntary guidelines such as NIST’s AI RMF exist, but we propose making these compulsory. Agencies should require contractors to document and verify the risk management practices in place for the contract. For agencies that do not have their own guidelines, the NIST AI RMF should be used. And the NSF should require documentation of the grantee’s compliance with the NIST AI RMF in grant applications for AI projects. This approach will ensure all federally funded AI initiatives maintain a high bar for risk management.
Recommendation 3: NSF should increase its funding for “trustworthy AI” R&D.
“Trustworthy AI” refers to AI systems that are reliable, safe, transparent, privacy-enhanced, and unbiased. While NSF is a key non-military funder of AI R&D in the U.S., our rough estimates indicate that its investment in fields promoting trustworthiness has remained relatively static, accounting for only 10-15% of all AI grants. Given its $800 million annual AI-related budget, we recommend that NSF direct a larger share of grants towards research in trustworthy AI.
To enable this shift, NSF could stimulate trustworthy AI research through specific solicitations; launch targeted programs in this area; and incorporate a “trustworthy AI” section in funding applications, prompting researchers to outline the trustworthiness of their projects. This would help evaluate AI project impacts and promote proposals with significant potential in trustworthy AI. Lastly, researchers could be requested or mandated to apply the NIST AI RMF during their studies.
Recommendation 4: FedRAMP should be broadened to cover AI applications contracted for by the federal government.
The Federal Risk and Authorization Management Program (FedRAMP) is a government-wide initiative that standardizes security protocols for cloud services. Given the rising utilization of AI services in federal operations, a similar system of security standards should apply to these services, since they are responsible for managing highly sensitive data related to national security and individual privacy.
Expanding FedRAMP’s mandate to include AI services is a logical next step in ensuring the secure integration of advanced technologies into federal operations. Applying a framework like FedRAMP to AI services would involve establishing robust security standards specific to AI, such as secure data handling, model transparency, and robustness against adversarial attacks. The expanded FedRAMP program would streamline AI integration into federal operations and avoid repetitive security assessments.
Recommendation 5: The Department of Homeland Security should establish an AI incidents database.
The Department of Homeland Security (DHS) needs to create a centralized AI Incidents Database, detailing AI-related breaches, failures and misuse across industries. Its existing authorization under the Homeland Security Act of 2002 makes DHS capable of this role. This database would increase understanding, mitigate risks, and build trust in AI systems’ safety and security.
Voluntary reporting from AI stakeholders should be encouraged while preserving data confidentiality. For effectiveness, anonymized or aggregated data should be shared with AI developers, researchers, and policymakers to better understand AI risks. DHS could use existing databases such as the one maintained by the Partnership on AI and Center for Security and Emerging Technologies, as well as adapt reporting methods from global initiatives like the Financial Services Information Sharing and Analysis Center.
Recommendation 6: OSTP should work with agencies to streamline the process of granting Interested Agency Waivers to AI researchers on J-1 visas.
The ongoing global competition in AI necessitates attracting and retaining a diverse, highly skilled talent pool. The US J-1 Exchange Visitor Program, often used by visiting researchers, requires some participants to return home for two years before applying for permanent residence.
Federal agencies can waive this requirement for certain individuals via an “Interested Government Agency” (IGA) request. Agencies should establish a transparent, predictable process for AI researchers to apply for such waivers. The OSTP should collaborate with agencies to streamline this process. Taking cues from the Department of Defense’s structured application process, including a dedicated webpage, application checklist, and sample sponsor letter, could prove highly beneficial for improving the transition of AI talent to permanent residency in the US.
Review the details of these proposals in our public comment.