The Federal Government Should Pilot a Decision Subject Representative Program for AI Systems
AI systems are regularly used to make decisions that directly impact individuals, from who gets a housing voucher to who gets a job, to bail—contexts with a long history of social disparities, facilitating encoded discrimination. The designs of these consequential AI decision systems are shaped by corporations and increasingly overseen by governments with little input from the public, specifically from users and individuals impacted by these decisions.
Executive branch agencies frequently engage the public in policy decisions via requests for comment and town halls. For decades, the Food and Drug Administration (FDA) has gone beyond traditional agency engagement processes via the Patient Representative Program (PRP), which recruits, trains, and embeds patients into oversight of the pharmaceutical industry, including decisions regarding clinical trial design, endpoints (evaluation metrics), risk/benefit analysis, product labeling, etc. This memo proposes creating a Decision Subject Representative Program inspired by the FDA’s Patient Representative Program.
While pharmaceutical drugs and consequential AI decision systems vary in scope and impact, both technologies need to be safe and effective to be trusted by the public and consumers. Public engagement has long been a tool for building trust and legitimacy in governance decisions while providing a complement to expertise associated with elite institutions. Three decades of FDA experience in systematizing patient engagement offer valuable inspiration for AI governance. Specifically, the General Services Administration (GSA) should pilot embedding Decision Subject Representatives into the procurement process for consequential AI decision systems, the National Institute of Standards and Technology (NIST) should pilot engaging Decision Subject Representatives in efforts to shape standards, and Congress could add a flexible Decision Subject Representatives Program (DSRP) to new regulatory proposals.
Challenge and Opportunity
Technologists have attempted to address concerns regarding bias and discrimination in consequential AI decision systems (AI systems that serve as a basis for a decision or judgment in consequential contexts such as education, employment, essential utilities, financial services, legal services, etc.) by analyzing statistical outcomes or applying fairness metrics. The challenge with this approach is that there are a variety of ways to conceptualize and measure fairness that can not be encoded at the same time. Additionally, fairness metrics often rely on the availability of sensitive category data, which may be restricted by privacy laws and historic human rights laws. Instead, scholars offer that the application context matters and those directly affected should be engaged in the selection and formation of fairness metrics.
More recently, scholars have advocated for a more holistic view of fairness that takes into account the sociotechnical context and the whole process of coming to a certain decision. This approach underscores the need for decision subjects to be included in the entire process of AI system design and deployment, with an emphasis on the assessment of risks and harms broadly, processes for contestability, and transparency measures. As consequential AI decision systems proliferate, it is imperative that the U.S. government pilot systems for engaging decision subjects.
Engaging decision subjects in AI governance faces many challenges. Efforts risk looking like participation washing or engaging decision subjects for theatrical purposes without real power or influence over the final decision. Participatory AI projects can also be inaccessible or exploitative—challenges the FDA’s Patient Representative Program has grappled with.
FDA’s Patient Representative Program
Officials at the FDA woke up to the power of lay expertise in pharmaceutical drug development in the wake of the AIDS epidemic when patients advocated to have their experiences considered in disputes pertaining to the design and methodology of drug trials.
In 1988, the FDA initiated the patient engagement process through the Office of AIDS Coordination. By 1993 the first Patient Representative served on an FDA Advisory Committee. Since then, the FDA has greatly expanded patient engagement with over 200 Patient Representatives, dedicated offices and programs, reporting systems, and regular public guidance aimed at incorporating patient experience data into regulatory decision-making.
The program has been largely implemented at the direction of FDA leadership. In 2012, Congress enacted the Safety and Innovation Act. The act’s language is the first “official” codification of the FDA Patient Representative Program and other patient liaison activities. The law provided greater stability to the program and opened the door for more staff and educational programs.
Patient Representatives include patients, patient advocacy group members, family and/or caregivers, and health care providers. The FDA recruits Patient Representatives through open applications, patient organizations, and staff outreach. Selected participants are vetted and onboarded as Special Government Employees (a category of federal worker for individuals that serve the government temporarily while maintaining employment elsewhere). They consult with FDA review divisions, serve on advisory committees, present at workshops, participate in the Patient Engagement Collaborative which shapes practices in clinical trials, and in other regulatory activities where the patient perspective is needed. Patient Representatives receive training in FDA regulatory processes and can work with FDA staff to prepare to meaningfully participate in advisory committees meetings along with other stakeholders. The FDA covers travel costs and forgone salary for Patient Representatives participating in meetings and training.
FDA employees describe Patient Representatives’ expertise as “a street sense” based on personal experience, describing their views as “a value judgment overlay on top of measurable, empirical clinical trial evidence.” Participants often ask “questions [that] would never be raised” and push for clarity. When Patient Representatives engage with expert stakeholders, learning “works both ways” with clinicians altering their way of thinking based on what Patient Representatives share, and Patient Representatives gaining a better understanding of the science which they can share with their communities.
Over the years, FDA’s Patient Representative program has faced challenges. The drug-specific nature of engagement makes it difficult for patients to engage on cross-cutting issues, conflict of interest rules have made it hard for patients to engage with both drug companies and the FDA, and patients have expressed concerns about not actually knowing the impact they have on decisions. The FDA continues to address these concerns. In 2017, it initiated a Patient Affairs Staff to centralize support for Patient Representatives, and has launched new communication and transparency efforts to help patients understand their influence. While the degree to which patient representative views should be weighted alongside clinical trial data and processes for measuring patient influence will long be contested, the FDA’s program represents a model in which Patient Representatives are remunerated for their time and continually shape FDA processes.
How may the FDA’s Patient Representative Program inspire similar efforts for AI governance?
AI systems are different from pharmaceutical drugs. They are deployed cross-sector and individuals can unknowingly be impacted by an automated decision, whereas drug patients often know they are taking a drug. These differences will impact the way a Decision Subject Representative Program would be deployed, but the types of decisions Patient Representatives consult on are analogous to important decisions in AI governance today. The table below describes the types of governing decisions Patient Representatives have engaged in via the FDA’s drug approval and monitoring process and how those areas correspond with governing consequential AI decision systems. Similar to the FDA, federal agencies could engage Decision Subject Representatives throughout the lifecycle of consequential AI decision systems development including pre-market approval (most comparable to procurement), and the development of testing, evaluation, and transparency standards for deployers and developers (both voluntary and mandated by regulation).
Similar to the way the FDA engages patients in specific drugs (or categories of drugs) as needed, decision subject engagement would work best on applied systems where they can be assessed within a sociotechnical context – particularly in contexts where fairness may be an issue (e.g., education, employment, essential utilities, financial services, legal services). Similar to the FDA, Decision Subject Representatives could consult directly with agency staff on decisions described in Table 1 or serve on advisory committees with other relevant stakeholders.
Recruitment for a Decision Subject Representative Program will vary based on the context and likely include partnering with civil society organizations and community groups that can recommend Decision Subject Representatives. For example, agencies governing (i.e., procuring, drafting standards, issuing testing mandates) hiring software could recruit workers who have experience navigating AI systems to obtain a job or individuals from communities that are historically discriminated against in hiring by partnering with workforce development programs (e.g., American Job Centers, local libraries). Whereas agencies governing (i.e., procuring, drafting standards, issuing mandates) education technology could recruit students, parents and teachers, particularly those from lower-income school districts, to serve as Decision Subject Representatives.
Similar to the FDA, a Decisions Subject Representative Program must include extensive training for Decision Subject Representatives that covers agency (e.g., GSA) processes, why decision subject views are important, and training to combat feeling intimidated by academic and industry expertise. Host agencies should also communicate the importance of decision subject perspectives to other stakeholders. Any pilot should be accompanied by an evaluation of the Decision Subject Representatives’ experience and impact on final decisions.
Plan of Action
Recommendation 1. GSA should consult Decision Subject Representatives when procuring consequential AI decision systems
Recent procurement guidance issued by the Trump administration and the Biden administration directs GSA to identify risks in high-impact AI systems (including what this memo refers to as consequential AI decision systems), conduct pre-award testing, and monitor performance, including quantitative success metrics. GSA should pilot onboarding Decision Subject Representatives as Special Government Employees to consult on these activities as they apply to consequential AI decision systems.
One benefit of engaging decision subjects in the procurement process is that private companies that build systems for the government and for industry may choose to adopt the practices and standards required to meet government requirements for their commercial offerings.
Decision Subject Representatives will have connections to a broader community of individuals impacted by consequential AI decision systems and can serve as a bridge to a wider set of experiences. Additionally, Decision Subject Representatives can consult on new agency programs aimed at engaging decision subjects more broadly over time.
Recommendation 2. NIST should engage Decision Subject Representatives in future Zero Draft development
Risk management, assessment, metrics, and documentation of AI systems will likely be shaped by international standards, especially as the International Standards Organization (ISO) responds to the European Union’s AI Act and similar efforts globally. International standards have traditionally focused on objective guidance and been shaped by industry actors. The need to consider context-specific harms in AI risk assessment has necessitated a recent shift towards sociotechnical standards, creating an imperative for broader stakeholder representation.
NIST, recognizing this shift, recently launched a “Zero Drafts” pilot with the express interest of engaging stakeholders in NIST proposals that are eventually submitted to standards bodies. The two initial topics: AI testing, evaluation, verification, and validation (TEVV) and transparency documentation, are horizontal standards (i.e., not specific to an applied AI system or specific AI use case) and therefore not as well suited to decision subject engagement. But the NIST zero draft program is designed to be responsive to AI workstreams within the international standards bodies, which means they should eventually work on context-specific risk guidance such as AI in hiring, AI in education, AI in financial systems, AI in criminal justice, etc.
As context specific or applied zero draft efforts begin, NIST should pilot engaging Decision Subject Representatives in stakeholder meetings and on edits to draft text. While standards are not regulatory, they can be referenced by regulators worldwide, including U.S. states. In this way, they represent a potential central point of influence over the design and assessment of consequential AI decision systems.
Recommendation 3. Congress should add flexible Decision Subject Representatives Programs to new regulatory proposals
With some exceptions, AI systems do not have to meet transparency or testing requirements to demonstrate they are safe or effective in order to enter or remain on the market. While transparency and testing guidance are currently the domain of NIST (and therefore voluntary), Congress is considering proposals to mandate risk assessment and transparency requirements for consequential AI decision systems (e.g., Algorithmic Accountability Act of 2025). Additionally, Congress has introduced proposals for a comprehensive new digital regulator that would issue regulations, oversee codes of conduct councils or advisory boards, and weigh in on decisions such as those listed in Table 1 (e.g., Digital Consumer Protection Commission Act, Digital Platform Commission Act).
Similar to the Food and Drug Administration Safety and Innovation Act (FDASIA) (2012) Section 1137, Congress could add legislative text to proposals such as those listed above that provides flexibility for agencies to onboard Decision Subject Representatives when they can contribute to decisions related to consequential AI decision systems.
An Example of Legislative Text
Inspired by FDASIA 2012 Section 1137
‘‘(a) IN GENERAL.—The Secretary [or Commission] shall develop and implement strategies to solicit the views of decision subjects during [procurement decisions] or [standards development] or [regulatory discussions] related to consequential AI decision systems including by –
(1) fostering participation of decision subjects who may serve as a special government employee in appropriate agency meetings with consequential AI decision systems developers, deployers, assessors, and investigators; and
(2) exploring means to provide for identification of decision subjects who do not have any, or have minimal, financial interests in companies that provide consequential AI decision systems”
Where DECISION SUBJECT means the person or party to whom the decision applies in a specific context.
Where CONSEQUENTIAL AI DECISION SYSTEMS means “any system, software, or process (including one derived from machine learning, statistics, or other data processing or artificial intelligence techniques and excluding passive computing infrastructure) that uses computation, the result of which serves as a basis for a decision or judgment” [followed by a list of critical contexts such as education, employment, essential utilities, financial services, legal services, etc)]
(definition inspired by the Algorithmic Accountability Act of 2022 and lineages therein, exact definition may be adjusted based on the bill context)
Conclusion
As AI systems are increasingly integrated into government and entrusted with decision-making roles, we risk further embedding bias and mistakes into AI-assisted decisions and outcomes. Existing tools from other domains, such as existing robust public engagement processes in drug development, when applied to AI deployment can help strengthen public trust in these systems and enhance perceptions of their legitimacy and the decisions they produce. Embedding Decision Subject Representatives in the procurement of consequential AI systems, regulatory processes, and agency decision-making represents a gold-standard approach. With minimal additional oversight and support, this practice can help drive the development of high-quality systems that are informed by real-world needs.
Similar to the pharmaceutical context, both companies designing and deploying consequential AI decision systems and governments procuring and overseeing consequential AI decision systems should engage decision subjects. Corbet and colleagues recently (2023) assessed participatory approaches to AI development and found that many projects struggle to provide decision subjects with meaningful influence over AI governance decisions.
As the FDA has worked to engage patients over the years, it has shared its learnings back with the pharma industry, leading to overall improvements in patient engagement related to both regulatory decisions and company drug development decisions. A Decision Subject Representative Program accompanied by rigorous evaluation could help inform best practices for industry public engagement efforts.
Engaging the public in science and technology policy involves building bridges between communities with different levels of power and access in society. It will be challenging, require time, financial resources, and rigorous evaluation. AI fairness advocates should push for these activities both in industry and within government agencies.
Commercial artificial intelligence tools have recently emerged that are able to produce police reports. If the resulting reports are inaccurate, incomplete or biased, or if the process leaks confidential information, this could undermine the criminal justice system and harm citizens.
Too often, affected patients, clinicians, and regulators cannot see how the system works, why a decision was made, or whether meaningful human oversight occurred.
Existing tools from other domains, such as existing robust public engagement processes in drug development, when applied to AI deployment can help strengthen public trust in these systems and enhance perceptions of their legitimacy and the decisions they produce.
With thoughtful policy action, it is still possible to build systems that are fair, transparent, and accountable, and to earn the public trust that will ultimately determine AI’s future. We hope policymakers are ready to act.