FairCare Verification Offers a Human-Centered Path for AI in Medicaid
A wheelchair user with complex care needs submits a prior-authorization request that her physician supports. An algorithm-generated denial arrives with no meaningful explanation, only that her condition “does not meet medical necessity.” Her appeal languishes for weeks. By the time a human reviewer sees the case, her condition has deteriorated, confirming the algorithm’s prediction that she was “high-risk.” This scenario reflects concerns documented across automated denial systems in commercial insurance and other coverage settings, where algorithms have been used to guide or accelerate utilization review and prior-authorization decisions. In Medicaid managed care, similar dynamics are playing out as algorithmic systems make high-stakes decisions about patient care. These decisions include prior authorizations, risk scoring, triage, and fraud detection. Too often, affected patients, clinicians, and regulators cannot see how the system works, why a decision was made, or whether meaningful human oversight occurred. Growing evidence suggests these systems may systematically disadvantage low-income patients, people with disabilities, and racial and ethnic minorities, perpetuating health inequities at scale. Their deployment is broadly unpopular with the general public and with frontline healthcare workers, particularly nurses, whose clinical judgment is routinely overridden by opaque automated systems.
This memo focuses primarily on Medicaid because it is where vulnerable beneficiaries are already exposed to opaque automated decision systems through managed care organizations (MCOs). It also focuses on Medicaid because existing federal and state authorities can be used now, without waiting for new legislation. The memo proposes a policy framework (“FairCare Verification”) built around two core reforms: Community Algorithmic Impact Statements (CAIS) and Nursing-Led AI Audit Brigades (N-LABs). These reforms would be supported by patient-facing appeal and explainability protections and by contract-based limits on exploitative secondary uses of safety-net data.
These reforms can be advanced through guidance from the Centers for Medicare & Medicaid Services (CMS), enforcement by the HHS Office for Civil Rights (OCR), certification standards from the Office of the National Coordinator for Health Information Technology (ONC), and state Medicaid managed care contracts. Because managed care organizations and vendors often build products to the highest applicable compliance standard, Medicaid-focused guardrails can shape broader market behavior across healthcare.
Challenge and Opportunity
This section addresses the central problem created by growing reliance on algorithmic tools in Medicaid managed care and explains why the current policy moment creates an opening for targeted intervention. The core issue is not simply that AI is entering healthcare; it is that high-impact decisions about coverage and care are increasingly being shaped by systems that affected patients, clinicians, and regulators cannot meaningfully scrutinize.
Accountability Structures
Current accountability frameworks were built for human decision-makers. They assume that decisions can be explained, questioned, appealed, and attributed to a responsible actor. Algorithmic systems strain each of those assumptions. Civil rights enforcement is still largely organized around individual complaints, but algorithmic harms often emerge at the population level through statistical patterns that are difficult for any single patient to detect or prove, a challenge well documented in emerging legal analyses of algorithmic discrimination.
Medicaid is especially important because it is a joint federal-state program in which the federal government sets baseline rules while states administer benefits and contract with managed care organizations. That structure creates multiple adoption points for algorithmic systems, but it also creates multiple intervention points. This memo therefore focuses on Medicaid managed care, especially prior authorization and utilization management. Similar issues are emerging elsewhere in healthcare. For example, CMS’s Wasteful and Inappropriate Service Reduction (WISeR) Model is a traditional Medicare model that uses enhanced technologies, including AI and machine learning, along with human clinical review, to review selected services before or around payment. That initiative is distinct from this memo’s principal Medicaid focus, but it illustrates why automated review is becoming a broader policy concern.
Automated Accountability Risks
AI tools are already embedded across prior authorization, risk scoring, triage, patient communications, and fraud detection. Proponents argue that these systems can speed processing, reduce administrative delays, standardize decision-making, and detect fraud more efficiently than manual review. In a system with substantial administrative waste, those goals are not trivial. But the case for efficiency is often asserted rather than independently validated, and internal safeguards are rarely transparent to the communities most affected.
The concern is not limited to explicit use of race, disability, or other protected traits. High-impact systems can generate discriminatory effects through proxy variables such as prior healthcare spending, geographic indicators, housing instability, or employment status. Under Section 1557 of the Affordable Care Act, Medicaid programs may not discriminate on the basis of race, color, national origin, age, disability, or sex, yet algorithmic systems may produce exactly such differential outcomes through indirect pathways. The Optum algorithm controversy showed how a widely used population health tool could systematically under-identify Black patients for additional care even when they were equally sick. Patients with disabilities may be especially vulnerable to extended automated reviews or wrongful denials when algorithms fail to account for the complexity and variability of disability-related care needs. Separately, reporting on automated insurance denials has raised concern that the speed and scale of algorithmic review can sideline meaningful clinical judgment.
Population Differences and High-Impact Use
A related concern is the gap between the population on which a system was trained and the population on which it is deployed. One problem arises when a model is trained on national claims data that does not capture the disease burden, disability prevalence, language needs, or care-access barriers of a specific state Medicaid population. A second problem arises when a model trained in one hospital system is deployed in a different care environment with different workflows, staffing patterns, and patient needs. Both problems should be treated as core governance issues, not afterthoughts. A central purpose of Community Algorithmic Impact Statements is to force disclosure of source populations, deployment settings, and subgroup validation before high-impact use.
Existing legal tools already provide a starting point. Section 1557 of the ACA, Medicaid managed care regulations under 42 CFR Part 438, health IT certification authorities, and state utilization-management oversight all create avenues for oversight. The problem is not total absence of authority. The problem is that existing authority has not been translated into a practical governance framework for algorithmic systems before those systems become entrenched.
Public trust is also fragile. A 2023 Pew Research Center survey found that 60 percent of Americans would feel uncomfortable if their healthcare provider relied on AI for diagnosis and treatment. Frontline workforce opposition reinforces this concern. In a 2024 National Nurses United survey of more than 2,300 registered nurses, many respondents reported that AI tools undermined patient safety, conflicted with clinical judgment, or could not be modified when nurses disagreed with the output. Nurses are therefore well positioned to identify when automated systems conflict with bedside realities, create avoidable delays, or shift burdens onto patients and care teams. For that reason, this memo treats nurses not just as affected stakeholders, but as central participants in accountability.
The Strategic Window
There is a near-term opportunity to act because state Medicaid contracts are periodically renewed and routinely used to add new performance, reporting, and quality requirements. Several large states, including Texas, Florida, Ohio, and Illinois, have active or upcoming Medicaid managed care procurement cycles. These cycles create natural insertion points for algorithmic transparency, audit cooperation, and appeal safeguards. States do not need to wait for Congress to begin acting through procurement and contract oversight.
At the same time, policymakers are paying closer attention to adjacent problems in Medicare. CMS’s WISeR prior-authorization model has heightened concern about automated review and delayed care, even though CMS describes the model as combining enhanced technology with human clinical review. Bipartisan congressional inquiries into automated denial systems in both Medicare Advantage and Medicaid also signal growing political interest in this space. This proposal remains centered on Medicaid managed care, where states and federal administrators have especially clear opportunities to set guardrails for high-impact systems already being used in coverage and care management.
The political environment also calls for a realistic implementation strategy. The current federal administration has expressed skepticism toward disparate-impact frameworks, and a proactive federal push framed solely in those terms may face resistance. For that reason, the most durable near-term pathway is to emphasize patient protection, clinical accountability, fair process, transparency, and state contract authority, while preserving civil-rights enforcement as an essential backstop rather than the only implementation lever. This proposal protects patient autonomy through the right to appeal and receive an explanation. It supports clinical judgment by empowering nurses to challenge opaque algorithms. It also creates accountability without expanding government bureaucracy by leveraging existing external review infrastructure and state authority. These principles resonate across the political spectrum.
The urgency of this window is heightened by the introduction of new Medicaid work-reporting requirements. States are rushing on expedited timelines to build algorithmic systems for eligibility and compliance determinations, creating additional risks of erroneous benefit terminations and increased vendor lock-in. While this proposal focuses on AI in clinical and utilization management decision-making rather than eligibility processing, the governance frameworks proposed here, particularly CAIS transparency requirements, could be extended to eligibility determination systems as well.
Plan of Action
The recommendations below are designed to be mutually reinforcing, but the memo places greatest weight on two core interventions: Community Algorithmic Impact Statements and Nursing-Led AI Audit Brigades. The remaining proposals are narrower supports intended to make those two primary reforms workable in practice.
Recommendation 1. Require Community Algorithmic Impact Statements and establish Nursing-Led AI Audit Brigades.
Any Medicaid managed care organization, subcontractor, or vendor should be required to file a public Community Algorithmic Impact Statement before deploying AI for high-impact Medicaid decisions. Covered uses should include prior authorization, utilization management, care coordination prioritization, fraud flagging, triage, and other decisions that materially affect access to care. The filing should occur before deployment and annually thereafter.
CAIS should be modeled in part on the logic of environmental impact review, which requires public assessment of potential effects, alternatives, and mitigation before major federal actions. The goal is not to create a generic disclosure form. A CAIS should require a plain-language description of the system, its intended use, the population affected, the decisions it can influence, the data sources on which it relies, the source population on which it was trained, known limitations, plausible risks of harm, mitigation steps, monitoring plans, and available alternatives. For high-impact uses, it should also disclose subgroup performance testing and whether performance was evaluated on a population meaningfully similar to the state Medicaid population in which the tool will be deployed.
CAIS should classify systems as high-impact, moderate-impact, or advisory. High-impact systems would include prior-authorization denials, utilization-management restrictions, fraud flagging with downstream care consequences, and triage systems that materially affect access. These risk tiers draw on existing frameworks such as the NIST AI Risk Management Framework, adapted to Medicaid decision-making. High-impact systems should require pre-deployment filing, state review, and a public comment period before use. Moderate-impact systems should require annual reporting and post-deployment monitoring. Advisory tools should still be documented, but with lighter obligations.
Testing for disparate impact across all protected characteristics presents measurement challenges, particularly for disability status. Unlike race and ethnicity, for which inference methodologies such as Bayesian Improved Surname Geocoding (BISG) exist when self-reported data is unavailable, no comparable inference methodology currently exists for disability status. RAND describes BISG as a method that combines surname and geocoded address information to estimate race and ethnicity when direct data are missing or incomplete. Medicaid claims data and eligibility categories may provide some basis for identifying disability-related disparities, but this remains an area requiring further methodological development. CAIS filings should document these measurement limitations transparently and describe the best available approaches for subgroup testing.
At the same time, states should establish Nursing-Led AI Audit Brigades, or N-LABs, as independent audit teams. These teams should include registered nurses but also data scientists, health law or civil-rights experts, and at least one patient advocate or community health worker with lived Medicaid experience. The American Association of Colleges of Nursing reports more than 5 million registered nurses in the United States, making nursing the nation’s largest healthcare profession. Recent nursing scholarship on clinical AI auditing similarly emphasizes that assurance frameworks should include nursing leadership, not merely technical validation. The purpose of the N-LAB model is not to have nurses perform technical validation alone, nor to have data scientists audit systems without clinical grounding. The point is to create a multidisciplinary audit process in which each discipline evaluates the same system from a different but complementary vantage point. Including patient advocates ensures that audit priorities and scorecard criteria reflect beneficiary perspectives alongside clinical and technical expertise; beneficiary input can also be channeled through existing Community Advisory Boards (required under 42 CFR 438.110) and Federally Qualified Health Center governing boards (which include patient majorities).
In practice, an N-LAB would operate in six steps. First, it would review the CAIS and underlying documentation. Second, it would obtain case samples, denial rationales, override data, and subgroup outcomes from the managed care organization or vendor. Third, data scientists would test for accuracy, subgroup disparities, calibration, and training-versus-deployment mismatch. Fourth, nurse auditors would assess clinical plausibility, workflow burden, appropriateness of overrides, and whether the system appears to displace rather than support professional judgment. Fifth, legal reviewers would analyze whether observed patterns raise concerns under Medicaid managed care rules, civil-rights obligations, grievance requirements, or contract terms. Sixth, the team would publish a public scorecard and, where necessary, require a corrective action plan.
N-LAB scorecards should rate systems on accuracy, subgroup performance, explainability, human-override capacity, documentation quality, and post-deployment monitoring. Systems rated “needs improvement” should be required to submit corrective action plans within 60 days. Systems rated “fails” for high-impact use should be suspended by the state Medicaid agency until corrective action is verified. State agencies, not N-LABs themselves, should retain final suspension authority.
Estimated cost remains modest relative to Medicaid program scale. A team costing roughly $500,000 to $750,000 annually, assuming roughly 1.25 to 1.9 million beneficiaries per audit team, would amount to approximately $0.40 per Medicaid beneficiary. This is an illustrative estimate based on comparable external quality review organization (EQRO) staffing models. A large state may require two to four teams depending on MCO and system volume. The stronger argument, however, is institutional: N-LABs translate abstract oversight into a repeatable operational process.
Implementation can proceed through existing authority. CMS can issue model CAIS guidance and encourage incorporation into managed care contracts. States can require filing and audit cooperation through requests for proposals and contract terms. ONC can reinforce these expectations by embedding documentation and audit-readiness requirements into certification-related pathways, building on ONC’s existing role coordinating EHR certification under the 21st Century Cures Act. The HHS Office for Civil Rights can use filings and scorecards as triggers for proactive review. If current political conditions make explicit Affordable Care Act Section 1557 (protecting individuals from sex discrimination) framing difficult, alternative language focusing on “differential outcomes based on personal characteristics” or “equitable access to care” can maintain legal force while being politically adaptive. These timelines are realistic based on comparable regulatory actions: CMS issued comprehensive Medicaid managed care rules (42 CFR Part 438) with 18-month implementation; ONC implemented 21st Century Cures Act certification criteria within 24 months; state insurance commissioners routinely issue bulletins with 6–12 month effective dates.
Recommendation 2. Add algorithm-specific explainability and appeal protections for high-impact adverse decisions.
Some Medicaid rules already require plans to provide denial reasons to requesting providers (42 CFR § 438.404). This proposal does not duplicate that baseline. It adds a more usable and enforceable framework for algorithmic decisions by requiring patient-facing explanations, clinician-usable explanation materials, auditor documentation, and an independent review pathway when an automated or algorithmically informed adverse decision affects care. The following standards are adapted from emerging model documentation and explanation frameworks, including model cards for model reporting and post-hoc explanation tools such as LIME (Local Interpretable Model-agnostic Explanations) and SHAP (SHapley Additive exPlanations).
For patients, explanations should be plain-language, translated as needed, and specific enough to explain what decision was made, what key factors drove it, and what information could change the outcome. Boilerplate language such as “not medically necessary” is not enough when the decision was materially shaped by an algorithmic system. Patients should receive a short contestability notice with the adverse decision that explains the decision, the right to appeal, the timeline, and how to submit additional information. For complex models, post-hoc counterfactual explanations using methods such as LIME or SHAP can translate opaque outputs into patient-accessible language.
For clinicians, the standard should be more operational. If a model materially shaped a denial or restriction, the responsible entity should be able to identify what inputs were used, which factors were most influential, what uncertainty surrounds the output, and how the case compares with relevant benchmarks or similar cases. The goal is not to force disclosure of source code or trade secrets. It is to provide enough information for meaningful clinical contestation and review.
For regulators and auditors, the system should be documented well enough to reconstruct individual decisions, evaluate subgroup performance, review validation methods, and identify known failure modes. A system that cannot generate meaningful patient-facing, clinician-facing, and auditor-facing explanations should not be used as the sole basis for a high-impact denial or restriction. That is the operative limit on insufficiently explainable systems. ONC should add these requirements to health IT certification where systems fall within certification-related pathways, specifying that certified systems must demonstrate capability to generate explanations meeting these standards.
This memo also establishes an enforceable appeal right for adverse algorithmic decisions, preserving at least the existing 60-day appeal period under federal Medicaid managed care rules while adding algorithm-specific notice and expedited independent review by an appropriately qualified clinician not employed by the plan. This mirrors the logic of existing external review structures, at comparable cost ($500-$700 per review), but makes clear that algorithmic involvement triggers heightened transparency, documentation, and reversal tracking. The process: (1) patient receives an adverse decision with a plain-language contestability card explaining the decision, the algorithmic factors involved, and appeal rights, available in multiple languages; (2) patient files appeal within the applicable appeal period; (3) independent clinician conducts de novo review; (4) binding decision issued within 15 business days; (5) reversals reported to the relevant N-LAB; (6) patterns of reversals for a particular system trigger automatic audit.
These appeal rights should be understood as complements to, not substitutes for, bright line regulatory rules where appropriate. A bright-line rule is a clear rule that draws a firm boundary and leaves little room for case-by-case balancing. For certain categories of decisions, such as prior authorizations for treatments meeting established clinical criteria, regulators should consider prohibiting automated denial without clinical review altogether, as has been proposed in the Medicare Advantage context. Audits and transparency mechanisms are most valuable for the many algorithmic applications where bright-line prohibitions are impractical.
Finally, safety-net data should not be treated as a free raw material for secondary commercial exploitation. Vendors using Medicaid and other safety-net data should be required to sign Community Data Covenants restricting secondary uses through purpose limitation, data minimization, retention limits, transparency, and benefit-sharing where appropriate. Existing education-sector data-use agreements provide a workable precedent for this kind of contractual approach, including the U.S. Department of Education’s Model Terms of Service and state-level models such as Alabama’s SDPC Alliance.
Recommendation 3. Use state Medicaid contracts as the primary near-term implementation pathway.
Because Medicaid is jointly governed, states do not need to wait for new federal actions. State Medicaid agencies can incorporate FairCare Verification (CAIS filing, N-LAB cooperation), minimum explanation requirements, and appeal obligations directly into managed care requests for proposals and contracts. These mechanisms fit naturally alongside existing reporting, quality, and utilization-management provisions. For example, a state could require: “Contractor shall file CAIS documents for all high-impact AI systems 90 days before deployment and cooperate with state-designated N-LAB audits. Failure to comply constitutes material breach.”
A state-based pathway is also more feasible in the current moment. It allows policymakers to frame the issue in terms of accountability, patient protection, fair process, and clinical integrity rather than relying exclusively on expansive new federal directives. It also creates opportunities for coalition-building among states that want to avoid fragmented vendor compliance and regulatory arbitrage.
To reduce patchwork harms, states should work through a consortium model. A coalition of large states could develop shared contract language, common filing templates, reciprocal recognition of audit findings, and aligned minimum standards for high-impact systems. This approach mirrors the National Association of Insurance Commissioners (NAIC) model law development process and multi-state pharmaceutical supplemental rebate agreements. Benefits include reducing compliance burden, increasing leverage over vendors, and accelerating de facto national standard-setting through market pressure. Where stronger requirements are desired, states can pass legislation, with Colorado’s SB 21-169 and Illinois’s AI Video Interview Act providing useful models.
Enforcement should sit inside this state-contract architecture rather than operate as a disconnected apparatus. Failure to file a required CAIS, cooperate with audit review, produce required explanation materials, or implement corrective action should count as a material contract breach. States can then use cure notices, financial penalties, suspension authority, and procurement consequences already familiar in managed care oversight. Federal agencies, including CMS through Medicaid matching-fund conditions, ONC through certification standards, and OCR through independent Section 1557 authority, should reinforce, not replace, this state-centered pathway.
Conclusion
The strongest immediate case for action is not that every healthcare AI use can be solved at once. It is that Medicaid beneficiaries are already exposed to high-impact algorithmic systems in coverage and care-management decisions, and that existing law and contract authority can be used now to make those systems more visible, more contestable, and more accountable.
“FairCare Verification”, or Community Algorithmic Impact Statements and Nursing-Led AI Audit Brigades, are the memo’s central reforms because they address the first-order governance problem: systems are being deployed without adequate public disclosure or credible independent review. Explainability rules, appeal rights, and data covenants matter, but they work best as supporting mechanisms around those two core interventions.No new legislation is required for initial implementation. CMS could issue initial guidance quickly; state Medicaid directors can add clauses at the next MCO contract renewal; state Medicaid agencies and MCOs can adopt policies at their next meeting. What is required is a practical governance strategy that fits current institutional realities, focuses clearly on Medicaid managed care, and gives patients, nurses, regulators, and community stakeholders structured ways to detect and correct harm before opaque systems become entrenched. The next 18-24 months may be a critical window before harmful practices become deeply embedded. The question is whether policymakers seize this opportunity or face far more difficult remediation after extensive harm accumulates.
Medicaid beneficiaries are especially vulnerable to opaque administrative systems and often have the least practical power to absorb delays, navigate appeals, or switch coverage arrangements. Similar concerns also exist in Medicare, including in traditional Medicare pilots, but Medicaid managed care offers especially clear and immediate levers for contract-based intervention. Medicaid requirements can also influence commercial insurers, shape vendor products, and set precedent across sectors because vendors and MCOs often build products to satisfy the strictest large-market contract standard rather than maintaining separate systems for every payer.
Existing requirements under 42 CFR § 438.404 generally address the obligation to provide a denial reason. This proposal goes further by requiring algorithm-specific disclosure, patient-facing contestability materials, clinician-usable explanation content, independent review for adverse algorithmic decisions, and systematic feedback from reversals into ongoing audit oversight.
The proposal does not require disclosure of source code or proprietary architecture. It requires disclosure of inputs, intended use, validation, subgroup performance, explanation capacity, and monitoring results—enough to evaluate what the system does and whether it can lawfully and safely be used in high-impact Medicaid decisions. Post-hoc explainability techniques (LIME, SHAP, counterfactual explanations) provide transparency without revealing trade secrets.
N-LAB audits cost approximately $0.40 per Medicaid beneficiary annually. CAIS filing requires staff time, not new technology. The appeal right leverages existing external review infrastructure and timelines. These costs are modest compared with remediating entrenched algorithmic discrimination years later.
Too often, affected patients, clinicians, and regulators cannot see how the system works, why a decision was made, or whether meaningful human oversight occurred.
Existing tools from other domains, such as existing robust public engagement processes in drug development, when applied to AI deployment can help strengthen public trust in these systems and enhance perceptions of their legitimacy and the decisions they produce.
With thoughtful policy action, it is still possible to build systems that are fair, transparent, and accountable, and to earn the public trust that will ultimately determine AI’s future. We hope policymakers are ready to act.
Procurement is not merely an administrative function—it is how AI enters government and the first line of defense for responsible AI in the public sector.