Driving Equitable Healthcare Innovations through an AI for Medicaid (AIM) Initiative

Artificial intelligence (AI) has transformative potential in the public health space – in an era when millions of Americans have limited access to high-quality healthcare services, AI-based tools and applications can enable remote diagnostics, drive efficiencies in implementation of public health interventions, and support clinical decision-making in low-resource settings. However, innovation driven primarily by the private sector today may be exacerbating existing disparities by training models on homogenous datasets and building tools that primarily benefit high socioeconomic status (SES) populations. 

To address this gap, the Center for Medicare and Medicaid Innovation (CMMI) should create an AI for Medicaid (AIM) Initiative to distribute competitive grants to state Medicaid programs (in partnership with the private sector) for pilot AI solutions that lower costs and improve care delivery for rural and low-income populations covered by Medicaid. 

Challenge & Opportunity

In 2022, the United States spent $4.5 trillion on healthcare, accounting for 17.3% of total GDP. Despite spending far more on healthcare per capita compared to other high-income countries, the United States has significantly worse outcomes, including lower life expectancy, higher death rates due to avoidable causes, and lesser access to healthcare services. Further, the 80 million low-income Americans reliant on state-administered Medicaid programs often have below-average health outcomes and the least access to healthcare services. 

AI has the potential to transform the healthcare system – but innovation solely driven by the private sector results in the exacerbation of the previously described inequities. Algorithms in general are often trained on datasets that do not represent the underlying population – in many cases, these training biases result in tools and models that perform poorly for racial minorities, people living with comorbidities, and people of low SES. For example, until January 2023, the model used to prioritize patients for kidney transplants systematically ranked Black patients lower than White patients – the race component was identified and removed due to advocacy efforts within the medical community. AI models, while significantly more powerful than traditional predictive algorithms, are also more difficult to understand and engineer, resulting in the likelihood of further perpetuating such biases. 

Additionally, startups innovating the digital health space today are not incentivized to develop solutions for marginalized populations. For example, in FY 2022, the top 10 startups focused on Medicaid received only $1.5B in private funding, while their Medicare Advantage (MA)-focused counterparts received over $20B. Medicaid’s lower margins are not attractive to investors, so digital health development targets populations that are already well-insured and have higher degrees of access to care.

The Federal Government is uniquely positioned to bridge the incentive gap between developers of AI-based tools in the private sector and American communities who would benefit most from said tools. Accordingly, the Center for Medicare and Medicaid Innovation (CMMI) should launch the AI for Medicaid (AIM) Initiative to incentivize and pilot novel AI healthcare tools and solutions targeting Medicaid recipients. Precedents in other countries demonstrate early success in state incentives unlocking health AI innovations – in 2023, the United Kingdom’s National Health Service (NHS) partnered with Deep Medical to pilot AI software that streamlines services by predicting and mitigating missed appointment risk. The successful pilot is now being adopted more broadly and is projected to save the NHS over $30M annually in the coming years. 

The AIM Initiative, guided by the structure of the former Medicaid Innovation Accelerator Program (IAP), President Biden’s executive order on integrating equity into AI development, and HHS’ Equity Plan (2022), will encourage the private sector to partner with State Medicaid programs on solutions that benefit rural and low-income Americans covered by Medicaid and drive efficiencies in the overall healthcare system. 

Plan of Action

CMMI will launch and operate the AIM Initiative within the Department of Health and Human Services (HHS). $20M of HHS’ annual budget request will be allocated towards the program. State Medicaid programs, in partnership with the private sector, will be invited to submit proposals for competitive grants. In addition to funding, CMMI will leverage the former structure of the Medicaid IAP program to provide state Medicaid agencies with technical assistance throughout their participation in the AIM Initiative. The programs ultimately selected for pilot funding will be monitored and evaluated for broader implementation in the future. 

Sample Detailed Timeline

Risks and Limitations

Conclusion

The AI for Medicaid Initiative is an important step in ensuring the promise of artificial intelligence in healthcare extends to all Americans. The initiative will enable the piloting of a range of solutions at a relatively low cost, engage with stakeholders across the public and private sectors, and position the United States as a leader in healthcare AI technologies. Leveraging state incentives to address a critical market failure in the digital health space can additionally unlock significant efficiencies within the Medicaid program and the broader healthcare system. The rural and low-income Americans reliant on Medicaid have too often been an afterthought in access to healthcare services and technologies – the AIM Initiative provides an opportunity to address this health equity gap.

This action-ready policy memo is part of Day One 2025 — our effort to bring forward bold policy ideas, grounded in science and evidence, that can tackle the country’s biggest challenges and bring us closer to the prosperous, equitable and safe future that we all hope for whoever takes office in 2025 and beyond.

Strategies to Accelerate and Expand Access to the U.S. Innovation Economy

In 2020, we outlined a vision for how the incoming presidential administration could strengthen the nation’s innovation ecosystem, encouraging the development and commercialization of science and technology (S&T) based ventures. This vision entailed closing critical gaps from lab to market, with an emphasis on building a broadly inclusive pipeline of entrepreneurial talent while simultaneously providing key support in venture development. 

During the intervening years, we have seen extraordinary progress, in good part due to ambitious legislation. Today, we propose innovative ways that the federal government can successfully build on this progress and make the most of new programs. With targeted policy interventions, we can efficiently and effectively support the U.S. innovation economy through the translation of breakthrough scientific research from the lab to the market. The action steps we propose are predicated on three core principles: inclusion, relevance, and sustainability. Accelerating our innovation economy and expanding access to it can make our nation more globally competitive, increase economic development, address climate change, and improve health outcomes. A strong innovation economy benefits everyone. 

Challenge

Our Day One 2020 memo began by pitching the importance of innovation and entrepreneurship: “Advances in scientific and technological innovations—and, critically, the ability to efficiently transform breakthroughs into scalable businesses—have contributed enormously to American economic leadership over the past century.” Now, it is widely recognized that innovation and entrepreneurship are key to both global economic leadership and addressing the challenges of changing climate. The question is no longer whether we must innovate but rather how effectively we can stimulate and expand a national innovation economy. 

Since 2020, the global and U.S. economies have gone through massive change and uncertainty.  The Global Innovation Index (GII) 2023 described the challenges involved in its yearly analysis of monitoring global innovation trends amid uncertainty brought on by a sluggish economic recovery from the COVID-19 pandemic, elevated interest rates, and geopolitical tensions. Innovation indicators like scientific publications, research and development (R&D), venture capital (VC) investments, and the number of patents rose to historic levels, but the value of VC investment declined by close to 40%. As a counterweight to this extensive uncertainty, the GII 2023 described the future of S&T innovation and progress as “the promise of Digital Age and Deep Science innovation waves and technological progress.” 

In the face of the pressures of global competitiveness, societal needs, and climate change, the clear way forward is to continue to innovate based on scientific and technical advancements. Meeting the challenges of our moment in history requires a comprehensive and multifaceted effort led by the federal government with many public and private partners.

Grow global competitiveness

Around the world, countries are realizing that investing in innovation is the most efficient way to transform their economies. In 2022, the U.S. had the largest R&D budget internationally, with spending growing by 5.6%, but China’s investment in R&D grew by 9.8%. For the U.S. to remain a global economic leader, we must continue to invest in innovation infrastructure, including the basic research and science, technology, engineering, and math (STEM) education that underpins our leadership, while we grow our investments in translational innovation. This includes reframing how existing resources are used as well as allocating new spending. It will require a systems change orientation and long-term commitments. 

Increase economic development

Supporting and growing an innovation economy is one of our best tools for economic development. From place-based innovation programs to investment in emerging research institutions (ERIs) and Minority-Serving Institutions (MSIs) to training S&T innovators to become entrepreneurs in I-Corps™, these initiatives stimulate local economies, create high-quality jobs, and reinvigorate regions of the country left behind for too long. 

Address climate change

In 2023, for the first time, global warming exceeded 1.5°C for an entire year. It is likely that all 12 months of 2024 will also exceed 1.5°C above pre-industrial temperatures. Nationally and internationally, we are experiencing the effects of climate change; climate mitigation, adaptation, and resilience solutions are urgently needed and will bring outsized economic and social impact.

Improve U.S. health outcomes

The COVID-19 pandemic was devastating, particularly impacting underserved and underrepresented populations, but it spurred unprecedented medical innovation and commercialization of new diagnostics, vaccines, and treatments. We must build on this momentum by applying what we’ve learned about rapid innovation to continue to improve U.S. health outcomes and to ensure that our nation’s health care needs across regions and demographics are addressed. 

Make innovation more inclusive

Representational disparities persist across racial/ethnic and gender lines in both access to and participation in innovation and entrepreneurship. This is a massive loss for our innovation economy. The business case for broader inclusion and diversity is growing even stronger, with compelling data tracking the relationship between leadership diversity and company performance. Inclusive innovation is more effective innovation: a multitude of perspectives and lived experiences are required to fully understand complex problems and create truly useful solutions. To reap the full benefits of innovation and entrepreneurship, we must increase access and pathways for all. 

Opportunity

With the new presidential administration in 2025, the federal government has a renewed opportunity to prioritize policies that will generate and activate a wave of powerful, inclusive innovation and entrepreneurship. Implementing such policies and funding the initiatives that result is crucial if we as a nation are to successfully address urgent problems such as the climate crisis and escalating health disparities. 

Our proposed action steps are predicated on three core principles: inclusion, relevance, and sustainability. 

Inclusion

One of this nation’s greatest and most unique strengths is our heterogeneity. We must leverage our diversity to meet the complexity of the substantial social and economic challenges that we face today. The multiplicity of our people, communities, identities, geographies, and lived experiences gives the U.S. an edge in the global innovation economy: When we bring all of these perspectives to the table, we better understand the challenges that we face, and we are better equipped to innovate to meet them. If we are to harness the fullness of our nation’s capacity for imagination, ingenuity, and creative problem-solving, entrepreneurship pathways must be inclusive, equitable, and accessible to all. Moreover, all innovators must learn to embrace complexity, think expansively and critically, and welcome perspectives beyond their own frame of reference. Collaboration and mutually beneficial partnerships are at the heart of inclusive innovation. 

Relevance

Innovators and entrepreneurs have the greatest likelihood of success—and the greatest potential for impact—when their work is purpose-driven, nimble, responsive to consumer needs, and adaptable to different applications and settings.  Research suggests that “breakthrough innovation” occurs when different actors bring complementary and independent skills to co-create interesting solutions to existing problems. Place-based innovation is one strategy to make certain that technology development is grounded in regional concerns and aspirations, leading to better outcomes for all concerned. 

Sustainability 

Multiple layers of sustainability should be integrated into the innovation and entrepreneurship landscape. First and most salient is supporting the development of innovative technologies that respond to the climate crisis and bolster national resilience. Second is encouraging innovators to incorporate sustainable materials and processes in all stages of research and development so that products benefit the planet and risks to the environment are mitigated through the manufacturing process, whether or not climate change is the focus of the technology. Third, it is vital to prioritize helping ventures develop sustainable business models that will result in long-term viability in the marketplace. Fourth, working with innovators to incorporate the potential impact of climate change into their business planning and projections ensures they are equipped to adapt to changing needs. All of these layers contribute to sustaining America’s social well-being and economic prosperity, ensuring that technological breakthroughs are accessible to all.

Proposed Action

Recommendation 1. Supply and prepare talent.

Continuing to grow the nation’s pipeline of S&T innovators and entrepreneurs is essential. Specifically, creating accessible entrepreneurial pathways in STEM will ensure equitable participation. Incentivizing individuals to become innovators-entrepreneurs, especially those from underrepresented groups, will strengthen national competitiveness by leveraging new, untapped potential across innovation ecosystems.

Expand the I-Corps model

By bringing together experienced industry mentors, commercial experts, research talent, and promising technologies, I-Corps teaches scientific innovators how to evaluate whether their innovation can be commercialized and how to take the first practical steps of bringing their product to market. Ten new I-Corps Hubs, launched in 2022, have expanded the network of engaged universities and collaborators, an important step toward growing an inclusive innovation ecosystem across the U.S. 

Interest in I-Corps far outpaces current capacity, and increasing access will create more expansive pathways for underrepresented entrepreneurs. New federal initiatives to support place-based innovation and to grow investment at ERIs and MSIs will be more successful if they also include lab-to-market training programs such as I-Corps. Federal entities should institute policies and programs that increase awareness about and access to sequenced venture support opportunities for S&T innovators. These opportunities should include intentional “de-risking” strategies through training, advising, and mentoring.

Specifically, we recommend expanding I-Corps capacity so that all interested participants can be accommodated. We should also strive to increase access to I-Corps so that programs reach diverse students and researchers. This is essential given the U.S. culture of entrepreneurship that remains insufficiently inclusive of women, people of color, and those from low-income backgrounds, as well as international students and researchers, who often face barriers such as visa issues or a lack of institutional support needed to remain in the U.S. to develop their innovations. Finally, we should expand the scope of what I-Corps offers, so that programs provide follow-on support, funding, and access to mentor and investor networks even beyond the conclusion of initial entrepreneurial training. 

I-Corps has already expanded beyond the National Science Foundation (NSF) to I-Corps at National Institutes of Health (NIH), to empower biomedical entrepreneurs, and Energy I-Corps, established by the Department of Energy (DOE) to accelerate the deployment of energy technologies. We see the opportunity to grow I-Corps further by building on this existing infrastructure and creating cohorts funded by additional science agencies so that more basic research is translated into commercially viable businesses. 

Close opportunity gaps by supporting emerging research institutions (ERIs) and Minority-Serving Institutions (MSIs)

ERIs and MSIs provide pathways to S&T innovation and entrepreneurship, especially for individuals from underrepresented groups. In particular, a VentureWell-commissioned report identified that “MSIs are centers of research that address the unique challenges and opportunities faced by BIPOC communities. The research that takes place at MSIs offers solutions that benefit a broad and diverse audience; it contributes to a deeper understanding of societal issues and drives innovation that addresses these issues.”

The recent codification of ERIs in the 2022 CHIPS and Science Act pulls this category into focus. Defining this group, which comprises thousands of higher education institutions,  was the first step in addressing the inequitable distribution of federal research funding. That imbalance has perpetuated regional disparities and impacted students from underrepresented groups, low-income students, and rural students in particular. Further investment in ERIs will result in more STEM-trained students, who can become innovators and entrepreneurs with training and engagement. Additional support that could be provided to ERIs includes increased research funding, access to capital/investment, capacity building (faculty development, student support services), industry partnerships, access to networks, data collection/benchmarking, and implementing effective translation policies, incentives, and curricula. 

Supporting these institutions—many of which are located in underserved rural or urban communities that experience underinvestment—provides an anchor for sustained talent development and economic growth. 

Recommendation 2. Support place-based innovation.

Place-based innovation not only spurs innovation but also builds resilience in vulnerable communities, enhancing both U.S. economic and national security. Communities that are underserved and underinvested in present vulnerabilities that hostile actors outside of the U.S. can exploit. Place-based innovation builds resilience: innovation creates high-quality jobs and brings energy and hope to communities that have been left behind, leveraging the unique strengths, ecosystems, assets, and needs of specific regions to drive economic growth and address local challenges.  

Evaluate and learn from transformative new investments

There have been historic levels of government investment in place-based innovation, funding the NSF’s Regional Innovation Engines awards and two U.S. Department of Commerce Economic Development Administration (EDA) programs: the Build Back Better Regional Challenge and Regional Technology and Innovation Hubs awards. The next steps are to refine, improve, and evaluate these initiatives as we move forward. 

Unify the evaluation framework, paired with local solutions

Currently, evaluating the effectiveness and outcomes of place-based initiatives is challenging, as benchmarks and metrics can vary by region. We propose a unified framework paired with solutions locally identified by and tailored to the specific needs of the regional innovation ecosystem. A functioning ecosystem cannot be simply overlaid upon a community but must be built by and for that community. The success of these initiatives requires active evaluation and incorporation of these learnings into effective solutions, as well as deep strategic collaboration at the local level, with support and time built into processes.   

Recommendation 3. Increase access to financing and capital.

Funding is the lifeblood of innovation. S&T innovation requires more investment and more time to bring to market than other types of ventures, and early-stage investments in S&T startups are often perceived as risky by those who seek a financial return. Bringing large quantities of early-stage S&T innovations to the point in the commercialization process where substantial private capital takes an interest requires nondilutive and patient government support. The return on investment that the federal government seeks is measured in companies successfully launched, jobs created, and useful technologies brought to market.

Disparities in access to capital by companies owned by women and underrepresented minority founders are well documented. The federal government has an interest in funding innovators and entrepreneurs from many backgrounds: they bring deep and varied knowledge and a multitude of perspectives to their innovations and to their ventures. This results in improved solutions and better products at a cheaper price for consumers. Increasing access to financing and capital is essential to our national economic well-being and to our efforts to build climate resilience. 

Expand SBIR/STTR access and commercial impact

The SBIR and STTR programs spur innovation, bolster U.S. economic competitiveness, and strengthen the small business sector, but barriers persist. In a recent third-party assessment of the SBIR/STTR program at NIH, the second largest administrator of SBIR/STTR funds, the committee found outreach from the SBIR/STTR programs to underserved groups is not coordinated, and there has been little improvement in the share of applications from or awards to these groups in the past 20 years. Further, NIH follows the same processes used for awarding R01 research grants, using the same review criteria and typically the same reviewers, omitting important commercialization considerations. 

To expand access and increase the commercialization potential of the SBIR/STTR program, funding agencies should foster partnerships with a broader group of organizations, conduct targeted outreach to potential applicants, offer additional application assistance to potential applicants, work with partners to develop mentorship and entrepreneur training programs, and increase the percentage of private-sector reviewers with entrepreneurial experience. Successful example programs of SBIR/STTR support programs include the NSF Beat-The-Odds Boot Camp, Michigan’s Emerging Technologies Fund, and the SBIR/STTR Innovation Summit. 

Provide entrepreneurship education and training

Initiatives like NSF Engines, Tech Hubs, Build-Back-Better Regional Challenge, the Minority Business Development Agency (MBDA) Capital Challenge, and the Small Business Administration (SBA) Growth Accelerator Fund expansion will all achieve more substantial results with supplemental training for participants in how to develop and launch a technology-based business. As an example of the potential impact, more than 2,500 teams have participated in I-Corps since the program’s inception in 2012. More than half of these teams, nearly 1,400, have launched startups that have cumulatively raised $3.16 billion in subsequent funding, creating over 11,000 jobs. Now is an opportune moment to widely apply similarly effective approaches. 

Launch a local investment education initiative

Angel investors are typically providing the first private funding available to S&T innovators and entrepreneurs. These very early-stage funders give innovators access to needed capital, networks, and advice to get their ventures off the ground. We recommend that the federal government expand the definition of an accredited investor and incentivize regionally focused initiatives to educate policymakers and other regional stakeholders about best practices to foster more diverse and inclusive angel investment networks. With the right approach and support, there is the potential to engage thousands more high-net-worth individuals in early-stage investing, contributing their expertise and networks as well as their wealth.

Encourage investment in climate solutions

Extreme climate-change-attributed weather events such as floods, hurricanes, drought, wildfire, and heat waves cost the global economy an average of $143 billion annually. S&T innovations have the potential to help address the impacts of climate change at every level:

Given the global scope of the problem and the shared resources of affected communities, the federal government can be a leader in prioritizing, collaborating, and investing in solutions to direct and encourage S&T innovation for climate solutions. There is no question whether climate adaptation technologies will be needed, but we must ensure that these solutions are technologies that create economic opportunity in the U.S. We encourage the expansion and regular appropriations of funding for successful climate programs across federal agencies, including the DoE Office of Technology Transitions’ Energy Program for Innovation Clusters, the National Oceanic and Atmospheric Administration’s (NOAA) Ocean-Based Climate Resilience Accelerators program, and the U.S. Department of Agriculture’s Climate Hubs. 

Recommendation 4. Shift to a systems change orientation.

To truly stimulate a national innovation economy, we need long-term commitments in policy, practice, and regulations. Leadership and coordination from the executive branch of the federal government are essential to continue the positive actions already begun by the Biden-Harris Administration.  

These initiatives include: 

Policy

Signature initiatives like the CHIPS and Science Act, Infrastructure Investment and Jobs Act, and the National Quantum Initiative Act are already threatened by looming appropriations shortfalls. We need to fully fund existing legislation, with a focus on innovative and translational R&D. According to a report by PricewaterhouseCoopers, if the U.S. increased federal R&D spending to 1% of GDP by 2030, the nation could support 3.4 million jobs and add $301 billion in labor income, $478 billion in economic value, and $81 billion in tax revenue. Beyond funding, we propose supporting innovative policies to bolster U.S. innovation capacity at the local and national levels. This includes providing R&D tax credits to spur research collaboration between industry and universities and labs, providing federal matching funds for state and regional technology transfer and commercialization efforts, and revising the tax code to support innovation by research-intensive, pre-revenue companies.

Practice

The University and Small Business Patent Procedures Act of 1980, commonly known as the Bayh-Dole Act, allows recipients of federal research funding to retain rights to inventions conceived or developed with that funding. The academic tech transfer system created by the Bayh-Dole Act (codified as amended at 35 U.S.C. §§ 200-212) generated nearly $1.3 trillion in economic output, supported over 4.2 million jobs, and launched over 11,000 startups. We should preserve the Bayh-Dole Act as a means to promote commercialization and prohibit the consideration of specific factors, such as price, in march-in determinations. 

In addition to the continual practice and implementation of successful laws such as Bayh-Dole, we must repurpose resources to support innovation and the high-value jobs that result from S&T innovation. We believe the new administration should allocate a share of federal funding to promote technology transfer and commercialization and better incentivize commercialization activities at federal labs and research institutes. This could include new programs such as mentoring programs for researcher entrepreneurs and student entrepreneurship training programs. Incentives include evaluating the economic impact of lab-developed technology by measuring commercialization outcomes in the annual Performance Evaluation and Management Plans of federal labs, establishing stronger university entrepreneurship reporting requirements to track and reward universities that create new businesses and startups, and incentivizing universities to focus more on commercialization activities as part of promotion and tenure of faculty, 

Regulations

A common cause of lab-to-market failure is the inability to secure regulatory approval, particularly for novel technologies in nascent industries. Regulation can limit potentially innovative paths, increase innovation costs, and create a compliance burden on businesses that stifle innovation. Regulation can also spur innovation by enabling the management of risk. In 1976 the Cambridge (Massachusetts) City Council became the first jurisdiction to regulate recombinant DNA, issuing the first genetic engineering license and creating the first biotech company. Now Boston/Cambridge is the world’s largest biotech hub: home to over 1,000 biotech companies, 21% of all VC biotech investments, and 15% of the U.S. drug development pipeline.

To advance innovation, we propose two specific regulatory actions:

Conclusion

To maintain its global leadership role, the United States must invest in the individuals, institutions, and ecosystems critical to a thriving, inclusive innovation economy. This includes mobilizing access, inclusion, and talent through novel entrepreneurship training programs; investing, incentivizing, and building the capacity of our research institutions; and enabling innovation pathways by increasing access to capital, networks, and resources.

Fortunately, there are  several important pieces of legislation recommitting the U.S. leadership to bold S&T goals, although much of the necessary resources are yet to be committed to those efforts. As a society, we benefit when federally supported innovation efforts tackle big problems that are beyond the scope of single ventures; notably, the many challenges arising from climate change. A stronger, more inclusive innovation economy benefits the users of S&T-based innovations, individual innovators, and the nation as a whole.

When we intentionally create pathways to innovation and entrepreneurship for underrepresented individuals, we build on our strengths. In the United States, our strength has always been our people, who bring problem-solving abilities from a multitude of perspectives and settings. We must unleash their entrepreneurial power and become, even more, a country of innovators.. 

Earlier memo contributors Heath Naquin and Shaheen Mamawala (2020) were not involved with this 2024 memo.

This action-ready policy memo is part of Day One 2025 — our effort to bring forward bold policy ideas, grounded in science and evidence, that can tackle the country’s biggest challenges and bring us closer to the prosperous, equitable and safe future that we all hope for whoever takes office in 2025 and beyond.

Policy Experiment Stations to Accelerate State and Local Government Innovation

The federal government transfers approximately $1.1 trillion dollars every year to state and local governments. Yet most states and localities are not evaluating whether the programs deploying these funds are increasing community well-being. Similarly, achieving important national goals like increasing clean energy production and transmission often requires not only congressional but also state and local policy reform. Yet many states and localities are not implementing the evidence-based policy reforms necessary to achieve these goals.

State and local government innovation is a problem not only of politics but also of capacity. State and local governments generally lack the technical capacity to conduct rigorous evaluations of the efficacy of their programs, search for reliable evidence about programs evaluated in other contexts, and implement the evidence-based programs with the highest chances of improving outcomes in their jurisdictions. This lack of capacity severely constrains the ability of state and local governments to use federal funds effectively and to adopt more effective ways of delivering important public goods and services. To date, efforts to increase the use of evaluation evidence in federal agencies (including the passage of the Evidence Act) have not meaningfully supported the production and use of evidence by state and local governments.

Despite an emerging awareness of the importance of state and local government innovation capacity, there is a shortage of plausible strategies to build that capacity. In the words of journalist Ezra Klein, we spend “too much time and energy imagining the policies that a capable government could execute and not nearly enough time imagining how to make a government capable of executing them.”

Yet an emerging body of research is revealing that an effective strategy to build government innovation capacity is to partner government agencies with local universities on scientifically rigorous evaluations of the efficacy of their programs, curated syntheses of reliable evaluation evidence from other contexts, and implementation of evidence-based programs with the best chances of success. Leveraging these findings, along with recent evidence of the striking efficacy of the national network of university-based “Agriculture Experiment Stations” established by the Hatch Act of 1887, we propose a national network of university-based “Policy Experiment Stations” or policy innovation labs in each state, supported by continuing federal and state appropriations and tasked with accelerating state and local government innovation.  

Challenge

Advocates of abundance have identified “failed public policy” as an increasingly significant barrier to economic growth and community flourishing. Of particular concern are state and local policies and programs, including those powered by federal funds, that do not effectively deliver critically important public goods and services like health, education, safety, clean air and water, and growth-oriented infrastructure.

Part of the challenge is that state and local governments lack capacity to conduct rigorous evaluations of the efficacy of their policies and programs. For example, the American Rescue Plan, the largest one-time federal investment in state and local governments in the last century, provided $350 billion in State and Local Fiscal Recovery Funds to state, territorial, local, and Tribal governments to accelerate post-pandemic economic recovery. Yet very few of those investments are being evaluated for efficacy. In a recent survey of state policymakers, 59% of those surveyed cited “lack of time for rigorous evaluations” as a key obstacle to innovation. State and local governments also typically lack the time, resources, and technical capacity to canvass evaluation evidence from other settings and assess whether a program proven to improve outcomes elsewhere might also improve outcomes locally. Finally, state and local governments often don’t adopt more effective programs even when they have rigorous evidence that these programs are more effective than the status quo, because implementing new programs disrupts existing workflows. 

If state and local policymakers don’t know what works and what doesn’t, and/or aren’t able to overcome even relatively minor implementation challenges when they do know what works, they won’t be able to spend federal dollars more effectively, or more generally to deliver critical public goods and services.

Opportunity

A growing body of research on government innovation is documenting factors that reliably increase the likelihood that governments will implement evidence-based policy reform. First, government decision makers are more likely to adopt evidence-based policy reforms when they are grounded in local evidence and/or recommended by local researchers. Boston-based researchers sharing a Boston-based study showing that relaxing density restrictions reduces rents and house prices will do less to convince San Francisco decision makers than either a San Francisco-based study, or San Francisco-based researchers endorsing the evidence from Boston. Proximity matters for government innovation.

Second, government decision makers are more likely to adopt evidence-based policy reforms when they are engaged as partners in the research projects that produce the evidence of efficacy, helping to define the set of feasible policy alternatives and design new policy interventions. Research partnerships matter for government innovation.

Third, evidence-based policies are significantly more likely to be adopted when the policy innovation is part of an existing implementation infrastructure, or when agencies receive dedicated implementation support. This means that moving beyond incremental policy reforms will require that state and local governments receive more technical support in overcoming implementation challenges. Implementation matters for government innovation. 

We know that the implementation of evidence-based policy reform produces returns for communities that have been estimated to be on the order of 17:1. Our partners in government have voiced their direct experience of these returns. In Puerto Rico, for example, decision makers in the Department of Education have attributed the success of evidence-based efforts to help students learn to the “constant communication and effective collaboration” with researchers who possessed a “strong understanding of the culture and social behavior of the government and people of Puerto Rico.” Carrie S. Cihak, the evidence and impact officer for King County, Washington, likewise observes, 

“It is critical to understand whether the programs we’re implementing are actually making a difference in the communities we serve. Throughout my career in King County, I’ve worked with  County teams and researchers on evaluations across multiple policy areas, including transportation access, housing stability, and climate change. Working in close partnership with researchers has guided our policymaking related to individual projects, identified the next set of questions for continual learning, and has enabled us to better apply existing knowledge from other contexts to our own. In this work, it is essential to have researchers who are committed to valuing local knowledge and experience–including that of the community and government staff–as a central part of their research, and who are committed to supporting us in getting better outcomes for our communities.” 

The emerging body of evidence on the determinants of government innovation can help us define a plan of action that galvanizes the state and local government innovation necessary to accelerate regional economic growth and community flourishing. 

Plan of Action 

An evidence-based plan to increase state and local government innovation needs to facilitate and sustain durable partnerships between state and local governments and neighboring universities to produce scientifically rigorous policy evaluations, adapt evaluation evidence from other contexts, and develop effective implementation strategies. Over a century ago, the Hatch Act of 1887 created a remarkably effective and durable R&D infrastructure aimed at agricultural innovation, establishing university-based Agricultural Experiment Stations (AES) in each state tasked with developing, testing, and translating innovations designed to increase agricultural productivity. 

Locating university-based AES in every state ensured the production and implementation of locally-relevant evidence by researchers working in partnership with local stakeholders. Federal oversight of the state AES by an Office of Experiment Stations in the US Department of Agriculture ensured that work was conducted with scientific rigor and that local evidence was shared across sites. Finally, providing stable annual federal appropriations for the AES, with required matching state appropriations, ensured the durability and financial sustainability of the R&D infrastructure. This infrastructure worked: agricultural productivity near the experiment stations increased by 6% after the stations were established.

Congress should develop new legislation to create and fund a network of state-based “Policy Experiment Stations.”

 The 119th Congress that will convene on January 3, 2025 can adapt the core elements of the proven-effective network of state-based Agricultural Experiment Stations to accelerate state and local government innovation. Mimicking the structure of 7 USC 14, federal grants to states would support university-based “Policy Experiment Stations” or policy innovation labs in each state, tasked with partnering with state and local governments on (1) scientifically rigorous evaluations of the efficacy of state and local policies and programs; (2) translations of evaluation evidence from other settings; and (3) overcoming implementation challenges. 

As in 7 USC 14, grants to support state policy innovation labs would be overseen by a federal office charged with ensuring that work was conducted with scientific rigor and that local evidence was shared across sites. We see two potential paths for this oversight function, paths that in turn would influence legislative strategy.

Pathway 1: This oversight function could be located in the Office of Evaluation Sciences (OES) in the General Services Administration (GSA). In this case, the congressional committees overseeing GSA, namely the House Committee on Oversight and Responsibility and the Senate Committee on Homeland Security and Governmental Affairs, would craft legislation providing for an appropriation to GSA to support a new OES grants program for university-based policy innovation labs in each state. The advantage of this structure is that OES is a highly respected locus of program and policy evaluation expertise. 

Pathway 2: Oversight could instead be located in the Directorate of Technology, Innovation, and Partnerships in the National Science Foundation (NSF TIP). In this case, the House Committee on Science, Space, and Technology and the Senate Committee on Commerce, Science, and Transportation would craft legislation providing for a new grants program within NSF TIP to support university-based policy innovation labs in each state. The advantage of this structure is that NSF is a highly respected grant-making agency. 

Either of these paths is feasible with bipartisan political will. Alternatively, there are unilateral steps that could be taken by the incoming administration to advance state and local government innovation. For example, the Office of Management and Budget (OMB) recently released updated Uniform Grants Guidance clarifying that federal grants may be used to support recipients’ evaluation costs, including “conducting evaluations, sharing evaluation results, and other personnel or materials costs related to the effective building and use of evidence and evaluation for program design, administration, or improvement.” The Uniform Grants Guidance also requires federal agencies to assess the performance of grant recipients, and further allows federal agencies to require that recipients use federal grant funds to conduct program evaluations. The incoming administration could further update the Uniform Grants Guidance to direct federal agencies to require that state and local government grant recipients set aside grant funds for impact evaluations of the efficacy of any programs supported by federal funds, and further clarify the allowability of subgrants to universities to support these impact evaluations.

Conclusion

Establishing a national network of university-based “Policy Experiment Stations” or policy innovation labs in each state, supported by continuing federal and state appropriations, is an evidence-based plan to facilitate abundance-oriented state and local government innovation. We already have impressive examples of what these policy labs might be able to accomplish. At MIT’s Abdul Latif Jameel Poverty Action Lab North America, the University of Chicago’s Crime Lab and Education Lab, the University of California’s California Policy Lab, and Harvard University’s The People Lab, to name just a few, leading researchers partner with state and local governments on scientifically rigorous evaluations of the efficacy of public policies and programs, the translation of evidence from other settings, and overcoming implementation challenges, leading in several cases to evidence-based policy reform. Yet effective as these initiatives are, they are largely supported by philanthropic funds, an infeasible strategy for national scaling.

In recent years we’ve made massive investments in communities through federal grants to state and local governments. We’ve also initiated ambitious efforts at growth-oriented regulatory reform which require not only federal but also state and local action. Now it’s time to invest in building state and local capacity to deploy federal investments effectively and to galvanize regional economic growth. Emerging research findings about the determinants of government innovation, and about the efficacy of the R&D infrastructure for agricultural innovation established over a century ago, give us an evidence-based roadmap for state and local government innovation.

This action-ready policy memo is part of Day One 2025 — our effort to bring forward bold policy ideas, grounded in science and evidence, that can tackle the country’s biggest challenges and bring us closer to the prosperous, equitable and safe future that we all hope for whoever takes office in 2025 and beyond.

Promoting American Resilience Through a Strategic Investment Fund

Critical minerals, robotics, advanced energy systems, quantum computing, biotechnology, shipbuilding, and space are some of the resources and technologies that will define the economic and security climate of the 21st century. However, the United States is at risk of losing its edge in these technologies of the future. For instance, China processes the vast majority of the world’s batteries and critical metals and has successfully launched a quantum communications satellite. The implications are enormous: the U.S. relies on its qualitative technological edge to fuel productivity growth, improve living standards, and maintain the existing global order. Indeed, the Inflation Reduction Act (IRA) and CHIPS Act were largely reactionary moves to shore up atrophied manufacturing capabilities in the American battery and semiconductor industries, requiring hundreds of billions in outlays to catch up. In an ideal world, critical industries would be sufficiently funded well in advance to avoid economically costly catch-up spending.

However, many of these technologies are characterized by long timelines, significant capital expenditures, and low and uncertain profit margins, presenting major challenges for private-sector investors who are required by their limited partners (capital providers such as pension funds, university endowments, and insurance companies) to underwrite to a certain risk-adjusted return threshold. This stands in contrast to technologies like artificial intelligence and pharmaceuticals: While both are also characterized by large upfront investments and lengthy research and development timelines, the financial payoffs are far clearer, incentivizing private sectors to play a leading role in commercialization. This issue for technologies in economically and geopolitically vital industries such as lithium processing and chips is most acute in the “valley of death,” when companies require scale-up capital for an early commercialization effort: the capital required is too large for traditional venture capital, yet too risky for traditional project finance.

The United States needs a strategic investment fund (SIF) to shepherd promising technologies in nationally vital sectors through the valley of death. An American SIF is not intended to provide subsidies, pick political winners or losers, or subvert the role of private capital markets. On the contrary, its role would be to “crowd in” capital by uniquely managing risk that no private or philanthropic entities have the capacity to do. In doing so, an SIF would ensure that the U.S. maintains an edge in critical technologies, promoting economic dynamism and national security in an agile, cost-efficient manner. 

Challenges

The Need for Private Investment 

A handful of resources and technologies, some of which have yet to be fully characterized, have the potential to play an outsized role in the future economy. Most of these key technologies have meaningful national security implications.

Since ChatGPT’s release in November 2022, artificial intelligence (AI) has experienced a commercial renaissance that has captured the public’s imagination and huge sums of venture dollars, as evidenced by OpenAI’s October 2024 $6.5 billion round at a $150 billion pre-money valuation. However, AI is not the only critical resource or technology that will power the future economy, and many of those critical resources and technologies may struggle to attract the same level of private investment. Consider the following:

Few sectors receive the level of consistent venture attention that software technology, most recently in AI, has gotten in the last 18 months. However, this does not make them unbackable or unimportant; on the contrary, technologies that increase mineral recovery yields or make drone engines cheaper should receive sufficient support to get to scale. While private-sector capital markets have supported the development of many important industries, they are not perfect and may miss important opportunities due to information asymmetries and externalities.

Overcoming the Valley of Death

Many strategically important technologies are characterized by high upfront costs and low or uncertain margins, which tends to dissuade investment by private-sector organizations at key inflection points, namely, the “valley of death.”

By their nature, innovative technologies are complex and highly uncertain. However, some factors make future economic value—and therefore financeability—more difficult to ascertain than others. For example, innovative battery technologies that enable long-term storage of energy generated from renewables would greatly improve the economics of utility-scale solar and wind projects. However, this requires production at scale in the face of potential competition from low-cost incumbents. In addition, there is the element of scientific risk itself, as well as the question of customer adoption and integration. There are many good reasons why technologies and companies that seem feasible, economical, and societally valuable do not succeed.

These dynamics result in lopsided investment allocations. In the early stages of innovation, venture capital is available to fund startups with the promise of outsized return driven partially by technological hype and partially by the opportunity to take large equity stakes in young companies. At the other end of the barbell, private equity and infrastructure capital are available to mature companies seeking an acquisition or project financing based on predictable cash flows and known technologies. 

However, gaps appear in the middle as capital requirements increase (often by an order of magnitude) to support the transition to early commercialization. This phenomenon is called the “valley of death” as companies struggle to raise the capital they need to get to scale given the uncertainties they face.

Figure 1. The “valley of death” describes the mismatch between existing financial structures and capital requirements in the crucial early commercialization phase. (Source: Maryland Energy Innovation Accelerator)

Shortcoming of Federal Subsidies

While the federal government has provided loans and subsidies in the past, its programs remain highly reactive and require large amounts of funding.

Aside from asking investors to take on greater risk and lower returns, there are several tools in place to ameliorate the valley of death. The IRA one such example: It appropriated some $370 billion for climate-related spending with a range of instruments, including tax subsidies for renewable energy production, low-cost loans through organizations such as the Department of Energy’s Loan Program Office (LPO), and discretionary grants.

On the other hand, there are major issues with this approach. First, funding is spread out across many calls for funding that tend to be slow, opaque, and costly. Indeed, it is difficult to keep track of available resources, funding announcements, and key requirements—just try searching for a comprehensive, easy-to-understand list of opportunities.

More importantly, these funding mechanisms are simply expensive. The U.S. does not have the financial capacity to support an IRA or CHIPS Act for every industry, nor should it go down that route. While one could argue that these bills reflect the true cost of achieving the stated policy aims of energy transition or securing the semiconductor supply chain, it is also the case that there both knowledge (engineering expertise) and capital (manufacturing facility) capabilities underpin these technologies. Allowing these networks to atrophy created greater costs down the road, which could have been prevented by targeted investments at the right points of development.

The Future Is Dynamic

The future is not perfectly knowable, and new technological needs may arise that change priorities or solve previous problems. Therefore, agility and constant re-evaluation are essential.

Technological progress is not static. Take the concept of peak oil: For decades, many of the world’s most intelligent geologists and energy forecasters believed that the world would quickly run out of oil reserves as the easiest to extract resources were extracted. In reality, technological advances in chemistry, surveying, and drilling enabled hydraulic fracturing (fracking) and horizontal drilling, creating access to “unconventional reserves” that substantially increased fossil fuel supply.

Figure 2. In 1956, M.K. Hubbert created “peak oil” theory, projecting that reserves would be exhausted around the turn of the millennium.

Fracking greatly expanded fossil fuel production in the U.S., increasing resource supply, securing greater energy independence, and facilitating the transition from coal to natural gas, whose expansion has proved to be a helpful bridge towards renewable energy generation. This transition would not have been possible without a series of technological innovations—and highly motivated entrepreneurs—that arose to meet the challenge of energy costs.

To meet the challenges of tomorrow, policymakers need tools that provide them with flexible and targeted options as well as sufficient scale to make an impact on technologies that might need to get through the valley of death. However, they need to remain sufficiently agile so as not to distort well-functioning market forces. This balance is challenging to achieve and requires an organizational structure, authorizations, and funding mechanisms that are sufficiently nimble to adapt to changing technologies and markets.

Opportunity

Given these challenges, it seems unlikely that solutions that rely solely on the private sector will bridge the commercialization gap in a number of capital-intensive strategic industries. On the other hand, existing public-sector tools, such as grants and subsidies, are too costly to implement at scale for every possible externality and are generally too retrospective in nature rather than forward-looking. The government can be an impactful player in bridging the innovation gap, but it needs to do so cost-efficiently.

An SIF is a promising potential solution to the challenges posed above. By its nature, an SIF would have a public mission focused on strategic technologies crossing the valley of death by using targeted interventions and creative financing structures that crowd in private investors. This would enable the government to more sustainably fund innovation, maintain a light touch on private companies, and support key industries and technologies that will define the future global economic and security outlook.

Plan of Action

Recommendation 1. Shepherd technologies through the valley of death. 

While the SIF’s investment managers are expected to make the best possible returns, this is secondary to the overarching public policy goal of ensuring that strategically and economically vital technologies have an opportunity to get to commercial scale.

The SIF is meant to crowd in capital such that we achieve broader societal gains—and eventually, market-rate returns—enabled by technologies that would not have survived without timely and well-structured funding. This creates tension between two competing goals: The SIF needs to act as if it will intend to make returns, or else there is the potential for moral hazard and complacency. However, it also has to be willing to not make market-rate returns, or even lose some of its principal, in the service of broader market and ecosystem development. 

Thus, it needs to be made explicitly clear from the beginning that an SIF has the intent of achieving market rate returns by catalyzing strategic industries but is not mandated to do so. One way to do this is to adopt a 501(c)(3) structure that has a loose affiliation to a department or agency, similar to that of In-Q-Tel. Excess returns could either be recycled to the fund or distributed to taxpayers.

The SIF should adapt the practices, structures, and procedures of established private-sector funds. It should have a standing investment committee made up of senior stakeholders across various agencies and departments (expanded upon below). Its day-to-day operations should be conducted by professionals who provide a range of experiences, including investing, engineering and technology, and public policy across a spectrum of issue areas. 

In addition, the SIF should develop clear underwriting criteria and outputs for each investment. These include, but are not limited to, identifying the broader market and investment thesis, projecting product penetration, and developing potential return scenarios based on different permutations of outcomes. More critically, each investment needs to create a compelling case for why the private sector cannot fund commercialization on its own and why public catalytic funding is essential.

Recommendation 2. The SIF should have a permanent authorization to support innovation under the Department of Commerce. 

The SIF should be affiliated with the Department of Commerce but work closely with other departments and agencies, including the Department of Energy, Department of Treasury, Department of Defense, Department of Health and Human Services, National Science Foundation, and National Economic Council.

Strategic technologies do not fall neatly into one sector and cut across many customers. Siloing funding in different departments misses the opportunity to capture funding synergies and, more importantly, develop priorities that are built through information sharing and consensus. Enter the Department of Commerce. In addition to administering the National Institute of Standards and Technology, they have a strong history of working across agencies, such as with the CHIPS Act.

Similar arguments can also be made for the Treasury, and it may even be possible to have Treasury and Commerce work together to manage an SIF. They would be responsible for bringing in subject matter experts (for example, from the Department of Energy or National Science Foundation) to provide specific inputs and arguments for why specific technologies need government-based commercialization funding and at what point such funding is appropriate, acting as an honest broker to allocate strategic capital.

To be clear: The SIF is not intended to supersede any existing funding programs (e.g., the Department of Energy’s Loan Program Office or the National Institute of Health’s ARPA-H) that provide fit-for-purpose funding to specific sectors. Rather, an SIF is intended to fill in the gaps and coordinate with existing programs while providing more creative financing structures than are typically available from government programs.

Recommendation 3. Create a clear innovation roadmap.

Every two years, the SIF should develop or update a roadmap of strategically important industries, working closely with private, nonprofit, and academic experts to define key technological and capability gaps that merit public sector investment. 

The SIF’s leaders should be empowered to make decisions on areas to prioritize but have the ability to change and adapt as the economic environment evolves. Although there is a long list of industries that an SIF could potentially support, resources are not infinite. However, a critical mass of investment is required to ensure adequate resourcing. One acute challenge is that this is not perfectly known in advance and changes depending on the technology and sector. However, this is precisely what the strategic investment roadmap is supposed to solve for: It should provide an even-handed assessment of the likely capital requirements and where the SIF is best suited to provide funding compared to other agencies or the private sector.

Moreover, given the ever-changing nature of technology, the SIF should frequently reassess its understanding of key use cases and their broader economic and strategic importance. Thus, after initial development of the SIF, it should be updated every two years to ensure that its takeaways and priorities remain relevant. This is no different than documents such as the National Security Strategy, which are updated every two to four years; in fact, the SIF’s planning documents should flow seamlessly into the National Security Strategy. 

To provide a sufficiently broad set of perspectives, the government should include the expertise and insights of outside experts to develop its plan. Existing bodies, such as the President’s Council of Advisors on Science and Technology and the National Quantum Initiative, provide some of the consultative expertise required. However, the SIF should also stand up subject matter specific advisory bodies where a need arises (for example, on critical minerals and mining) and work internally to set specific investment areas and priorities.

Recommendation 4. Limit the SIF to financing.

The government should not be an outsized player in capital markets. As such, the SIF should receive no governance rights (e.g., voting or board seats) in the companies that it invests in.

Although the SIF aims to catalyze technological and ecosystem development, it should be careful not to dictate the future of specific companies. Thus, the SIF should avoid information rights beyond financial reporting. Typical board decks and stockholder updates include updates on customers, technologies, personnel matters, and other highly confidential and specific pieces of information that, if made public through government channels, would play a highly distortionary role in markets. Given that the SIF is primarily focused on supporting innovation through a particularly tricky stage to navigate, the SIF should receive the least amount of information possible to avoid disrupting markets.

Recommendation 5. Focus on providing first-loss capital.

First-loss capital should be the primary mechanism by which the SIF supports new technologies, providing greater incentives for private-sector funders to support early commercialization while providing a means for taxpayers to directly participate in the economic upside of SIF-supported technologies.

Consider the following stylized example to demonstrate a key issue in the valley of death. A promising clean technology company, such as a carbon-free cement or long-duration energy storage firm, is raising $100mm of capital for facility expansion and first commercial deployment. To date, the company has likely raised $30 – $50mm of venture capital to enable tech development, pilot the product, and grow the team’s engineering, R&D, and sales departments.

However, this company faces a fundraising dilemma. Its funding requirements are now too big for all but the largest venture capital firms, who may or may not want to invest in projects and companies like these. On the other hand, this hypothetical company is not mature enough for private equity buyouts nor is it a good candidate for typical project-based debt, which typically require several commercial proof points in order to provide sufficient risk reduction for an investor whose upside is relatively limited. Hence, the “valley of death.”

First-loss capital is an elegant solution to this issue: A prospective funder could commit to equal (pro rata) terms as other investors, except that this first-loss funder is willing to use its investment to make other investors whole (or at least partially offset losses) in the event that the project or company does not succeed. In this example, a first-loss funder would commit to $33.5 million of equity funding (roughly one-third of the company’s capital requirement). If the company succeeds, the first-loss funder makes the same returns as the other investors. However, if the company is unable to fully meet these obligations, the first-loss funder’s $33.5 million would be used to pay the other investors back (the other $66.5 million that was committed). This creates a floor on losses for the non-first-loss investors: Rather than being at risk of losing 100% of their principal, they are at risk of losing 50% of their principal.

The creation of a first-loss layer has a meaningful impact on the risk-reward profile for non-first-loss investors, who now have a floor on returns (in the case above, half their investment). By expanding the acceptable potential loss ratio, growth equity capital (or another appropriate instrument, such as project finance) can fill the rest, thereby crowding in capital. 

From a risk-adjusted returns standpoint, this is not a free lunch for the government or taxpayers. Rather, it is intended to be a capital-efficient way of supporting the private-sector ecosystem in developing strategically and economically vital technologies. In other words, it leverages the power of the private sector to solve externalities while providing just enough support to get them to the starting line in the first place.

Conclusion

Many of tomorrow’s strategically important technologies face critical funding challenges in the valley of death. Due to their capital intensity and uncertain outcomes, existing financing tools are largely falling short in the critical early commercialization phases. However, a nimble, properly funded SIF could bridge key gaps while allowing the private sector to do most of the heavy lifting. The SIF would require buy-in from many stakeholders and well-defined sources of funding, but these can be solved with the right mandates, structures, and pay-fors. Indeed, the stakes are too high, and the consequences too dire, to not get strategic innovation right in the 21st century.

This action-ready policy memo is part of Day One 2025 — our effort to bring forward bold policy ideas, grounded in science and evidence, that can tackle the country’s biggest challenges and bring us closer to the prosperous, equitable and safe future that we all hope for whoever takes office in 2025 and beyond.

Frequently Asked Questions
If first-loss capital is such a powerful mechanism, why doesn’t the market already provide it?

Put simply, there needs to be an entity that is actually willing and able to absorb lower returns, or even lose some of its principal, in the service of building an ecosystem. Even if the “median” outcome is a market-rate return of capital, the risk-adjusted returns are in effect far lower because the probability of a zero outcome for first-loss providers is substantially nonzero. Moreover, it’s not clear exactly what the right probability estimate should be; therefore, it requires a leap of faith that no economically self-interested private market actor would be willing to take. While some quasi-social-sector organizations can play this role (for example, Bill Gates’s Breakthrough Energy Ventures for climate tech), their capacity is finite, nor is there a guarantee that such a vehicle will appear for every sector of interest. Therefore, a publicly funded SIF is an integral solution to bridging the valley of death.

Does the SIF always have to use first-loss structures?

No, the SIF would not always have to use first-loss structures. However, it is the most differentiated structure that is available to the U.S. government; otherwise, a private-sector player is likely able—and better positioned—to provide funding.

What type of financial instruments (e.g., debt, equity) would the SIF use?

The SIF should be able to use the full range of instruments, including project finance, corporate debt, convertible loans, and equity capital, and all combinations thereof. The instrument of choice should be up to the judgment of the applicant and SIF investment team. This is distinct from providing first-loss capital: Regardless of the financial instrument used, the SIF’s investment would be used to buffer other investors against potential losses.

What is the target rate of return?

The target return rate should be commensurate with that of the instrument used. For example, mezzanine debt should target 13–18% IRR, while equity investments should aim for 20–25% IRR. However, because of the increased risk of capital loss to the SIF given its first loss position, the effective blended return should be expected to be lower. 


The SIF should be prepared to lose capital on each individual investment and, as a blended portfolio, to have negative returns. While it should underwrite such that it will achieve market-rate returns if successful in crowding in other capital that improves the commercial prospects of technologies and companies in the valley of death, the SIF has a public goal of ecosystem development for strategic domains. Therefore, lower-than-market-rate returns, and even some principal degradation, is acceptable but should be avoided as much as possible through the prudence of the investment committee.

What protections will the SIF be offered, if they do not get board seats or representation?

By and large, the necessary public protections are granted through CFIUS, which requires regulatory approval for export or foreign ownership stakes with voting rights above 25% of critical technologies. The SIF can also enact controls around information rights (e.g., customer lists, revenue, product roadmaps) such that they have a veto on parties that can receive such information. However, given its catalytic mission, the SIF does not need board seats or representation and should focus on ensuring that critical technologies and assets are properly protected.

What is the overall investment committee composition, and what are the processes for approval?

In most private investment firms, the investment committee is made up of the most senior individuals in the fund. These individuals can cross asset classes, sectors of expertise, and even functional backgrounds. However, the investment committee represents a wide breadth of expertise and experiences that, when brought together, enable intellectual honesty and the application of collective wisdom and judgment to the opportunity at hand.


Similarly, the SIF’s investment committee could include the head of the fund and representatives from various departments and agencies in alignment with its strategic priorities. The exact size of the investment committee should be defined by these priorities, but approval should be driven by consensus, and unanimity (or near unanimity) should be expected for investments that are approved.


Given the fluid nature of investment opportunities, the committee should be called upon whenever needed to evaluate a potential opportunity. However, given the generally long process times for investments discussed above (6–12 months), the investment committee should have been briefed multiple times before a formal decision is made.

What check size is envisioned? Is there a strict cutoff for the amount of funding that can be disbursed?

Check sizes can be flexible to the needs of the investment opportunity. However, as an initial guiding principle, first loss capital should likely make up 20–35% of capital invested so as to require private-sector investors to have meaningful skin in the game. Depending on the fundraise size, this could imply investments of $25 million to $100 million.


Target funding amounts should be set over multiyear timeframes, but the annual appropriations process implies that there will likely be a set cap in any given year. In order to meet the needs of the market, there should be mechanisms that enable emergency draws, up to a cap (e.g., 10% of the annual target funding amount, which will need to be “paid for” by reducing future outlays). 

How will the SIF be funded?

An economically efficient way to fund a government program in support of a positive externality is a Pigouvian tax on negative externalities (such as carbon). However, carbon taxes are as politically unappealing as they are economically sensible and need to be packaged into other policy goals that could potentially support such legislation. Notwithstanding the questionable economic wisdom of tariffs in general, some 56% of voters support a 10% tax on all imports and 60% tariffs on China. Rather than using tariffs harmfully, they could be used more productively. One such proposal is a carbon import tariff that taxes imports on the carbon emitted in the production and transportation of goods into the U.S.


The U.S. would not be a first mover: in fact, the European Union has already implemented a similar mechanism called the Carbon Border Adjustment Mechanism (CBAM), which is focused on heavy industry, including cement, iron and steel, aluminum, fertilizers, electricity, and hydrogen, with chemicals and polymers potentially to be included after 2026. At full rollout in 2030, the CBAM is expected to generate roughly €10–15 billion of tax revenue. Tax receipts of a similar size could be used to fund an SIF or, if Congress authorizes an upfront amount, could be used to nullify the incremental deficit over time.

If the carbon innovation fee is implemented, how would levies be assessed and exemptions provided?

The EU’s CBAM phased in its reporting requirements over several years. Through July 2024, companies were allowed to use default amounts per unit of production without an explanation as to why actual data was not used. Until January 1, 2026, companies can make estimates for up to 20% of goods; thereafter, the CBAM requires reporting of actual quantities and embedded greenhouse gas emissions.


The U.S. could use a similar phase-in, although given the challenges of carbon reporting, could allow companies to use the lower of actual, verified emissions or per-unit estimates. Under a carbon innovation fee regime, exporters and countries could apply for exemption on a case-by-case basis to the Department of Commerce, which they could approve in line with other goals (e.g., economic development in a region).

Besides a carbon innovation fee, what are other ways to fund the SIF?

The SIF could also be funded by repurposing other funding and elevating their strategic importance. Potential candidates include the Small Business Innovation Research (SBIR) and Small State Business Credit Initiative (SSBCI), which could play a bigger role if moved into the SIF umbrella. For example, the SBIR program, whose latest reporting data is as of FY2019, awarded $3.3 billion in funding that year and $54.6 billion over its lifespan. Moreover, the SSBCI, a $10 billion fund that already provides loan guarantees and other instruments similar to those described above, can be used to support technologies that fall into the purview of the SIF.

What funding options are available if reallocation of existing funds isn’t an option?

Congress could also assess reallocating dollars towards an SIF from spending reforms that are likely inevitable given the country’s fiscal position. In 2023, the Congressional Budget Office (CBO) published a report highlighting potential solutions for reducing the budget deficit. Some potential solutions, like establishing caps on Medicaid federal spending, while fiscally promising, seem unlikely to pass in the near future. However, others are more palatable, especially those that eliminate loopholes or ask higher-income individuals to pay their fair share.


For instance, increasing the amount subject to Social Security taxes above the $250,000 threshold has the potential to raise up to $1.2 trillion over 10 years; while this can be calibrated, an SIF would take only a small fraction of the taxes raised. In addition, the CBO found that federal matching funds for Medicaid frequently ended up getting back to healthcare providers in the form of higher reimbursement rates; eliminating what are effectively kickbacks could reduce the deficit by up to $525 billion over 10 years.

Promoting Fairness in Medical Innovation

There is a crisis within healthcare technology research and development, wherein certain groups due to their age, gender, or race and ethnicity are under-researched in preclinical studies, under-represented in clinical trials, misunderstood by clinical practitioners, and harmed by biased medical technology. These issues in turn contribute to costly disparities in healthcare outcomes, leading to losses of $93 billion a year in excess medical-care costs, $42 billion a year in lost productivity, and $175 billion a year due to premature deaths. With the rise of artificial intelligence (AI) in healthcare, there’s a risk of encoding and recreating existing biases at scale.

The next Administration and Congress must act to address bias in medical technology at the development, testing and regulation, and market-deployment and evaluation phases. This will require coordinated effort across multiple agencies. In the development phase, science funding agencies should enforce mandatory subgroup analysis for diverse populations, expand funding for under-resourced research areas, and deploy targeted market-shaping mechanisms to incentivize fair technology. In the testing and regulation phase, the FDA should raise the threshold for evaluation of medical technologies and algorithms and expand data-auditing processes. In the market-deployment and evaluation phases, infrastructure should be developed to perform impact assessments of deployed technologies and government procurement should incentivize technologies that improve health outcomes.

Challenge and Opportunity

Bias is regrettably endemic in medical innovation. Drugs are incorrectly dosed to people assigned female at birth due to historical exclusion of women from clinical trials. Medical algorithms make healthcare decisions based on biased health data, clinically disputed race-based corrections, and/or model choices that exacerbate healthcare disparities. Much medical equipment is not accessible, thus violating the Americans with Disabilities Act. And drugs, devices, and algorithms are not designed with the lifespan in mind, impacting both children and the elderly. Biased studies, technology, and equipment inevitably produce disparate outcomes in U.S. healthcare.

The problem of bias in medical innovation manifests in multiple ways: cutting across technological sectors in clinical trials, pervading the commercialization pipeline, and impeding equitable access to critical healthcare advances.

Bias in medical innovation starts with clinical research and trials

The 1993 National Institutes of Health (NIH) Revitalization Act required federally funded clinical studies to (i) include women and racial minorities as participants, and (ii) break down results by sex and race or ethnicity. As of 2019, the NIH also requires inclusion of participants across the lifespan, including children and older adults. Yet a 2019 study found that only 13.4% of NIH-funded trials performed the mandatory subgroup analysis, and challenges in meeting diversity targets continue into 2024 . Moreover, the increasing share of industry-funded studies are not subject to Revitalization Act mandates for subgroup analysis. These studies frequently fail to report differences in outcomes by patient population as a result. New requirements for Diversity Action Plans (DAPs), mandated under the 2023 Food and Drug Omnibus Reform Act, will ensure drug and device sponsors think about enrollment of diverse populations in clinical trials. Yet, the FDA can still approve drugs and devices that are not in compliance with their proposed DAPs, raising questions around weak enforcement. 

The resulting disparities in clinical-trial representation are stark: African Americans represent 12% of the U.S. population but only 5% of clinical-trial participants, Hispanics make up 16% of the population but only 1% of clinical trial participants, and sex distribution in some trials is 67% male. Finally, many medical technologies approved prior to 1993 have not been reassessed for potential bias. One outcome of such inequitable representation is evident in drug dosing protocols: sex-aware prescribing guidelines exist for only a third of all drugs.

Bias in medical innovation is further perpetuated by weak regulation

Algorithms

Regulation of medical algorithms varies based on end application, as defined in the 21st Century Cures Act. Only algorithms that (i) acquire and analyze medical data and (ii) could have adverse outcomes are subject to FDA regulation. Thus, clinical decision-support software (CDS) is not regulated even though these technologies make important clinical decisions in 90% of U.S. hospitals. The FDA has taken steps to try and clarify what CDS must be considered a medical device, although these actions have been heavily criticized by industry. Finally, the lack of regulatory frameworks for generative AI tools is leading to proliferation without oversight.

Even when a medical algorithm is regulated, regulation may occur through relatively permissive de novo pathways and 510(k) pathways. A de novo pathway is used for novel devices determined to be low to moderate risk, and thus subject to a lower burden of proof with respect to safety and equity. A 510(k) pathway can be used to approve a medical device exhibiting “substantial equivalence” to a previously approved device, i.e., it has the same intended use and/or same technological features. Different technical features can be approved so long as there are no questions raised around safety and effectiveness.

Medical algorithms approved through de novo pathways can be used as predicates for approval of devices through 510(k) pathways. Moreover, a device approved through a 510(k) pathway can remain on the market even if its predicate device was recalled. Widespread use of 510(k) approval pathways has generated a “collapsing building” phenomenon, wherein many technologies currently in use are based on failed predecessors. Indeed, 97% of devices recalled between 2008 to 2017 were approved via 510(k) clearance. 

While DAP implementation will likely improve these numbers, for the 692 AI-ML enabled medical devices, only 3.6% reported race or ethnicity, 18.4% reported age, and only .9% include any socioeconomic information. Further, less than half did detailed analysis of algorithmic performance and only 9% included information on post-market studies, raising the risk of algorithmic bias following approvals and broad commercialization.

Even more alarming is evidence showing that machine learning can further entrench medical inequities. Because machine learning medical algorithms are powered by data from past medical decision-making, which is rife with human error, these algorithms can perpetuate racial, gender, and economic bias. Even algorithms demonstrated to be ‘unbiased’ at the time of approval can evolve in biased ways over time, with little to no oversight from the FDA. As technological innovation progresses, especially generative AI tools, an intentional focus on this problem will be required.

Medical devices

Currently, the Medical Device User Fee Act requires the FDA to consider the least burdensome appropriate means for manufacturers to demonstrate the effectiveness of a medical device or to demonstrate a device’s substantial equivalence. This requirement was reinforced by the 21st Century Cures Act, which also designated a category for “breakthrough devices” subject to far less-stringent data requirements. Such legislation shifts the burden of clinical data collection to physicians and researchers, who might discover bias years after FDA approval. This legislation also makes it difficult to require assessments on the differential impacts of technology.

Like medical algorithms, many medical devices are approved through 510(k) exemptions or de novo pathways. The FDA has taken steps since 2018 to increase requirements for 510(k) approval and ensure that Class III (high-risk) medical devices are subject to rigorous pre-market approval, but problems posed by equivalence and limited diversity requirements remain. 

Finally, while DAPs will be required for many devices seeking FDA approval, the recommended number of patients in device testing is shockingly low. For example, currently, only 10 people are required in a study of any new pulse oximeter’s efficacy and only 2 of those people need to be “darkly pigmented”. This requirement (i) does not have the statistical power necessary to detect differences between demographic groups, and (i) does not represent the composition of the U.S. population. The standard is currently under revision after immense external pressure. FDA-wide, there are no recommended guidelines for addressing human differences in device design, such as pigmentation, body size, age, and pre-existing conditions.

Pharmaceuticals

The 1993 Revitalization Act strictly governs clinical trials for pharmaceuticals and does not make recommendations for adequate sex or genetic diversity in preclinical research. The results are that a disproportionately high number of male animals are used in research and that only 5% of cell lines used for pharmaceutical research are of African descent. Programs like All of Us, an effort to build diverse health databases through data collection, are promising steps towards improving equity and representation in pharmaceutical research and development (R&D). But stronger enforcement is needed to ensure that preclinical data (which informs function in clinical trials) reflects the diversity of our nation. 

Bias in medical innovation are not tracked post-regulatory approval

FDA-regulated medical technologies appear trustworthy to clinicians, where the approval signals safety and effectiveness. So, when errors or biases occur (if they are even noticed), the practitioner may blame the patient for their lifestyle rather than the technology used for assessment. This in turn leads to worse clinical outcomes as a result of the care received.

Bias in pulse oximetry is the perfect case study of a well-trusted technology leading to significant patient harm. During the COVID-19 pandemic, many clinicians and patients were using oximeter technology for the first time and were not trained to spot factors, like melanin in the skin, that cause inaccurate measurements and impact patient care. Issues were largely not attributed to the device. This then leads to underreporting of adverse events to the FDA — which is already a problem due to the voluntary nature of adverse-event reporting. 

Even when problems are ultimately identified, the federal government is slow to respond. The pulse oximeter’s limitations in monitoring oxygenation levels across diverse skin tones was identified as early as the 1990s. 34 years later, despite repeated follow-up studies indicating biases, no manufacturer has incorporated skin-tone-adjusted calibration algorithms into pulse oximeters. It required the large Sjoding study, and the media coverage it garnered around delayed care and unnecessary deaths, for the FDA to issue a safety communication and begin reviewing the regulation.

Other areas of HHS are stepping up to address issues of bias in deployed technologies. A new ruling by the HHS Office of Civil Rights (OCR) on Section 1557 of the Affordable Care Act requires covered providers and institutions (i.e. any receiving federal funding) to identify their use of patient care decision support tools that directly measure race, color, national origin, sex, age, or disability, and to make reasonable efforts to mitigate the risk of discrimination from their use of these tools. Implementation of this rule will depend on OCR’s enforcement, and yet it provides another route to address bias in algorithmic tools.

Differential access to medical innovation is a form of bias

Americans face wildly different levels of access to new medical innovations. As many new innovations have high cost points, these drugs, devices, and algorithms exist outside the price range of many patients, smaller healthcare institutions and federally funded healthcare service providers, including the Veterans Health Administration, federally qualified health centers and the Indian Health Service. Emerging care-delivery strategies might not be covered by Medicare and Medicaid, meaning that patients insured by CMS cannot access the most cutting-edge treatments. Finally, the shift to digital health, spurred by COVID-19, has compromised access to healthcare in rural communities without reliable broadband access. 

Finally, the Advanced Research Projects Agency for Health (ARPA-H) has a commitment to have all programs and projects consider equity in their design. To fulfill ARPA-H’s commitment, there is a need for action to ensure that medical technologies are developed fairly, tested with rigor, deployed safely, and made affordable and accessible to everyone.

Plan of Action

The next Administration should launch “Healthcare Innovation for All Americans” (HIAA), a whole of government initiative to improve health outcomes by ensuring Americans have access to bias-free medical technologies. Through a comprehensive approach that addresses bias in all medical technology sectors, at all stages of the commercialization pipeline, and in all geographies, the initiative will strive to ensure the medical-innovation ecosystem works for all. HIAA should be a joint mandate of Health and Human Services (HHS) and the Office of Science Technology and Policy (OSTP) to work with federal agencies on priorities of equity, non-discrimination per Section 1557 of the Affordable Care Act and increasing access to medical innovation, and initiative leadership should sit at both HHS and OSTP. 

This initiative will require involvement of multiple federal agencies, as summarized in the table below. Additional detail is provided in the subsequent sections describing how the federal government can mitigate bias in the development phase; testing, regulation, and approval phases; and market deployment and evaluation phases.

Three guiding principles should underlie the initiative:

  1. Equity and non-discrimination should drive action. Actions should seek to improve the health of those who have been historically excluded from medical research and development. We should design standards that repair past exclusion and prevent future exclusion. 
  2. Coordination and cooperation are necessary. The executive and legislative branches must collaborate to address the full scope of the problem of bias in medical technology, from federal processes to new regulations. Legislative leadership should task the Government Accountability Office (GAO) to engage in ongoing assessment of progress towards the goal of achieving bias-free and fair medical innovation.
  3. Transparent, evidence-based decision making is paramount. There is abundant peer-reviewed literature that examines bias in drugs, devices, and algorithms used in healthcare settings — this literature should form the basis of a non-discrimination approach to medical innovation. Gaps in evidence should be focused on through deployed research funding. Moreover, as algorithms become ubiquitous in medicine, every effort should be made to ensure that these algorithms are trained on representative data of those experiencing a given healthcare condition.
AgencyRole
Advanced Research Projects Agency for Health (ARPA-H)ARPA-H has committed to tackling health equity in biomedical research, and to aligning each project it undertakes with that goal. As such, ARPA-H should lead the charge in developing processes for equity in medical technology — from idea conceptualization to large-scale rollout — and serve as a model for other federally funded healthcare programs.
National Institute of Health (NIH)NIH should fund research that addresses health-data gaps, investigates algorithmic and data bias, and assesses bias embedded in medical technical tools. Simultaneously, NIH should create standards for diversity in samples and/or datasets for preclinical research. Finally, NIH must strongly enforce the 1993 NIH Revitalization Act’s diversity provisions.
National Science Foundation (NSF)NSF should collaborate with NIH on cross-agency programs that fund R&D specific to mitigating bias of technologies like AI.
Food and Drug Administration (FDA)FDA should take a more active role in uncovering bias in medical innovation, given its role as a regulatory checkpoint for all new medical technologies. This should include more rigorous evaluation protocols as well as better tracking of emergent bias in medical technologies post-approval.
Assistant Secretary for Technology Policy (ASTP)ASTP publishes standards for effective use of healthcare information technology that ensure quality care delivery. Their standards-setting should offer solutions for compliance with Section 1557 for novel AI/ML algorithms.
Centers for Medicare and Medicaid (CMS)CMS oversees the coordination of coverage, coding, and payment processes with respect to new technologies and procedures. Thus, CMS should focus on ensuring all new technologies developed through federal funding, like those that will be built by ARPA-H and its industry partners, are covered by Medicare and Medicaid. In addition, CMS and its accrediting partners can require compliance with federal regulatory standards, which should be extended to assess medical technologies. Finally, CMS should assess how flawed medical technologies are being used to decide on medical care provision, and update guidelines.
Federal Trade Commission (FTC)FTC should protect America’s medical technology consumers by auditing high-risk medical innovations, such as decision-making algorithms.
Agency for Healthcare Research and Quality (AHRQ)The AHRQ, a component of HHS, should identify areas where technology bias is leading to disparate healthcare outcomes and report its findings to Congress, the White House, and agency leaders for immediate action.
Centers for Disease Control and Prevention (CDC)CDC’s expertise in health-data collection should be mobilized to identify research and development gaps.
Department of Commerce (DOC)Given its role in enforcing U.S. trade laws and regulations, DOC can do much to incentivize equity in medical device design and delivery. The National Institute of Standards and Technology (NIST) should play a key role in crafting standards for identifying and managing bias across key medical-technology sectors.
Department of Education (ED)ED should work with medical schools to develop and implement learning standards and curricula on bias in medical technology.
Department of Defense (DOD)DOD has formalized relationships with FDA to expedite medical products useful to American military personnel. As a DOD priority is to expand diversity and inclusion in the armed forces, these medical products should be assessed for bias that limits safety and efficacy.
Health Resources and Services Administration (HRSA)HRSA should coordinate with federally qualified health centers on digital health technologies, taking advantage of the broadband expansion outlined in the Bipartisan Infrastructure Bill.
Veterans Affairs (VA) and the Veterans Health Administration (VHA)The VA should work with ARPA-H and its industry partners to establish cost-effective rollout of new innovations to VA-run hospitals. VA should also use its procurement power to require diversity in the clinical trials of the drugs, devices, and algorithms it procures. VA could also use prize challenges to spur innovation.
Government Accountability Office (GAO)The GAO should prepare a comprehensive roadmap for addressing bias endemic to the cycle of medical technology development, testing, and deployment, with a focus on mitigating bias in “black box” algorithms used in medical technology.
Office of Management and Budget (OMB)OMB should work with HIAA leadership to design a budget for HIAA implementation, including for R&D funding, personnel for programmatic expansion, data collectives, education, and regulatory enforcement.
Office of Science Technology and Policy (OSTP)OSTP should develop processes and standards for ensuring that individual rights are not violated by biased medical technologies. This work can build on the AI Bill of Rights Initiative.

Addressing bias at the development phase

The following actions should be taken to address bias in medical technology at the innovation phase:

Addressing bias at the testing, regulation, and approval phases

The following actions should be taken to address bias in medical innovation at the testing, regulation, and approval phases:

Addressing bias at the market deployment and evaluation phases 

A comprehensive road map is needed

The GAO should conduct a comprehensive investigation of “black box” medical technologies utilizing algorithms that are not transparent to end users, medical providers, and patients. The investigation should inform a national strategic plan for equity and non-discrimination in medical innovation that relies heavily on algorithmic decision-making. The plan should include identification of noteworthy medical technologies leading to differential healthcare outcomes, creation of enforceable regulatory standards, development of new sources of research funding to address knowledge gaps, development of enforcement mechanisms for bias reporting, and ongoing assessment of equity goals.

Timeline for action

Realizing HIAA will require mobilization of federal funding, introduction of regulation and legislation, and coordination of stakeholders from federal agencies, industry, healthcare providers, and researchers around a common goal of mitigating bias in medical technology. Such an initiative will be a multi-year undertaking and require funding to enact R&D expenditures, expand data capacity, assess enforcement impacts, create educational materials, and deploy personnel to staff all the above.

Near-term steps that can be taken to launch HIAA include issuing a public request for information, gathering stakeholders, engaging the public and relevant communities in conversation, and preparing a report outlining the roadmap to accomplishing the policies outlined in this memo.

Conclusion

Medical innovation is central to the delivery of high-quality healthcare in the United States. Ensuring equitable healthcare for all Americans requires ensuring that medical innovation is equitable across all sectors, phases, and geographies. Through a bold and comprehensive initiative, the next Administration can ensure that our nation continues leading the world in medical innovation while crafting a future where healthcare delivery works for all.

This action-ready policy memo is part of Day One 2025 — our effort to bring forward bold policy ideas, grounded in science and evidence, that can tackle the country’s biggest challenges and bring us closer to the prosperous, equitable and safe future that we all hope for whoever takes office in 2025 and beyond.

Frequently Asked Questions
How will the success of HIAA be evaluated?

HIAA will be successful when medical policies, projects, and technologies yield equitable health care access, treatment, and outcomes. For instance, success would yield the following outcomes:



  • Representation in preclinical and clinical research equivalent to the incidence of a studied condition in the general population.

  • Research on a disease condition funded equally per affected patient.

  • Existence of data for all populations facing a given disease condition.

  • Medical algorithms that have equal efficacy across subgroup populations.

  • Technologies that work equally well in testing as they do when deployed to the market.

  • Healthcare technologies made available and affordable to all care facilities.

Why does this memo propose an expansive multi-agency effort instead of just targeting the FDA?

Regulation alone cannot close the disparity gap. There are notable gaps in preclinical and clinical research data for women, people of color, and other historically underrepresented groups that need to be filled. There are also historical biases encoded in AI/ML decision making algorithms that need to be studied and rectified. In addition, the FDA’s role is to serve as a safety check on new technologies — the agency has limited oversight over technologies once they are out on the market due to the voluntary nature of adverse reporting mechanisms. This means that agencies like the FTC and CMS need to be mobilized to audit high-risk technologies once they reach the market. Eliminating bias in medical technology is only possible through coordination and cooperation of federal agencies with each other as well as with partners in the medical device industry, the pharmaceutical industry, academic research, and medical care delivery.

What challenges might the Administration encounter from industry in launching this initiative?

A significant focus of the medical device and pharmaceutical industries is reducing the time to market for new medical devices and drugs. Imposing additional requirements for subgroup analysis and equitable use as part of the approval process could work against this objective. On the other hand, ensuring equitable use during the development and approval stages of commercialization will ultimately be less costly than dealing with a future recall or a loss of Medicare or Medicaid eligibility if discriminatory outcomes are discovered.

Is there bipartisan support to secure the funding for this initiative?

Healthcare disparities exist in every state in America and are costing billions a year in economic growth. Some of the most vulnerable people live in rural areas, where they are less likely to receive high-quality care because costs of new medical technologies are too high for the federally qualified health centers that serve one in five rural residents as well as rural hospitals. Furthermore, during continued use, a biased device creates adverse healthcare outcomes that cost taxpayers money. A technology functioning poorly due to bias can be expensive to replace. It is economically imperative to ensure technology works as expected, as it leads to more effective healthcare and thus healthier people.

Scaling Effective Methods across Federal Agencies: Looking Back at the Expanded Use of Incentive Prizes between 2010-2020

Policy entrepreneurs inside and outside of government, as well as other stakeholders and advocates, are often interested in expanding the use of effective methods across many or all federal agencies, because how the government accomplishes its mission is integral to what the government is able to produce in terms of outcomes for the public it serves. Adoption and use of promising new methods by federal agencies can be slowed by a number of factors that discourage risk-taking and experimentation, and instead encourage compliance and standardization, too often as a false proxy for accountability. As a result, many agency-specific and government-wide authorities for promising methods go under-considered and under-utilized. 

Policy entrepreneurs within center-of-government agencies (e.g., Executive Office of the President) are well-positioned to use a variety of policy levers and actions to encourage and accelerate federal agency adoption of promising and effective methods. Some interventions by center-of-government agencies are better suited to driving initial adoption, others to accelerating or maintaining momentum, and yet others to codifying and making adoption durable once widespread. Therefore, a policy entrepreneur interested in expanding adoption of a given method should first seek to understand the “adoption maturity” of that method and then undertake interventions appropriate for that stage of adoption. The arc of agency adoption of new methods can be long—measured in years and decades, not weeks and months. Policy entrepreneurs should be prepared to support adoption over similar timescales. In considering adoption maturity of a method of interest, policy entrepreneurs can also reference the ideas of Tom Kalil in a July 2024 Federation of American Scientists blog post on “Increasing the ‘Policy Readiness of Ideas,” which offers sample questions to ask about “the policy landscape surrounding a particular idea.”

As a case study for driving federal adoption of a new method, this paper looks back at actions that supported the widespread adoption of incentive prizes by most federal agencies over the course of fiscal years 2010 through 2020. Federal agency use of prizes increased from several incentive prize competitions being offered by a handful of agencies in the early 2000s to more than 2,000 prize competitions offered by over 100 federal agencies by the end of fiscal year 2022. These incentive prize competitions have helped federal agencies identify novel solutions and technologies, establish new industry benchmarks, pay only for results, and engage new talent and organizations. 

A summary framework below includes types of actions that can be taken by policy entrepreneurs within center-of-government agencies to support awareness, piloting, and ongoing use of new methods by federal agencies in the years ahead. (Federal agency program and project managers who seek to scale up innovative methods within their agencies are encouraged to reference related resources such as this article by Jenn Gustetic in the Winter 2018 Issues in Science and Technology: “Scaling Up Policy Innovations in the Federal Government: Lessons from the Trenches.”) 

Efforts to expand federal capacity through new and promising methods are worthwhile to ensure the federal government can use a full and robust toolbox of tactics to meet its varied goals and missions. 

OPPORTUNITIES AND CHALLENGES IN FEDERAL ADOPTION OF NEW METHODS

Opportunities for federal adoption and use of promising and effective methods

To address national priorities, solve tough challenges, or better meet federal missions to serve the public, a policy entrepreneur may aim to pilot, scale, and make lasting federal use of a specific method. 

A policy entrepreneur’s goals might include new ways for federal agencies to, for example:

To support these and other goals, an array of promising methods exist and have been demonstrated, such as in other sectors like philanthropy, industry, and civil society, in state, local, Tribal, or territorial governments and communities, or in one or several federal agencies—with promise for beneficial impact if more federal agencies adopted these practices. Many methods are either specifically supported or generally allowable under existing government-wide or agency-specific authorities. 

Center-of-government agencies include components of the Executive Office of the President (EOP) like the Office of Management and Budget (OMB) and the Office of Science and Technology Policy (OSTP), as well as the Office of Personnel Management (OPM) and the General Services Administration (GSA). These agencies direct, guide, convene, support, and influence the implementation of law, regulation, and the President’s policies across all Federal agencies, especially the executive departments. An August 2016 report by the Partnership for Public Service and the IBM Center for the Business of Government noted that, “The Office of Management and Budget and other “center of government” agencies are often viewed as adding processes that inhibit positive change—however, they can also drive innovation forward across the government.”

A policy entrepreneur interested in expanding adoption of a given method through actions driven or coordinated by one or more center-of-government agencies should first seek to understand the “adoption maturity” of a given method of interest by assessing: (1) the extent that adoption of the method has already occurred across the federal interagency; (2) any real or perceived barriers to adoption and use; and (3) the robustness of existing policy frameworks and agency-specific and government-wide infrastructure and resources that support agency use of the method.

Challenges in federal adoption and use of new methods

Policy entrepreneurs are usually interested in expanding federal adoption of new methods for good reason: a focus on supporting and expanding beneficial outcomes. Effective leaders and managers across sectors understand the importance of matching appropriate and creative tactics with well-defined problems and opportunities. Ideally, leaders are picking which tactic or tool to use based on their expert understanding of the target problem or opportunity, not using a method solely because it is novel or because it is the way work has always been done in the past. Design of effective program strategies is supported by access to a robust and well-stocked toolbox of tactics. 

However, many currently authorized and allowable methods for achieving federal goals are generally underutilized in the implementation strategies and day-to-day tactics of federal agencies. Looking at the wide variety of existing authorities in law and the various flexibilities allowed for in regulation and guidance, one might expect agency tactics for common activities like acquisition or public comment to be varied, diverse, iterative, and even experimental in nature, where appropriate. In practice, however, agency methods are often remarkably homogeneous, repeated, and standardized.   

This underutilization of existing authorities and allowable flexibilities is due to factors such as:

Strategies for addressing challenges in federal adoption and use of new methods

Attention and action by center-of-government agencies often is needed to address the factors cited above that slow the adoption and use of new methods across federal agencies and to build momentum. The following strategies are further explored in the case study on federal use of incentive prizes that follows: 

Additional strategies can be deployed within federal agencies to address agency-level barriers and scale promising methods—see, for example, this article by Jenn Gustetic in the Winter 2018 Issues in Science and Technology: “Scaling Up Policy Innovations in the Federal Government: Lessons from the Trenches.” 

LOOKING BACK: A DECADE OF POLICY ACTIONS SUPPORTING EXPANDED FEDERAL USE OF INCENTIVE PRIZES

The use of incentive prizes is one method for open innovation that has been adopted broadly by most federal agencies, with extensive bipartisan support in Congress and with White House engagement across multiple administrations. In contrast to recognition prizes, such as the Nobel Prize or various presidential medals, which reward past accomplishments, incentive prizes specify a target, establish a judging process (ideally as objective as possible), and use a monetary prize purse and/or non-monetary incentives (such as media and online recognition, access to development and commercialization facilities, resources, or experts, or even qualification for certain regulatory flexibility) to induce new efforts by solvers competing for the prize. 

The use of incentive prizes by governments (and by high net worth individuals) to catalyze novel solutions certainly is not new. In 1795, Napoleon offered 12,000 francs to improve upon the prevailing food preservation methods of the time, with a goal of better feeding his army. Fifteen years later, confectioner Nicolas François Appert claimed the prize for his method involving heating, boiling and sealing food in airtight glass jars — the same basic technology still used to can foods. Dava Sobel’s book Longitude details how the rulers of Spain, the Netherlands, and Britain all offered separate prizes, starting in 1567, for methods of figuring out longitude at sea, and finally John Harrison was awarded Britain’s top longitude prize in 1773. In 1919, Raymond Orteig, a French-American hotelier, aviation enthusiast, and philanthropist, offered a $25,000 prize for the first person who could perform a nonstop flight between New York and Paris. The prize offer initially expired by 1924 without anyone claiming it. Given technological advances and a number of engaged pilots involved in trying to win the prize, Orteig extended the deadline by 5 years. By 1926, nine teams had come forward to formally compete, and the prize went to a little-known aviator named Charles Lindbergh, who attempted the flight in a custom-built plane known as the “Spirit of St. Louis.”

The U.S. Government did not begin to adopt the use of incentive prizes until the early 21st century, following a 1999 National Academy of Engineering workshop about the use of prizes as an innovation tool. In the first decade of the 2000s, the Defense Advanced Research Projects Agency (DARPA), the National Aeronautics and Space Administration (NASA), and the Department of Energy conducted a small number of pilot prize competitions. These early agency-led prizes focused on autonomous vehicles, space exploration, and energy efficiency, demonstrating a range of benefits to federal agency missions. 

Federal use of incentive prizes did not accelerate until, in the America COMPETES Reauthorization Act of 2010, Congress granted all federal agencies the authority to conduct prize competitions (15 USC § 3719). With that new authority in place, and with the support of a variety of other policy actions, federal use of incentive prizes reached scale, with over 2,000 prize competitions offered on Challenge.gov by over 100 federal agencies between the fiscal years 2010 and 2022. 

There certainly remains extensive opportunity to improve the design, rigor, ambition, and effectiveness of federal prize competitions. That said, there are informative lessons to be drawn from how incentive prizes evolved in the United States from a method used primarily outside of government, with limited pilots among a handful of early-adopter federal agencies, to a method being tried by many civil servants across an active interagency community of practice and lauded by administration leaders, bipartisan members of Congress, and external stakeholders alike. 

A summary follows of the strategies and tactics used by policy entrepreneurs within the EOP—with support and engagement from Congress as well as program managers and legal staff across federal agencies—that led to increased adoption and use of incentive prizes in the federal government.

role of philanthropy

Summary of strategies and policy levers supporting expanded use of incentive prizes

In considering how best to expand awareness, adoption, and use among federal agencies of promising methods, policy entrepreneurs might consider utilizing some or all of the strategies and policy levers described below in the incentive prizes example. Those strategies and levers are summarized generally in the table that follows. Some of the listed levers can advance multiple strategies and goals. This framework is intended to be flexible and to spark brainstorming among policy entrepreneurs, as they build momentum in the use of particular innovation methods. 

Policy entrepreneurs are advised to consider and monitor the maturity level of federal awareness, adoption, and use, and to adjust their strategies and tactics accordingly. They are encouraged to return to earlier strategies and policy levers as needed, should adoption and momentum lag, should agency ambition in design and implementation of initiatives be insufficient, or should concerns regarding risk management be raised by agencies, Congress, or stakeholders. 

Stage of Federal AdoptionStrategyTypes of Center-of-Government Policy Levers
Early – No or few Federal agencies using methodUnderstand federal opportunities to use method, and identify barriers and challenges* Connect with early adopters across federal agencies to understand use of agency-specific authorities, identify pain points and lessons learned, and capture case studies (e.g., 2000-2009)

* Engage stakeholder community of contractors, experts, researchers, and philanthropy

* Look to and learn from use of method in other sectors (such as by philanthropy, industry, or academia) and document (or encourage third-party documentation of) that use and its known benefits and attributes (e.g., April 1999, July 2009)

* Encourage research, analysis, reports, and evidence-building by National Academies, academia, think tanks, and other stakeholders (e.g., April 1999, July 2009, June 2014)

* Discuss method with OMB Office of General Counsel and other relevant agency counsel

* Discuss method with relevant Congressional authorizing committee staff

* Host convenings that connect interested federal agency representatives with experts

* Support and connect nascent federal “community of interest”
Early – No or few Federal agencies using methodBuild interest among federal agencies* Designate primary policy point of contact/dedicated staff member in the EOP (e.g., 2009-2017, 2017-2021)

* Designate a primary implementation point of contact/dedicated staff at GSA and/or OPM

* Identify leads in all or certain federal agencies

* Connect topic to other administration policy agendas and strategies

* Highlight early adopters within agencies in communications from center-of-government agencies to other federal agencies (and to external audiences)

* Offer congressional briefings and foster bipartisan collaboration (e.g., 2015)
Early – No or few Federal agencies using methodEstablish legal authorities and general administration policy * Engage OMB Office of OMB General Counsel and OMB Legislative Review Division, as well as other relevant OMB offices and EOP policy councils

* Identify existing general authorities and regulations that could support federal agency use of method (e.g., March 2010)

* Establish general policy guidelines, including by leveraging Presidential authorities through executive orders or memoranda (e.g., January 2009)

* Issue OMB directives on specific follow-on agency actions or guidance to support agency implementation (“M-Memos” or similar) (e.g., December 2009, March 2010, August 2011, March 2012)

* Provide technical assistance to Congress regarding government-wide or agency-specific authority (or authorities) (e.g., June-July 2010, January 2011)

* Delegate existing authorities within agencies (e.g., October 2011)

* Encourage issuance of agency-specific guidance (e.g., October 2011, February 2014)

* Include direction to agencies as part of broader Administration policy agendas (e.g., September 2009, 2011-2016)
Early – No or few Federal agencies using methodRemove barriers and “make it easier”* Create a central government website with information for federal agency practitioners (such as toolkits, case studies, and trainings) and for the public (e.g., September 2010)

* Create dedicated GSA schedule of vendors (e.g., July 2011)

* Establish an interagency center of excellence (e.g., September 2011)

* Encourage use of interagency agreements on design or implementation of pilot initiatives (e.g., September 2011)

* Request agency budget submissions to OMB to support pilot use in President’s budget (e.g., December 2013)
Adoption well underway – Many federal agencies have begun to use methodConnect practitioners* Launch a federal “community of practice” with support from GSA for meetings, listserv, and collaborative projects (e.g., April 2010, 2016, June 2019)

* Host regular events, workshops, and conferences with federal agency and, where appropriate and allowable, seek philanthropic or nonprofit co-hosts (e.g., April 2010, June 2012, April 2015, March 2018, May 2022)
Adoption well underway – Many federal agencies have begun to use methodStrengthen agency infrastructure* Foster leadership buy-in through briefings from White House/EOP to agency leadership, including members of the career senior executive service

* Encourage agencies to dedicate agency staff and invest in prize design support within agencies

* Encourage agencies to create contract vehicles as needed to support collaboration with vendors/ experts

* Encourage agencies to develop intra-agency networks of practitioners and to provide external communications support and platforms for outreach

* Request agency budget submissions to OMB for investments in agency infrastructure and expansion of use, to include in the President's budget where needed (e.g., 2012-2013), and request agencies otherwise accommodate lower-dollar support (such as allocation of FTEs) where possible within their budget toplines
Adoption well underway – Many federal agencies have begun to use methodClarify existing policies and authorities* Issue updated OMB, OSTP, or agency-specific policy guidance and memoranda as needed based on engagement with agencies and stakeholders (e.g.,: August 2011, March 2012)

* Provide technical assistance to Congress on any needed updates to government-wide or agency-specific authorities (e.g., January 2017)
Adoption prevalent – Most if not all federal agencies have adopted, with a need to maintain use and momentum over timeHighlight progress and capture lessons learned* Require regular reporting from agencies to EOP (OSTP, OMB, or similar) (e.g., April 2012, May 2022)

* Require and take full advantage of regular reports to Congress (e.g., April 2012, December 2013, May 2014, May 2015, August 2016, June 2019, May 2022, April 2024)

* Continue to capture and publish federal-use case studies in multiple formats online (e.g., June 2012)

* Undertake research, evaluation, and evidence-building

* Co-develop practitioner toolkit with federal agency experts (e.g., December 2016)

* Continue to feature promising examples on White House/EOP blogs and communication channels (e.g., October 2015, August 2020)

* Engage media and seek both general interest and targeted press coverage, including through external awards/honorifics (e.g., December 2013)
Adoption prevalent – Most if not all federal agencies have adopted, with a need to maintain use and momentum over timePrepare for presidential transitions and document opportunities for future administrations* Integrate go-forward proposals and lessons learned into presidential transition planning and transition briefings (e.g., June 2016-January 2017)

* Brief external stakeholders and Congressional supporters on progress and future opportunities

* Connect use of method to other, broader policy objectives and national priorities (e.g., August 2020, May 2022, April 2024)

Phases and timeline of policy actions advancing the adoption of incentive prizes by federal agencies

  1. Growing number of incentive prizes offered outside government (early 2000s)

At the close of the 20th century, federal use of incentive prizes to induce activity toward targeted solutions was limited, though the federal government regularly utilized recognition prizes to reward past accomplishment. In October 2004, the $10 million Ansari XPRIZE—which was first announced in May 1996—was awarded by the XPRIZE Foundation for the successful flights of Spaceship One by Scaled Composites. Following the awarding of the Ansari XPRIZE and the extensive resulting news coverage, philanthropists and high net worth individuals began to offer prize purses to incentivize action on a wide variety of technology and social challenges. A variety of new online challenge platforms sprung up, and new vendors began offering consulting services for designing and hosting challenges, trends that lowered the cost of prize competition administration and broadened participation in prize competitions among thousands of diverse solvers around the world. This growth in the use of prizes by philanthropists and the private sector increased the interest of the federal government in trying out incentive prizes to help meet agency missions and solve national challenges. Actions during this period to support federal use of incentive prizes include:

  1. Obama-Biden Administration Seeks to Expand Federal Prizes Through Administrative Action (2009-2010)

From the start of the Obama-Biden Administration, OSTP and OMB took a series of policy steps to expand the use of incentive prizes across federal agencies and build federal capacity to support those open-innovation efforts. Bipartisan support in Congress for these actions soon led to new legislation to further advance agency adoption of incentive prizes. Actions during this period to support federal use of incentive prizes include:

  1. Implementing New Government-Wide Prizes Authority Provided by the America COMPETES Act (2011-2016)

During this period of expansion in the federal use of incentive prizes supported by new government-wide prize authority provided by Congress, the Obama-Biden Administration continued to emphasize its commitment to the model, including as a key method for accomplishing administration priorities, including priorities related to open government and evidence-based decision making. Actions during this period to support federal use of incentive prizes include:

toolkit
  1. Maintaining Momentum in New Presidential Administrations

Support for federal use of incentive prizes continued beyond the Obama-Biden Administration foundational efforts. Leadership by federal agency prize leads was particularly important to support this momentum from administration to administration. Actions during the Trump-Pence and Biden-Harris Administrations to support federal use of incentive prizes include:

Harnessed American ingenuity through increased use of incentive prizes. Since 2010, more than 80 Federal agencies have engaged 250,000 Americans through more than 700 challenges on Challenge.gov to address tough problems ranging from fighting Ebola, to decreasing the cost of solar energy, to blocking illegal robocalls. These competitions have made more than $220 million available to entrepreneurs and innovators and have led to the formation of over 275 startup companies with over $70 million in follow-on funding, creating over 1,000 new jobs.

In addition, in January 2017, the Obama-Biden Administration OSTP mentioned the use of incentive prizes in its public “exit memo” as a key “pay-for-performance” method in agency science and technology strategies that “can deliver better results at lower cost for the American people,” and also noted:

Harnessing the ingenuity of citizen solvers and citizen scientists. The Obama Administration has harnessed American ingenuity, driven local innovation, and engaged citizen solvers in communities across the Nation by increasing the use of open-innovation approaches including crowdsourcing, citizen science, and incentive prizes. Following guidance and legislation in 2010, over 700 incentive prize competitions have been featured on Challenge.gov from over 100 Federal agencies, with steady growth every year.

By the end of fiscal year 2022, federal agencies had hosted over 2,000 prize competitions on Challenge.gov, since its launch in 2010. OSTP, GSA, and NASA CoECI had provided training to well over 2,000 federal practitioners during that same period. 

Number of Federal Prize Competitions by Authority FY14-FY22

Source: Office of Science and Technology Policy. Biennial Report on “IMPLEMENTATION OF FEDERAL PRIZE AND CITIZEN SCIENCE AUTHORITY: FISCAL YEARS 2021-22.” April 2024.

Federal Agency Practices to Support the Use of Prize Competitions

Source: Office of Science and Technology Policy. Biennial Report on “IMPLEMENTATION OF FEDERAL PRIZE AND CITIZEN SCIENCE AUTHORITY: FISCAL YEARS 2019-20.” March 2022. 

CONCLUSION

Over the span of a decade, incentive prizes had moved from a tool used primarily outside of the federal government to one used commonly across federal agencies, due to a concerted, multi-pronged effort led by policy entrepreneurs and incentive prize practitioners in the EOP and across federal agencies, with bipartisan congressional support, crossing several presidential administrations. And yet, the work to support the use of prizes by federal agencies is not complete–there remains extensive opportunity to further improve the design, rigor, ambition, and effectiveness of federal prize competitions; to move beyond “ideas challenges” to increase the use of incentive prizes to demonstrate technologies and solutions in testbeds and real-world deployment scenarios; to train additional federal personnel on the use of incentive prizes; to learn from the results of federal incentive prizes competitions; and to apply this method to address pressing and emerging challenges facing the nation.

In applying these lessons to efforts to expand the use of other promising methods in federal agencies, policy entrepreneurs in center-of-government federal agencies should be strategic in the policy actions they take to encourage and scale method adoption, by first seeking to understand the adoption maturity of that method (as well as the relevant policy readiness) and then by undertaking interventions appropriate for that stage of adoption. With attention and action by policy entrepreneurs to address factors that discourage risk-taking, experimentation, and piloting of new methods by federal agencies, it will be possible for federal agencies to utilize a further-expanded strategic portfolio of methods to catalyze the development, demonstration, and deployment of technology and innovative solutions to meet agency missions, solve long-standing problems, and address grand challenges facing our nation. 

Photo by Nick Fewings

Automating Scientific Discovery: A Research Agenda for Advancing Self-Driving Labs

Despite significant advances in scientific tools and methods, the traditional, labor-intensive model of scientific research in materials discovery has seen little innovation. The reliance on highly skilled but underpaid graduate students as labor to run experiments hinders the labor productivity of our scientific ecosystem. An emerging technology platform known as Self-Driving Labs (SDLs), which use commoditized robotics and artificial intelligence for automated experimentation, presents a potential solution to these challenges.

SDLs are not just theoretical constructs but have already been implemented at small scales in a few labs. An ARPA-E-funded Grand Challenge could drive funding, innovation, and development of SDLs, accelerating their integration into the scientific process. A Focused Research Organization (FRO) can also help create more modular and open-source components for SDLs and can be funded by philanthropies or the Department of Energy’s (DOE) new foundation. With additional funding, DOE national labs can also establish user facilities for scientists across the country to gain more experience working with autonomous scientific discovery platforms. In an era of strategic competition, funding emerging technology platforms like SDLs is all the more important to help the United States maintain its lead in materials innovation.

Challenge and Opportunity

New scientific ideas are critical for technological progress. These ideas often form the seed insight to creating new technologies: lighter cars that are more energy efficient, stronger submarines to support national security, and more efficient clean energy like solar panels and offshore wind. While the past several centuries have seen incredible progress in scientific understanding, the fundamental labor structure of how we do science has not changed. Our microscopes have become far more sophisticated, yet the actual synthesizing and testing of new materials is still laboriously done in university laboratories by highly knowledgeable graduate students. The lack of innovation in how we historically use scientific labor pools may account for stagnation of research labor productivity, a primary cause of concerns about the slowing of scientific progress. Indeed, analysis of scientific literature suggests that scientific papers are becoming less disruptive over time and that new ideas are getting harder to find. The slowing rate of new scientific ideas, particularly in the discovery of new materials or advances in materials efficiency, poses a substantial risk, potentially costing billions of dollars in economic value and jeopardizing global competitiveness. However, incredible advances in artificial intelligence (AI) coupled with the rise of cheap but robust robot arms are leading to a promising new paradigm of material discovery and innovation: Self-Driving Labs. An SDL is a platform where material synthesis and characterization is done by robots, with AI models intelligently selecting new material designs to test based on previous experimental results. These platforms enable researchers to rapidly explore and optimize designs within otherwise unfeasibly large search spaces.

Today, most material science labs are organized around a faculty member or principal investigator (PI), who manages a team of graduate students. Each graduate student designs experiments and hypotheses in collaboration with a PI, and then executes the experiment, synthesizing the material and characterizing its property. Unfortunately, that last step is often laborious and the most time-consuming. This sequential method to material discovery, where highly knowledgeable graduate students spend large portions of their time doing manual wet lab work, rate limits the amount of experiments and potential discoveries by a given lab group. SDLs can significantly improve the labor productivity of our scientific enterprise, freeing highly skilled graduate students from menial experimental labor to craft new theories or distill novel insights from autonomously collected data. Additionally, they yield more reproducible outcomes as experiments are run by code-driven motors, rather than by humans who may forget to include certain experimental details or have natural variations between procedures.

Self-Driving Labs are not a pipe dream. The biotech industry has spent decades developing advanced high-throughput synthesis and automation. For instance, while in the 1970s statins (one of the most successful cholesterol-lowering drug families) were discovered in part by a researcher testing 3800 cultures manually over a year, today, companies like AstraZeneca invest millions of dollars in automation and high-throughput research equipment (see figure 1). While drug and material discovery share some characteristics (e.g., combinatorially large search spaces and high impact of discovery), materials R&D has historically seen fewer capital investments in automation, primarily because it sits further upstream from where private investments anticipate predictable returns. There are, however, a few notable examples of SDLs being developed today. For instance, researchers at Boston University used a robot arm to test 3D-printed designs for uniaxial compression energy adsorption, an important mechanical property for designing stronger structures in civil engineering and aerospace. A Bayesian optimizer was then used to iterate over 25,000 designs in a search space with trillions of possible candidates, which led to an optimized structure with the highest recorded mechanical energy adsorption to date. Researchers at North Carolina State University used a microfluidic platform to autonomously synthesize >100 quantum dots, discovering formulations that were better than the previous state of the art in that material family.

 AstraZeneca automation equipment

These first-of-a-kind SDLs have shown exciting initial results demonstrating their ability to discover new material designs in a haystack of thousands to trillions of possible designs, which would be too large for any human researcher to grasp. However, SDLs are still an emerging technology platform. In order to scale up and realize their full potential, the federal government will need to make significant and coordinated research investments to derisk this materials innovation platform and demonstrate the return on capital before the private sector is willing to invest it.

Other nations are beginning to recognize the importance of a structured approach to funding SDLs: University of Toronto’s Alan Aspuru-Guzik, a former Harvard professor who left the United States in 2018, has created an Acceleration Consortium to deploy these SDLs and recently received $200 million in research funding, Canada’s largest ever research grant. In an era of strategic competition and climate challenges, maintaining U.S. competitiveness in materials innovation is more important than ever. Building a strong research program to fund, build, and deploy SDLs in research labs should be a part of the U.S. innovation portfolio.

Plan of Action

While several labs in the United States are working on SDLs, they have all received small, ad-hoc grants that are not coordinated in any way. A federal government funding program dedicated to self-driving labs does not currently exist. As a result, the SDLs are constrained to low-hanging material systems (e.g., microfluidics), with the lack of patient capital hindering labs’ ability to scale these systems and realize their true potential. A coordinated U.S. research program for Self-Driving Labs should:

Initiate an ARPA-E SDL Grand Challenge: Drawing inspiration from DARPA’s previous grand challenges that have catalyzed advancements in self-driving vehicles, ARPA-E should establish a Grand Challenge to catalyze state-of-the-art advancements in SDLs for scientific research. This challenge would involve an open call for teams to submit proposals for SDL projects, with a transparent set of performance metrics and benchmarks. Successful applicants would then receive funding to develop SDLs that demonstrate breakthroughs in automated scientific research. A projected budget for this initiative is $30 million1, divided among six selected teams, each receiving $5 million over a four-year period to build and validate their SDL concepts. While ARPA-E is best positioned in terms of authority and funding flexibility, other institutions like National Science Foundation (NSF) or DARPA itself could also fund similar programs.

Establish a Focused Research Organization to open-source SDL components: This FRO would be responsible for developing modular, open-source hardware and software specifically designed for SDL applications. Creating common standards for both the hardware and software needed for SDLs will make such technology more accessible and encourage wider adoption. The FRO would also conduct research on how automation via SDLs is likely to reshape labor roles within scientific research and provide best practices on how to incorporate SDLs into scientific workflows. A proposed operational timeframe for this organization is five years, with an estimated budget of $18 million over that time period. The organization would work on prototyping SDL-specific hardware solutions and make them available on an open-source basis to foster wider community participation and iterative improvement. A FRO could be spun out of the DOE’s new Foundation for Energy Security (FESI), which would continue to establish the DOE’s role as an innovative science funder and be an exciting opportunity for FESI to work with nontraditional technical organizations. Using FESI would not require any new authorities and could leverage philanthropic funding, rather than requiring congressional appropriations.

Provide dedicated funding for the DOE national labs to build self-driving lab user facilities, so the United States can build institutional expertise in SDL operations and allow other U.S. scientists to familiarize themselves with these platforms. This funding can be specifically set aside by the DOE Office of Science or through line-item appropriations from Congress. Existing prototype SDLs, like the Argonne National Lab Rapid Prototyping Lab or Berkeley Lab’s A-Lab, that have emerged in the past several years lack sustained DOE funding but could be scaled up and supported with only $50 million in total funding over the next five years. SDLs are also one of the primary applications identified by the national labs in the “AI for Science, Energy, and Security” report, demonstrating willingness to build out this infrastructure and underscoring the recognized strategic importance of SDLs by the scientific research community.

Frequently Asked Questions
What factors determine whether an SDL is appropriate for materials innovation?

As with any new laboratory technique, SDLs are not necessarily an appropriate tool for everything. Given that their main benefit lies in automation and the ability to rapidly iterate through designs experimentally, SDLs are likely best suited for:



  • Material families with combinatorially large design spaces that lack clear design theories or numerical models (e.g., metal organic frameworks, perovskites)

  • Experiments where synthesis and characterization are either relatively quick or cheap and are amenable to automated handling (e.g., UV-vis spectroscopy is relatively simple, in-situ characterization technique)

  • Scientific fields where numerical models are not accurate enough to use for training surrogate models or where there is a lack of experimental data repositories (e.g., the challenges of using density functional theory in material science as a reliable surrogate model)


While these heuristics are suggested as guidelines, it will take a full-fledged program with actual results to determine what systems are most amenable to SDL disruption.

What aren’t SDLs?

When it comes to exciting new technologies, there can be incentives to misuse terms. Self-Driving Labs can be precisely defined as the automation of both material synthesis and characterization that includes some degree of intelligent, automated decision-making in-the-loop. Based on this definition, here are common classes of experiments that are not SDLs:



  • High-throughput synthesis, where synthesis automation allows for the rapid synthesis of many different material formulations in parallel (lacks characterization and AI-in-the-loop)

  • Using AI as a surrogate trained over numerical models, which is based on software-only results. Using an AI surrogate model to make material predictions and then synthesizing an optimal material is also not a SDL, though certainly still quite the accomplishment for AI in science (lacks discovery of synthesis procedures and requires numerical models or prior existing data, neither of which are always readily available in the material sciences).

Will SDLs “automate” away scientists? How will they change the labor structure of science?

SDLs, like every other technology that we have adopted over the years, eliminate routine tasks that scientists must currently spend their time on. They will allow scientists to spend more time understanding scientific data, validating theories, and developing models for further experiments. They can automate routine tasks but not the job of being a scientist.


However, because SDLs require more firmware and software, they may favor larger facilities that can maintain long-term technicians and engineers who maintain and customize SDL platforms for various applications. An FRO could help address this asymmetry by developing open-source and modular software that smaller labs can adopt more easily upfront.

FAS Senior Fellow Jen Pahlka testifies on Using AI to Improve Government Services

Jennifer Pahlka (@pahlkadot) is a FAS Senior Fellow and the author of Recoding America: Why Government is Failing in the Digital Age and How We Can Do Better. Here is Pahlka’s testimony about artificial intelligence presented today, January 10, 2024, to the full Senate Committee on Homeland Security and Government Affairs hearing on “Harnessing AI to Improve Government Services and Customer Experience”. More can be found here, here, and here.

How the U.S. government chooses to respond to the changes AI brings is indeed critical, especially in its use to improve government services and customer experience. If the change is going to be for the better (and we can’t afford otherwise) it will not be primarily because of how much or how little we constrain AI’s use. Constraints are an important conversation, and AI safety experts are better suited to discuss these than me. But we could constrain agencies significantly and still get exactly the bad outcomes that those arguing for risk mitigation want to avoid. We could instead direct agencies to dive headlong into AI solutions, and still fail to get the benefit that the optimists expect. The difference will come down to how much or how little capacity and competency we have to deploy these technologies thoughtfully.

There are really two ways to build capacity: having more of the right people doing the right things (including but not limited to leveraging technology like AI) and safely reducing the burdens we place on those people. AI, of course, could help reduce those burdens, but not without the workforce we need – one that understands the systems we have today, the policy goals we have set, and the technology we are bringing to bear to achieve those goals. Our biggest priority as a government should be building that capacity, working both sides of that equation (more people, less burden.)

Building that capacity will require bodies like the US Senate to use a wide range of the tools at its disposal to shape our future, and use them in a specific way. Those tools can be used to create mandates and controls on the institutions that deliver for the American people, adding more rules and processes for administrative agencies and others to comply with. Or they can be used to enable these institutions to develop the capacity they so desperately need and to use their judgment in the service of agreed-upon goals, often by asking what mandates and controls might be removed, rather than added. This critical AI moment calls for enablement.

The recent executive order on AI already provides some new controls and safeguards. The order strikes a reasonable balance between encouragement and caution, but I worry that some of its guidance will be applied inappropriately. For example, some government agencies have long been using AI for day to day functions like handwriting recognition on envelopes or improved search to retrieve evidence more easily, and agencies may now subject these benign, low-risk uses to red tape based on the order. Caution is merited in some places, and dangerous in others, where we risk moving backwards, not forward. What we need to navigate these frameworks of safeguard and control are people in agencies who can tell the difference, and who have the authority to act accordingly.

Moreover, in many areas of government service delivery, the status quo is frankly not worth protecting. We understandably want to make sure, for instance, that applicants for government benefits aren’t unfairly denied because of bias in algorithms. The reality is that, to take just one benefit, one in six determinations of eligibility for SNAP is substantively incorrect today. If you count procedural errors, the rate is 44%. Worse are the applications and adjudications that haven’t been decided at all, the ones sitting in backlogs, causing enormous distress to the public and wasting taxpayer dollars. Poor application of AI in these contexts could indeed make a bad situation worse, but for people who are fed up and just want someone to get back to them about their tax return, their unemployment insurance check, or even their company’s permit to build infrastructure, something has to change. We may be able to make progress by applying AI, but not if we double down on the remedies that failed in the Internet Age and hope they somehow work in the age of AI. We must finally commit to the hard work of building digital capacity.

Read the full testimony here.

What Works in Boston, Won’t Necessarily Work in Birmingham: 4 Pragmatic Principles for Building Commercialization Capacity in Innovation Ecosystems

Just like crop tops, flannel, and some truly unfortunate JNCO jeans that one of these authors wore in junior high, the trends of the 90’s are upon us again. In the innovation world, this means an outsized focus on tech-based economic development, the hottest new idea in economic development, circa 1995. This takes us back in time to fifteen years after the passage of the Bayh Dole Act, the federal legislation that granted ownership of federally funded research to universities. It was a time when the economy was expanding, dot-com growth was a boom, not a bubble, and we spent more time watching Saved by the Bell than thinking about economic impact. 

After the creation of tech transfer offices across the country and the benefit of time, universities were just starting to understand how much the changes wrought by Bayh-Dole would impact them (or not). A raft of optimistic investments in venture development organizations and state public-private partnerships swept the country, some of which (like Ben Franklin Technology Partners and BioSTL) are still with us today, and some of which (like the Kansas Technology Enterprise Center) have flamed out in spectacular fashion. All of a sudden, research seemed like a process to be harnessed for economic impact. Out of this era came the focus on “technology commercialization” that has captured the economic development imagination to this day. 

Commercialization, in the context of this piece, describes the process through which universities (or national labs) and the private sector collaborate to bring to the market technologies that were developed using federal funding. Unlike sponsored research and development, in which industry engages with universities from the beginning to fund and set a research agenda, commercialization brings in the private sector after the technology has been conceptualized. Successful commercialization efforts have now grown across the country, and we believe they can be described by four practical principles: 

Principle 1: A strong research enterprise is a necessary precondition to building a strong commercialization pipeline.

The first condition necessary to developing a commercialization pipeline is a reasonably advanced research enterprise. While not every region in the U.S. has access to a top-tier research university, there are pockets of excellent research at most major U.S. R1 and R2 institutions. However, because there is natural attrition at each stage of the commercialization process (much like the startup process) a critical mass of novel, leading, and relevant research activity must exist in a given University. If that bar is assumed to be the ability to attract $10 million in research funding (the equivalent of winning 20-25 SBIR Phase 1 grants annually), that limits the number of schools that can run a fruitful commercialization pipeline to approximately 350 institutions, based on data from the NSF NCSES. A metro area should have at least one research institution that meets this bar in order to secure federal funding for the development of lab-to-market programs, though given the co-location of many universities, it is possible for some metro areas to have several such research institutions or none at all.

Principle 2: Commercialization via established businesses creates different economic impacts than commercialization via startups; each pathway requires fundamentally different support.

When talking about commercialization, it is also important to differentiate between whether a new technology is brought to market by a large, incumbent company or start-up. The first half of the commercialization process is the same for both: technology is transferred out of universities, national labs, and other research institutions through the process of registering, patenting, and licensing new intellectual property (IP). Once licensed, though, the commercialization pathway branches into two. 

With an incumbent company, whether or not it successfully brings new technology to the market is largely dependent on the company’s internal goals and willingness to commit resources to commercializing that IP. Often, incumbent companies will license patents as a defensive strategy in order to prevent competition with their existing product lines. As a result, license of a technology by an incumbent company cannot be assumed to represent a guarantee of commercial use or value creation.

The alternative pathway is for universities to license their IP to start-ups, which may be spun out of university labs. Though success is not guaranteed, licensing to these new companies is where new programs and better policies can actually make an impact. Start-ups are dependent upon successful commercialization and require a lot of support to do so. Policies and programs that help meet their core needs can play a significant role in whether or not a start-up succeeds. These core needs include independent space for demonstrating and scaling their product, capital for that work and commercialization activities (e.g. scouting customers and conducting sales), and support through mentorship programs, accelerators, and in-kind help navigating regulatory processes (especially in deep tech fields). 

Principle 3: Local context matters; what works in Boston won’t necessarily work in Birmingham. 

Unfortunately, many universities approach their tech transfer programs with the goal of licensing their technology to large companies almost exclusively. This arises because university technology transfer offices (TTOs) are often understaffed, and it is easier to license multiple technologies to the same large company under an established partnership than to scout new buyers and negotiate new contracts for each patent. The Bayh-Dole Act, which established the current tech transfer system, was never intended to subsidize the R&D expenditures of our nation’s largest and most profitable companies, nor was it intended to allow incumbents to weaponize IP to repel new market entrants. Yet, that is how it is being used today in practical application.

Universities are not necessarily to blame for the lack of resources, though. Universities spend on average 0.6% of their research expenditures on their tech transfer programs. However, there is a large difference in research expenditures between top universities that can attract over a billion in research funding and the average research university, and thus a large difference in the staffing and support of TTOs. State government funding for the majority of public research universities have been declining since 2008, though there has been a slight upswing since the pandemic, while R&D funding at top universities continues to increase. Only a small minority of TTOs bring in enough income from licensing in order to be self-sustaining, often from a single “blockbuster” patent, while the majority operate at a loss to the institution. 

To successfully develop innovation capacity in ecosystems around the country through increased commercialization activity, one must recognize that communities have dramatically different levels of resources dedicated to these activities, and thus, “best practices” developed at leading universities are seldom replicable in smaller markets. 

Principle 4: Successful commercialization pipelines include interventions at the individual, institutional, and ecosystem level.

As we’ve discussed at length in our FAS “systems-thinking” blog series, which includes a post on innovation ecosystems, a systems lens is fundamental to how we see the world. Thinking in terms of systems helps us understand the structural changes that are needed to change the conditions that we see playing out around us every day. When thinking about the structure of commercialization processes, we believe that intervention at various structural levels of a system is necessary to create progres on challenges that seem insurmountable at first—such as changing the cultural expectations of “success” that are so influential in the academic systems. Below we have identified some good practices and programs for supporting commercialization at the individual, institutional, and ecosystem level, with an emphasis on pathways to start-ups and entrepreneurship.

Practices and Programs Targeted at Individuals

University tech transfer programs are often reliant on individuals taking the initiative to register new IP with their TTOs. This requires individuals to be both interested enough in commercialization and knowledgeable enough about the commercialization potential of their research to pursue registration. Universities can encourage faculty to be proactive in pursuing commercialization through recognizing entrepreneurial activities in their hiring, promotion and tenure guidelines and encouraging faculty to use their sabbaticals to pursue entrepreneurial activities. An analog to the latter at national laboratories are Entrepreneurial Leave Programs that allow staff scientists to take a leave of up to three years to start or join a start-up before returning to their position at the national lab.

Faculty and staff scientists are not the only source of IP though; graduate students and postdoctoral researchers produce much of the actual research behind new intellectual property. Whether or not these early-career researchers pursue commercialization activities is correlated with whether they have had research advisors who were engaged in commercialization. For this reason, in 2007, the National Research Foundation of Singapore established a joint research center with the Massachusetts Institute of Technology (MIT) such that by working with entrepreneurial MIT faculty members, researchers at major Singaporean universities would also develop a culture of entrepreneurship. Most universities likely can’t establish programs of this scale, but some type of mentorship program for early-career scientists pre-IP generation can help create a broader culture of translational research and technology transfer. Universities should also actively support graduate students and postdoctoral researchers in putting forward IP to their TTO. Some universities have even gone so far as to create funds to buy back the time of graduate students and postdocs from their labs and direct that time to entrepreneurial activities, such as participating in an iCorps program or conducting primary market research.  

Student at work in the NOAA CIGLR Lab at the University of Michigan School for Environment and Sustainability

Some universities have even gone so far as to create funds to buy back the time of graduate students and postdocs from their labs and direct that time to entrepreneurial activities, such as participating in an iCorps program or conducting primary market research.

Once IP has been generated and licensed, many universities offer mentorship programs for new entrepreneurs, such as MIT’s Venture Mentorship Services. Outside of universities, incubators and accelerators provide mentorship along with funding and/or co-working spaces for start-ups to grow their operation. Hardware-focused start-ups especially benefit from having a local incubator or accelerator, since hard-tech start-ups attract significantly less venture capital funding and support than digital technology start-ups, but require larger capital expenditures as they scale. Shared research facilities and testbeds are also crucial for providing hard-tech start-ups with the lab space and equipment to refine and scale their technologies.

For internationally-born entrepreneurs, an additional consideration is visa sponsorship. International graduate students and postdocs that launch start-ups need visa sponsors in order to stay in the United States as they transition out of academia. Universities that participate in the Global Entrepreneur in Residence program help provide H-1B visas for international entrepreneurs to work on their start-ups in affiliation with universities. The university benefits in return by attracting start-ups to their local community that then generate economic opportunities and help create an entrepreneurial ecosystem.

Practices and Programs Targeted at Institutions

As mentioned in the beginning, one of the biggest challenges for university tech transfer programs is understaffed TTOs and small patent budgets. On average, TTOs have only four people on staff, who can each file a handful of patents a year, and budgets for the legal fees on even fewer patents. Fully staffing TTOs can help universities ensure that new IP doesn’t slip through the cracks due to a lack of capacity for patenting or licensing activities. Developing standard term sheets for licensing agreements can also reduce administrative burden and make it easier for TTOs to establish new partnerships.

Instead of TTOs, some universities have established affiliated technology intermediaries, which are organizations that take on the business aspects of technology commercialization. For example, the Wisconsin Alumni Research Foundation (WARF) was launched as an independent, nonprofit corporation to manage the University of Wisconsin–Madison’s vitamin D patents and invest the resulting revenue into future research at the university. Since its inception 90 years ago, WARF has provided $2.3 billion in grants to the university and helped establish 60 start-up companies. 

In general, universities need to be more consistent about collecting and reporting key performance indicators for TTOs outside of the AUTM framework, such as the number of unlicensed patents and the number of products brought to the market using licensed technologies. In particular, universities should disaggregate metrics for licensing and partnerships between companies less than five years old and those greater than five years old so that stakeholders can see whether there is a difference in commercialization outcomes between incumbent and start-up licensees.

Practices and Programs Targeted at Innovation Ecosystems

Innovation ecosystems are made up of researchers, entrepreneurs, corporations, the workforce, government, and sources of capital. Geographic proximity through co-locating universities, corporations, start-ups, government research facilities, and other stakeholder institutions can help foster both formal and informal collaboration and result in significant technology-driven economic growth and benefits. Co-location may arise organically over time or result from the intentional development of research parks, such as the NASA Research Park. When done properly, the work of each stakeholder should advance a shared vision. This can create a virtuous cycle that attracts additional talent and stakeholders to the shared vision and can integrate with more traditional attraction and retention efforts. One such example is the co-location of the National Bio- and Agro-Defense Facility in Manhattan, KS, near the campus of Kansas State University. After securing that national lab, the university made investments in additional BSL-2, 3 and 3+ research facilities including the Biosecurity Research Institute and its Business Development Module. The construction and maintenance of those facilities required the creation of new workforce development programs to train HVAC technicians that manage the independent air handling capabilities of the labs and train biomanufacturing workers, which was then one of the selling points for the successful campaign for the relocation of corporation Scorpius Biologics to the region. At best, all elements of an innovation ecosystem are fueled by a research focus and the commercialization activity that it provides. 

For regions that find themselves short of the talent they need, soft-landing initiatives can help attract domestic and international entrepreneurs, start-ups, and early-stage firms to establish part of their business in a new region or to relocate entirely. This process can be daunting for early-stage companies, so soft-landing initiatives aim to provide the support and resources that will help an early-stage company acclimatize and thrive in a new place. These initiatives help to expand the reach of a community, create a talent base, and foster the conditions for future economic growth and benefits.

Alongside the creation of innovation ecosystems should be the establishment of “scale-up ecosystems” focused on developing and scaling new manufacturing processes necessary to mass produce the new technologies being developed. This is often an overlooked aspect of technology development in the United States, and supply chain shocks over the past few years have shone a light on the need to develop more local manufacturing supply chains. Fostering the growth of manufacturing alongside technology innovation can (1) reduce the time cycling between product and process development in the commercialization process, (2) capture the “learning by doing” benefits from scaling the production of new technologies, and (3) replenish the number of middle-income jobs that have been outsourced over the past few decades. 

Any way you slice it, commercialization capacity is one clear and critical input to a successful innovation ecosystem. However, it’s not the only element that’s important. A strong startup commercialization effort, standing alone, without the corporate, workforce, or government support that it needs to build a vibrant ecosystem around its entrepreneurs, might wane with time or simply be very successful at shipping spinouts off to a coastal hotspot. Building a commercialization pipeline is not, nor has it ever been, a one-size-fits-all solution for ecosystem building. 

It may even be something we’ve over-indexed on, given the widespread adoption of tech-based economic development strategies. One significant reason for this is the fact that entrepreneurship via commercialization is most open to those who already have access to a great deal of privilege–who have attained, or are on the path to, graduate degrees in STEM fields critical to our national competitiveness. If you’ve already earned a Ph.D. in machine learning, chances are your future is looking pretty bright—with or without entrepreneurial opportunity involved. To truly reap the economic benefits of commercialization activity (and the startups it creates), we need to aggressively implement programs, training, and models that change the demographics of who gets to commercialize technology, not just how they do it. To shape this, we’ll need to change the conditions for success for early-career researchers and reconsider the established model of how we mentor and train the next generation of scientists and engineers–you’ll hear more from us on these topics in future posts!

Risk and Reward in Peer Review

This article was written as a part of the FRO Forecasting project, a partnership between the Federation of American Scientists and Metaculus. This project aims to conduct a pilot study of forecasting as an approach for assessing the scientific and societal value of proposals for Focused Research Organizations. To learn more about the project, see the press release here. To participate in the pilot, you can access the public forecasting tournament here.

The United States federal government is the single largest funder of scientific research in the world. Thus, the way that science agencies like the National Science Foundation and the National Institutes of Health distribute research funding has a significant impact on the trajectory of science as a whole. Peer review is considered the gold standard for evaluating the merit of scientific research proposals, and agencies rely on peer review committees to help determine which proposals to fund. However, peer review has its own challenges. It is a difficult task to balance science agencies’ dual mission of protecting government funding from being spent on overly risky investments while also being ambitious in funding proposals that will push the frontiers of science, and research suggests that peer review may be designed more for the former rather than the latter. We at FAS are exploring innovative approaches to peer review to help tackle this challenge.

Biases in Peer Review

A frequently echoed concern across the scientific and metascientific community is that funding agencies’ current approach to peer review of science proposals tends to be overly risk-averse, leading to bias against proposals that entail high risk or high uncertainty about the outcomes. Reasons for this conservativeness include reviewer preferences for feasibility over potential impact, contagious negativity, and problems with the way that peer review scores are averaged together.

This concern, alongside studies suggesting that scientific progress is slowing down, has led to a renewed effort to experiment with new ways of conducting peer review, such as golden tickets and lottery mechanisms. While golden tickets and lottery mechanisms aim to complement traditional peer review with alternate means of making funding decisions — namely individual discretion and randomness, respectively — they don’t fundamentally change the way that peer review itself is conducted. 

Traditional peer review asks reviewers to assess research proposals based on a rubric of several criteria, which typically include potential value, novelty, feasibility, expertise, and resources. These criteria are given a score based on a numerical scale; for example, the National Institutes of Health uses a scale from 1 (best) to 9 (worst). Reviewers then provide an overall score that need not be calculated in any specific way based on the criteria scores. Next, all of the reviewers convene to discuss the proposal and submit their final overall scores, which may be different from what they submitted prior to the discussion. The final overall scores are averaged across all of the reviewers for a specific proposal. Proposals are then ranked based on their average overall score and funding is prioritized for those ranked before a certain cutoff score, though depending on the agency, some discretion by program administrators is permitted.  

The way that this process is designed allows for the biases mentioned at the beginning—reviewer preferences for feasibility, contagious negativity, and averaging problems—to influence funding decisions. First, reviewer discretion in deciding overall scores allows them to weigh feasibility more heavily than potential impact and novelty in their final scores. Second, when evaluations are discussed reviewers tend to adjust their scores to better align with their peers. This adjustment tends to be greater when correcting in the negative direction than in the positive direction, resulting in a stronger negative bias. Lastly, since funding tends to be quite limited, cutoff scores tend to be quite close to the best score. This means that even if almost all of the reviewers rate a proposal positively, one very negative review can potentially bring the average below the cutoff.

Designing a New Approach to Peer Review

In 2021, the researchers Chiara Franzoni and Paula Stephan published a working paper arguing that risk in science results from three sources of uncertainty: uncertainty of research outcomes, uncertainty of the probability of success, and uncertainty of the value of the research outcomes. To comprehensively and consistently account for these sources of uncertainty, they proposed a new expected utility approach to peer review evaluations, in which reviewers are asked to

  1. Identify the primary expected outcome of a research proposal and, optionally, a potential secondary outcome;
  2. Assess the probability between 0 to 1 of achieving each expected outcome (P(j); and
  3. Assess the value of achieving each expected outcome (uj) on a numerical scale (e.g., 0 to 100).

From this, the total expected utility can be calculated for each proposal and used to rank them.1 This systematic approach addresses the first bias we discussed by limiting the extent to which reviewers’ preferences for more feasible proposals would impact the final score of each proposal.

We at FAS see a lot of potential in Franzoni and Stephan’s expected value approach to peer review, and it inspired us to design a pilot study using a similar approach that aims to chip away at the other biases in review.

To explore potential solutions for negativity bias, we are taking a cue from forecasting by complementing the peer review process with a resolution and scoring process. This means that at a set time in the future, reviewers’ assessments will be compared to a ground truth based on the actual events that have occurred (i.e., was the outcome actually achieved and, if so, what was its actual impact?). Our theory is that if implemented in peer review, resolution and scoring could incentivize reviewers to make better, more accurate predictions over time and provide empirical estimates of a committee’s tendency to provide overly negative (or positive) assessments, thus potentially countering the effects of contagion during review panels and helping more ambitious proposals secure support. 

Additionally, we sought to design a new numerical scale for assessing the value or impact of a research proposal, which we call an impact score. Typically, peer reviewers are free to interpret the numerical scale for each criteria as they wish; Franzoni and Stephan’s design also did not specify how the numerical scale for the value of the research outcome should work. We decided to use a scale ranging from 1 (low) to 10 (high) that was base 2 exponential, meaning that a proposal that receives a score of 5 has double the impact of a proposal that receives a score of 4, and quadruple the impact of a proposal that receives a score of 3.

Plot demonstrating the exponential nature of the impact score: a score of 1 shows an impact of zero, while a score of 10 shows an impact for 1000.
Figure 1. Plot demonstrating the exponential nature of the impact score.
Table 1. Example of how to interpret the impact score.
ScoreImpact
1None or negative
2Minimal
3Low or mixed
4Moderate
5High
6Very high
7Exceptional
8Transformative
9Revolutionary
10Paradigm-shifting

The choice of an exponential scale reflects the tendency in science for a small number of research projects to have an outsized impact (Figure 2), and provides more room at the top end of the scale for reviewers to increase the rating of the proposals that they believe will have an exceptional impact. We believe that this could help address the last bias we discussed, which is that currently, bad scores are more likely to pull a proposal’s average below the cutoff than good scores are likely to pull a proposal’s average above the cutoff.

Figure 2. Citation distribution of accepted and rejected journal articles

Source: Siler, Lee, and Bero (2014)

We are now piloting this approach on a series of proposals in the life sciences that we have collected for Focused Research Organizations, a new type of non-profit research organization designed to tackle challenges that neither academia or industry is incentivized to work on. The pilot study was developed in collaboration with Metaculus, a forecasting platform and aggregator, and will be hosted on their website. We welcome subject matter experts in the life sciences — or anyone interested! — to participate in making forecasts on these proposals here. Stay tuned for the results of this pilot, which we will publish in a report early next year.

Collaboration for the Future of Public and Active Transportation

Summary

Public and active transportation are not equally accessible to all Americans. Due to a lack of sufficient infrastructure and reliable service for public transportation and active modes like biking, walking, and rolling, Americans must often depend on personal vehicles for travel to work, school, and other activities. During the past two years, Congress has allocated billions of dollars to equitable infrastructure, public transportation upgrades, and decreasing greenhouse gas pollution from transportation across the United States. The Department of Transportation (DOT) and its agencies should embrace innovation and partnerships to continue to increase active and public transportation across the country. The DOT should require grant applications for funding to discuss cross-agency collaborations, partner with the Department of Housing and Urban Development (HUD) to organize prize competitions, encourage public-private partnerships (P3s), and work with the Environmental Protection Agency (EPA) to grant money for transit programs through the Greenhouse Gas Reduction Fund. 

Challenge and Opportunity

Historically, U.S. investment in transportation has focused on expanding and developing highways for personal vehicle travel. As a result, 45% of Americans do not have access to reliable and safe public transportation, perpetuating the need for single-use vehicles for almost half of the country. The EPA reports that transportation accounts for 29% of total U.S. greenhouse gas emissions, with 58% of those emissions coming from light-duty cars. This large share of nationwide emissions from personal vehicles has short- and long-term climate impacts. 

Investments in green public and active transit should be a priority for the DOT in transitioning away from a personal-vehicle-dominated society and meeting the Biden Administration’s “goals of a 100% clean electrical grid by 2035 and net-zero carbon emissions by 2050.” Public and active transportation infrastructure includes bus systems, light rail, bus rapid transit, bike lanes, and safe sidewalks. Investments in public and active transportation should go towards a combination of electrifying existing public transportation, such as buses; improving and expanding public transit to be more reliable and accessible for more users; constructing bike lanes; developing community-owned bike share programs; and creating safe walking corridors. 

In addition to reducing carbon emissions, improved public transportation that disincentivizes personal vehicle use has a variety of co-benefits. Prioritizing public and active transportation could limit congestion on roads and lower pollution. Fewer vehicles on the road result in less tailpipe emissions, which “can trigger health problems such as aggravated asthma, reduced lung capacity, and increased susceptibility to respiratory illnesses, including pneumonia and bronchitis.” This is especially important for the millions of people who live near freeways and heavily congested roads. 

Congestion can also be financially costly for American households; the INRIZ Global Traffic Scorecard reports that traffic congestion cost the United States $81 billion in 2022. Those costs include vehicle maintenance, fuel cost, and “lost time,” all of which can be reduced with reliable and accessible public and active transportation. Additionally, the American Public Transportation Association reports that every $1 invested in public transportation generates $5 in economic returns, measured by savings in time traveled, reduction in traffic congestion, and business productivity. Thus, by investing in public transportation, communities can see improvements in air quality, economy, and health.

Public transportation is primarily managed at the local and state level; currently, over  6000 local and state transportation agencies provide and oversee public transportation in their regions. Public transportation is funded through federal, state, and local sources, and transit agencies receive funding from “passenger fares and other operating receipts.” The Federal Transit Administration (FTA) distributes funding for transit through grants and loans and accounts for 15% of total income for transit agencies, including 31% of capital investments in transit infrastructure. Local and state entities often lack sufficient resources to improve public transportation systems because of the uncertainty of ridership and funding streams.

Public-private partnerships can help alleviate some of these resource constraints because contracts can allow the private partner to operate public transportation systems. Regional and national collaboration across multiple agencies from the federal to the municipal level can also help alleviate resource barriers to public transit development. Local and state agencies do not have to work alone to improve public and active transportation systems. 

The following recommendations provide a pathway for transportation agencies at all levels of government to increase public and active transportation, resulting in social, economic, and environmental benefits for the communities they serve. 

Plan of Action

Recommendation 1. The FTA should require grant applicants for programs such as the Rebuilding American Infrastructure with Sustainability and Equity (RAISE) to define how they will work collaboratively with multiple federal agencies and conduct community engagement. 

Per the National Blueprint for Transportation Decarbonization, FTA staff should prioritize funding for grant applicants who successfully demonstrate partnerships and collaboration. This can be demonstrated, for example, with letters of support from community members and organizations for transit infrastructure projects. Collaboration can also be demonstrated by having applicants report clear goals, roles, and responsibilities for each agency involved in proposed projects. The FTA should: 

  1. Develop a rubric for evaluating partnerships’ efficiency and alignment with national transit decarbonization goals. 
  2. Create a tiered metrics system within the rubric that prioritizes grants for projects based on collaboration and reduction of greenhouse gas emissions in the transit sector.
  3. Add a category to their Guidance Center on federal-state-local partnerships to provide insight on how they view successful collaboration. 

Recommendation 2. The DOT and HUD should collaborate on a prize competition to design active and/or public transportation projects to reduce traffic congestion. 

Housing and transportation costs are related and influence one another, which is why HUD is a natural partner. Funding can be sourced from the Highway Trust Fund, which the DOT has the authority to allocate up to “1% of the funds for research and development to carry out . . . prize competition program[s].”

This challenge should call on local agency partners to provide a design challenge or opportunity that impedes their ability to adopt transit-oriented infrastructure that could reduce traffic congestion. Three design challenges should be selected and publicly posted on the Challenge.gov website so that any individual or organization can participate. 

The goal of the prize competition is to identify challenges, collaborate, and share resources across agencies and communities to design transportation solutions. The competition would connect the DOT with local and regional planning and transportation agencies to solicit solutions from the public, whether from individuals, teams of individuals, or organizations. The DOT and HUD should work collaboratively to design the selection criteria for the challenge and select the winners. Each challenge winner would be provided with a financial prize of $250,000, and their idea would be housed on the DOT website as a case study that can be used for future planning decisions. The local agencies that provide the three design challenges would be welcome to implement the winning solutions.

Recommendation 3. Federal, state, and local government should increase opportunities for public-private partnerships (P3s). 

The financial investment required to develop active and public transportation infrastructure is a hurdle for many agencies. To address this issue, we make the following recommendations: 

Conclusion

The road to decarbonizing the transportation sector requires public and active transportation. Federal agencies can allocate funding for public and active transit more effectively through the recommendations above. It’s time for the government to recognize public and active transportation as the key to equitable decarbonization of the transportation sector throughout the United States.

Frequently Asked Questions
What are examples of transit public-private partnerships (P3s)?

Most P3s in the United States are for highways, bridges, and roads, but there have been a few successful public transit P3s. In 2018 the City of Los Angeles joined LAX and LAX Integrated Express Solutions in a $4.9 billion P3 to develop a train system within the airport. This project aims to launch in 2024 to “enhance the traveler experience” and will “result in 117,000 fewer vehicle miles traveled per day” to the airport. This project is a prime example of how P3s can help reduce traffic congestion and enable and encourage the use of public transportation.

How can P3s be further supported through federal policy beyond the recommendations in this memo?

In 2021, the Congressional Research Service released a report about public-private partnerships (3Ps) that highlights the role the federal government can play by making it easier for agencies to participate in P3s.

What are examples of existing green banks and infrastructure banks?

The state of Michigan has a long history with its Michigan Saves program, the nation’s first nonprofit green bank, which provides funding for projects like rooftop solar or energy efficiency programs.


In California the California Alternative Energy and Advanced Transportation Financing Authority works “collaboratively with public and private partners to provide innovative and effective financing solutions” for renewable energy sources, energy efficiency, and advanced transportation and manufacturing technologies.


The Rhode Island Infrastructure Bank provides funding to municipalities, businesses, and homeowners for projects “including water and wastewater, roads and bridges, energy efficiency and renewable energy, and brownfield remediation.”

Applying ARPA-I: A Proven Model for Transportation Infrastructure

Executive Summary

In November 2021, Congress passed the Infrastructure Investment and Jobs Act (IIJA), which included $550 billion in new funding for dozens of new programs across the U.S. Department of Transportation (USDOT). Alongside historic investments in America’s roads and bridges, the bill created the Advanced Research Projects Agency-Infrastructure (ARPA-I). Building on successful models like the Defense Advanced Research Projects Agency (DARPA) and the Advanced Research Program-Energy (ARPA-E), ARPA-I’s mission is to bring the nation’s most innovative technology solutions to bear on our most significant transportation infrastructure challenges.

ARPA-I must navigate America’s uniquely complex infrastructure landscape, characterized by limited federal research and development funding compared to other sectors, public sector ownership and stewardship, and highly fragmented and often overlapping ownership structures that include cities, counties, states, federal agencies, the private sector, and quasi-public agencies. Moreover, the new agency needs to integrate the strong culture, structures, and rigorous ideation process that ARPAs across government have honed since the 1950s. This report is a primer on how ARPA-I, and its stakeholders, can leverage this unique opportunity to drive real, sustainable, and lasting change in America’s transportation infrastructure.

How to Use This Report

This report highlights the opportunity ARPA-I presents; orients those unfamiliar with the transportation infrastructure sector to the unique challenges it faces; provides a foundational understanding of the ARPA model and its early-stage program design; and empowers experts and stakeholders to get involved in program ideation. However, individual sections can be used as standalone tools depending on the reader’s prior knowledge of and intended involvement with ARPA-I.

An Opportunity for Transportation Infrastructure Innovation

In November 2021, Congress passed the Infrastructure Investment and Jobs Act (IIJA) authorizing the U.S. Department of Transportation (USDOT) to create the Advanced Research Projects Agency-Infrastructure (ARPA-I), among other new programs. ARPA-I’s mission is to advance U.S. transportation infrastructure by developing innovative science and technology solutions that:

ARPA-I will achieve this goal by supporting research projects that:

ARPA-I is the newest addition to a long line of successful ARPAs that continue to deliver breakthrough innovations across the defense, intelligence, energy, and health sectors. The U.S. Department of Defense established the pioneering Defense Advanced Research Projects Agency (DARPA) in 1958 in response to the Soviet launch of the Sputnik satellite to develop and demonstrate high-risk, high-reward technologies and capabilities to ensure U.S. military technological superiority and confront national security challenges. Throughout the years, DARPA programs have been responsible for significant technological advances with implications beyond defense and national security, such as the early stages of the internet, the creation of the global positioning system (GPS), and the development of mRNA vaccines critical to combating COVID-19. 

In light of the many successful advancements seeded through DARPA programs, the government replicated the ARPA model for other critical sectors, resulting in the Intelligence Advanced Research Projects Activity (IARPA) within the Office of the Director of National Intelligence, the Advanced Research Projects Agency-Energy within the Department of Energy, and, most recently, the Advanced Research Projects Agency-Health (ARPA-H) within the Department of Health and Human Services.

Now, there is the opportunity to bring that same spirit of untethered innovation to solve the most pressing transportation infrastructure challenges of our time. The United States has long faced a variety of transportation infrastructure-related challenges, due in part to low levels of federal research and development (R&D) spending in this area; the fragmentation of roles across federal, state, and local government; risk-averse procurement practices; and sluggish commercial markets. These challenges include:

The Fiscal Year 2023 Omnibus Appropriations Bill awarded ARPA-I its initial appropriation in early 2023. Yet even before that, the Biden-Harris Administration saw the potential for ARPA-I-driven innovations to help meet its goal of net-zero GHG emissions by 2050, as articulated in its Net-Zero Game Changers Initiative. In particular, the Administration identified smart mobility, clean and efficient transportation systems, next-generation infrastructure construction, advanced electricity infrastructure, and clean fuel infrastructure as “net-zero game changers” that ARPA-I could play an outsize role in helping develop.

For ARPA-I programs to reach their full potential, agency stakeholders and partners need to understand not only how to effectively apply the ARPA model but how the unique circumstances and challenges within transportation infrastructure need to be considered in program design.

Unique Challenges of the Transportation Infrastructure Landscape

Using ARPA-I to advance transportation infrastructure breakthroughs requires an awareness of the most persistent challenges to prioritize and the unique set of circumstances within the sector that can hinder progress if ignored. Below are summaries of key challenges and considerations for ARPA-I to account for, followed by a deeper analysis of each challenge.

Lower Federal R&D Spending in Transportation Infrastructure 

Federal R&D expenditures in transportation infrastructure lag behind those in other sectors. This gap is particularly acute because, unlike for some other sectors, federal transportation R&D expenditures often fund studies and systems used to make regulatory decisions rather than technological innovation. The table below compares actual federal R&D spending and sector expenditures for 2019 across defense, healthcare, energy, and transportation as a percentage of each sector’s GDP. The federal government spends orders of magnitude less on transportation than other sectors: energy R&D spending as a percentage of sector GDP is nearly 15 times higher than transportation, while health is 13 times higher and defense is nearly 38 times higher.

Agency ^1Actual federal R&D spending, 2019Value added by industry and % of U.S. GDP, 20192019 federal R&D spending as % of sector GDP
Defense$54.69 billion$732 billion (3.4%)7.5%
Health and Human Services$38.51 billion$1,452 billion (6.8%) ^22.7%
Energy$18.27 billion$607 billion (2.8%) ^33.0%
Transportation$1.10 billion$610 billion (2.9%) ^40.2%
Expand Footnotes

1. The comparison of federal R&D spending and sector expenditures for 2019 is similar to those for the years 2020 and 2021.

2. Excludes GDP value-adds relating to Social Assistance

3. Includes GDP value-adds relating to oil and gas extraction, utilities, and petroleum and coal products

4. Excludes GDP value adds relating to Warehousing

Public Sector Dominance Limits Innovation Investment 

Since 1990, total investment in U.S. R&D has increased by roughly 9 times. When looking at the source of R&D investment over the same period, the private and public sectors invested approximately the same amount of R&D funding in 1982, but today the rate of R&D investment is nearly 4 times greater for the private industry than the government. 

While there are problems with the bulk of R&D coming from the private sector, such as innovations to promote long-term public goods being overlooked because of more lucrative market incentives, industries that receive considerable private R&D funding still see significant innovation breakthroughs. For example, the medical industry saw $161.8 billion in private R&D funding in 2020 compared to only $61.5 billion from federal funding. More than 75% of this private industry R&D occurred within the biopharmaceutical sector where corporations have profit incentives to be at the cutting edge of advancements in medicine.

The transportation sector has one robust domain for private R&D investment: vehicle and aircraft equipment manufacturing. In 2018, total private R&D was $52.6 billion. Private sector transportation R&D focuses on individual customers and end users, creating better vehicles, products, and efficiencies. The vast majority of that private sector R&D does not go toward infrastructure because the benefits are largely public rather than private. Put another way, the United States invests more than 50 times the amount of R&D into vehicles than the infrastructure systems upon which those vehicles operate. 

Market Fragmentation across Levels of Government

Despite opportunities within the public-dominated transportation infrastructure system, market fragmentation is a persistent obstacle to rapid progress. Each level of government has different actors with different objectives and responsibilities. For instance, at the federal level, USDOT provides national-level guidance, policy, and funding for transportation across aviation, highway, rail, transit, ports, and maritime modes. Meanwhile, the states set goals, develop transportation plans and projects, and manage transportation networks like the interstate highway system. Metropolitan planning organizations take on some of the planning functions at the regional level, and local governments often maintain much of their infrastructure. There are also local individual agencies that operate facilities like airports, ports, or tollways organized at the state, regional, or local level. Programs that can use partnerships to cut across this tapestry of systems are essential to driving impact at scale. 

Local agencies have limited access and capabilities to develop cross-sector technologies. They have access to limited pools of USDOT funding to pilot technologies and thus generally rely on commercially available technologies to increase the likelihood of pilot success. One shortcoming of this current process is that both USDOT and infrastructure owner-operators (IOOs) play a more passive role in developing innovative technologies, instead depending on merely deploying market-ready technologies. 

Multiple Modes, Customers, and Jurisdictions Create Difficulties in Efficiently Allocating R&D Resources

The transportation infrastructure sector is a multimodal environment split across many modes, including aviation, maritime, pipelines, railroads, roadways (which includes biking and walking), and transit. Each mode includes various customers and stakeholders to be considered. In addition, in the fragmented market landscape federal, state, and local departments of transportation have different—and sometimes competing—priorities and mandates. This dynamic creates difficulties in allocating R&D resources and considering access to innovation across these different modes.

Customer identification is not “one size fits all” across existing ARPAs. For example, DARPA has a laser focus on delivering efficient innovations for one customer: the Department of Defense. For ARPA-E, it is less clear; their customers range from utility companies to homeowners looking to benefit from lower energy costs. ARPA-I would occupy a space in between these two cases, understanding that its end users are IOOs—entities responsible for deploying infrastructure in many cases at the local or regional level. 

However, even with this more direct understanding of its customers, a shortcoming of a system focused on multiple modes is that transportation infrastructure is very broad, occupying everything from self-healing concrete to intersection safety to the deployment of electrified mobility and more. Further complicating matters is the rapid evolution of technologies and expectations across all modes, along with the rollout of entirely new modes of transportation. These developments raise questions about where new technologies and capabilities fit in existing modal frameworks, what actors in the transportation infrastructure market should lead their development, and who the ultimate “customers” or end users of innovation are.

Having a matrixed understanding of the rapid technological evolution across transportation modes and their potential customers is critical to investing in and building infrastructure for the future, given that transportation infrastructure investments not only alter a region’s movement of people and goods but also fundamentally impact its development. ARPA-I is poised to shape learnings across and in partnership with USDOT’s modes and various offices to ensure the development and refinement of underlying technologies and approaches that serve the needs of the entire transportation system and users across all modes.

Core Tenets of ARPA Success

Success using the ARPA model comes from demonstrating new innovative capabilities, building a community of people (an “ecosystem”) to carry the progress forward, and having the support of key decision-makers. Yet the ARPA model can only be successful if its program directors (PDs), fellows, stakeholders, and other partners understand the unique structure and inherent flexibility required when working to create a culture conducive to spurring breakthrough innovations. From a structural and cultural standpoint, the ARPA model is unlike any other agency model within the federal government, including all existing R&D agencies. Partners and other stakeholders should embrace the unique characteristics of an ARPA.

Cultural Components

ARPAs should take risks. 

An ARPA portfolio may be the closest thing to a venture capital portfolio in the federal government. They have a mandate to take big swings so should not be limited to projects that seem like safe bets. ARPAs will take on many projects throughout their existence, so they should balance quick wins with longer-term bets while embracing failure as a natural part of the process.

ARPAs should constantly evaluate and pivot when necessary.

An ARPA needs to be ruthless in its decision-making process because it has the ability to maneuver and shift without the restriction of initial plans or roadmaps. For example, projects around more nascent technology may require more patience, but if assessments indicate they are not achieving intended outcomes or milestones, PDs should not be afraid to terminate those projects and focus on other new ideas.

ARPAs should stay above the political fray. 

ARPAs can consider new and nontraditional ways to fund innovation, and thus should not be caught up in trends within their broader agency. As different administrations onboard, new offices get built and partisan priorities may shift, but ARPAs should limit external influence on their day-to-day operations.

ARPA team members should embrace an entrepreneurial mindset. 

PDs, partners, and other team members need to embrace the creative freedom required for success and operate much like entrepreneurs for their programs. Valued traits include a propensity toward action, flexibility, visionary leadership, self-motivation, and tenacity.

ARPA team members must move quickly and nimbly.

Trying to plan out the agency’s path for the next two years, five years, 10 years, or beyond is a futile effort and can be detrimental to progress. ARPAs require ultimate flexibility from day to day and year to year. Compared to other federal initiatives, ARPAs are far less bureaucratic by design, and forcing unnecessary planning and bureaucracy on the agency will slow progress.

Collegiality must be woven into the agency’s fabric.

With the rapidly shifting and entrepreneurial nature of ARPA work, the federal staff, contractors, and other agency partners need to rely on one another for support and assistance to seize opportunities and continue progressing as programs mature and shift. 

Outcomes matter more than following a process.

ARPA PDs must be free to explore potential program and project ideas without any predetermination. The agency should support them in pursuing big and unconventional ideas unrestricted by a particular process. While there is a process to turn their most unconventional and groundbreaking ideas into funded and functional projects, transformational ideas are more important than the process itself during idea generation.

ARPA team members welcome feedback.

Things move quickly in an ARPA, and decisions must match that pace, so individuals such as fellows and PDs must work together to offer as much feedback as possible. Constructive pushback helps avoid blind alleys and thus makes programs stronger.

Structural Components

The ARPA Director sets the vision.

The Director’s vision helps attract the right talent and appropriate levels of ambition and focus areas while garnering support from key decision-makers and luminaries. This vision will dictate the types and qualities of PDs an ARPA will attract to execute within that vision.

PDs can make or break an ARPA and set the technical direction.

Because the power of the agency lies within its people, ARPAs are typically flat organizations. An ARPA should seek to hire the best and most visionary thinkers and builders as PDs, enable them to determine and design good programs, and execute with limited hierarchical disruption. During this process, PDs should engage with decision-makers in the early stages of the program design to understand the needs and realities of implementers.

Contracting helps achieve goals.

The ARPA model allows PDs to connect with universities, companies, nonprofits, organizations, and other areas of government to contract necessary R&D. This allows the program to build relationships with individuals without needing to hire or provide facilities or research laboratories. 

Interactions improve outcomes. 

From past versions of ARPA that attempted remote and hybrid environments, it became evident that having organic collisions across an ARPA’s various roles and programs is important to achieving better outcomes. For example, ongoing in-person interactions between and among PDs and technical advisors are critical to idea generation and technical project and program management. 

Staff transitions must be well facilitated to retain institutional knowledge. 

One of ARPA’s most unique structural characteristics is its frequent turnover. PDs and fellows are term-limited, and ARPAs are designed to turn over those key positions every few years as markets and industries evolve, so having thoughtful transition processes in place is vital, including considering the role of systems engineering and technical assistance (SETA) contractors in filling knowledge gaps, cultivating an active alumni network, and staggered hiring cycles so that large numbers of PDs and fellows are not all exiting their service at once.

Scaling should be built into the structure.

It cannot be assumed that if a project is successful, the private sector will pick that technology up and help it scale. Instead, an ARPA should create its own bridge to scaling in the form of programs dedicated to funding projects proven in a test environment to scale their technology for real-world application. 

Technology-to-market advisors play a pivotal role.

Similarly to the dedicated funding for scaling described above, technology-to-market advisors are responsible for thinking about how projects make it to the real world. They should work hand in hand with PDs even in the early stages of program development to provide perspectives on how projects might commercialize and become market-ready. Without this focus, technologies run the risk of dying on the vine—succeeding technically, but failing commercially. 

A Primer on ARPA Ideation

Tackling grand challenges in transportation infrastructure through ARPA-I requires understanding what is unique about its program design. This process begins with considering the problem worth solving, the opportunity that makes it a ripe problem to solve, a high-level idea of an ARPA program’s fit in solving it, and a visualization of the future once this problem has been solved. This process of early-stage program ideation requires a shift in one’s thinking to find ideas for innovative programs that fit the ARPA model in terms of appropriate ambition level and suitability for ARPA structure and objectives. It is also an inherently iterative process, so while creating a “wireframe” outlining the problem, opportunity, program objectives, and future vision may seem straightforward, it can take months of refining. 

Common Challenges

No clear diagnosis of the problem

Many challenges facing our transportation infrastructure system are not defined by a single problem; rather, they are a conglomeration of issues that simultaneously need addressing. An effective program will not only isolate a single problem to tackle, but it will approach it at a level where something can be done to solve it through root cause analysis.

Thinking small and narrow

On the other hand, problems being considered for ARPA programs can be isolated down to the point that solving them will not drive transformational change. In this situation, narrow problems would not cater to a series of progressive and complementary projects that would fit an ARPA.

Incorrect framing of opportunities:

When doing early-stage program design, opportunities are sometimes framed as “an opportunity to tackle a problem.” Rather, an opportunity should reflect a promising method, technology, or approach already in existence but which would benefit from funding and resources through an advanced research agency program.

Approaching solutions solely from a regulatory or policy angle

While regulations and policy changes are a necessary and important component of tackling challenges in transportation infrastructure, approaching issues through this lens is not the mandate of an ARPA. ARPAs focus on supporting breakthrough innovations in developing new methods, technologies, capabilities, and approaches. Additionally, regulatory approaches to problem-solving can often be subject to lengthy policy processes.

No explicit ARPA role

An ARPA should pursue opportunities to solve problems where, without its intervention, breakthroughs may not happen within a reasonable timeframe. If the public or private sector already has significant interest in solving a problem, and they are well on their way to developing a transformational solution in a few years or less, then ARPA funding and support might provide a higher value-add elsewhere.

Lack of throughline

The problems identified for ARPA program consideration should be present as themes throughout the opportunities chosen to solve them as well as how programs are ultimately structured. Otherwise, a program may lack a targeted approach to solving a particular challenge.

Forgetting about end users

Human-centered design should be at the heart of how ARPA programs are scoped, especially when considering the scale at which designers need to think about how solving a problem will provide transformational change for everyday users.

Being solutions-oriented

Research programs should not be built with predetermined solutions in mind; they should be oriented around a specific problem to ensure that any solutions put forward are targeted and effective.

Not being realistic about direct outcomes of the program

Program objectives should not simply restate the opportunity, nor should they jump to where the world will be many years after the program has run its course. They should separate the tactical elements of a program and what impact they will ultimately drive. Designers should consider their program as one key step in a long arc of commercialization and adoption, with a firm sense of who needs to act and what needs to happen to make a program objective a reality.

Keeping these common mistakes in mind throughout the design process ensures that programs are properly scoped, appropriately ambitious, and in line with the agency’s goals. With these guideposts in mind, idea generators should begin their program design in the form of a wireframe.

Wireframe Development 

The first phase in ARPA program development is creating a program wireframe, which is an outline of a potential program that captures key components for consideration to assess the program’s fit and potential impact. The template below shows the components characteristic of a program wireframe.

Wireframe template

To create a fully fleshed-out wireframe, program directors work backward by first envisioning a future state that would be truly transformational for society and across sectors if it were to be realized. Then, they identify a clearly-articulated problem that needs solving and is hindering progress toward this transformational future state. During this process, PDs need to conduct extensive root cause analysis to consider whether the problem they’ve identified is exacerbated by policy, regulatory, or environmental complications—as opposed to those that technology can already solve. This will inform whether a problem is something that ARPA-I has the opportunity to impact fundamentally. 

Next, program directors identify a promising opportunity—such as a method, approach, or technology—that, if developed, scaled, and implemented, would solve the problem they articulated and help achieve their proposed future state. When considering a promising opportunity, PDs must assess whether it front-runs other potential technologies that would also need developing to support it and whether it is feasible to achieve concrete results within three to five years and with an average program budget. Additionally, it is useful to think about whether an opportunity considered for program development is part of a larger cohort of potential programs that lie within an ARPA-I focus area that could all be run in parallel.

Most importantly, before diving into how to solve the problem, PDs need to articulate what has prevented this opportunity from already being solved, scaled, and implemented, and what explicit role or need there is for a federal R&D agency to step in and lead the development of technologies, methods, or approaches to incentivize private sector deployment and scaling. For example, if the private sector is already incentivized to, and capable of, taking the lead on developing a particular technology and it will achieve market readiness within a few years, then there is less justification for an ARPA intervention in that particular case. On the other hand, the prescribed solution to the identified problem may be so nascent that what is needed is more early-stage foundational R&D, in which case an ARPA program would not be a good fit. This area should be reserved as the domain of more fundamental science-based federal R&D agencies and offices.

One example to illustrate this maturity fit is DARPA investment in mRNA. While the National Institutes of Health contributed significantly to initial basic research, DARPA recognized the technological gap in being able to quickly scale and manufacture therapeutics, prompting the agency to launch the Autonomous Diagnostics to Enable Prevention and Therapeutics (ADEPT) program to develop technologies to respond to infectious disease threats. Through ADEPT, in 2011 DARPA awarded a fledgling Moderna Therapeutics with $25 million to research and develop its messenger RNA therapeutics platform. Nine years later, Moderna became the second company after Pfizer-BioNTech to receive an Emergency Use Authorization for its COVID-19 vaccine.

Another example is DARPA’s role in developing the internet as we know it, which was not originally about realizing the unprecedented concept of a ubiquitous, global communications network. What began as researching technologies for interlinking packet networks led to the development of ARPANET, a pioneering network for sharing information among geographically separated computers. DARPA then contracted BBN Technologies to build the first routers before becoming operational in 1969. This research laid the foundation for the internet. The commercial sector has since adopted ARPANET’s groundbreaking results and used them to revolutionize communication and information sharing across the globe.

Wireframe Refinement and Iteration

To guide program directors through successful program development, George H. Heilmeier, who served as the director of DARPA from 1975 to 1977, used to require that all PDs answer the following questions, known as the Heilmeier Catechism, as part of their pitch for a new program. These questions should be used to refine the wireframe and envision what the program could look like. In particular, wireframe refinement should examine the first three questions before expanding to the remaining questions.

Alongside the Heilmeier Catechism, a series of assessments and lines of questioning should be completed to pressure test and iterate once the wireframe has been drafted. This refinement process is not one-size-fits-all but consistently grounded in research, discussions with experts, and constant questioning to ensure program fit. The objective is to thoroughly analyze whether the problem we are seeking to solve is the right one and whether the full space of opportunities around that problem is ripe for ARPA intervention.

One way to think about determining whether a wireframe could be a program is by asking, “Is this wireframe science or is this science fiction?” In other words, is the proposed technology solution at the right maturity level for an ARPA to make it a reality? There is a relatively broad range in the middle of the technological maturity spectrum that could be an ARPA program fit, but the extreme ends of that spectrum would not be a good fit, and thus those wireframes would need further iteration or rejection. On the far left end of the spectrum would be basic research that only yields published papers or possibly a prototype. On the other extreme would be a technology that is already developed to the point that only full-scale implementation is needed. Everything that falls between could be suitable for an ARPA program topic area.  

Once a high-impact program has been designed, the next step is to rigorously pressure test and develop a program until it resembles an executable ARPA program.

Applying ARPA Frameworks to Transportation Infrastructure Challenges

By using this framework, any problem or opportunity within transportation infrastructure can be evaluated for its fit as an ARPA-level idea. Expert and stakeholder idea generation is essential to creating an effective portfolio of ARPA-I programs, so idea generators must be armed with this framework and a defined set of focus areas to develop promising program wireframes. An initial set of focus areas for ARPA-I includes safety, climate and resilience, and digitalization, with equity and accessibility as underlying considerations within each focus area. 

There are hundreds of potential topic areas that ARPA-I could tackle; the two wireframes below represent examples of early-stage program ideas that would benefit from further pressure testing through the program design iteration cycle.

Note: The following wireframes are samples intended to illustrate ARPA ideation and the wireframing process, and do not represent potential research programs or topics under consideration by the U.S. Department of Transportation.

Next-Generation Resilient Infrastructure Management

A Digital Inventory of Physical Infrastructure and Its Uses

Wireframe Development Next Steps

After initial wireframe development, further exploration is needed to pressure test an idea and ensure that it can be developed into a viable program to achieve “moonshot” ambitions. Wireframe authors should consider the following factors when iterating:

Wireframes are intended to be a summary communicative of a larger plan to follow. After further iteration and exploration of the factors outlined above, what was first just a raw program wireframe should develop into more detailed documents. These should include an incisive diagnosis of the problem and evidence and citations validating opportunities to solve it. Together, these components should lead to a plausible program objective as an outcome.

Conclusion

The newly authorized and appropriated ARPA-I presents a once-in-a-generation opportunity to apply a model that has been proven successful in developing breakthrough innovations in other sectors to the persistent challenges facing transportation infrastructure.

Individuals and organizations that would work within the ARPA-I network need to have a clear understanding of the unique circumstances, challenges, and opportunities of this sector, as well as how to apply this context and the unique ARPA program ideation model to build high-impact future innovation programs. This community’s engagement is critical to ARPA-I’s success, and the FAS is looking for big thinkers who are willing to take on this challenge by developing bold, innovative ideas.

To sign up for future updates on events, convenings, and other opportunities for you to work in support of ARPA-I programs and partners, click here.

To submit an advanced research program idea, click here.

initiatives
See all
Advanced Research at DOT

We’re scoping ambitious ideas to help ARPA-I begin executing projects that improve transportation across the country.

read more