ARPA-I: Get Involved
FAS is seeking to engage experts from across the transportation infrastructure community who are the right kind of big thinkers to get involved in developing solutions to transportation moonshots.
Widespread engagement of this diverse network is critical to ensuring ARPA-I’s success. So whether you are an academic researcher, startup CEO, safe streets activist, or have experience with federal R&D programs–we are looking for your insights and expertise.
To be considered for opportunities to support future efforts around transportation infrastructure moonshots, please fill out this form and a member of our team will be in touch as opportunities to get involved arise.
ARPA-I: Share an Idea
Do you have ideas that could inform an ambitious Advanced Research Projects Agency-Infrastructure (ARPA-I) portfolio at the U.S. Department of Transportation (DOT)? We’re looking for your boldest infrastructure moonshots.
The Federation of American Scientists (FAS) is seeking to engage experts across the transportation policy space who can leverage their expertise to help FAS identify a set of grand solutions around transportation infrastructure challenges and advanced research priorities for DOT to consider. Priority topic areas include but are not limited to metropolitan safety, rural safety, resilient and climate-prepared infrastructure, digital infrastructure, expediting “mega projects,” and logistics. You can read more about these topic areas in depth here.
What We’re Looking For and How to Submit
We are looking for experts to develop and submit an initial program design in the form of a wireframe that could inform a future advanced research portfolio at DOT. A wireframe is an outline of a potential program that captures key components that need to be considered in order to assess the program’s fit and potential impact. The template below reflects the components of a program wireframe. Wireframes can be submitted by email here. Please include all four sections of the wireframe shown in the template below in the body of your email submission.
![](https://fas.org/wp-content/uploads/2023/01/Untitled-presentation.png)
When writing your wireframe, we ask you aim to avoid the following common challenges to ensure that ideas are properly scoped, appropriately ambitious, and are in line with the agency’s goals:
- No clear diagnosis of the problem: Many challenges facing our transportation infrastructure are not defined by a single problem; rather, they are an ecosystem of issues that simultaneously need addressing. An effective program will not only isolate a single “problem” to tackle, but it will approach it at a level where something can actually be done to solve it through root cause analysis.
- Thinking small and narrow: On the other hand, problems being considered for advanced research programs can be isolated down to the point that solving them will not drive transformational change. In this situation, narrow problems would not cater to a series of progressive and complimentary projects that would fit an ARPA.
- Incorrect framing of opportunities: When doing early-stage program design, opportunities are sometimes framed as “an opportunity to tackle a problem.” Rather, an opportunity should reflect a promising method, technology, or approach that is already in existence but would benefit from funding and resources through an advanced research agency program.
- Approaching solutions solely from a regulatory or policy angle: While regulations and policy changes are a necessary and important component of tackling challenges in transportation infrastructure, approaching issues through this lens is not the mandate of an ARPA. ARPAs focus on supporting breakthrough innovations across methods, technologies, and approaches. Additionally, regulatory approaches to problem solving can often be subject to lengthy policy processes.
- No explicit ARPA role: An ARPA should pursue opportunities to solve problems where, without its intervention, breakthroughs may not happen within a reasonable timeframe. If solving a problem already has significant interest from the private or public sector, and they are well on their way to developing a transformational solution in a few years time, then ARPA funding and support might provide a higher value-add elsewhere.
- Lack of throughline: The problems identified for ARPA program consideration should be present as themes throughout the opportunities chosen to solve them as well as how programs are ultimately structured–otherwise, a program may lack a targeted approach to solving a particular challenge.
- Forgetting about end-users: Human-centered design should be at the heart of how ARPA programs are scoped, especially when thinking about the scale at which designers need to think about how solving a problem will provide transformational change for everyday users of transportation infrastructure.
- Being solutions-oriented: Research programs should not be built with pre-determined solutions already in mind; they should be oriented around a specific problem in order to ensure that any solutions put forward are targeted and effective.
For a more detailed primer on ARPA program ideation, please read our publication, “Applying ARPA-I: A Proven Model for Transportation.”
Sample Idea
Informed by input from non-federal subject matter experts
Problem
Urban and suburban environments are complex, with competing uses for public space across modes and functions – drivers, transit users, cyclists, pedestrians, diners, etc. Humans are prone to erratic, unpredictable, and distracted driving behavior, and when coupled with speed, vehicle size, and infrastructure design, such behaviors can cause injury, death, property damage, and transportation system disruption. A decade-old study from NHTSA – at a time when roadway fatalities were approximately 25% lower than current levels – found that the total value of societal harm from crashes in 2010 was $836 billion.
Opportunity
What if the relationships between the driver, the environment (including pedestrians), and the vehicle could be personalized?
- Driver-to-Vehicle: Using novel data gathered from cars’ sensors, driver smartphones, and other collectible data, design a feedback loop that customizes Advanced Driver Assistance Systems (ADAS) to unique driving behavior signatures.
- Vehicle-to-Environment: Using V2I/V2X and geofencing technologies to govern and harmonize speed and lane operations that optimize max speeds for safety in unique street contexts.
- Driver-to-Environment: Blending both D2V and V2E technologies, develop integrated awareness of the surrounding environment that alerts drivers of potential risks in parked (e.g., car door opening to a bike lane) and moving states (e.g., approaching car).
Program Objective
- Driver-to-Vehicle: (1) Identify the totality of usable driver data within the vehicular environment, from car sensors to phone usage; (2) develop a series of driver profiles that will build the foundation for human-centered, personalized ADAS that can both intervene in an emergency and nudge behavior change through informational updates, intuitive behavioral feedback, or modifying vehicle operations (e.g., acceleration); (3) develop dynamic, intelligent ADAS systems that customize to driver signatures based on preset profiles and experiential, local training of the algorithm; (4) establish this as a proof of concept for a novel, personalized ADAS and architect a grand-challenge for industry to improve upon this personalized, human-centered ADAS with key target metrics; (5) create a regulatory framework mandating Original Equipment Manufacturers (OEMs) to include a baseline level of ADAS, given the results of the grand challenge.
- Vehicle-to-Environment: (1) Design the universal mobile application or geofence trigger that will contour virtual boundaries for a set of diverse, transferrable streets (e.g., school zones) and characteristics (e.g., bike lanes); (2) engage OEMs to design and integrate the geofence triggers with the human-centered ADAS and/or another vehicle-based receiver within a test fleet of different car types to modify vehicle responses to the geofence criteria as outlined by the pilot cities; (3) broker partnerships with 10 cities to identify a menu of geofence criteria, pilot the use of them, and establish a mechanism to measure before-and-after outcomes and comparisons from neighboring regions;
- Driver-to-Environment: integrate ADAS with the geofence trigger to develop an advanced and dynamic situational awareness environment for drivers that is customized to their profile and based on built environment conditions such as bike lanes and school zones, as well as weather, high traffic, and time of day.
Future
Digital transportation networks can communicate personalized information with drivers through their cars in a uniform medium and with a goal of augmenting safety in each of the nation’s largest metropolitan areas.
USDOT Workshop: Transportation, Mobility, and the Future of Infrastructure
On December 8th, 2022, the U.S. Department of Transportation hosted a workshop, “Transportation, Mobility, and the Future of Infrastructure,” in collaboration with the Federation of American Scientists.
The goal for this event was to bring together innovative thinkers from various sectors of infrastructure and transportation to scope ideas where research, technology, and innovation could drive meaningful change for the Department of Transportation’s strategic priorities.
To provide framing for the day, participants heard from Secretary of Transportation Pete Buttigieg and Deputy Assistant Secretary for Research and Technology Robert Hampshire, who both underscored the potential for a new agency – The Advanced Research Projects Agency – Infrastructure (ARPA-I) to accelerate transformative solutions for the transportation sector. Then, a panel featuring Kei Koizumi, Jennifer Gerbi, and Erwin Gianchandani focused on Federal Research and Development (R&D) explored federal advanced research models that drive innovation in complex sectors and explored how such approaches may accelerate solutions to key priorities in the transportation system.
![](https://fas.org/wp-content/uploads/2022/12/Screen-Shot-2023-05-22-at-2.11.17-PM.png)
Workshop participants listening to remarks from U.S. Transportation Secretary Pete Buttigieg.
Participants then participated in separate breakout sessions organized around: 1) safety; 2) digitalization; and 3) climate and resilience. During the breakouts sessions, participants were asked to build on pre-work they had completed before the Workshop by brainstorming future vision statements and using them as the foundation to come up with innovative federal R&D program designs. Participants then regrouped and ended the day by discussing the most promising ideas from their respective breakout sessions, and where their ideas could go next.
The Workshop inspired participants to dig deep to surface meaningful challenges and innovative solutions for USDOT to tackle, whether through ARPA-I or other federal R&D mechanisms, and represents an initial step of a broader process to identify topics and domains in which stakeholders can drive transformational progress for our infrastructure and transportation system. Such an effort will require continued engagement and buy-in from a diverse community of experts.
As such, FAS is seeking to engage experts from across the transportation infrastructure community who are willing to “think big” and creatively about solutions to transportation moonshots. If you’re interested in supporting future efforts around transportation infrastructure moonshots, please visit our “Get Involved” page; if you’re ready to submit an initial program design in the form of a wireframe that could inform a future advanced research portfolio at DOT, please visit our “Share an Idea” page.
Building Momentum for Equity in Medical Devices
Just over a year ago, I found myself pausing during a research lab meeting. “Why were all the subjects in our studies of wearable devices white? And what were the consequences of exclusion?”
This question stuck with me long after the meeting. Digging into the evidence, I was alarmed to find paper after paper signaling embedded biases in key medical technologies.
One device stuck out amongst the rest – the pulse oximeter. Because of its crucial role in diagnosing COVID-19, it had caught the attention of a diverse group of stakeholders: clinicians looking to understand the impacts on patient care, engineers working to build more equitable devices, social scientists tracing the history of device and examining colorism in pulse oximetry, policymakers seeking solutions for their constituents, and the FDA, which was examining racial bias in medical technologies for the first time. But what I found as I scoped out this policy area is that these stakeholders weren’t talking to one another, at the expense of coordinated progress towards equity in pulse oximetry.
With all eyes directed towards the FDA’s Advisory Committee meeting on November 1st, 2022, FAS convened a half-day session of stakeholders on November 2nd to chart a research and policy agenda for near-term mitigation of inequities in pulse oximetry and other medical technologies. Eight experts from medicine, engineering, sociology, and anthropology shared insights with an audience of 60 participants from academia, the private sector, and federal government. Collectively, we developed several key insights for future progress on this issue and outlined a path forward for achieving equity now. You can access the full readout here. We’ll dive into the key highlights below:
Key Insights
Through discussions with experts during the forum, three key themes rose to the surface:
- Racial bias in pulse oximetry cannot be fixed by focusing on “race” alone. Existing evidence suggests reducing bias in pulse oximetry requires replacing devices with less-biased ones. This will take time as new devices are developed and will be a significant cost.
- Better calibration for skin tone is vital, but measurement is complicated. The crux of the problem is a comprehensive standard for quantifying the full range of skin pigmentation. This is vital to understanding how pulse oximeter accuracy varies by melanin content.
- Proactively identifying and addressing bias in medical devices will require system-wide efforts. Identification of bias in medical devices has been piecemeal rather than the outcome of proactive, deliberative efforts. Further efforts to address bias in medical devices should engage diverse stakeholders to establish best practices for ensuring equity in medical devices.
Resolving the problem of bias in pulse oximeter devices will likely take several years. But in the meantime, this issue will continue negatively impacting patients. Our participants urged that we need to think about actions that can be initiated this next year that will advance more equitable care with existing pulse oximeters.
![](https://fas.org/wp-content/uploads/2023/04/IMG_4885-2048x1536-1.jpeg)
Motivating Action for Equity Now
While a daunting problem, a collaborative, multi-stakeholder effort can bring us closer to solutions. We can work together to advance equity in standards of care by:
- Gathering evidence on existing pulse oximeter devices and their use in care [ASAP, start early 2023]. More evidence is required to identify the best approaches to equitable care with existing devices. This evidence gathering process should be initiated over the next year to inform clinicians on
- Establishing consensus to advance the standard of care [start early 2024]. After growing the body of evidence, there will be a need to convene around key conclusions derived from the evidence. Evidence synthesis will need to be generated and care societies will need to make decisions on how clinicians should use pulse oximeters in their care practice.
- Taking action to ensure equitable care nationwide [2024 onwards]. Once the care standards are changed, there is a need for system-wide efforts to communicate these to clinicians nationwide, inform procurement across federal hospitals, and re-evaluate insurance reimbursement standards.
![](https://fas.org/wp-content/uploads/2023/04/Screen-Shot-2022-12-02-at-8.10.36-AM.png)
Looking Ahead
This won’t be easy, but it’s 30 years overdue. We believe correcting the bias will pioneer a model that can be readily applied to combatting biases across the medical device ecosystem, something already underway in the United Kingdom with their Equity in Medical Devices Independent Review. Through a systematic approach, stakeholders can work to close racial disparities in the near-term and advance health equity.
An Overdue Fix: Racial Bias and Pulse Oximeters
The invention of pulse oximeters in the 1980s reshaped healthcare. While tracking blood oxygen content (commonly recognized as the “fifth vital sign”) once required a painful blood draw and time-delayed analysis, pulse oximeters deliver nearly instantaneous data by simply sending a pulse of light through the skin. Today, pulse oximeters today are ubiquitous: built into smartwatches, purchased at pharmacies for home health monitoring, and used by clinicians to inform treatment of everything from asthma to heart failure to COVID-19. Emerging algorithms are even incorporating pulse ox data to predict future illness.
There is a huge caveat. Pulse oximeters are medically transformative, but racially biased. The devices work less accurately on dark-skinned populations because melanin, the chemical which gives skin pigment, interferes with light-based pulse ox measurements. This means that dark-skinned individuals can exhibit normal pulse ox readings, but be suffering from hypoxemia or other critical conditions.
But because regulations to this day do not require diversity in medical device evaluation, many pulse ox manufacturers don’t test their devices on diverse populations. And because the Food and Drug Administration (FDA) has created streamlined pathways to approve new medical devices based on technology that is “substantially similar” to already-approved technology, the racial bias embedded in ‘80s-era pulse ox technology continues to pervade pulse oximeters on the market today.
COVID-19 illustrated, in devastating fashion, the consequences of this problem. Embedded bias in pulse oximeters demonstrably worsened outcomes for patient populations already disproportionately impacted by COVID-19. Studies show, for instance, that Black COVID-19 patients have been 29% less likely to receive supplemental oxygen on time and three times as likely to suffer occult hypoxemia during the pandemic.
Similar inequities persist across the health-innovation ecosystem. Women suffer from lack of sex-aware prescription drug dosages. Minorities increasingly suffer from biased health risk-assessment algorithms. Children and those with varying body types suffer from medical equipment not built for their physical characteristics. Across the board, inequities create greater risks of morbidity and mortality and contribute to ballooning national healthcare costs.
This need not be the status quo. If health stakeholders—including patient advocates, medtech companies, clinicians, researchers, and policymakers—collectively commit to systematic evaluation and remediation of bias in health technology, change is possible.
An excellent example is eGFR algorithms. These algorithms, used to assess kidney functionality, previously used faulty “correction factors” to account for patient race. But this correction did not actually correlate with biological realities—and instead of treating patients more effectively, it increased disparities in care. Motivated by the data, advocacy and industry organizations issued broad recommendations to avoid using the eGFR calculation. Hospitals and medical systems listened, dropping eGFR from practice, and the National Institutes of Health (NIH) is now committing funding to investigate alternative calculations.
We as a society must continue to root out bias in health technology, from development to testing to deployment.
When we develop new medical tools, we should consider all the populations who could ultimately need them.
When we test tools, we should rigorously evaluate outcomes across subgroup populations, looking for groups that might fare better or worse from its use in care.
And when we deploy technologies, we need to be ready to track the outcomes of their use at scale.
Engineers, researchers, and clinicians can support these goals by designing medical devices with equity in mind. The UK just launched its evidence-gathering process on equity in medical devices, looking into the impacts of bias and ways to build more equitable solutions. The FDA’s meeting reviewing the evidence on pulse oximetry is a start to auditing technologies for their performance on different populations.
Advocacy organizations can support these goals by providing input to ongoing policy processes. The Federation of American Scientists (FAS), alongside the University of Maryland Medical System, submitted a public comment to the FDA to call for regulations that will encourage the development of low-bias and bias-free tools. FAS is also convening a Forum on Bias in Pulse Oximetry to examine the consequences of bias, build an evidence base for bias-free pulse oximetry, and look ahead to approaches to build more equitable devices.
“Do no harm”, a central oath in medicine, is becoming exceedingly difficult in our technological age. Yet, with an evidence-based approach that ensures technologies equitably serve all groups in a population and works to correct them when they do not, we can come closer to achieving this age-old goal.
What we learned in Mexico City
Moonshots seem impossible—until they’ve hit their target. This was the mantra of our in-person accelerator workshop, hosted with our partners at Unlock Aid in Mexico City. The workshop was just one part of our larger accelerator process where we’re working with innovators to develop moonshots around global development targets. FAS’s largest policy-development convening to date brought together over 70 participants (representing 40+ organizations, 25+ countries, and six continents) to think through creative approaches for achieving the United Nations Sustainable Development Goals by 2030.
![](https://fas.org/wp-content/uploads/2022/09/ROD_0417-300x200-1.jpeg)
After years of Zoom calls, phone conferences, and emails, meeting our accelerator cohort in person was a refreshing change of pace. Connecting IRL enabled free-flowing collaboration on big issues like global water security, access, and safety. We saw convergence across organizations: City Taps, Drinkwell, and Evidence Action came together to create the “WaterShot”—a new approach to solving water access with an outcomes marketplace framework. We were reminded anew that policy is powered by people—and that strong interpersonal connections inevitably lead to better and more creative policy ideas.
Brainstorming at the event was inspired by remarks from global development leaders. Project Drawdown spoke about the Drawdown Framework for climate solutions, NPX Advisors demonstrated how to drive better outcomes with advanced market commitments, and Nasra Ismail, a leader in global development strategy, talked about the power of coalition building.
![](https://fas.org/wp-content/uploads/2022/09/ROD_0582-300x200-1.jpeg)
The workshop was an initial opportunity to expose global development experts to the idea that policy, like seed funding or infrastructure investment, is an input that supports scaling. Most individual innovators are understandably hyper-focused on scaling up their individual ideas or products. But good policy is needed to build a flourishing global development environment—a rising tide that lifts all entrepreneurial ships. An underlying theme of the Mexico City workshop was the importance of policy as a growth enabler.
Now that we’re back in DC and over our jetlag, the accelerator continues, and we’re working with workshop participants to inform policymakers on key priorities for global development policy. We’re thinking about pain points in the field and opportunities for systems change, including earmarking funds for innovation, uplifting and incorporating community voices to policy, and setting new standards that focus on results.
The field of global development can be individualized and competitive—grants are few and far between, which doesn’t always foster shared best practices. But achieving the SDGs by 2030 must be a collaborative effort. Problems like climate change and food security are more pressing than ever, and they require an entirely new way of thinking about global development—finding and building on opportunities from proven results. Later this fall, look out for our participants’ moonshot memos as we roll them out. And if you have an idea about meeting the SDGs with a moonshot—or something else—why not submit it? Aiming at the moon is one thing, but getting there takes dedicated and sustained collaboration, and we’re so honored that these daring organizations want to work with us to do just that.
![](https://fas.org/wp-content/uploads/2022/09/ROD_0816-1024x683-1.jpeg)
The Day One Project is going international!
The Day One Project is going international! My colleague Josh Schoop and I will be spending this week in Mexico City with our partners at Unlock Aid, where we’ll be co-hosting the Reimagining the Future of Global Development Moonshot Accelerator. This will be our eighth accelerator cohort, and the very first in-person group.
We’re convening a group of 70 entrepreneurs, innovators, policymakers, and funders from around the world to think big about the future of global development and how government, business, industry, and aid can meet the UN’s Sustainable Development Goals (SDGs) by 2030. With the recent passage of a historic investment in the transition away from fossil fuels, the time is now to act on the biggest threats facing humanity. Sitting squarely within ‘the decade of delivery’—the remaining years we have to achieve the SDGs— this moonshot accelerator serves as an important call for action.
Our goal for this accelerator is to disrupt. We aim to generate new models to seed, scale, and implement catalytic solutions. The inspiration for the design of this accelerator comes from the “moonshot” model that the Kennedy administration pioneered to put the first man on the moon. A moonshot has since come to mean solving a daunting problem in an accelerated time period, requiring breakthrough, innovative, and radical thinking. If the past decades have not delivered the necessary change, then we have to shoot for the moon.
Over the course of the week, we’ll be working with the accelerator cohort to develop their own moonshots in health, the green economy, biodiversity, food and water insecurity, and more. These moonshots will be the building blocks for a Global Development Outcomes Marketplace, pitching funders and policymakers on new ways to unlock innovation in global development. Innovation here involves so much more than just new technology – it means new systems, processes, cooperation, organizing, and change, while ensuring that diverse perspectives are leading the way forward.
We’re really invested in these ideas and the people driving them, so here’s a sneak peek at a couple innovative groups joining us in Mexico City: Instiglio, headquartered in Bogotá, Colombia, is experimenting with new global development systems, like innovative financing methods, while SwipeRx in Indonesia is dedicated to revolutionizing the pharmacy industry with a tech-based solution. There are, of course, so many more groups who’ll be discussing and testing their ideas in-person, and the goal is to channel this cohort’s diverse expertise to create high-level, actionable recommendations for funders and multilateral organizations that center equity and outcomes.
I’m most excited to hear from experts and entrepreneurs from all over the world on what has been holding back progress despite attempted solutions, and I look forward to collaborating on what a new set of systems could look like. I’m excited to learn from our inspiring cohort, and to build mutual understanding across sectors to find ways to improve current solutions and break ground on a new path forward.I have high hopes that this accelerator will reinvigorate global development and uplift new voices and ideas in order to build a more prosperous planet. I resonate strongly with the voices of young people demanding urgent climate action worldwide–it’s time to harness this momentum to make sustainable, transformative change. Follow the journey of our first in-person accelerator on Twitter.
A Convening on The Future of U.S. Infrastructure Innovation
Background and Purpose
On July 26, 2022, MIT Mobility Initiative, MIT Washington Office, and The Engine hosted a workshop with leaders from the U.S. Department of Transportation (DOT) and infrastructure stakeholders — industry veterans, startup founders, federal, state and local policymakers and regulators, academics and investors.
The purpose of this convening was to engage a broad, diverse set of stakeholders in a series of ideation exercises to imagine what a set of ambitious advanced research programs could focus on to remake the future of American infrastructure. This read-out builds on a partnership FAS and the Day One Project have with the Department of Transportation to support solutions-based research and development. You can learn more about our work here.
The workshop consisted of two sessions. In the first working session, attendees discussed key challenges in infrastructure and possible research priority areas for ARPA-I. In the second half of the first session, participants were asked to come up with priority program areas that ARPA-I could focus on
During the second working session, participants considered the barriers that prevent the translation of breakthrough science and engineering into infrastructure reality, and opportunities for ARPA-I to smooth some of those frictions as an institution.
Resulting Recommendations
While some of the recommendations below may ultimately fall outside of ARPA-I’s mandate, or may require further Congressional authorization, they emphasize the need for ARPA-I to be strategically coordinating future deployment at scale even at the earliest stages of a project.
Deploying capital strategically
- Use existing or new authorities, such as consortium Other Transactions Authority, prize challenges, and public-private capital matching to ensure maximum flexibility and capital availability to fund complex, capital-intensive infrastructure investments.
- Create an Office of Scale-up within ARPA-I to ensure coordination across all “valleys of death” from early-stage basic research to full scale deployment. Mechanisms can range from early-stage open seed topics to later-stage Scale-up and deployment loan contracting mechanisms.
Establishing development and test infrastructure:
- The costs of infrastructure testbeds are prohibitive for most innovators. Creating government-sponsored testbeds, with participation from standards and regulatory bodies, would decrease the need for private capital and could create early linkages between innovators and those in charge of deployment. Existing national labs may have relevant expertise and equipment. Opening up access to these facilities through consortia, lighter weight Cooperative Research and Development Agreements (CRADAs), or other low-friction mechanisms may also have similar effects.
Catalyzing stakeholder collaboration:
- For early-stage researchers, create a community to share, discuss, and reflect upon the innovative landscape of infrastructure projects. To help reduce later-stage friction, create an Office of Strategic Engagement that would report to the ARPA-I director. This office would coordinate ARPA-I investment areas with external stakeholders, including academia, corporate partners, regulatory bodies, and, perhaps, even local community deployment advocacy.
Towards a Solution for Broadening the Geography of NSF Funding
Congressional negotiations over the massive bipartisan innovation bill have stumbled over a controversial proposal to expand the geographic footprint of National Science Foundation (NSF) funding. That proposal, in the Senate-passed U.S. Innovation and Competition Act (USICA), mandates that 20% of NSF’s budget be directed to a special program to help institutions in the many states that receive relatively few NSF dollars.
Such a mandate would represent a dramatic expansion of the Established Program to Stimulate Competitive Research (EPSCoR), which currently receives less than 3% of NSF’s budget. Major EPSCoR expansion is popular among legislators who would like to see the research institutions they represent become more competitive within the NSF portfolio. Some legislators have said their support of the overall innovation package is contingent on such expansion.
But the proposed 20% set-aside for EPSCoR is being met with fierce opposition on Capitol Hill. 96 other legislators recently co-authored a letter warning, “Arbitrarily walling off a sizable percentage of a science agency’s budget from a sizable majority of the country’s research institutions would fundamentally reduce the entire nation’s scientific capacity and damage the research profiles of existing institutions.”
Both proponents and opponents of the 20% set-aside make good points. Those in favor want to see more equitable distribution of federal research dollars, while those against are concerned that the mandatory set-aside is too massive and blunt an instrument for achieving that goal. Fortunately, we believe compromise is achievable—and well worth pursuing. Here’s how.
What is EPSCoR?
First, some quick background on the program at the heart of the controversy: ESPCoR. The program was established in 1979 with the admirable goal of broadening the geographic distribution of NSF research dollars, which even then were disproportionately concentrated in a handful of states.
EPSCoR provides eligible jurisdictions with targeted support for research infrastructure, development activities like workshops, and co-funding for project proposals submitted to other parts of NSF. A jurisdiction is eligible to participate in EPSCoR if its most recent five-year level of total NSF funding is equal to or less than 0.75% of the total NSF budget (excluding EPSCoR funding and NSF funding to other federal agencies). Currently, 25 states plus Puerto Rico, Guam, and the U.S. Virgin Islands qualify for EPSCoR. Yet the non-EPSCoR states still accounted for nearly 90% of NSF awards in FY 2021.
Why is expansion controversial?
As mentioned above, the Senate-passed USICA (S. 1260) would require NSF to devote 20% of its budget to EPSCoR (including research consortia led by EPSCoR institutions). The problem is that EPSCoR received only 2.4% of NSF’s FY 2022 appropriation. This means that to achieve the 20% mandate without cutting non-EPSCoR funding, Congress would have to approve nearly $2 billion in new appropriations for NSF in FY 2023, representing a 22% year-over-year increase, devoted entirely to EPSCoR. This is, to be blunt, wildly unlikely.
On the other hand, achieving a 20% budget share for EPSCoR under a more realistic FY 2023 appropriation for NSF would require cutting funding for non-EPSCoR programs on the order of 15%: a cataclysmic proposition for the research community.
Neither pathway for a 20% EPSCoR set-aside seems plausible. Still, key legislators have said that the 20% target is a must-have. So what can be done?
A path forward
We think a workable compromise is possible. The following three revisions to the Senate-proposed set-aside that everyone might accept:
- Specify that the 20% mandate applies to institutions in EPSCoR states rather than the EPSCoR program itself. While specific funding for the EPSCoR program accounts for less than 3% of the total NSF budget, institutions in current EPSCoR states actually receive about 13% of NSF research dollars. In other words, a substantial portion of NSF funding is allocated to EPSCoR institutions through the agency’s normal competitive-award opportunities. Given this fact, there’s a clear case to be made for focusing the 20% ramp-up on EPSCoR-eligible institutions rather than the EPSCoR program.
- Specify that the mandate only applies to extramural funding, not to agency operations and administrative appropriations. This is simply good government. If EPSCoR funding is tied to administrative appropriations, it may create an incentive to bloat the administrative line items. Further, if the mandate is applied to the entirety of the NSF budget and administrative costs must increase for other reasons (for instance, to cover future capital investments at NSF headquarters), then NSF may be forced to “balance the books” by cutting non-EPSCoR extramural funding to maintain the 20% ESPCoR share.
- Establish a multi-year trajectory to achieve the 20% target. As mentioned above, a major year-over-year increase in the proportion of NSF funding directed to either EPSCoR or EPSCoR-eligible institutions could cripple other essential NSF programs from which funding would have to be pulled. Managing the deluge of new dollars could also prove a challenge for EPSCoR-eligible institutions. Phasing in the 20% target over, say, five years would (i) enable federal appropriators to navigate pathways for increasing EPSCoR funding while avoiding drastic cuts elsewhere at NSF, and (ii) give EPSCoR-eligible institutions time to build out the capacities needed to maximize return on new research investments.
Crunching the numbers
To illustrate what this proposed compromise could mean fiscally, let’s say Congress mandates that NSF funding for EPSCoR-eligible institutions rises from its current ~13% share of total research dollars to 20% in five years. To achieve this target, the share of NSF funding received by EPSCoR-eligible states would have to rise by approximately 9% per year for five years.
Under this scenario, if NSF achieves 3% annual increases in appropriations (which is close to what it’s done since the FY 2013 “sequestration” year), then we’d see about 13% annual growth in NSF research dollars funneled to EPSCoR states due to the escalating set-aside. NSF research dollars funneled to non-EPSCoR states would increase by about 1% annually over the same time period. By the end of the five-year period, EPSCoR-eligible institutions would have seen a more than 80% increase in funding.
Annual increases in NSF appropriations of 2% would be enough to achieve the 20% set-aside without cutting funding for institutions in non-EPSCoR states, but wouldn’t allow any growth in funding for those institutions either. In other words, the appropriations increases would have to be entirely directed to the rising EPSCoR set-aside.
Finally, annual increases in NSF appropriations of 5% would be enough to achieve the 20% set-aside for EPSCoR-eligible institutions while also enabling non-EPSCoR-eligible institutions to enjoy continued 3% annual increases in funding growth.
The next step
U.S. strength in innovation is predicated on the scientific contributions from all corners of the nation. There is hence a clear and compelling reason to ensure that all U.S. research institutions have the resources they need to succeed, including those that have historically received a lower share of support from federal agencies.
he bipartisan innovation package offers a chance to achieve this, but it must be done carefully. The three-pronged compromise on EPSCoR outlined above is a prudent way to thread the needle. It should also be supported by sustained, robust increases in NSF funding as a whole. Congress should therefore couple this compromise with an explicit, bipartisan commitment to support long-term appropriations growth for NSF—because such growth would benefit institutions in every state.
The bipartisan innovation package offers enormous potential upside along several dimensions for U.S. science, innovation, and competitiveness. To enable that upside, an EPSCoR compromise is worth pursuing.
Regulating Use of Mobile Sentry Devices by U.S. Customs and Border Protection
Summary
Robotic and automated systems have the potential to remove humans from dangerous situations, but their current intended use as aids or replacements for human officers conducting border patrols raises ethical concerns if not regulated to ensure that this use “promot[es] the safety of the officer/agent and the public” (emphasis added). U.S. Customs and Border Protection (CBP) should update its use-of-force policy to cover the use of robotic and other autonomous systems for CBP-specific applications that differ from the military applications assumed in existing regulations. The most relevant existing regulation, Department of Defense Directive 3000.09, governs how semi-autonomous weapons may be used to engage with enemy combatants in the context of war. This use case is quite different from mobile sentry duty, which may include interactions with civilians (whether U.S. citizens or migrants). With robotic and automated systems about to come into regular use at CBP, the agency should proactively issue regulations to forestall adverse effects—specifically, by only permitting use of these systems in ways that presume all encountered humans to be non-combatants.
Challenge and Opportunity
CBP is currently developing mobile sentry devices as a new technology to force-multiply its presence at the border. Mobile sentry devices, such as legged and flying robots, have the potential to reduce deaths at the border by making it easier to locate and provide aid to migrants in distress. According to an American Civil Liberties Union (ACLU) report, 22% of migrant deaths between 2010 and 2021 that involved an on-duty CBP agent or officer were caused by medical distress that began before the agent or officer arrived on the scene. However, the eventual use cases, rules of engagement, and functionalities of these robots are unclear. If not properly regulated, mobile sentry devices could also be used to harm or threaten people at the border—thereby contributing to the 44% of deaths that occurred as a direct result of vehicular or foot pursuit by a CBP agent. Regulations on mobile sentry device use—rather than merely acquisition—are needed because even originally unarmed devices can be weaponized after purchase. Devices that remain unarmed can also harm civilians using a limb or propeller.
Existing Department of Homeland Security (DHS) regulations governing autonomous systems seek to minimize technological bias in artificially intelligent risk-assessment systems. Existing military regulations seek to minimize risks of misused or misunderstood capabilities for autonomous systems. However, no existing federal regulations govern how uncrewed vehicles, whether remotely controlled or autonomous, can be used by CBP. The answer is not as simple as extending military regulations to the CBP. Military regulations governing autonomous systems assume that the robots in question are armed and interacting with enemy combatants. This assumption does not apply to most, if not all, possible CBP use cases.
With the CBP already testing robotic dogs for deployment on the Southwestern border, the need for tailored regulation is pressing. Recent backlash over the New York Police Department testing similar autonomous systems makes this topic even more timely. While the robots used by CBP are currently unarmed, the same company that developed the robots being tested by CBP is working with another company to mount weapons on them. The rapid innovation and manufacturing of these systems requires implementation of policies governing their use by CBP before CBP has fully incorporated such systems into its workflows, and before the companies that build these systems have formed a powerful enough lobby to resist appropriate oversight.
Plan of Action
CBP should immediately update its Use of Force policy to include restrictions on use of force by mobile sentry devices. Specifically, CBP should add a chapter to the policy with the following language:
- A “Mobile Sentry Device” should be defined as any remotely controlled, autonomous, or semi-autonomous mobile technology used for surveillance. Examples of Mobile Sentry Devices include self-driving cars, legged robots, or quadcopter drones.
- No amount of force may be determined “reasonable” if administered by a Mobile Sentry Device, whether the Device is (i) completely controlled by an agent or officer, or (ii) operating in an autonomous or semi-autonomous mode.
- No Mobile Sentry Device may be authorized to administer Lethal Force, Less-Lethal Force, or any type of force applied directly by contact with the Device (i.e., contact equivalent to an “Empty Hand” technique). For example, a legged robot may not be used to discharge a firearm, disperse Oleoresin Capsicum spray (pepper spray), or strike a human with a limb.
- A Mobile Sentry Device may not be used as a Vehicular Immobilization Device (or used to deploy such a device), whether the Mobile Sentry Device is (i) completely controlled by an agent or officer, or (ii) operating in an autonomous or semi-autonomous mode.
- When powered on, Mobile Sentry Devices must maintain a distance of at least two feet from any humans not authorized to operate the Device. The Device and its operator are responsible for maintaining this distance.
- Mobile Sentry Devices may not be used to detain or perform arrests, nor to threaten or intimidate with the implicit threat of detainment or arrest.
- A Mobile Sentry Device may be used to administer humanitarian aid or provide a two-way visual or auditory connection to a CBP officer or agent.
- When approaching people to offer humanitarian aid, the Device must use de-escalation techniques to indicate that it is not a threat. These techniques will necessarily vary based on the specific technology. Some examples might include a flying device landing and immediately unfolding a screen playing a non-threatening video, or a legged device sitting with its legs underneath it and cycling through non-threatening audio recordings in multiple languages.
- When used for humanitarian purposes, the Device may not touch its human target(s) or request them to touch it. To transfer an item (such as food, water, or emergency medical supplies) to the target(s), the Device must drop the package with the items while maintaining at least two feet of distance from the closest person.
- When used to provide a two-way visual or auditory connection with a CBP officer or agent, the Device must indicate that such a connection is about to be formed and indicate when the connection is broken. For example, the Device could use an audio clip of a ringing phone to signal that a two-way audio connection to a CBP officer is about to commence.
These regulations should go into effect before Mobile Sentry Devices are moved from the testing phase to the deployment phase. Related new technology, whether it increases capabilities for surveillance or autonomous mobility, should undergo review by a committee that includes representatives from the National Use of Force Review Board, migrant rights groups, and citizens living along the border. This review should mirror the process laid out in the Community Control over Police Surveillance project, which has already been successfully implemented in multiple cities.
Conclusion
U.S. Customs and Border Patrol (CBP) is developing an application for legged robots as mobile sentry devices at the southwest border. However, the use cases, functionality, and rules of engagement for these robots remain unclear. New regulations are needed to forestall adverse effects of autonomous robots used by the federal government for non-military applications, such as those envisioned by CBP. These regulations should specify that mobile sentry devices can only be used as humanitarian aids, and must use de-escalation methods to indicate that they are not threatening. Regulations should further mandate that mobile sentry devices maintain clear distance from human targets, that use of force by mobile sentry devices is never considered “reasonable,” and that mobile sentry devices may never be used to pursue, detain, or arrest humans. Such regulations will help ensure that the legged robots currently being tested as mobile sentry devices by CBP—as well as any future mobile sentry devices—are used ethically and in line with CBP’s goals, alleviating concerns for migrant advocates and citizens along the border.
Regulations on purchasing are not sufficient to prevent mobile sentry device technology from being weaponized after it is purchased. However, DHS could certainly also consider updating its acquisition regulations to include clauses resulting in fines when mobile sentry devices acquired by the CBP are not used for humanitarian purposes.
DOD Directive 3000.09 regulates the use of autonomous weapons systems in the context of war. For an autonomous, semi-autonomous, or remotely controlled system that is deployed with the intention to be a weapon in an active battlefield, this regulation makes sense. But applications of robotic and automated systems currently being developed by DHS are oriented towards mobile sentry duty along stretches of American land where civilians are likely to be found. This sentry duty is likely to be performed by uncrewed ground robots following GPS breadcrumb trails along predetermined regular patrols along the border. Applying Directive 3000.09, the use of a robot to kill or harm a person during a routine patrol along the border would not be a violation as long as a human had “meaningful control” over the robot at that time. The upshot is that mobile sentry devices used by CBP should be subject to stricter regulations.
Most companies selling legged robots in the United States have explicit end-user policies prohibiting the use of their machines to harm or intimidate humans or animals. Some companies selling quadcopter drones have similar policies. But these policies lack any enforcement mechanism. As such, there is a regulatory gap that the federal government must fill.
No, but it is an immediately actionable strategy. An alternative—albeit more time-consuming—option would be for CBP to form a committee comprising representatives from the National Use of Force Review Board, the military, migrant-rights activist groups, and experts on ethics to develop a directive for CBP’s use of mobile sentry devices. This directive should be modeled after DoD Directive 3000.09, which regulates the use of lethal autonomous weapons systems by the military. As the autonomous systems in DOD Directive 3000.09 are assumed to be interacting with enemy combatants while CBP’s jurisdiction consists mostly of civilians, the CBP directive should be considerably more stringent than Directive 3000.09.
The policies proposed in this memo govern what mobile sentry devices are and are not permitted to do, regardless of the extent to which humans are involved in device operation and/or the degree of autonomy possessed by the technology in question. The policies proposed in this memo could therefore be applied consistently as the technology continues to be developed. AI is always changing and improving, and by creating policies that are tech-agnostic, CPB can avoid updating regulations as mobile sentry device technology evolves.
Combating Bias in Medical Innovation
There is a crisis within healthcare technology research and development, wherein historically marginalized groups are under-researched in preclinical studies, under-represented in clinical trials, misunderstood by clinical practitioners, and harmed by biased medical technology. These issues in turn contribute to costly disparities in healthcare outcomes, leading to losses of $93 billion a year in excess medical-care costs, $42 billion a year in lost productivity, and $175 billion a year due to premature deaths. COVID-19 put these disparities into especially sharp focus. In December 2020, pulse oximeters, critical for healthcare monitoring during the pandemic, were shown to be much less accurate in patients with darker skin, thereby putting those patients at a greater risk of organ damage. The Food and Drug Administration (FDA) responded by issuing a safety communication, but not with any changes to regulation of pulse oximeters.
Especially for an administration that has embedded equity throughout its policy agenda, this situation is unacceptable. The Biden-Harris Administration must act to address bias in medical technology at the development, testing and regulation, and market-deployment and evaluation phases. This will require coordinated effort across multiple agencies. In the development phase, science-funding agencies should crack down on federally funded studies that do not conduct mandatory subgroup analysis for diverse populations. Funding agencies should also expand funding for under-resourced research areas. In the testing and regulation phase, the FDA should raise the threshold for evaluation of medical technologies, make diversity requirements binding, and expand data-auditing processes. In the market-deployment and evaluation phases, the FDA should strengthen reporting mechanisms for adverse outcomes, the Federal Trade Commission (FTC) should require impact assessments of deployed technologies, and the Agency for Healthcare Research and Quality (AHRQ) should identify technologies that could address healthcare disparities.
Challenge and Opportunity
Bias is regrettably endemic in medical innovation. Drugs are incorrectly dosed to people assigned female at birth due to historical exclusion of women from clinical trials. Medical algorithms make healthcare decisions based on biased health data, clinically disputed race-based corrections, and/or model choices that exacerbate healthcare disparities. Much medical equipment is not accessible, thus violating the Americans with Disabilities Act. Biased studies, technology, and equipment inevitably produce disparate outcomes in U.S. healthcare.
The problem of bias in medical innovation manifests in multiple ways: cutting across technological sectors in clinical trials, pervading the commercialization pipeline, and impeding equitable access to critical healthcare advances.
Bias in medical innovation cuts across technology sectors
The 1993 National Institutes of Health (NIH) Revitalization Act required federally funded clinical studies to (i) include women and racial minorities as participants, and (ii) break down results by sex and race or ethnicity. Yet a 2019 study found that only 13.4% of NIH-funded trials performed the mandatory subgroup analysis. Moreover, the increasing share of industry-funded studies are not subject to Revitalization Act mandates — they are only governed by non-binding FDA recommendations for clinical-trial diversity. These studies frequently fail to report differences in outcomes by patient population as a result. The resulting disparities in clinical-trial representation are stark: African Americans represent 12% of the U.S. population but only 5% of clinical-trial participants, Hispanics make up 16% of the population but only 1% of clinical trial participants, and sex distribution in some trials is 67% male. Finally, many medical technologies approved prior to 1993 have not been reassessed for potential bias. One outcome of such inequitable representation is evident in drug dosing protocols: sex-aware prescribing guidelines exist for only a third of all drugs.
Bias in U.S. medical innovation — perpetuated by weak or weakly enforced federal regulations — extends beyond clinical trials. As explained below, bias pervades medical algorithms, medical devices, and the pharmaceutical sector as well.
Algorithms
Regulation of medical algorithms varies based on end application, as defined in the 21st Century Cures Act. Only algorithms that (i) acquire and analyze medical data and (ii) could have adverse outcomes are subject to FDA regulation. Thus, clinical decision-support software is not regulated even though these technologies make important clinical decisions in 90% of U.S. hospitals.
Even when a medical algorithm is regulated, regulation may occur through relatively permissive de novo pathways and 510(k) pathways. A de novo pathway is used for novel devices determined to be low to moderate risk, and thus subject to a lower burden of proof with respect to safety and equity. A 510(k) pathway can be used to approve a medical device exhibiting “substantial equivalence” to a previously approved device, i.e., it has the same intended use and/or same technological features. Different technical features can be approved so long as there are not questions raised around safety and effectiveness.
Medical devices approved through de novo pathways can be used as predicates for approval of devices through 510(k) pathways. Moreover, a device approved through a 510(k) pathway can remain on the market even if its predicate device was recalled. Widespread use of 510(k) approval pathways has generated a “collapsing building” phenomenon, wherein many technologies currently in use are based on failed predecessors. Indeed, 97% of devices recalled between 2008 to 2017 were approved via 510(k) clearance.
Even more alarming is evidence showing that machine learning can further entrench medical inequities. Because machine learning medical algorithms are powered by data from past medical decision-making, which is rife with human error, these algorithms can perpetuate racial, gender, and economic bias. Even algorithms demonstrated to be unbiased at the time of approval can evolve in biased ways over time, with little to no oversight from the FDA. As technological innovation progresses, an intentional focus on this problem will be required.
Finally, there is not a list of approved medical algorithms on the market, making it difficult for researchers to assess them for bias.
Medical devices
Currently, the Medical Device User Fee Act requires the FDA to consider the least burdensome appropriate means for manufacturers to demonstrate the effectiveness of a medical device or to demonstrate a device’s substantial equivalence. This requirement was reinforced by the 21st Century Cures Act, which also designated a category for “breakthrough devices” subject to far less-stringent data requirements. Such legislation shifts the burden of clinical data collection to physicians and researchers, who might discover bias years after FDA approval. This legislation also makes it difficult to require assessments on the differential impacts of technology.
Like medical algorithms, many medical devices are approved through 510(k) exemptions or de novopathways. The FDA has taken steps since 2018 to increase requirements for 510(k) approval and ensure that Class III (high-risk) medical devices are subject to rigorous pre-market approval, but problems posed by equivalence and limited diversity requirements remain.
Pharmaceuticals
The 1993 Revitalization Act strictly governs clinical trials for pharmaceuticals and does not make recommendations for adequate sex or genetic diversity in preclinical research. The results are that a disproportionately high number of male animals are used in research and that only 5% of cell lines used for pharmaceutical research are of African descent. Programs like All of Us, an effort to build diverse health databases through data collection, are promising steps towards improving equity and representation in pharmaceutical research and development (R&D). But stronger enforcement is needed to ensure that preclinical data (which informs function in clinical trials) reflects the diversity of our nation.
Bias in medical innovation exists throughout the commercialization pipeline
Bias occurs not only in multiple medical innovation sectors, but also across the development, testing and regulation, and market-deployment and evaluation phases of the medical innovation pipeline. This can be understood through the example of pulse oximeters.
Development
Pulse oximetry was developed by Biox and given FDA approval in 1980. The technology works by shining a light through the skin and measuring the difference in light absorbance to estimate arterial oxygen saturation. Melanin absorbs visual and infrared light and will interfere at all wavelengths. No algorithm has yet been developed to account for melanin attenuation. Hence pulse oximeter calibration data does not accurately reflect Black patients.
Testing and regulation
The first pulse oximeter was approved by the FDA at a time when clinical trials did not require gender and racial diversity. Thus, the foundational, 1980s-era pulse oximeter technology upon which subsequent 510(k) clearance for pulse oximeters has been granted is one that was tested almost exclusively on white, male patient populations.
With the 510(k) clearance, only 10 people are required in a study of any new pulse oximeter’s efficacy. The FDA states that pulse oximetry study populations should have a range of skin pigmentations and must include at least two darkly pigmented individuals or 15% of the participant pool, whichever is larger. But the FDA does not provide an objective standard for “darkly pigmented”. Moreover, this requirement (i) does not have the statistical power necessary to detect differences between demographic groups, and (i) does not represent the composition of the U.S. population. Finally, FDA guidance is silent on how pulse oximetry technology should be calibrated — it does not, for instance, specifically recommend studies on melanin interference.
Market deployment and evaluation
To clinical practitioners, pulse oximeters are a metaphorical “black box”, with oxygenation calculations hidden by proprietary algorithms. When errors or biases occur in oximeter data (if they are even noticed), the practitioner may blame the patient for their lifestyle rather than the technology used for assessment. This in turn leads to worse clinical outcomes for patients with darker skin tones, as they are at greater risk of becoming sicker before receiving care. The problem is exacerbated by the fact that clinicians who use oximeter technology for the first time (as was the case during COVID-19) generally are not trained to spot factors that cause inaccurate measurements. This leads to underreporting of adverse events to the FDA — which is already a problem due to the voluntary nature of adverse-event reporting. When problems are ultimately identified during market deployment and evaluation of a given technology, government can be slow to respond. The pulse oximeter’s limitations in monitoring oxygenation levels across diverse skin tones was identified as early as the 1990s. 31 years later, despite repeated follow-up studies indicating biases, no manufacturer has incorporated skin-tone-adjusted calibration algorithms into pulse oximeters. It required the large Sjoding study, and the media coverage it garnered, for the FDA to issue a safety communication. Even then, the safety communication has not been followed with any additional regulatory action.
Inequitable access to medical innovation represents a form of bias
Americans face wildly different levels of access to new medical innovations. As many new innovations have high cost points, these devices exist outside the price range of many smaller healthcare institutions and/or federally funded healthcare services, including Veterans Affairs, health centers, and the Indian Health Service. Emerging care-delivery strategies might not be covered by Medicare and Medicaid, meaning that patients under those systems cannot access the most cutting-edge treatments. Finally, the shift to digital health in response to COVID-19 has compromised access to healthcare in rural communities without reliable broadband access.
Finally, the Advanced Research Projects Agency for Health (ARPA-H) has a commitment to have all programs and projects consider equity in their design. To fulfill ARPA-H’s commitment, there is a need for action across the federal government to ensure that medical technologies are developed fairly, tested with rigor, deployed safely, and made affordable and accessible to everyone.
Plan of Action
The Biden-Harris Administration should launch “Healthcare Technology for All Americans” (HTAA), a government-wide initiative to address systemic inequities in U.S. healthcare wrought by biased medical technology. Through a comprehensive approach that addresses bias in all medical sectors, at all stages of the commercialization pipeline, and in all geographies, the initiative will strive to ensure unbiased, equitable care delivery across the entire medical-innovation ecosystem. HTAA should be a joint mandate of Health and Human Services (HHS) and the Office of Science Technology and Policy (OSTP) to work with federal agencies on priorities of health equity, and initiative leadership should sit at both HHS and OSTP.
This initiative will require involvement of multiple federal agencies, as summarized in the table below. Additional detail is provided in the subsequent sections describing how the federal government can mitigate bias in the development phase; testing, regulation, and approval phases; and market deployment and evaluation phases.
Three guiding principles should underlie the initiative:
- Equity should drive action. Actions should seek to improve the health of those who have been historically excluded from medical research and development. We should design standards that repair past exclusion and prevent future exclusion.
- Coordination and cooperation are necessary. The executive and legislative branches must collaborate to address the full scope of the problem of bias in medical technology, from federal processes to new regulations. Legislative leadership should task the Government Accountability Office (GAO) to engage in ongoing assessment of progress towards the goal of achieving equity in medical innovation.
- Transparent, evidence-based decision making is paramount. There is abundant peer-reviewed literature that examines bias in drugs, devices, and algorithms used in healthcare settings — this literature should form the basis of an equity-driven approach to medical innovation. Gaps in evidence should be focused on through deployed research funding. Moreover, as algorithms become ubiquitous in medicine, every effort should be made to ensure that these algorithms are trained on representative data of those experiencing a given healthcare condition.
Addressing bias at the development phase
The following actions should be taken to address bias in medical technology at the innovation phase:
- Enforce parity in government-funded research. For clinical research, NIH should examine the widespread lack of adherence to regulations requiring that government-funded clinical trials report sex and racial or ethnicity breakdown of trial participants. Funding should be reevaluatedfor non-compliant trials. For preclinical research, NIH should require gender parity in animal models and representation of diverse cell lines used in federally funded studies.
- Deploy funding to address research gaps. Where data sources for historically marginalized people are lacking, such as for women’s cardiovascular health, NIH should deploy strategic, targeted funding programs to fill these knowledge gaps. This should include resources for underrepresented groups to participate in research and clinical trials. Results should be added to a publicly available database so they can be accessed by designers of new technologies. Funding programs should also be created to fill gaps in technology, such as in diagnostics and treatmentsfor high-prevalence and high-burden uterine diseases like endometriosis (found in 10% of reproductive–aged people with uteruses).
- Invest in research into healthcare algorithms and databases. Given the explosion of algorithms in healthcare decision-making, NIH and NSF should launch a new research program focused on the study, evaluation, and application of algorithms in healthcare delivery, and on how artificial intelligence and machine learning (AI/ML) can exacerbate healthcare inequities. The initial request for proposals should focus on design strategies for medical algorithms that mitigate bias from data or model choices.
- Task ARPA-H with developing metrics for equitable medical technology development. ARPA-H should prioritize developing a set of procedures and metrics for equitable development of medical technology. Once developed, these processes should be rapidly deployed across ARPA-H, as well as published for potential adoption by additional federal agencies, industry, and other stakeholders. ARPA-H could also collaborate with NIST on standards setting with NIST and ONC on relevant standards setting. For instance, NIST is developing a standard for managing bias in AI, and the ONC engages in setting standards that achieve equity by design. CMS could use resultantstandards for Medicare and Medicaid reimbursements.
Addressing bias at the testing, regulation, and approval phases
The following actions should be taken to address bias in medical innovation at the testing, regulation, and approval phases:
- Raise the threshold for FDA evaluation of devices and algorithms. Equivalency necessary to receive 510(k) clearance should be narrowed. For algorithms, this would involve consideration of whether the datasets or machine learning tactics used by the new device and its predicate are similar. For devices (including those that use algorithms), this would require tightening the definition of “same intended use” (currently defined as a technology having the same functionality as one previously approved by the FDA) as well as eliminating the approval of new devices with “different technological characteristics” (the application of one technology to a new area of treatment in which that technology is untested).
- Evaluate FDA’s guidance on specific technology groups for equity. Requirements for the safety of a given drug, medical device, or algorithm should have the statistical power necessary to detect differences between demographic groups and represent all end-users of the technology.
- Make FDA guidance on diversity in testing binding. Currently, FDA guidance makes ensuring diversity of clinical trials and datasets a voluntary step in industry-funded development of a new medical technology. FDA should make this guidance mandatory for all medical technologies it regulates.
- Establish a data bank for auditing medical algorithms. The newly established Office of Digital Transformation within the FDA should create a “data bank” of healthcare images and datasets representative of the U.S. population. Medical technology developers could use the data bank to assess the performance of medical algorithms across patient populations. Regulators could use the data bank to ground claims made by those submitting a technology for FDA approval.
- Allow data submitted to the FDA to be examined by the broader scientific community. Currently, data submitted to the FDA as part of its regulatory-approval process is kept as a trade secret and not released pre-authorization to researchers. Releasing the data via an FDA-invited “peer review” step in the regulation of high-risk technologies, like automated decision-making algorithms, Class III medical devices, and drugs, will ensure that additional, external rigor is applied to the technologies that could cause the most harm due to potential biases.
- Establish an AI Bill of Rights. The federal government and Congress should create protections for necessary uses of artificial intelligence (AI) identified by OSTP. Federally funded healthcare centers, like facilities part of the Veterans Health Administration, could refuse to buy software or technology products that violate this “AI Bill of Rights” through changes to federal acquisition regulation (FAR).
Addressing bias at the market deployment and evaluation phases
- Strengthen reporting mechanisms at the FDA. Healthcare providers, who are often closest to the deployment of medical technologies, should be made mandatory reporters to the FDA of all witnessed adverse events related to bias in medical technology. In addition, the FDA should require the inclusion of unique device identifiers (UDIs) in adverse-response reporting. Using this data, Congress should create a national and publicly accessible registry that uses UDIs to track post-market medical outcomes and safety.
- Train physicians to identify bias in medical technologies and identify new areas of specialization. ED should work with medical schools to develop curricula training physicians to identify potential sources of bias in medical technologies and ensuring that physicians understand how to report adverse events to the FDA. In addition, ED should consider creating new medical specialties that work at the intersection of technology and care delivery.
- Require impact assessments of deployed technologies. Congress must establish systems of accountability for medical technologies, like algorithms, that can evolve over time. Such work could be done by passing the 2022 Algorithmic Accountability Act (an update of the 2019 Algorithmic Accountability Act), which would require companies that create “high-risk automated decision systems” to conduct impact assessments reviewed by the FTC as frequently as necessary (yearly at minimum). Impact assessments should be expanded to medical devices — specifically, to include requirements for companies to assess clinical outcomes in diverse patient populations one year post-implementation in the market, using UDIs as a tracking mechanism.
- Assess disparities in patient outcomes to direct technical auditing. AHRQ should be given the funding needed to fully investigate patient-outcome disparities that could be caused by biases in medical technology, such as further investigation into algorithms identified in the agency’s March 2021 Request for Information. The results of this research should be used to identify technologies that the FDA should audit post-market for efficacy. CMS and its accrediting agencies can monitor these technologies and assess whether they should receive Medicare funding.
- Ensure that technologies developed by ARPA-H have an enforceable access plan. ARPA-H will produce cutting-edge technologies that must be made accessible to all Americans. ARPA-H should collaborate with the Center for Medicare and Medicaid Innovation to develop strategies for equitable delivery of these new technologies. A cost-effective deployment strategy must be identified to service Veterans Affairs hospitals, federally funded health centers, and facilities that are part of the Indian Health Service, among others.
- Create a fund to support digital health technology infrastructure in rural hospitals. To capitalize on the $65 billion expansion of broadband access allocated in the bipartisan infrastructure bill, the HRSA should deploy strategic funding to federally qualified health centers and rural health clinics to support digital health strategies — such as telehealth and mobile health monitoring — and patient education for technology adoption.
A comprehensive road map is needed
In January 2021, Senators Elizabeth Warren, Cory Booker, and Ron Wyden called for an FDA review of pulse oximetry measurements and their skin tone bias, citing the lack of understanding about clinical outcomes of this bias in their call to action. The GAO should go a step beyond this call to action and conduct a comprehensive investigation of “black box” medical technologies utilizing algorithms that are not transparent to end users, medical providers, and patients. The investigation should inform a national strategic plan for equity and inclusion in medical innovation that relies heavily on algorithmic decision-making. The plan should include identification of noteworthy medical algorithms exacerbating inequities, creation of enforceable regulatory standards, development of new sources of research funding to address knowledge gaps, development of enforcement mechanisms for bias reporting, and ongoing assessment of equity goals.
Timeline for action
Realizing HTAA will require mobilization of federal funding, introduction of regulation and legislation, and coordination of stakeholders from federal agencies, industry, healthcare providers, and researchers around a common goal of mitigating bias in medical technology. Such an initiative will be a multi-year undertaking and require funding to enact R&D expenditures, expand data capacity, assess enforcement impacts, create educational materials, and deploy personnel to staff all the above.
Near-term steps that can be taken to launch HTAA include issuing a public request for information, gathering stakeholders, engaging the public and relevant communities in conversation, and preparing a report outlining the roadmap to accomplishing the policies outlined in this memo.
Conclusion
Medical innovation is central to the delivery of high-quality healthcare in the United States. Ensuring equitable healthcare for all Americans requires ensuring that medical innovation is equitable across all sectors, phases, and geographies. Through a bold and comprehensive initiative, the Biden-Harris Administration can ensure that our nation continues leading the world in medical innovation while crafting a future where healthcare delivery works for all.
HTAA will be successful when medical policies, projects, and technologies yield equitable health care access, treatment, and outcomes. For instance, success would yield the following outcomes:
- Representation in preclinical and clinical research equivalent to the incidence of a studied condition in the general population.
- Research on a disease condition funded equally per affected patient.
- Existence of data for all populations facing a given disease condition.
- Medical algorithms that have equal efficacy across subgroup populations.
- Technologies that work equally well in testing as they do when deployed to the market.
- Healthcare technologies made available and affordable to all care facilities.
Regulation alone cannot close the disparity gap. There are notable gaps in preclinical and clinical research data for women, people of color, and other historically marginalized groups that need to be filled. There are also historical biases encoded in AI/ML decision-making algorithms that need to be studied and rectified. In addition, the FDA’s role is to serve as a safety check on new technologies — the agency has limited oversight over technologies once they are out on the market due to the voluntary nature of adverse reporting mechanisms. This means that agencies like the FTC and CMS need to be mobilized to audit high-risk technologies once they reach the market. Eliminating bias in medical technology is only possible through coordination and cooperation of federal agencies with each other as well as with partners in the medical device industry, the pharmaceutical industry, academic research, and medical care delivery.
Working together to address the enormous challenge of bias in medical innovation will require communication, coordination, and collaboration. ARPA-H provides the essential platform for these three tasks. As an agency bridging academic research and industry, ARPA-H will focus on developing technologies that address some of the greatest healthcare challenges facing Americans, including inequities existing in healthcare. By committing to consider equity in every project, ARPA-H provides the basis for practice of technological development that is inclusive, responsible, and accountable. ARPA-H’s deep relationships with industry will spur medical device companies to align with ARPA-H’s processes.
Offices like the OMH do necessary work in identifying disparities in care and pointing out solutions. For example, the call for digital infrastructure improvements to improve care access for vulnerable populations has been echoed by OMH. But these offices lack the ability to operationalize multi-agency collaborations needed to address cross-cutting challenges related to medical bias. A new initiative led by the White House in close partnership with HHS leadership is needed to ensure that the broad scope of the plan outlined in this memo is actualized.
A significant focus of the medical device and pharmaceutical industries is reducing the time to market for new medical devices and drugs. Imposing additional requirements for subgroup analysis and equitable use as part of the approval process could work against this objective. On the other hand, ensuring equitable use during the development and approval stages of commercialization will ultimately be less costly than dealing with a future recall or a loss of Medicare or Medicaid eligibility if inequitable outcomes are discovered. FAR regulation can also be employed to incentivize companies to adhere to equity standards in order to receive federal contracts.
Healthcare disparities exist in every state in America and are costing billions a year in economic growth. Some of the most vulnerable people live in rural areas, where they are less likely to receive high-quality care because costs of new medical technologies are too high for the federally qualified health centers that serve one in five rural residents as well as rural hospitals. Furthermore, during continued use, a biased device creates adverse healthcare outcomes that cost taxpayers money. A technology functioning poorly due to bias can be expensive to replace. It is economically imperative to ensure technology works as expected, as it leads to more effective healthcare and thus healthier people.
An Earthshot for Clean Steel and Aluminum
Summary
The scale of mobilization and technological advancement required to avoid the worst effects of climate change has recently led U.S. politicians to invoke the need for a new, 21st century “moonshot.” The Obama Administration launched the SunShot Initiative to dramatically reduce the cost of solar energy and, more recently, the Department of Energy (DOE) announced a series of “Earthshots” to drive down the cost of emerging climate solutions, such as long-duration energy storage.
While DOE’s Earthshots to date have been technology-specific and sector-agnostic, certain heavy industrial processes, such as steel and concrete, are so emissions- intensive and fundamental to modern economies as to demand an Earthshot unto themselves. These products are ubiquitous in modern life, and will be subject to increasing demand as we seek to deploy the clean energy infrastructure necessary to meet climate goals. In other words, there is no reasonable pathway to preserving a livable planet without developing clean steel and concrete production at mass scale. Yet the sociotechnical pathways to green industry – including the mix of technological solutions to replace high-temperature heat and process emissions, approaches to address local air pollutants, and economic development strategies – remain complex and untested. We urgently need to orient our climate innovation programs to the task.
Therefore, this memo proposes that DOE launch a Steel Shot to drive zero-emissions iron, steel, and aluminum production to cost-parity with traditional production within a decade. In other words, zero dollar difference for zero-emissions steel in ten years, or Zero for Zero in Ten.
Challenge and Opportunity
As part of the Biden-Harris Administration’s historic effort to quadruple federal funding for clean energy innovation, DOE has launched a series of “Earthshots” to dramatically slash the cost of emerging technologies and galvanize entrepreneurs and industry to hone in on ambitious but achievable goals. DOE has announced Earthshots for carbon dioxide removal, long-duration storage, and clean hydrogen. New programs authorized by the Infrastructure Investment and Jobs Act, such as hydrogen demonstration hubs, provide tools to help DOE to meet the ambitious cost and performance targets set in the Earthshots. The Earthshot technologies have promising applications for achieving net-zero emissions economy-wide, including in sectors that are challenging to decarbonize through clean electricity alone.
One such sector is heavy industry, a notoriously challenging and emissions-intensive sector that, despite contributing to nearly one-third of U.S. emissions, has received relatively little focus from federal policymakers. Within the industrial sector, production of iron and steel, concrete, and chemicals are the biggest sources of CO2 emissions, producing climate pollution not only from their heavy energy demands, but also from their inherent processes (e.g., clinker production for cement).
Meanwhile, global demand for cleaner versions of these products – the basic building blocks of modern society – is on the rise. The International Energy Agency (IEA) estimates that CO2 emissions from iron and steel production alone will need to fall from 2.4 Gt to 0.2 Gt over the next three decades to meet a net-zero emissions target economy-wide, even as overall steel consumption increases to meet our needs for clean energy buildout. Accordingly, by 2050, global investment in clean energy and sustainable infrastructure materials will grow to $5 trillion per year. The United States is well-positioned to seize these economic opportunities, particularly in the metals industry, given its long history of metals production, skilled workforce, the initiation of talks to reach a carbon emissions-based steel and aluminum trade agreement, and strong labor and political coalitions in favor of restoring U.S. manufacturing leadership.
“The metals industry is foundational to economic prosperity, energy infrastructure, and national security. It has a presence in all 50 states and directly employs more than a half million people. The metals industry also contributes 10% of national climate emissions.”
Department of Energy request for information on a new Clean Energy Manufacturing Institute, 2021
However, the exact solutions that will be deployed to decarbonize heavy industry remain to be seen. According to the aforementioned IEA Net-Zero Energy (NZE) scenario, steel decarbonization could require a mix of carbon capture, hydrogen-based, and other innovative approaches, as well as material efficiency gains. It is likely that electrification – and in the case of steel, increased global use of electric arc furnaces – will also play a significant role. While technology research funding should be increased, traditional “technology-push” efforts alone are unlikely to spur rapid and widespread adoption of a diverse array of solutions, particularly at low-margin, capital-intensive manufacturing facilities. This points to the potential for creative technology-neutral policies, such as clean procurement programs, which create early markets for low-emissions production practices without prescribing a particular technological pathway.
Therefore, as a complement to its Earthshots that “push” promising clean energy technologies down the cost curve, DOE should also consider adopting technology-neutral Earthshots for the industrial sector, even if some of the same solutions may be found in other Earthshots (e.g., hydrogen). It is important for DOE to be very disciplined in identifying one or two essential sectors, where the opportunity is large and strategic, to avoid creating overly balkanized sectoral strategies. In particular, DOE should start with the launch of a Steel Shot to buy down the cost of zero-emissions iron, steel, and aluminum production to parity with traditional production within a decade, while increasing overall production in the sector. In other words, zero dollar difference for zero-emissions steel in ten years, or Zero for Zero in Ten.
The Steel Shot can bring together applied research and demonstration programs, public-private partnerships, prizes, and government procurement, galvanizing public energy around a target that enables a wide variety of approaches to compete. These efforts will be synergistic with technology-specific Earthshots seeking dramatic cost declines on a similar timeline.
Plan of Action
Develop and launch a metals-focused Earthshot:
- Design and announce the Steel Shot in close partnership with industry, labor, and communities. DOE should hold a series of roundtables with industry, labor, and communities to define and calculate the gap between zero-emissions and traditional production, often called the “green premium,” for clean steel and aluminum. This should incorporate measures to achieve near-zero carbon dioxide emissions as well as deep reductions in other harmful air and water pollutants to achieve a “Zero for Zero in Ten” goal – zero dollar difference for zero emissions steel within one decade. DOE should launch the Steel Shot with pledges from major steelmakers and steel purchasers, such as automakers.
- Calculate targets along the way to the decadal goal and define how success will be measured. After launching the new Earthshot, DOE should release a Request for Information (RFI) and use an initial Steel Shot Summit to compile projections for anticipated cost parity milestones along the way to the decadal target. DOE should plan to update assessments of the current “green premium” on a regular basis to ensure that research, development, and demonstration efforts are targeted at continued reductions in the cost of clean steel – not just improvements over the original baseline. To assess the emissions footprint of various steel production processes, DOE should work closely with the White House’s Buy Clean Task Force, which was tasked with developing recommendations for improving transparency and reporting around embodied emissions, particularly through environmental product declarations.
- Hold an annual Steel Shot Summit to bring together technologists, industry, and financiers to share solutions and develop projects. DOE should hold an annual Steel Shot event to help to highlight existing innovation efforts underway and connect stakeholders. This summit will build on existing Eartshot stakeholder gathering efforts underway, such as the Hydrogen Shot Summit and the Long Duration Storage Shot Summit.
Invest in domestic clean steelmaking capacity:
- Stand up the seventh Clean Energy Manufacturing Institute with funding for cooperative applied R&D and a demonstration facility. Last year, AMO put out a Request for Information on the establishment of a seventh Manufacturing USA institute on industrial decarbonization. The RFI had a particular focus on metals manufacturing. In 2022, DOE should formally issue a funding opportunity for the institute, with a requirement that the institute conduct cooperative R&D in industrial decarbonization practices and operate a manufacturing demonstration and workforce development facility for low- and zero-emissions manufacturing processes.
- Launch an annual competition for entrepreneurs and companies demonstrating low- and zero-emissions processes that reduce the green premium. Modeled after the SunShot’s American Made Solar Prize, AMO could issue a series of smaller-scale prize competitions targeted at challenges for clean metals. Prizes are particularly effective for challenges where the desired end target is defined and clearly measurable, but the optimal solution to achieve this target is not yet known. The variety of potential solutions for steel decarbonization makes the sector an excellent candidate for a prize program with multiple rounds and awardees. DOE could consider subprograms within the Steel Shot prize that align with reducing key sources of emissions – EPA identifies the three sources of emissions as 1) process emissions, 2) direct fuel combustion, and 3) indirect emissions from electricity consumption.
- Pass legislation to directly invest in deployment of commercial-scale solutions. While a prize program can promote prototype and pilot-stage technologies, real-world demonstration and deployment will buy down the cost of clean steel. These investments should pursue a range of decarbonization opportunities across blast-oxygen furnaces, electric arc furnaces, and emerging direct reduction approaches. They should also ensure that federal funds go to projects with strong labor standards, building on a long legacy of quality U.S. steelmaking jobs. The original American Jobs Plan released by President Biden proposed ten “pioneer facilities” to demonstrate clean industrial processes, including steel. Several proposals included in House-passed bills, such as the Build Back Better Act and the America COMPETES Act, would provide new authorities to DOE to fund commercial-scale retrofits and first-of-a-kind facilities employing clean steelmaking technologies. For instance, an amendment to America COMPETES expands the industrial decarbonization RD&D program authorized in the Energy Act of 2020 to include “commercial deployment projects.” Should these provisions pass, they can be leveraged to rapidly retrofit facilities and achieve the goals of the Steel Shot.
Create demand for “green steel” through market pull mechanisms:
- Match innovators and steelmakers with private purchasers to generate demand for clean metals. Demand-pull incentives can reduce risk for U.S. steelmakers and move the innovations that emerge from DOE R&D and prize programs into commercial adoption, which is critical for additional “learning-by-doing” at scale. DOE can work with domestic industries that are major purchasers of steel to develop sector-based advanced market commitments as part of the Earthshot launch. For instance, DOE should leverage its relationships with major automakers with ambitious climate goals, such as Ford and GM, to spur auto sector commitments to purchasing clean steel. In developing these advanced market commitments, DOE can work with the First Movers Coalition, a consortium of private sector buyers of innovative, clean products, launched by the State Department and the World Economic Forum in Glasgow in 2021. They included both steel and aluminum in their initial round of target products.
- Use federal procurement power to favor “green steel” for government-funded projects, including infrastructure and defense. AMO and DOE’s Federal Energy Management Program should advise the General Services Administration, Department of Defense, Department of Transportation, and other major federal procurers as they execute federal sustainability plans and procurement working groups, including the Buy Clean Task Force announced in December 2021. For instance, DOE can utilize the Earthshot to provide recommendations on reasonable costs for steel included in a Buy Clean program, and provide technical assistance to innovators to access federal clean procurement efforts.
The lower technology prices targeted by the Hydrogen Earthshot and the Carbon Negative Shot are necessary but not sufficient to guarantee that these technologies are deployed in the highest emissions producing sectors, such as steel, cement, and chemicals. The right combination of approaches to achieve price reduction remains uncertain and can vary by plant, location, process, product, as noted in a recent McKinsey study on decarbonization challenges across the industrial sector. Additionally, there is a high upfront cost to deploying novel solutions, and private financers are reluctant to take a risk on untested technologies. Nonetheless, to avoid creating overly balkanized sectoral strategies, it will be important for DOE to be very disciplined in identifying one or two essential sectors, such as metals, where the opportunity is large and strategic.
These products are ubiquitous and increasingly crucial for deploying the clean energy infrastructure necessary to reach net-zero. The United States of America has a long history of metals production, a skilled workforce, and strong labor and political coalitions in favor of restoring U.S. manufacturing leadership. Additionally, carbon-intensive steel from China has become a growing concern for U.S. manufacturers and policymakers; China produces 56% of global crude steel, followed by India (6%), Japan (5%), and then the U.S. (4%). The U.S. already maintains a strong competitive advantage in clean steel, and the technologies needed to double-down and fully decarbonize steel are close to commercialization, but still require government support to achieve cost parity.
U.S. steel production is already less polluting than many foreign sources, but that typically comes with additional costs. Reducing the “green premium” will help to keep US metal producers competitive, while preparing them for the needs of buyers, who are increasingly seeking out green steel products. End users such as Volkswagen are aiming for zero emissions across their entire value chain by 2050, while Mercedes-Benz and Volvo have already begun sourcing low-emissions steel for new autos. Meanwhile, the EU is preparing to implement a carbon border adjustment mechanism that could result in higher prices for steel and aluminum-produced products from the United States. The ramifications of the carbon border tax are already being seen in steel agreements, such as the recent US-EU announcement to drop punitive tariffs on each other’s steel and aluminum exports and to begin talks on a carbon-based trade agreement.
Breakthrough Energy estimated that the “green premium” for steel using carbon capture is approximately 16% – 29% higher than “normally” produced steel. Because there are a variety of processes that could be used to reduce emissions, and thus contribute to the “green premium,” there may not be a single number that can be estimated for the current costs. However, wherever possible, we advocate for using real-world data of “green” produced steel to estimate how close DOE is to achieving its benchmark targets in comparison to “traditional” steel.