It has been 10 years since the National Academies of Sciences (NAS) published its 2015 consensus report on Enhancing the Effectiveness of Team Science, recognizing then that the growing trend toward collaborative and team-based research required guidance on organizational and funding structures, leadership, and team composition. In the decade that followed, the U.S. scientific enterprise experimented with different models for delivering on the promise of “team science”—from university-based centers to new mission-focused government agencies (e.g. ARPAs) to more recently Focused Research Organizations (FROs). Each approach has converged on both the challenge and the opportunity—that team-based science is a critical engine of discovery for complex national problems—but its success depends far more on incentives, leadership, governance, and institutional reward systems than on scientific talent alone.

Today, the Directorate for Technology, Innovation and Partnerships (TIP) of the National Science Foundation (NSF) announced its plan for an ambitious pilot “Tech Labs program,” a $10 to 50M per year initiative designed to harness a team-based approach to tackle national scientific priorities, such as quantum technology, artificial intelligence, critical materials, semiconductor manufacturing, and biotechnology. Though we now have multiple examples of philanthropic and venture capital-funded entities, NSF’s Tech Labs will be the first predominantly federally funded entity, similar to a Focused Research Organization (FRO), structured to harness foundational science in high leverage teams in ‘independent organizations’ to execute on outcomes-based initiatives aligned with a federal science agenda. This initiative promises to fill a critical gap in the federal science and innovation ecosystem, making a space for rapid, high-risk, system-level capabilities that could prove foundational in building national technical capabilities—things that markets, universities, and philanthropy cannot do on their own. The Federation of American Scientists applauds this significant milestone for the U.S. scientific enterprise, but also offers several considerations to ensure that diverse scientific talent, from universities and the private sector, can fully see themselves in this emerging vision.

A central feature for NSF-funded Tech Labs is that they must be “independent,” meaning that they must have organizational and operational autonomy, as well as the freedom to act outside of academic or corporate constraints. This is an intentional design feature that ensures teams can operate like a mission-driven applied R&D organization, with their own authority, hiring, IP strategy, and operational systems. Universities have a key role to play. They can serve as a key source of ideas, facilities, personnel, and future talent, but their ingrained “publish or perish” culture and the existing incentive structure for tenure and promotion may systematically limit participation of university faculty and students in a potential NSF Tech Lab project. At a time when universities are already facing intense pressure to re-envision their role in the science and technology (S&T) ecosystem, we encourage both universities and the NSF to take steps to ensure complementarity and co-benefit, providing pathways for ambitious research acceleration that can leverage expertise and infrastructure that universities can contribute. We offer the following ideas to consider:

Create Intentional Pathways for Talent Mobility between Universities and Tech Labs 

• Students. Tech Labs could become a rich training ground for future scientists and entrepreneurs and students could become critical components of “portable research teams.” For example, graduate students and postdocs could be embedded full-time at a Tech Lab but tuition could still flow to the university and salaries be co-funded. Joint deliverables could include academic publications and also milestone-based efforts that align with research goals.
• Faculty. Similar to the NSF Rotator model, consider the creation of a short-term appointment category (e.g. “Tech Lab Program Scientist”) that creates a hybrid tour-of-service model for tenured faculty to participate in a Tech Lab team. 

Align financial incentives and share critical infrastructure

Independent NSF Tech Labs could develop milestone-based, block grant, subcontracts to university labs to leverage the use of critical equipment and physical space. This will help reduce administrative burden to universities and counter potential loss of indirect expenses.

With the comment period that opens today, NSF is inviting broad stakeholder input into this ambitious vision. This launch of the first federally funded FRO could mark a transformational moment for the U.S. science enterprise, changing what kinds of science and engineering become possible at all and enabling the execution of innovation at scale. But success will depend on broad buy-in and willingness to test this model. Please share your thoughts by January 20th about how to set this bold experiment up to deliver on the decade-long promise of “team science” and set the scientific research enterprise up to embark on a new era of innovation.

Today, the Directorate for Technology, Innovation and Partnerships (TIP) of the National Science Foundation (NSF) announced its plan for an ambitious pilot Tech Labs program, a “new initiative designed to launch and scale a new generation of independent research organizations” for up to $1 billion. FAS CEO Daniel Correa recently spoke with Adam Marblestone and Sam Rodriques, former FAS fellows who developed the idea for FROs and advocated for their use in a 2020 policy memo. This conversation has been lightly edited.

Daniel Correa: In 2020, you both published the initial blueprint for Focused Research Organizations (FROs). A lot has changed since then. How far has this idea come in the years since? 

Adam Marblestone: FROs are now a commonly discussed concept among scientists thinking about how to structure and fund new tools, systems, dataset or scientific infrastructure moonshots. This has arguably increased the level of ambition in the community and created more incentive to imagine and roadmap larger, more systematic and ambitious projects.

The high level of interest in FROs across many fields has shown us that they can help fill an important category of gap or bottleneck in the scientific landscape.  

Convergent Research, a nonprofit incubator and parent organization for FROs, which I lead, has launched 10 FROs and counting in fields as diverse as brain mapping, non-model microbes, ocean carbon modeling, software for math, and faster-cheaper-better astronomy. 

There are also other FROs that have emerged outside Convergent, such as Bind Research in the UK. Also in the UK, ARIA has funded Convergent to run a FRO residency, which will likely lead to 1-2 new UK based FROs in 2026. We are really excited to see the NSF announcement! 

DC: What are some of the biggest successes?

AM: Early FROs are in full swing doing science. 

They have released major datasets such as the largest map of the “pharm-ome” (drug target interactions) and a global atlas of ocean alkalinity enhancement efficiency. 

They’ve published many preprints on new tools and methods, including new self-driving labs for DNA delivery to speed research in novel organisms and new approaches for capturing biological protocols for translation to robotics. 

They’ve improved the cost-performance of proteomics by 27x already, and built software that has underpinned new industries like AI for math where there are now unicorn companies emerging partly as a result of these tools. 

They’ve introduced new methods for brain circuit mapping that could help make mapping entire mammalian brains cost effective and put new neural interface devices in human studies with unprecedented speed and safety. 

A small number of FROs have transitioned into new nonprofits as well, and we will also soon see the first major commercial spinouts of their technologies. 

Perhaps most importantly, extremely talented scientists, engineers, operators and leaders have been joining FROs, proving that these distinct institutions can indeed attract and retain top talent. These people span a huge range of academic and industrial backgrounds and career stages. Much more top talent has reached out with interest in starting their own ambitious FROs. A lot of that top talent wants to work on big goals in a tight-knit, fast moving team setting. 

In the process, more than 30 philanthropic organizations of many kinds, and several government organizations, have put funding into FROs at various stages. 

DC: Let’s back up. Can you remind us about where the independent labs model is most potent – the kinds of applications where we should expect to see CEO-led, team-based, time-bound focused research will deliver breakthrough results? I’m talking about about those that are not properly incentivized by more traditional research structures. Infrastructural technologies that enable other research, for example. What else?

Sam Rodriques: As I covered in an overview of the private lab landscape, there are actually many kinds of independent labs, some focused and milestone driven like FROs, others more exploratory, and with many different sizes. FROs are one sub-type of this menagerie. (FutureHouse, for example, which I co-founded, is a private lab but is a bit different from a canonical FRO.)

This is also clear from IFP’s excellent X-Labs proposal.

“X02 X-Labs” sound a bit like how we see FROs: they are execution focused and develop well defined tools, systems, datasets or other products that can un-block progress in an entire field. Things like 100x cheaper proteomics for biology, or a verifiable programming language for expressing mathematics. These projects are too large and coordinated for the individual-focused incentive structures of academia, and require professional technical and operational staff all working together on the same system with tight engineering coordination. They function in many ways like deep tech startups, except with greater focus on public goods creation, longer timescales and a primary goal to accelerate R&D versus make money for investors.  

But there are other kinds of proposed X-Labs that would be more like open ended institutes concentrating talent and resources on a broader problem area and generating more serendipitous discovery versus goal directed system building. This is more like the “X01 category” in the IFP X-Labs proposal. 

There is also a need for meta-level organizations like Convergent Research which are more like “X03s”, they roadmap fields, identify and incubate promising teams to solve bottlenecks, and oversee and act as parent organizations or regrantors for a set of projects. 

DC: It should be obvious, but is still important to note, that not everything should be an FRO. There is still a very key role for academic research and training, for startups and companies, for individual fellowship support, for ARPA style coordination, for other kinds of labs and institutes, and so on. The key is to have a diverse ecosystem containing many complementary strengths. In no way are FROs “better” than other mechanisms in any generic sense – they just fill one important category of gap in the system as it otherwise exists. 

SR: There is also a very real sense in which the FRO model is still early in its evolution and still an experiment, or really many experiments. 

DC: Despite all of the progress, federal support for independent lab models to date has been limited, though it’s been exciting to see their inclusion in America’s AI Action Plan from earlier this year. This week’s announcement suggests that’s changing. What role can federal funding play in the evolving ecosystem?

SR: Philanthropy has prototyped the model, and we are beginning to see federal support. 

Federal support can change the game in a few ways, beyond the obvious but important aspects of capitalizing the emerging ecosystem at a larger scale to allow more to be done overall. 

First, while philanthropic funding can be fast, federal funding can be more systematic and predictable. Rather than a somewhat ad hoc process of “match making” between goals, projects, teams and philanthropists, federal agencies can back broad open calls for FROs in specific areas or across fields. Clear backing to be made available at the end of an open and regimented selection process will ultimately drive creation of the highest quality FROs, providing predictability and clear timelines and milestones for all involved stakeholders.  

Second, the convening power of the federal government is huge, which is important for setting up FROs from day one with top partnerships, scientific community input and scalable dissemination paths. We’d love to see FROs nucleated deliberately as a step on the way to larger government programs and projects, e.g., a technology development FRO could reduce the cost of key data collection for a subsequent federal data collection or foundational model training moonshots. 

Federal agencies could also approach FROs programmatically, funding directed projects to remove not just one but many key bottlenecks to progress in an area systematically.

It would be great to see federal agencies back not only individual FROs, but also some FRO creation and support organizations, in line with the idea of “X03s” in the X-Labs proposal. 

DC: Any advice for NSF and other federal agencies hoping to play a catalytic role? What kinds of funding and support can the federal government most usefully provide? How much of a culture change will this require for funders who typically operate on a peer-review, hypothesis and publication-centric model?

AM: Convergent has learned that the role of the “X03 style” meta level organizations can be a potent one. It provides operational support, stable and experienced governance, best practices, and a strong community of other FROs, among other things. New potential FROs don’t have to do it on their own. They can work with others who have done it before to help get started and to manage change. 

Convergent recently published some “field notes” on learnings from running an incubator and parent organization that has launched many of the early FROs.

Tactical learnings include the importance of balancing specificity and flexibility in internal and funder facing milestones; the dynamic nature of startup-like founding teams and the benefits of good governance structures; the importance of having a scaled revolution in mind beyond the FRO itself with a strong theory of change for transformation of an entire field; the need for executive coaching for project leads, and so on. 

FROs are what would typically be called “high risk, high reward” projects (although we think this terminology has some conceptual problems), and they should not be designed by committee nor can risk be eliminated from them. “Empowered program manager” models could potentially be helpful in implementing FRO programs to get at sharp, non-consensus ideas and people. But we have in fact extensively peer reviewed all of Convergent’s FROs and received valuable input in the process – the question is how peer review feedback is used and whether the peer reviewers and program officers understand the nature of these ambitious and radical projects. 

Certainly this requires going beyond what we currently think of as the standard hypothesis driven research funding model. FROs aim for broad technological enablement of entire fields, allowing more rapid search through wide spaces of hypotheses, rather than answering narrow or incremental questions. We’ve tried to illustrate this broad category with the Convergent “Gap Map” where we summarized conversations with many scientists into an incomplete and preliminary list of potential opportunity areas for this kind of project. 

Traditional journal publication is probably too slow a mechanism to be integral to a FRO lifecycle during its active sprint. FROs are using preprints and other forms of dissemination a lot in practice, to get things moving and out there faster. 

DC: NSF’s Tech Labs announcement contemplates a variety of potential applications. Do you have any reflections on where NSF’s  support could  be most useful?

SR: There are a lot of efforts to drive commercialization and entrepreneurship. A FRO mechanism should allow maximum catalysis of a field through advanced research and technology, even if that is deeply pre-commercial or open source and public goods focused. That will likely be most distinct from what the private sector is doing well already.

DC: Any advice you’d give to someone hoping to start one of these independent team-based science organizations, such as mistakes to avoid? 

AM: Try to be as concrete as possible about what you’ll build, and about the theory of change for how that will be used – a very clear North Star. Don’t just propose to solve a general area, investigate a question or form a strong collaboration. What will you build? Why is it massively and disruptively better than the state of the art? How will this reach at scale adoption and ultimately drive profound and otherwise unachievable transformation?

Carve out the FRO shaped problem. If it can be done in an existing institution, just with more money, it probably isn’t a FRO. If it is too broad, it probably isn’t either. 

Think about creating a well rounded founding team with scientific, technological, management and other expertise. Don’t come in with too many presumptions about exactly what the roles will be within it as the team gels. There will be a lot of entrepreneurial learning at the individual and group levels. 

Importantly, make sure everyone on your founding team has absolute clarity about what the North Star of the project is and is “all in” on the endeavor. It’s a big commitment. Don’t dilute the concept by making it generic, and make sure what you’re proposing has the potential to be as disruptive as, say, next-generation sequencing was for biology or the ImageNet dataset was for deep learning. 

And consider reaching out to others who have started to pursue the FRO path.

SR: From my perspective, the most important thing is that the form of the organization must be derived from the organization’s mission. When we wrote down the FRO proposal originally, we enumerated some best practices, such as the notion that FROs should be funded for 5 years up-front, and that they should spin down or spin out at that point. These are good general guidelines that are useful for funders to orient around. However, founders need to figure out what works for them and their mission: if your mission is going to take three years or seven  years, that’s fine. If your mission requires three separate projects rather than a single unified effort, also fine. Funders should have flexibility to adapt their funding to the needs of the project.

Another common failure mode I observe among FRO founders is the notion that the alternative form of funding that FROs receive means that they can just put their heads down and focus on research, rather than worrying about publishing their results, fundraising, or so on. I strenuously disagree with this. FROs should publish early and often, and should engage regularly with funders, even when they do not immediately need funding. Publishing your work and talking to funders are two ways you get feedback on the work you’re doing. The faster you get feedback, the more quickly you can iterate to higher quality.

Finally, FROs are indeed closely inspired by startups. You should learn about and consider all of the standard advice given to startup founders. Iterate quickly. Remember that execution is virtually always more important than concept. And, in particular: avoid big egos on your team. They are very challenging to manage, and are also very challenging when it comes to building team cohesion. This is especially important advice for FRO founders who may come from and have strong ties to academia, which is sometimes home to big egos. 

Below, we analyze AI R&D grants from the National Science Foundation’s Computer and Information Science and Engineering (NSF CISE) directorate, estimating those supporting “trustworthy AI” research. NSF hasn’t offered an overview of specific funding for such studies within AI. Through reviewing a random sample of granted proposals 2018-2022, we estimate that ~10-15% of annual AI funding supports trustworthy AI research areas, including interpretability, robustness, privacy-preservation, and fairness, despite an increased focus on trustworthy AI in NSF’s strategic plan as well as public statements by key NSF and White House officials. Robustness receives the most allocation (~6% annually), while interpretability and fairness each obtain ~2%. Funding for privacy-preserving machine learning has seen a significant rise, from .1% to ~5%. We suggest NSF increases funding towards responsible AI, incorporating specific programs and solicitations addressing critical AI trustworthiness issues. We also clarify that NSF should consider trustworthiness in all AI grant application assessments and prioritize projects enhancing the safety of foundation models.

Background on Federal AI R&D

Federal R&D funding has been critical to AI research, especially a decade ago when machine learning (ML) tools had less potential for wide use and received limited private investment. Much of the early AI development occurred in academic labs that were mainly federally funded, forming the foundation for modern ML insights and attracting large-scale private investment. With private sector investments outstripping public ones and creating notable AI advances, federal funding agencies are now reevaluating their role in this area. The key question lies in how public investment can complement private finance to advance AI research that is beneficial for American wellbeing.

Figure 1.

Inspiration for chart from from Our World in Data

The Growing Importance of Trustworthy AI R&D

A growing priority within the discourse of national AI strategy is the advancement of “trustworthy AI”. Per the National Institutes of Standards and Technology, Trustworthy AI refers to AI systems that are safe, reliable, interpretable, robust, demonstrate respect for privacy, and have harmful biases mitigated. Though terms such as “trustworthy AI”, “safe AI”, “responsible AI”, and “beneficial AI” are not precisely defined, they are an important part of the government’s characterization of high-level AI R&D strategy. We aim to elucidate these concepts further in this report, focusing on specific research directions aimed at bolstering the desirable attributes in ML models. We will start by discussing an increasing trend we observe in governmental strategies and certain program solicitations emphasizing such goals.

This increased focus has been reflected in many government strategy documents in recent years. Both the 2016 National AI R&D Strategic Plan and its 2019 update from the National Science and Technology Council pinpointed trustworthiness in AI as a crucial objective. This was reiterated even more emphatically in the recent 2023 revision, which stressed ensuring confidence and reliability of AI systems as especially significant objectives. The plan also underlined how burgeoning numbers of AI models have necessitated urgent efforts towards enhancing safety parameters in AIs. Public feedback regarding previous versions of this plan highlight an expanded priority across academia, industry and society at large for AI models that maintain safety codes, transparency protocols, and equitable improvements without trespassing privacy norms. The NSF’s FY2024 budget proposal submission articulated its primary intention in advancing “the frontiers of trustworthy AI“, deviating from earlier years’ emphasis on sowing seeds for future advancements across various realms of human pursuits.

Concrete manifestations of this increasing emphasis on trustworthy AI can be seen not only in high-level discussions of strategy, but also through specific programs designed to advance trustworthiness in AI models. One of the seven new NSF AI institutes established recently focuses exclusively on “trustworthy AI“. Other programs like NSF’s Fairness in Artificial Intelligence and Safe-Learning Enabled Systems focus chiefly on cultivating dimensions of trustworthy AI research.

Despite their value, these individual programs focused on AI trustworthiness form only a small fragment of total funding allocated for AI R&D by the NSF; at around $20 million per year against nearly $800 million per year in funding towards AI R&D. It remains unclear how much this mounting concern surrounding trustworthy and responsible AI influences NSF’s funding commitments towards responsible AI research. In this paper, we aim to provide an initial investigation of this question by estimating the proportion of grants over the past five fiscal years (FY 2018-2022) from NSF’s CISE directorate (the primary funder of AI R&D within NSF) which support a few key research directions within trustworthy AI: interpretability, robustness, fairness, and privacy-preservation.

Please treat our approximations cautiously; these are neither exact nor conclusive responses to this question. Our methodology heavily relies upon individual judgments categorizing nebulous grant types within a sample of the overall grants. Our goal is to offer an initial finding into federal funding trends directed towards trustworthy AI research.

Methodology

We utilized NSF’s online database of granted awards from the CISE directorate to facilitate our research. Initially, we identified a representative set of AI R&D-focused grants (“AI grants”) funded by NSF’s CISE directorate across certain fiscal years 2018-2022. Subsequently, we procured a random selection of these grants and manually classified them according to predetermined research directions relevant to trustworthy AI. An overview of this process is given below, with details on each step of our methodology provided in the Appendix.

1. Search: Using NSF’s online award search feature, we extracted a near comprehensive collection of abstracts of grant applications approved by NSF’s CISE directorate during fiscal years 2018-2022. Since the search function relies on keywords, we focused on high recall in the search results over high precision, leading to an overly encompassing result set yielding close to 1000 grants annually. It is believed that this initial set encompasses nearly all AI grants from NSF’s CISE directorate while also incorporating numerous non-AI-centric R&D awards.
2. Sample: For each fiscal year, a representative random subset of 100 abstracts was drawn (approximating 10% of the total abstracts extracted). This sample size was chosen as it strikes a balance between manageability for manual categorization and sufficient numbers for reasonably approximate funding estimations.
3. Sort: Based on prevailing definitions of trustworthy AI, four clusters were conceptualized for research directions: i) interpretability/explainability, ii) robustness/safety, iii) fairness, iv) privacy-preservation. To furnish useful contrasts with trustworthy AI funding numbers, additional categories were designated: v) capabilities and vi) applications of AI. Herein, “capabilities” corresponds to pioneering initiatives in model performance and “application of AI” refers to endeavors leveraging extant AI techniques for progress in other domains. Non-AI-centric grants were sorted out of our sample and marked as “other” in this stage. Each grant within our sampled allotment was manually classified into one or more of these research directions based on its primary focus and possible secondary or tertiary objectives where applicable—additional specifics regarding this sorting process are delineated in the Appendix.

Findings

Based on our sorting process, we estimate the proportion of AI grant funds from NSF’s CISE directorate which are primarily directed at our trustworthy AI research directions.

Figure 2.

As depicted in Figure 2, the collective proportion of CISE funds allocated to trustworthy AI research directions usually varies from approximately 10% to around 15% of the total AI funds per annum. However, there are no noticeable positive or negative trends in this overall metric, indicating that over the five-year period examined, there were no dramatic shifts in the funding proportion assigned to trustworthy AI projects. 

Considering secondary and tertiary research directions

As previously noted, several grants under consideration appeared to have secondary or tertiary focuses or seemed to strive for research goals which bridge different research directions. We estimate that over the five-year evaluation period, roughly 18% of grant funds were directed to projects having at least a partial focus on trustworthy AI.

Figure 3.

Specific Research Directions

Robustness/safety

Presently, ML systems tend to fail unpredictably when confronted with situations considerably different from their training scenarios (non-iid settings). This failure propensity may induce detrimental effects, especially in high-risk environments. With the objective of diminishing such threats, robustness or safety-related research endeavors aim to enhance system reliability across new domains and mitigate catastrophic failure when facing untrained situations.¹ Additionally, this category encompasses projects addressing potential risks and failure modes identification for further safety improvements.

Over the past five years, our analysis shows that research pertaining to robustness is typically the most funded trustworthy AI direction, representing about 6% of the total funds allocated by CISE to AI research. However, no definite trends have been identified concerning funding directed at robustness over this period.

Figure 4.

Interpretability/explainability

Explaining why a machine learning model outputs certain predictions for a given input is still an unsolved problem.² Research on interpretability or explainability aspires to devise methods for better understanding the decision-making processes of machine learning models and designing more easily interpretable decision systems.

Over the investigated years, funding supporting interpretability and explainability doesn’t show substantial growth, averagely accounting for approximately 2% of all AI funds.

Figure 5.

Fairness/non-discrimination

ML systems often reflect and exacerbate existing biases present in their training data. To circumvent these issues, research focusing on fairness or non-discrimination purposes works towards creating systems that sidestep such biases. Frequently this area of study involves exploring ways to reduce dataset biases and developing bias-assessment metrics for current models along with other bias-reducing strategies for ML models.³

The funding allocated to this area also generally accounts for around 2% of annual AI funds. Our data did not reveal any discernible trend related to fairness/non-discrimination orientated fundings throughout the examined period.

Figure 6.

Privacy-preservation

AI systems training typically requires large volumes of data that can include personal information; therefore privacy preservation is crucial. In response to this concern, privacy-preserving machine learning research aims at formulating methodologies capable of safeguarding private information.⁴

Throughout the studied years, funding for privacy-preserving machine learning exhibits significant growth from under 1% in 2018 (the smallest among our examined research directions) escalating to over 6% in 2022 (the largest among our inspect trustworthy AI research topics). This increase flourishes around fiscal year 2020; however, its cause remains indeterminate.

Figure 7.

Recommendations

NSF should continue to carefully consider the role that its funding can play in an overall AI R&D portfolio, taking into account both private and public investment. Trustworthy AI research presents a strong opportunity for public investment. Many of the lines of research within trustworthy AI may be under-incentivized within industry investments, and can be usefully pursued by academics. Concretely, NSF could: 

• Build on its existing work by introducing more focused programs and solicitations for specific problems in trustworthy AI, and scaling these programs to be a significant fraction of its overall AI budget. 
• Include the consideration of trustworthy and responsible AI as a component of the “broader impacts” for NSF AI grants. NSF could also consider introducing a separate statement for every application for funding for an AI project, which explicitly asks researchers to identify how their project contributes to trustworthy AI. Reviewers could be instructed to favor work which offers potential benefits on some of these core trustworthy AI research directions. 
• Publish a Dear Colleague Letter (DCL) inviting proposals and funding requests for specific trustworthy AI projects, and/or a DCL seeking public input on potential new research directions in trustworthy AI.
• NSF could also encourage or require researchers to follow the NIST AI RMF when conducting their research.
• In all of the above, NSF should consider a specific focus on supporting the development of techniques and insights which will be useful in making large, advanced foundation models, such as GPT-4, more trustworthy and reliable. Such systems are advancing and proliferating, and government funding could play an important role in helping to develop techniques which proactively guard against risks of such systems.

---

Appendix

Methodology

For this investigation, we aim to estimate the proportion of AI grant funding from NSF’s CISE directorate which supports research that is relevant to trustworthy AI. To do this, we rely on publicly-provided data of awarded grants from NSF’s CISE directorate, accessed via NSF’s online award search feature. We first aim to identify, for each of the examined fiscal years, a set of AI-focused grants (“AI grants”) from NSF’s CISE directorate. From this set, we draw a random sample of grants, which we manually sort into our selected trustworthy AI research directions. We go into more detail on each of these steps below. 

How did we choose this question? 

We touch on some of the motivation for this question in the introduction above. We investigate NSF’s CISE directorate because it is the primary directorate within NSF for AI research, and because focusing on one directorate (rather than some broader focus, like NSF as a whole) allows for a more focused investigation. Future work could examine other directorates within NSF or other R&D agencies for which grant awards are publicly available. 

We focus on estimating trustworthy AI funding as a proportion of total AI funding, with our goal being to analyze how trustworthy AI is prioritized relative to other AI work, and because this information could be more action-guiding for funders like NSF who are choosing which research directions within AI to prioritize.

Search (identifying a list of AI grants from NSF’s CISE Directorate)

To identify a set of AI grants from NSF’s CISE directorate, we used the advanced award search feature on NSF’s website. We conducted the following search:

• For the NSF organization window, we selected “CSE – Direct for Computer & Info Science”
• For “Keyword”, we entered the following list of terms:
  • AI, “computer vision”, “Artificial Intelligence”, “Machine Learning”, ML, “Natural language processing”, NLP, “Reinforcement learning”, RL
• We included both active and expired awards.
• We set the range for each search to capture the fiscal years of interest (e.g. 10/01/2017 to 09/30/2018 for FY18, 10/01/2018 to 9/30/2019 for FY19, and so on). 

This search yielded a set of ~1000 grants for each fiscal year. This set of grants was over-inclusive, with many grants which were not focused on AI. This is because we aimed for high recall, rather than high precision when choosing our key words; our focus was to find a set of grants which would include all of the relevant AI grants made by NSF’s CISE directorate. We aim to sort out false positives, i.e. grants not focused on AI, in the subsequent “sorting” phase. 

Sampling

We assigned a random number to each grant returned by our initial search, and then sorted the grants from smallest to largest. For each year, we copied the 100 grants with the smallest randomly assigned numbers and into a new spreadsheet which we used for the subsequent “sorting” step. 

We now had a random sample of 500 grants (100 for each FY) from the larger set of ~5000 grants which we identified in the search phase. We chose this number of grants for our sample because it was manageable for manual sorting, and we did not anticipate massive shifts in relative proportions were we to expand from a ~10% sample to say, 20% or 30%. 

Identifying Trustworthy AI Research Directions

We aimed to identify a set of broad research directions which would be especially useful for promoting trustworthy properties in AI systems, which could serve as our categories in the subsequent manual sorting phase. We consulted various definitions of trustworthy AI, relying most heavily on the definition provided by NIST: “characteristics of trustworthy AI include valid and reliable, safe, secure and resilient, accountable and transparent, explainable and interpretable, privacy-enhanced, and fair with harmful bias managed.” We also consulted some lists of trustworthy AI research directions, identifying research directions which appeared to us to be of particular importance for trustworthy AI. Based on the above process, we identify the following clusters of trustworthy AI research:

• Interpretability/explainability
• Privacy-preserving machine learning
• Robustness/safety
• Fairness and non-discrimination

It is important to note here that none of these research areas are crisply defined, but we thought that these clusters provided a useful, high-level, way to break trustworthy AI research down into broad categories. 

In the subsequent steps, we aim to compare the amount of grant funds that are specifically aimed at promoting the above trustworthy AI research directions with the amount of funds which are directed towards improving AI systems’ capabilities in general, or simply applying AI to other classes of problems.

Sorting

For our randomly sampled set of 500 grants, we aimed to sort each grant according to its intended research direction. 

For each grant, we a) read the title and the abstract of the grant and b) assigned the grant a primary research direction, and if applicable, a secondary and tertiary research direction. Secondary and tertiary research directions were not selected for each grant, but were chosen for some grants which stood out to us as having a few different objectives. We provide examples of some of these “overlapping” grants below.

We sorted grants into the following categories:

1. Capabilities
   1. This category was used for projects that are primarily aimed at advancing the capabilities of AI systems, by making them more competent at some task, or for research which could be used to push forward the frontier of capabilities for AI systems. 
   2. This category also includes investments in resources that are generally useful for AI research, e.g. computing clusters at universities. 
   3. Example: A project which aims to develop a new ML model which achieves SOTA performance on a computer vision benchmark.
2. Application of AI/ML.
   1. This category was used for projects which apply existing ML/AI techniques to research questions in other domains. 
   2. Example: A grant which uses some machine learning techniques to analyze large sets of data on precipitation, temperature, etc. to test a hypothesis in climatology.
3. Interpretability/explainability.
   1. This category was used for projects which aim to make AI systems more interpretable or explainable, by allowing for a better understanding of their decision-making process. Here, we included both projects which offer methods for better interpreting existing models, and also on projects which offer new training methods that are easier to interpret.
   2. Example: A project which determines the features of a resume that make it more or less likely to be scored positively by a resume-ranking algorithm.
4. Robustness/safety
   1. This category was used for projects which aim to make AI systems more robust to distribution shifts and adversarial inputs, and more reliable in unfamiliar circumstances. Here, we include both projects which introduce methods for making existing systems more robust, and those which introduce new techniques that are more robust in general. 
   2. Example: A project which explores new methods for providing systems with training data that causes a computer vision model to learn robustly useful patterns from data, rather than spurious ones. 
5. Fairness/non-discrimination
   1. This category was used for projects which aim to make AI systems less likely to entrench or reflect harmful biases. Here, we focus on work directly geared at making models themselves less biased. Many project abstracts described efforts to include researchers from underrepresented populations in the research process, which we chose not to include because of our focus on model behavior.
   2. Example: A project which aims to design techniques for “training out” certain undesirable racial or gender biases.
6. Privacy preservation
   1. This category was used for projects which aim to make AI systems less privacy-invading. 
   2. Example: A project which provides a new algorithm that allows a model to learn desired behavior without using private data. 
7. Other
   1. This category was used for grants which are not focused on AI. As mentioned above, the random sample included many grants which were not AI grants, and these could be removed as “other.”

Some caveats and clarifications on our sorting process

This sorting focuses on the apparent intentions and goals of the research as stated in the abstracts and titles, as these are the aspects of each grant the NSF award search feature makes readily viewable. Our process may therefore miss research objectives which are outlined in the full grant application (and not within the abstract and title). 

A focus on specific research directions

We chose to focus on specific research agendas within trustworthy and responsible AI, rather than just sorting grants between a binary of “trustworthy” or “not trustworthy” in order to bring greater clarity to our grant sorting process. We still make judgment calls with regards to which individual research agendas are being promoted by various grants, but we hope that such a sorting approach will allow greater agreement.

As mentioned above, we also assigned secondary and tertiary research directions to some of these grants. You can view the grants in the sample and how we sorted each here. Below, we offer some examples of the kinds of grants which we would sort into these categories.

Examples of Grants with Multiple Research Directions

• Primary: Capabilities, Secondary: Application of ML.
  • A project which aims to introduce a novel ML approach that is useful for making progress on a research problem in another domain would be categorized as having a primary purpose of Capabilities and a Secondary purpose of Application of ML.
• Primary: Application of ML, Secondary: Capabilities
  • This is similar to the above, except that the “application” seems more central to the research objective than the novel capabilities do. Of course, this weighing of which research objectives were most and least central is subjective and ultimately our decisions on many were judgment calls.
• Primary: Capabilities, Secondary: Interpretability
  • A project which introduces a novel method that achieves better performance on some benchmark while also being more interpretable.
• Primary: Interpretability, Secondary: Robustness
  • A project which aims to introduce methods for making AI systems both more interpretable and more robust. 

To summarize: in the sorting phase, we read the title and abstract of each grant in our random sample, and assigned these grants to a research direction. Many grants received only a “primary” research direction, though some received secondary and tertiary research directions as well. This sorting was based on our understanding of the main goals of the project, based on the description provided by the project title and abstract.