Piloting and Evaluating NSF Science Lottery Grants: A Roadmap to Improving Research Funding Efficiencies and Proposal Diversity

This memo was jointly produced by the Federation of American Scientists & the Institute for Progress

Summary

The United States no longer leads the world in basic science. There is growing recognition of a gap in translational activities — the fruits of American research do not convert to economic benefits. As policymakers consider a slew of proposals that aim to restore American competitiveness with once-in-a-generation investments into the National Science Foundation (NSF), less discussion has been devoted to improving our research productivity — which has been declining for generations. Cross-agency data indicates that this is not the result of a decline in proposal merit, nor of a shift in proposer demographics, nor of an increase (beyond inflation) in the average requested funding per proposal, nor of an increase in the number of proposals per investigator in any one year. As the Senate’s U.S. Innovation and Competition Act (USICA) and House’s America COMPETES Act propose billions of dollars to the NSF for R&D activities, there is an opportunity to bolster research productivity but it will require exploring  new, more efficient ways of funding research. 

The NSF’s rigorous merit review process has long been regarded as the gold standard for vetting and funding research. However, since its inception in the 1950s, emergent circumstances — such as the significant growth in overall population of principal investigators (PIs) — have introduced a slew of challenges and inefficiencies to the traditional peer-review grantmaking process: The tax on research productivity as PIs submit about 2.3 proposals for every award they receive and spend an average of 116 hours grant-writing per NSF proposal (i.e., “grantsmanship”), corresponding to a staggering loss of nearly 45% of researcher time; the orientation of grantsmanship towards incremental research with the highest likelihood of surviving highly-competitive, consensus-driven, and points-based review (versus riskier, novel, or investigator-driven research); rating bias against interdisciplinary research or previously unfunded researchers as well as reviewer fatigue. The result of such inefficiencies is unsettling: as fewer applicants are funded as a percentage of the increasing pool, some economic analysis suggests that the value of the science that researchers forgo for grantsmanship may exceed the value of the science that the funding program supports.

Our nation’s methods of supporting new ideas should evolve alongside our knowledge base.

Our nation’s methods of supporting new ideas should evolve alongside our knowledge base. Science lotteries — when deployed as a complement to the traditional peer review grant process — could improve the systems’ overall efficiency-cost ratio by randomly selecting a small percentage of already-performed, high quality, yet unfunded grant proposals to extract value from. Tested with majority positive feedback from participants in New Zealand, Germany, and Switzerland, science lotteries would introduce an element of randomness that could unlock innovative, disruptive scholarship across underrepresented demographics and geographies. 

This paper proposes an experimental NSF pilot of science lotteries and the Appendix provides illustrative draft legislation text. In particular, House and Senate Science Committees should consider the addition of tight language in the U.S. Innovation and Competition Act (Senate) and the America COMPETES Act (House) that authorizes the use of “grant lotteries” across all NSF directorates, including the Directorate of Technology and Innovation. This language should carry the spirit of expanding the geography of innovation and evidence-based reviews that test what works.

Challenge and Opportunity

A recent NSF report pegged the United States as behind China in key scientific metrics, including the overall number of papers published and patents awarded. The numbers are sobering but reflect the growing understanding that America must pick which frontiers of knowledge it seeks to lead. One of these fields should be the science of science — in other words not just what science & technology innovations we hope to pursue, but in discovering new, more efficient ways to pursue them. 

Since its inception in 1950, NSF has played a critical role in advancing the United States’ academic research enterprise, and strengthened our leadership in scientific research across the world. In particular, the NSF’s rigorous merit review process has been described as the gold standard for vetting and funding research. However, growing evidence indicates that, while praiseworthy, the peer review process has been stretched to its limits. In particular, the growing overall population of researchers has introduced a series of burdens on the system. 

One NSF report rated nearly 70% of proposals as equally meritorious, while only one-third received funding. With a surplus of competitive proposals, reviewing committees often face tough close calls. In fact, empirical evidence has found that award decisions change nearly a quarter of the time when re-reviewed by a new set of peer experts. In response, PIs spend upwards of 116 hours on each NSF proposal to conform to grant expectations and must submit an average of 2.3 proposals to receive an award — a process known as “grantsmanship” that survey data suggests occupies nearly 45% of top researchers’ time. Even worse, this grantsmanship is oriented towards writing proposals on incremental research topics (versus riskier, novel, or investigator-driven research) which has a higher likelihood of surviving a consensus-driven, points-based review. On the reviewer side, data supports a clear rating bias against interdisciplinary research or previously unfunded researchers PIs, while experts increasingly are declining invitations to review proposals in the interests of protecting their winnowing time (e.g., reviewer fatigue). 

These tradeoffs in the current system appear quite troubling and merit further investigation of alternative and complementary funding models. At least one economic analysis suggests that as fewer applicants are funded as a percentage of the increasing pool, the value of the science that researchers forgo because of grantsmanship often exceeds the value of the science that the funding program supports. In fact, despite dramatic increases in research effort, America has for generations been facing dramatic declines in research productivity. And empirical analysis suggests this is notnecessarily the result of a decline in proposal merit, nor of a shift in proposer demographics, nor of an increase (beyond inflation) in the average requested funding per proposal, nor of an increase in the number of proposals per investigator in any one year. 

As the Senate’s U.S. Innovation and Competition Act (USICA) and House’s America COMPETES Act propose billions of dollars to the NSF for R&D activities, about 96% of which will be distributed via the peer review, meritocratic grant awards process, now is the time to apply the scientific method to ourselves in the experimentation of alternative and complementary mechanisms for funding scientific research. 

Science lotteries, an effort tested in New Zealand, Switzerland, and Germany, represent one innovation particularly suited to reduce the overall taxes on research productivity while uncovering new, worthwhile initiatives for funding that might otherwise slip through the cracks. In particular, modified science lotteries, as those proposed here, select a small percentage of well-qualified grant applications at random for funding. By only selecting from a pool of high-value projects, the lottery supports additional, quality research with minimal comparative costs to the researchers or reviewers. In a lottery, the value to the investigator of being admitted to the lottery scales directly with the number of awards available. 

These benefits translate to favorable survey data from PIs who have gone through science lottery processes. In New Zealand, for example, the majority of scientists supported a random allocation of 2% total research expenditures. Sunny Collings, chief executive of New Zealand’s Health Research Council, recounted: 

“Applications often have statistically indistinguishable scores, and there is a degree of randomness in peer review selection anyway. So why not formalize that and try to get the best of both approaches?”

By establishing conditions for entrance into the lottery — such as selecting for certain less funded or represented regions — NSF could also over-index for those applicants less prepared for “grantsmanship”.

What we propose, specifically, is a modified “second chance” lottery, whereby proposals that are deemed meritorious by the traditional peer-review process, yet are not selected for funding are entered into a lottery as a second stage in the funding process. This modified format ensures a high level of quality in the projects selected by the lottery to receive funding while still creating a randomized baseline to which the current system can be compared.

The use of science lotteries in the United States as a complement to the traditional peer-review process is likely to improve the overall system.  However, it is possible that selecting among well-qualified grants at random could introduce unexpected outcomes. Unfortunately, direct, empirical comparisons between the NSF’s peer review process and partial lotteries do not exist. Through a pilot, the NSF has the opportunity to evaluate to what extent the mechanism could supplement the NSF’s traditional merit review process. 

By formalizing a randomized selection process to use as a baseline for comparison, we may discover surprising things about the make up of and process that leads to successful or high-leverage research with reduced costs to researchers and reviewers. For instance, it may be the case that younger scholars who come from non-traditional backgrounds end up having as much or more success in terms of research outcomes through the lottery program as the typical NSF grant, but are selected at higher rates when compared to the traditional NSF grantmaking process. If this is the case, then there will be some evidence that something in the selection process is unfairly penalizing non-traditional candidates. 

Alternatively, we may discover that the average grant selected through the lottery is mostly indistinguishable from the average grant selected through the traditional meritorious selection, which would provide some evidence that existing administrative burdens to select candidates are too stringent. Or perhaps, we will discover that randomly selected winners, in fact, produce fewernoteworthy results than candidates selected through traditional means, which would be evidence that the existing process is providing tangible value in filtering funding proposals.By providing a baseline for comparison, a lottery would offer an evidence-based means of assessing the efficacy of the current peer-review system. Any pilot program should therefore make full use of a menu of selection criteria to toggle outcomes, while also undergoing evaluations from internal and external, scientific communities.

Plan of Action

Recommendation 1: Congress should direct the NSF to pilot experimental lotteries through America COMPETES and the U.S. Innovation and Competition Act, among other vehicles. 

In reconciling the differing House America COMPETES and Senate USICA, Congress should add language that authorizes a pilot program for “lotteries.” 

We recommend opting for signaling language and follow-on legislation that adds textual specificity. For example, in latest text of the COMPETES Act, the responsibilities of the Assistant Director of the Directorate for Science and Engineering Solutions could be amended to include “lotteries”: 

Sec. 1308(d)(4)(E). developing and testing diverse merit-review models and mechanisms, including lotteries, for selecting and providing awards for use-inspired and translational research and development at different scales, from individual investigator awards to large multi-institution collaborations;

Specifying language should then require the NSF to employ evidence-based evaluation criteria and grant it the flexibility to determine timeline of the lottery intake and award mechanisms, with broader goals of timeliness and supporting the equitable distribution among regional innovation contenders. 

The appendix contains one example structure of a science lottery in bill text (incorporated into the new NSF Directorate established by the Senate-passed United States Innovation and Competition Act), which includes the following key policy choices that Congress should consider: 

Recommendation 2: Create a “Translational Science of Science” Program within the new NSF Technology, Innovation and Partnerships Directorate that pilots the use of lotteries with evidence-based testing: 

First, the NSF Office of Integrative Activities (OIA) should convene a workshop with relevant stakeholders including representatives from each directorate, the research community including NSF grant recipients, non-recipients, and SME’s on programmatic implementation from New Zealand, Germany, and Switzerland in order to temperature- and pressure-test key criteria for implementing piloted science lotteries across directorates. 

Appendix: Bill Text

Note: Please view attached PDF for the formatted bill text

H. ______

To establish a pilot program for National Science Foundation grant lotteries.

In the House of Representatives of the United States

February 2, 2022

______________________________

A BILL

Title: To establish a pilot program for National Science Foundation grant lotteries.

Be it enacted by the Senate and House of Representatives of the United States of America in Congress assembled, 

SEC. _____. Pilot Program to Establish National Science Foundation Grant Lotteries

Right to Review.—Nothing in this section shall affect an applicant’s right to review, appeal, or contest an award decision.

Reforming Federal Rules on Corporate-Sponsored Research at Tax-Exempt University Facilities

Improving university/corporate research partnerships is key to advancing U.S. competitiveness. Reform of the IRS rules surrounding corporate sponsored research taking place in university facilities funded by tax-exempt bonds has long been sought by the higher education community and will stimulate more public-private partnerships. With Congress considering new ways to fund research through a new NSF Technology Directorate and the possibility of a large infrastructure package, an opportunity is now open for Congress to address these long-standing reforms in IRS rules.

Challenge and Opportunity

Research partnerships between private companies and universities are critical to U.S. technology competitiveness. China and other countries are creating massive, government-funded research centers in artificial intelligence, robotics, quantum computing, biotechnology, and other critical sectors, threatening our nation’s international technology advantage. The United States has responded with initiatives such as the corporate research and development (R&D) tax credit, the SBIR and STTR programs, Manufacturing USA institutes, and numerous other programs and policies to assist tech development and encourage public-private collaborations. States and cities have mirrored these efforts, helping to build a network of innovation hubs in communities across the nation

The U.S. Innovation and Competition Act recently passed by the Senate is designed to build on this progress. A key provision of the Act is the establishment of a National Science Foundation (NSF) Directorate for Science and Technology that would “identify and develop opportunities to reduce barriers for technology transfer, including intellectual property frameworks between academia and industry, nonprofit entities, and the venture capital communities.”

One such barrier is the suite of “private use” rules surrounding corporate research taking place in university facilities financed with tax-exempt bonds. Tax-exempt bonds are a preferred financing option for university research facilities as they carry lower interest rates and more favorable terms. But the Internal Revenue Service (IRS) prohibition on “private business use“ of facilities financed using tax-exempt bonds have the unfortunate consequence of hamstringing the U.S. research enterprise. Current IRS rules place universities wishing to avoid concerns about sponsoring research from outside organizations to hold the rights to almost all intellectual property (IP) generated within their research facilities, even when the research is sponsored by private corporations. This can lead U.S. corporations wishing to retain IP rights to partner with universities overseas instead of U.S. universities institutions. Small technology companies whose business plans depend on their claim to IP rights may similarly avoid partnerships with universities.

Though the IRS has issued policies that aim to address these problems (e.g., Revenue Procedure 2007-47), such policies are so narrow in scope that most research partnerships between companies and universities are still considered private uses. As a result, universities engaged in cutting-edge, industry-relevant research face an unenviable choice: they must either (i) forego promising partnerships with the many companies unwilling to completely cede claims to IP rights, (ii) dedicate substantial time and administrative resources to track and report all specific instances of corporate-sponsored research occurring in facilities financed by tax-exempt bonds, or (iii) use funding that would otherwise be available for research to finance facilities through taxable bonds.

Forcing this choice upon universities further exacerbates a system of “haves” and “have-nots”. Large and/or well-endowed universities may have the financial resources to avoid relying on tax-exempt financing for research facilities, or to hire sophisticated and expensive legal expertise for help structuring financing in a way that complies with IRS rules. But for many — perhaps most — universities, the more viable solution is to avoid corporate-sponsored research altogether.

Complex federal rules governing intellectual property and private business use are widely acknowledged as an issue. A memo from the Association of American Universities (AAU), which represents the leading research universities in North America, notes that “[m]any universities believe that the remaining [IRS] private use regulations are overly restrictive” and “[limit] their ability to conduct certain cooperative research.” Similarly, the website of the Carnegie Mellon University Office of Sponsored Programs warns:

“While colleges and universities have lobbied the Internal Revenue Service to reconsider its position with respect to sponsored research in bond financed facilities, they have not, as yet, been successful. Consequently, if the University does not receive fair market royalties from the sponsors of sponsored research, it risks having its tax-exempt bonds become taxable, with all of the concomitant consequences.”

At a 2013 hearing on “Improving Technology Transfer at Universities, Research Institute and National Laboratories” before the U.S. House of Representatives Committee on Science, Space and Technology, several university witnesses and members of Congress commented on the complications that federal rules present for cooperative research conducted by universities working in partnership with corporations.

In 2014, Congress introduced H.R. 5829 to amend the Internal Revenue Code to provide an exception from the “business use” test for certain public-private research arrangements, but it did not pass as a stand-alone bill.

In June 2021, the American Council of Education and Association of American Universities released a letter to Congress on behalf of over 20 higher education organizations asking Congress to modernize rules on tax exempt bond financing.

Overly restrictive federal rules may hamstring bipartisan efforts by the new administration and Congress to accelerate tech commercialization and enhance U.S. competitiveness in science and technology (S&T). The recent U.S. Innovation and Competition Act, passed by the Senate, for instance, aims to support public-private partnerships, cross-sectoral innovation hubs, and other multistakeholder initiatives in priority S&T areas. But such initiatives may run afoul of rules on facilities financed by tax-exempt bonds…unless reforms are adopted.

Plan of Action

The administration should implement the following two reforms to clarify and update rules governing use of facilities financed by tax-exempt bonds:

  1. Eliminate the requirement that universities must retain ownership to all IP generated in university-owned facilities financed by tax-exempt bonds. Instead, universities and corporations should be allowed to negotiate their own terms of IP ownership before entering a research partnership.
  2. Broaden applicability of IRS safe-harbor provisions. IRS revenue procedures include safe-harbor provisions that exempt “basic research agreements” from restrictions on private business use. The IRS defines basic research as “any original investigation for the advancement of scientific knowledge not having a specific commercial objective.” This definition is too narrow. But especially today, the lines between “basic” and applied research are blurry — and virtually nonexistent when it comes to cutting-edge fields such as digitalization, biosciences, and quantum computing. The IRS should broaden the applicability of its safe-harbor provisions to include all research activities, not just ‘basic research’.

Together, these reforms would support new public-private initiatives by the federal government (such the research hubs funded under the U.S. Innovation and Competition Act); help emerging research universities (including minority-serving institutions such as historically black colleges and universities (HBCUs) and Hispanic-serving institutions (HSIs)) grow their profiles and better compete for talent and resources; and repatriate corporate research to the United States. Moreover — since other countries do not have similarly onerous restrictions on research activities conducted in facilities financed with tax-exempt bonds — these reforms are needed for the U.S. tech economy to remain competitive on an international scale.

These reforms require changes to tax laws, but do not require a direct outlay of federal appropriations. Reforms could be implemented as part of several tech-commercialization legislative packages expected to be considered by this Congress, including the U.S. Innovation and Competition Act or the proposed US Infrastructure bill.

Conclusion

As the Congress and the Administration explore ways to make the U.S. more technologically competitive, ensuring robust university-industry partnerships should be a key factor in any strategy. Reforming the current rules concerning corporate research performed in university facilities needs to be considered, given that the IRS rules have not been updated in over 30 years. The debate over the infrastructure bill or other competitiveness initiatives provides such an opportunity to make these reforms. Now is the time.

A Profile of Defense Science & Tech Spending

Annual spending on defense science and technology has “grown substantially” over the past four decades from $2.3 billion in FY1978 to $13.4 billion in FY2018 or by nearly 90% in constant dollars, according to a new report from the Congressional Research Service.

Defense science and technology refers to the early stages of military research and development, including basic research (known by its budget code 6.1), applied research (6.2) and advanced technology development (6.3).

“While there is little direct opposition to Defense S&T spending in its own right,” the CRS report says, “there is intense competition for available dollars in the appropriations process,” such that sustained R&D spending is never guaranteed.

Still, “some have questioned the effectiveness of defense investments in R&D.”

CRS takes note of a 2012 article published by the Center for American Progress which argued that military spending was an inefficient way to spur innovation and that the growing sophistication of military technology was poorly suited to meet some low-tech threats such as improvised explosive devices (IEDs) in Iraq and Afghanistan (as discussed in an earlier article in the Bulletin of the Atomic Scientists).

The new CRS report presents an overview of the defense science and tech budget, its role in national defense, and questions about its proper size and proportion. See Defense Science and Technology Funding, February 21, 2018,

Other new and updated reports from the Congressional Research Service include the following.

Armed Conflict in Syria: Overview and U.S. Response, updated February 16, 2018

Jordan: Background and U.S. Relations, updated February 16, 2018

Bahrain: Reform, Security, and U.S. Policy, updated February 15, 2018

Potential Options for Electric Power Resiliency in the U.S. Virgin Islands, February 14, 2018

U.S. Manufacturing in International Perspective, updated February 21, 2018

Methane and Other Air Pollution Issues in Natural Gas Systems, updated February 15, 2018

Where Can Corporations Be Sued for Patent Infringement? Part I, CRS Legal Sidebar, February 20, 2018

How Broad A Shield? A Brief Overview of Section 230 of the Communications Decency Act, CRS Legal Sidebar, February 21, 2018

Russians Indicted for Online Election Trolling, CRS Legal Sidebar, February 21, 2018

Hunting and Fishing on Federal Lands and Waters: Overview and Issues for Congress, February 14, 2018