Experimenting with Science and Structure in Government
This piece is the first of a series of conversations hosted by FAS CEO Daniel Correa aimed at highlighting issues ripe for deeper discussion, and bringing together thinkers with valuable perspectives on how to drive positive change in public policy. The following transcript has been lightly edited for concision and clarity.
Daniel Correa, FAS CEO: Thank you both so much for joining me for this conversation. Seemay, I’m going to start with you. So you’ve practiced science. You’ve taught and now you are running several organizations that have as their mission a desire to encourage more experimentation, and to provide more funding and support for ambitious, scientific and useful breakthroughs. At the same time, you’ve done a decent amount of writing about how research institutions that perform and support science need to be more experimental.
So I would actually just like to start there. Why is the point about experimental, at the institutional level, so important?
Seemay Chou, co-founder of Arcadia Science and Astera Institute: For a really long time in the U.S., we’ve had just one main type of [research] institution. It’s very monolithic. And I think experimentation is important for a couple of reasons. One is that we don’t actually know: what are the right solutions? And the opposite of what people do is not necessarily correct. Also, I would argue that there’s not one solution. We need to learn a lot more about the heterogeneity of solutions that could exist that are fit for purpose. It’s very difficult to even explore that space, without just trying things. And what does it mean to do an experiment?
Is there some level of uncertainty where it could fail? Number one, that framing allows you to try many more things that in each case don’t have to hedge as much. Because if you do a bunch of things that hedge, you actually don’t shoot yourself into any new space to actually test some new hypothesis.
There’s so much we don’t know still, because of how few things we’ve tried.
You have to go far out enough [from institutions] where people actually behave differently, and do something differently, and that creates a level of risk for everybody involved. And then you have to find the kinds of funding sources where they’re okay if it doesn’t go well.
So when are the moments in history where [experimentation] might be the most important? I would argue right now. This is the most important time to do it because the technological and sociological contexts in which we established these institutions has completely blown up.
Correa: This very much resonates with me. I would love an example of what it looks like to do meaningful experimentation within scientific institutions.
Chou: Why don’t I just draw from my own example? But I’ll caveat it by saying the reason I will draw from my own examples is because I know the most about them. It is not because I think that they’ve fully succeeded, because I think that we’ve gone through a bunch of iterations where most of the experiments that got initiated in the last decade, we don’t actually have enough information to know if they’re full success.
Correa: Caveat accepted.
Chou: At Arcadia, one hypothesis we’re trying to test is that we think that there’s a whole lot of basic science that could be happening in the upper right hand quadrant of Pasteur’s quadrant, which is like a combination between basic and applied.
That is actually why we have set Arcadia up as a for-profit. If we have enough capital and different investment metrics, it’s basically self-funded to want to see a return on investment way later – like we can give it at least a decade. That’s very different from how capital usually flows today in the for-profit sphere.
Then the question is, can we see a win-win here that’s actually both good for science? Because you see more money coming in from the for-profit space, you see better connection with for-profit outcomes, like product outcomes that have true utility in the world.
It’s very different from what you see in current institutions. It’s not quite “company” and not quite “academia”. There’s no “Oh, let’s start out as a nonprofit and then if it goes well, we’ll become a for profit,” right?
We have to go for it. This is the thesis. We’re going to have to make sure we find the right capital that’s aligned for it, and then we can really try it. And the jury’s still out on whether we’re going to succeed. But what I can tell you is that, I think we would have behaved very differently if we hadn’t done it exactly in this way. We would have recruited a very different set of people, the outcomes would have been different in terms of the activities that we’re doing and the behaviors we’re seeing.
If you talk to actual academic scientists, their ultimate deliverable right now is a journal article. That deliverable drives a whole lot of behaviors and activities that happen, because that is the incentive. Besides money, there’s journal articles and prestige.
And so that was a test at Arcadia: no more journal articles. It’s not even an option. How would that fundamentally shake up how people are approaching their science? Incentives alone may not change people – but the most important thing this did was filter for the right people who are willing to approach things differently.
If you look at my track record, I’m not like an open science warrior. Before this, I was definitely publishing in journals. I knew from my own experience and the scientists around me that this was extremely influential to how we did things, sometimes subconsciously.
So [at Arcadia] we took that off the table, and actually it was remarkable. It was surprising to me the degree to which it changed the game. Even more so than funding – this was number one.
For one, the thing that was the simplest to remove was the ranked authorship hierarchy that creates an incredibly zero-sum prestige game that really changes the way you think about your cost-benefit analysis of where you’re going to put your time. You want to make sure you own things, you want to make sure you only work on things where you have the potential to be a higher up author.
There’s no ranked order; it is just listed, alphabetically. Overnight, every single scientist at Arcadia told me that was the biggest game changer for them, because then when they thought about working together as a team – it was truly as a team.
And I’ve seen this over and over even with other things we funded outside of Arcadia. When I take [journal publishing] off the table, people then actually have to go back to the drawing board and think, ‘All right, like, what should we do then?’ Because if that’s not part of the calculus, they literally come back and propose something entirely different. That’s maybe the strongest piece of evidence I have.
And I do want to clarify: I don’t think these are the types of experiments government should take the first step on because government has a huge responsibility to think about lasting infrastructure. This is where I feel like philanthropy or private public partnerships are really helpful to be like, ‘What are the things that we need to try out for the first time?’
There’s a bunch of de-risking that should happen to develop conviction and tactical strategies and infrastructure and all that before you ask the entire ecosystem to do it.
Correa: You put a lot on the table. So let me just synthesize a little bit. So, institutional innovations and institutional experimentation – that is actually downstream of a broader exercise that you described, which is finding intervention points in the entire ecosystem of what it would take to change the system. I think it’s relevant to how federal agencies should think about their missions and their work.
Because the condition you put on publication in journals is born out of an experiment you wanted to run on an insight about what was wrong with the system broadly. Then you also built an institution that was itself designed to be experimental as an intervention. And these things go together, but they come from a common systems level, thinking about what’s needed to drive change.
Chou: Yes, that’s absolutely right. What is worth building incentives around? You really need to identify those ‘first principles’. And then the closer you can stick to those ‘first principles, the better off you’ll be.
Correa: Loren, I want you to react. Are our federal institutions capable of this type of experimentation?
Loren DeJonge Schulman, FAS Director of Government Capacity: I think the answer is: maybe.
The simplest question is, ‘What do you want federal institutions to be there for?’ They could do this, but is that actually the role that we want them to play?
And that’s very related to something that I often think as people are talking about federal innovation or reform, no matter what it is: ‘Who is your customer? What are you actually trying to do? And who are you trying to do it for?’
Because odds are, as you’re thinking about incentives and trying to manage them, you’re not actually thinking about who else is going to come in and mess with the dynamics in your ecosystem no matter what kind of incentives you try to build.
For federal science institutions, you’ve got a lot of customers, and it’s often hard to consider what those incentives are to serve each. It’s something where we need to make more active choices and fewer tacit choices. You’ve got Congress, you’ve got the scientific community, you’ve got the public – hopefully, the public matters! And then you have industry that you’re trying to actually keep alive, and you have universities.
But we are not usually explicit about the master that we’re trying to serve most; or we try to serve them all, and serve all of them badly. And as a consequence, you end up designing incentives around things that are either more convenient or match the master of the time.
The other challenge I see in federal institutions is so often we turn to: “We’ll bring in a new leader that will make everything better.” Which is not a wrong choice short term, but all the experimentation that you are trying to drive with that new leadership change will probably end when they leave, unless you’ve actually rearchitected the incentives and the understanding of what you’re trying to innovate around – change management.
I think that your point on budget is an interesting one, because most of the people outside of the federal space tend to think of there being a lot more funding in the federal government to invest in things than there actually is. Or, if they’re right about the amount, they’re wrong about the flexibility.
So much of [federal funding] is already sort of pre-baked. So the amount of resources and flexibility you have to do new tiny things goes into a smaller and smaller and smaller box. And with that comes all of the administrative burden requirements that we lump on top, and in government that’s often, “Well, since you’re wanting to innovate in this particular way, can you also do it also in this region of the country, with these particular innovators in mind ,and with also these sorts of outcomes?”
[Public servants] tend to get to the end of that process and feel like, “I’m not really innovating, I’m just sitting here treading water in place,” even if it’s all very, extremely well-intentioned.
That’s where I think talent ends up being like the final constraint on innovation. The ways that we bring people in and out of the federal space, and particularly in the science space, is all very well-intentioned in terms of particular rotator programs, and particular civil service programs. But none of which allows you to accumulate the kind of knowledge you need in order to drive change consistently and sustainably, and also makes it so these people are incentivized to do the least innovative thing possible.
All that said I believe the federal government can also be a really good vehicle for carving a space that no one else could before – and so I’ll guess I put it back to you two, should I be thinking about this in a drastically different way – that federal government should be carving a space no one else can or that the federal government should be the slightly conservative infrastructure behind the scenes?
Correa: I have thoughts on this, but I’d love to hear from [Seemay Chou].
Chou: The thing you said about talent really resonates with me.
But the thing I have found most useful in, like, any of my interactions with government so far with our experiments is thinking of the government in one way as my customer. So I’m thinking, “What do I need to test to de-risk this [institutional experiment]?”
In some cases, the government has, actually, enormous power to sort of anoint something as potentially acceptable. Like preprints is a good example of that, right? Like when the government started basically paying attention and changing certain language and policy to basically say, ‘we accept this,’ that can move the Overton window. That is actually really helpful.
Your comment about talent is very related to that. What I have found in every arena that I’ve worked in is that there is no training people how to be risk-tolerant.
If you want to do the crazy experiments, you need to find crazy people, because otherwise it’s going to continue coming towards the middle. And the people who are least crazy are the ladder climbers and so institutionalist. So now, when I hire for stuff like this, the more radical of a thing that we’re trying, the more I push on this with people – I’m trying to get a sense of them innately.
Do they have a stomach for this? Because otherwise, they’re not the right person to lead this. And this is part of the reason why we have such a sharp line on the open science stuff about being outside journals. It’s a sharp line that only a maniac would cross, and that’s the right kind of maniac for me.
Correa: I’m imagining three categories of experimentation for government and for research institutions. I think one is where we started, which is government itself is not actually doing anything experimental at all. Things are being de-risked elsewhere. So I think a good example of this is the way in which the philanthropic community directs new funding models or new vectors of research.
Focused research organizations (FROs) are a good example – de-risked by philanthropic dollars, as a model for team-based time-bound science that now government is adding to its portfolio, at least at a modest size. An earlier example of that was, the work that HHMI (Howard Hughes Medical Institute) did to fund people, not projects, [to the point] where we now see NIH adding it to the portfolio.
We can quibble about whether those are the right level of adoption amid the portfolio in the government. But you can see that this move where someone else experiments, gets some success, and government can go take it and say, ‘Hey, this thing benefits from scale that only government can deliver.’ That is a move.
Category two is explicit experimental function. So we’re talking now about things like a metascience unit at the National Science Foundation. There are antecedents to this in government through the Evidence Act, that gave agencies a mandate to ask questions and to evaluate whether the answers and have them inform the way that they work, and other movements before that.
There’s a piece of that that’s deeply relevant to the state of the field. And what we understand about how to fund breakthrough science and how to organize it and what works. We actually don’t know enough.
And then the third piece is where you want to empower talented people to be the living embodiment of a vibrant, experimental approach to shaping a field. Our institutions on the research side are not organized this way.
It’s a very different structure, talent model, and level of empowerment that would get us to that. You don’t even need to call it experimentation. It just looks like empowering field leaders and tackling bottlenecks as a substantial part of what government research does.
DeJonge Schulman: The thing that struck me most immediately as you were talking is the explicit experimental functions that government can create space for: the metascience unit, the Evidence Act, the ARPAs, and things like that. Government needs to get better at being able to provide the behind the scenes infrastructure for that kind of work, and to acknowledge that sometimes it’s going to start a new team, a new organization, a new ARPA, whatever it is, and that’s going to last for like three years.
Not necessarily because it was a bad idea, but because momentum will fade – the right leader is not there. You may have the right moment in time for a thing, and it doesn’t need to be forever. But what you need to get good at is starting it, and making sure that whatever incentive you’re trying, you’re getting that perfect, whether that be the get-the-money-out-the-door function, which is hard in government, or the hiring function that gets you the right kind of people in quickly.
If I said, “I’m going to go start a totally new experimental research institution in government,” people might say, “Okay, cool. That’s going to be like ten years from now,” because it just takes forever to do.
Let’s get better at doing the shorter-term stuff with the acknowledgment that it may be shorter term.
Chou: There has to be an on and off rate. Otherwise there’s not really innovation. I have often found that the startup space does way better than the nonprofit space: if you talk to founders, they’re not that nervous if their first company fails, because that just happens.
Then they found a new company or find a new job. There’s more of a market around that. So none of it’s that scary. The metascience stuff would benefit from that, and it makes sense why the government would benefit from it too.
Things fail and things also expire as the world changes.
Correa: It’s not like this is a novel thing that only science is grappling with. We sit here and talk about Vannevar Bush – that was 75 years ago, but think about how long the Department of Interior has been around. Everyone’s grappling with aging institutions and mechanisms and trying to fit them to a changing mission.
Loren, are there any examples outside of our research institutions that we should be grabbing on to, for how our federal reasons from some situations can do this better?
DeJonge Schulman: Some of the newer institutions that have kicked off in government have done reasonably well: U.S. Digital Service, Consumer Financial Protection Bureau and a few others. They have tried to find the systems and authorities that made the most sense for them to operate, as opposed to just accepting whatever authority was handed over to them.
So whether that be around the amount that you could pay people, how quickly you can hire them, the way you do procurement, your enforcement power – one key thing is being able to mine the authorities that are available to you. You hire a bunch of genius lawyers who are ‘yes’ people, ‘yes-and’ people, to find what is available for you to actually go out and do. You either build those into the system that you’re trying to create, or be as flexible as you can in order to adopt them in some way.
Something else these sorts of organizations have done really well is be really purposeful about learning and adapting. In the federal space, that’s a really nice-to-have phrase. But what I mean by that is that there’s a cyclical moment in those organizations where they’re saying, “Hey, we set up this process that really sucked for everybody. Let’s redo it, not just on an every-year basis, but iterate every couple of weeks, every week or so.” Check on things that you have power over in those moments, as opposed to putting them off.
You can’t just rely on your superstar celebrity leader being the person who takes you over the innovation curve. You’ve probably got to have something like them at a key stage, but you also have to build up the cadre of, like, the most boring administrative functions you can think of who make their vision happen. They’ve got to be your champions, too.
When Seemay was talking about, ‘you need the crazy people,’ – I think you also need the crazy people who are building the safe space for that crazy person. I don’t mean ‘safe space’ like I’m going to limit you, but let me give you the infrastructure around you to make sure you succeed. So that they are always there to kind of have an internal absorption rate of the crazy [innovators] that come in, and so that they are not maybe safe, but like they are not likely to totally go off the rails.
The ‘point five’ to that last point is having really, really, really strong recruitment pipelines. Federal government doesn’t invest in that. They just wait for people to apply. Having and investing in people whose job it is to recruit and find the weirdest and most amazing possible talent for you is something that would get us so much gain in the federal space.
Correa: And so you could imagine something coming into focus that is a more experimental, agile form of federal research institution. Can you all help me understand what the core components of an agenda like that might be in your mind? Where do we start?
Chou: There are existing examples to draw on outside of government. I think a lot about pharma because I’m in biotech. There’s a whole bunch of innovation labs within companies like Johnson and Johnson, and Pfizer. They hire the right people, with guardrails.
But they’re really trying to figure out how to increase the surface area and touch points between them and the startup ecosystem. It can be anything from them going out and literally funding startups, to just advising them.
[Eli] Lilly is doing this with the new Lilly Tunelab. They provide basically the [AI-driven drug development] models and the data, or they provide the models and the infrastructure. And then outside folks come in from all over, whether it’s startups, academia, and sometimes they can have access to the models, but they at least introduce data. It’s a win-win. It’s also a great way to ‘date’ and see who’s out there, doing what.
Is there something like that that the government can help with, through data infrastructure or something like that, that actually creates like an awesome scaffold to have people trying lots of different things that [government] may not even need to directly fund, but that draws people in that are innovating in different arenas and for different problems to kind of learn from that?
DeJonge Schulman: To build on that, something that we talked a lot about in the tech talent space is that for a while – actually probably still – government would hire incredible technology talent into the federal government, and they would be brought in and it would take three months to get them an actual working computer, or the printer would never work.
Then it took ten minutes to load up the software in the morning to get them to log in. And the analogy we use for that was, ‘you’re bringing in World Cup players and asking them to play on a concrete field at an elementary school.’ Don’t even think the mission is going to be possible until you’re willing to invest in whatever that playing field is.
I think some of the other obvious [elements for success]: independence of some kind, whatever that means when you’re operating in a political context because it’s always going to be political in some way. And figuring out ways to have failure tolerance, and being really clear upfront all the time in every possible conversation about what that means, not only with Congress and the appropriators, but also with leaders, and the people that are doing your hiring for you.
We talked about talent, we’ve talked about hiring authority a whole lot. But we should also be really clear on not just the kinds of folks you want to bring in and how long you want to have them there, and how you want to grow them – but what a healthy organization looks like to you. How do you want people to feel at the end of the day? Do they have what they need to succeed? We’re not thinking about that a lot in the federal space. It’s just more, ‘Here’s your cube. Show up.’
We would have a far more productive, and often experimental, federal government if we were more focused on the health of the organization – not just how a few people feel about it, but how you know you’re actually doing the right work and are actually building that environment around it.
Correa: Going back to where we started the conversation, there’s a north star for what the National Science Foundation should be doing, and what its role in the research system is or can be. And this is like the bleeding edge of pushing the frontier, and there are a vast number of ways to effectuate that, but there are some common sense things that can get us closer to what it might look like to do that really well.
A couple of examples that I can think of are: in the global development context, for a long time, there’s there’s a nonprofit organization called UnlockAid. Back in the days of USAID, this nonprofit was both a source of programmatic ideas and accountability for actual evidence and ROI of what USAID was doing.
There’s no ‘Unlock NSF’, for example, where it is like a friendly partner on the outside that has capacity and expertise in how to think about how to make what you’re doing even better, given that some of those functions are hard to endure in government.
That strikes me as a little bit similar to the models that Seemay was just describing. By the same token, an aligned, philanthropic foundation like what DOE (Department of Energy) has and a bunch of other research institutions have. NSF doesn’t have one where you can imagine resources being deployed in a really aligned way.
In addition to some of the core attributes that we would want to include, just to continue to pull this thread for the National Science Foundation: it is up for a reauthorization just next year. So there’s an actual opportunity to kind of interrogate the DNA of the institution at a moment.
As Seemay pointed out, we are in one of these founding moments for federal institutions, by choice or not, where you can actually pull a number of different levers that could shape how an organization like that functions [and make sure that it’s] rooted in a more experimental approach.
Chou: One thing that really strikes me is just how difficult it is to move that process in conversation, because it’s getting really conflated with political polarization in this moment. The way I think about an institution like NSF, I actually feel like the shift in the technological landscape is actually bigger, even though the political landscape is louder.
If we can try and move past some of the political polarization to actually talk about what needs to happen, because there needs to be a lot of change, actually. And it’s kind of a once in a lifetime opportunity for that.
Correa: I think that’s why ‘experimental’ is the right Northstar. It’s not in the political sense. But like in reality, as a metric, an agile, self-learning institution is one that is equal to the challenge that our science endeavors present.
What excites me is that it’s very tempting to be very discouraged, and say, ‘Oh, we’ve got these archaic institutions that are calcified and you could never change them.’ But I think we’re in the middle of a technological revolution that will upend lots of things, and does provide a window.
But we have to be really intentional about building that experimental capability into the institutions, and it gives me hope.
DeJonge Schulman: I think this NSF moment is both good unto itself, but also it’s a good moment for us to think about how to do this on any number of levels with federal institutions. One of the things that is true of every federal institution is there’s just a ton of mythology built up around it, some of it kind of close to true, and some of it maybe not as much true.
There’s probably a lot of mythology at NSF around what is the right kind of instrument mix. What is it that we actually know about? What works in terms of funding mechanics and getting money out of the door, not just at NSF, but in the overall R&D enterprise. We know a little bit about that, and we have a lot of biases about it.
But that’s an example of something where we really need to interrogate our mythology around NSF, and around similar institutions before we go in and say, here’s where this has to be. Something that often happens when Congress gets a chance to relook at something – their instinct is, ‘What do I want to protect and keep, and constrain and keep in a box?’
We need to do as much as possible to get [Congress] to think about what they’re trying to empower and incentivize overall. That’s a much better platform to start from. But it’s harder. It requires looking at questions like: ‘What slowed us down before that we had wanted to do? What is it that we always want to be true about this institution? What is it we would want a chance to be able to revisit on a regular basis as the science changes?’
Because reauthorizations may end up taking forever because of how Congress works. They just delay things and kick it down the road. It’s something where there’s a lot of trust and mistrust that is resident in this space right now.
Rethinking how public accountability, public feedback, public participation in science and design work. It doesn’t have to be a radical rethinking, but at least opening the door to that is really important, as is thinking about what the public wants to know about boundaries within R&D. As we look at artificial intelligence, like in health care, or look at quantum, or biosecurity, any number of issues where there’s a lot of fear in this space and there doesn’t necessarily have to be. There is an opportunity to have a dialogue, and we miss those chances when we don’t take this moment to say, ‘How might this work better?
Correa: I think it’s a great place to leave it. Thank you both.
X-Labs seek to expand on what FROs have shown is possible: the generation of foundational infrastructure for entire new fields of research science.
The United States has never lacked for scientific ambition. What we need now is a renewed civic commitment to ensuring that talent is harnessed for the benefit of all people. Science can work for everyone. Join us as we build a broader coalition committed to that vision.