The concept of forecasting is pretty familiar to anyone who’s flipped on their local news to get a sense of the week’s weather. But the broader science of forecasting, which is being applied to policy-relevant topics such as epidemiology, energy, technology progress, and many more topics – – has never been in a more exciting place. In just a few weeks, FAS, along with Metaculus, will hold a forecasting tournament (The Climate Tipping Points Tournament) aimed at demonstrating just how powerful a tool forecasting can be for policymakers trying to effect change.
Metaculus is a forecasting platform – their unique system aggregates and scores forecasts and forecasters. Metaculus’ global forecasting community correctly predicted the outbreak of Russia’s Ukraine invasion, and its models have also helped state governments make better real-time decisions regarding COVID-19 response.
Metaculus’ CEO, Gaia Dempsey, sat down with us recently to discuss her organization’s work, and why forecasting holds such promise for better public policy.
FAS: Gaia, thank you for making the time for this. To start, could you talk about why you think forecasting, which at first blush can seem like peering into a crystal ball – not scientific at all – is actually very aligned with public policy based on scientific evidence?
Gaia Dempsey: At Metaculus we care a lot about ‘epistemics’ and epistemology. Epistemology is a branch of philosophy that deals with knowledge itself, and how we form beliefs – how we come to believe something to be true. Forecasting is a way of continuously improving our epistemics. It’s connected to the fundamental basis of the scientific method itself, which is this essential idea that our trust in any given theory about the world should increase when it’s able to predict the result of an experiment or the future state of the given system or environment that’s being studied. So this agreement between the theory and the experiment – that’s predictive accuracy. That’s the gold standard for what we trust as a valid explanation for anything in the world – any given phenomenon.
What we do on our platform is bring that mindset into more complex environments — outside of the laboratory and into society, into public policy.
FAS: You discuss Metaculus as both a forecasting platform and a community. What makes your platform unique?
GD: You can think of it as a network of citizen scientists, or a decentralization of the role of the analyst in a way that rewards and gives credence to analysis on the basis of its accuracy rather than the person’s position or title. Everyone has the ability to be an astute observer of the world. Metaculus is kind of like infrastructure that facilitates the collaboration of thousands of people and aggregates their insights. Our platform has a set of scoring rules – every time you make a forecast, you get rigorous, quantitative feedback. When you’re making a prediction, you have to take into account all of the factors that affect the outcome. If you haven’t, your score is going to let you know that you’ve missed something.
FAS: In your community you’ve said there are casual forecasters and hardcore forecasters – and recently you’ve even assembled a team of 30 Pro Forecasters – people with a track record of accuracy and prescience in their forecasts. Aside from a familiarity with and facility with data science, you’ve also said that humility is a good trait for a forecaster. Can you elaborate?
GD: I’d say it’s ‘epistemic humility’: you need to be able to update on new evidence. If you hold on really tightly to a belief, you’re going to be biased by that desire to believe that the world works one way, while the evidence is actually telling you something else.
FAS: That makes sense. In terms of why forecasting is such an exciting field right now, it seems as if we’re reaching this point in forecasting science because of the confluence of human learning coupled with new technologies that allow us to aggregate information at these incredible scales. Is that a fair way to think about it?
GD: Yes, I would say so. Metaculus mimics the structure of a neural network. We aggregate statistically independent forecasts, and our scores and tournament prizes serve as reward functions. It really is sort of like a hive mind. Within this system, accurate signals are cross-validated, while errors cancel each other out, and since we’re in effect running hundreds of trials at any given time, the system is designed to get more and more accurate over time. Our aggregation algorithm gives more weight to forecasters who have a track record of accuracy. We are basically weighting the probability of a forecaster being more accurate based on their past performance, so in a way it’s actually quantitatively controlling for cognitive biases across a population of forecasters.
FAS: Our upcoming Climate Tipping Points Tournament is aimed at showing how useful forecasting can be for policymakers, looking at questions like “What will the Zero-Emission Vehicle Adoption Rate be in X years?” or “If X policy is implemented, what will the charging infrastructure look like in year X?” It’s focusing on something known as the “conditional approach” to forecasting. How is this different from what Metaculus has done in the past?
GD: Conditional forecasts give you the delta – the difference – between taking an action, or not taking it. It’s going to be the first time we’re really doing this at scale like this. The thought process behind this particular project was: how can we leverage the talent and the analytical capacity of forecasters to actually be as useful as possible to the policy community? We want to help answer questions like, “Which policy actions are really feasible?” or “What may actually happen in the time period of interest that policymakers are concerned about?” If you’re asking, “What would happen if we just outlawed all cars tomorrow?” Yes, we could produce an estimate that could tell you what would happen to global CO2 emissions, but it wouldn’t be very useful. So it’s really about considering the policy interventions that are realistic, and developing a quantitative analysis that can tell us what the likely impact will be of these policy actions on the outcomes we really care about, such as the reduction of CO2 emissions. Our conditional forecasting methodology will make the relative expected value of various policy actions unmistakably clear, using an empirically grounded methodology.
FAS: And as the title of the tournament suggests, it will also highlight a fairly new idea to U.S. climate policy: positive tipping points. FAS’ own Erica Goldman is very excited about this area of study, but how does it relate to forecasting and the work that Metaculus does?
GD: It’s something I find really exciting about this particular project; it brings together research from the University of Exeter on positive tipping points, FAS’ policy expertise, and our forecasting methodology. The goal of this tournament is to assess what climate policies may present positive tipping points toward decarbonization. We know that tipping points are a real phenomenon in lots of different complex systems. The work that our partners at Exeter University did to pinpoint and really identify positive tipping points in our current understanding of climate science is so exciting. But we want to look at how we can then leverage those tipping points to achieve goals in terms of reducing the impact of climate change—and maybe even reversing it.
FAS: Before we let you go, could you talk about how to participate in a forecasting tournament? Once the questions and topics are released, how much expertise is required to engage with this?
GD: Laymen can totally participate – it just requires intellectual curiosity and a willingness to engage in the forecasting process. Most forecasters benefit from building some kind of model or by explicitly implementing Bayes’ rule. People who are already familiar with data science techniques, people who are familiar with modeling, people who have some kind of science background – they tend to be able to jump right in and feel comfortable, but you don’t necessarily need to have that background at the start. It’s something where you become a part of a community. If you comment on the public forecasting questions, people will engage with you very sincerely. It’s just like if you’re a new software developer: People don’t expect you to get everything right away. But if you’re genuinely putting in the effort, they will come and support you.
FAS: On that inviting note – we’ll let you go prepare those tournament questions! Thanks.
GD: Thank you!