Attempts to predict the occurrence or the likelihood of extreme acts of terrorist violence on the scale of 9/11 should be discouraged because the available data are too sparse to permit the reliable modeling of such “rare events,” according to a new report to the Pentagon (pdf) from the JASON defense advisory panel.
In a nutshell, “it is simply not possible to validate (evaluate) predictive models of rare events that have not occurred, and unvalidated models cannot be relied upon.”
On the other hand, the JASONs said, it may be possible and useful to assume that rare events are correlated with more frequent, observable events which can be reliably modeled. If one assumes that “rare events events occur on a continuum with more frequent events,” then the latter can be used to help predict the former.
In this way, the JASONs calculated that the probability of another 9/11-scale event in the world could be about 7% in the next ten years. But for reasons they went on to enumerate, the underlying assumption of continuity between rare and frequent events is not demonstrably correct.
“Much of the work on [anticipating] rare terrorist events seems to take for granted that ‘the truth is out there’ and we can discover it in a sufficiently timely fashion with the right mixture of motivational assessment, social network analysis, capability measures, etc.” This may not be true, they indicated.
The JASONs offered suggestions for improving the modeling process, and they stressed the need for “good, large datasets of [terrorist] events and incident data” that currently do not exist or are not widely available. It is “surprisingly hard to obtain primary datasets” even on “straightforward” questions of terrorist event frequency and magnitude.
They cautioned that the complexity of the problem and the presumed urgency of the threat have “led some to advocate the suspension of normal standards of scientific hypothesis testing, in order to press [predictive] models quickly into operational service.” But “while appreciating the urgency, JASON believes such advice to be misguided…. Experience in the development of many other scientific fields shows the importance of adhering to rigorous scientific standards, so that small successes are tested, communicated, critically examined, reproduced, and built upon.”
“Although patient husbandry of a long-term research program may fall short of addressing the immediate operational needs, JASON believes it is the best way forward for success in the long term.” A copy of the new JASON report was obtained by Secrecy News. See “Rare Events,” October 2009.
If the government wants a system of learning and adaptation that improves results in real time, it has to treat translation, utilization, and adaptation as core functions of governance rather than as afterthoughts.
Coordination among federal science agencies is essential to ensure government-wide alignment on R&D investment priorities. However, the federal R&D enterprise suffers from egregious siloization.
Don’t like the Chinese-backed EVs that are undercutting your market? Start with a well-designed statute to strengthen market oversight and competition while also providing American companies with support.
Cities and states are best positioned to design policies to accelerate clean energy, innovation, and economic development because they can design approaches that work in different social, political, and economic contexts.