The increasing capability of high-resolution military and intelligence sensors is producing ever growing quantities of data that could overwhelm the capacity to analyze them without new approaches to data management and analysis, according to a newly released report (pdf) from the JASON defense advisory panel.
“As the amount of data captured by these sensors grows, the difficulty in storing, analyzing, and fusing the sensor data becomes increasingly significant,” the report said.
Extrapolating from current trends, data production could hypothetically reach the Yottabyte range by 2015. (The Yotta- prefix means ten raised to the twenty-fourth power. Mega- means ten to the sixth power, Giga- means ten to the ninth power, and Tera- is ten to the twelfth power.) If one byte of data were used to image one square meter of the Earth’s surface, then 1.6 Yottabytes would be generated by imaging the entire surface of the Earth every second for a hundred years, the report explained.
While the data management challenge is daunting, it is not unmanageable in principle, the JASONs said, nor is it entirely unprecedented. “Important parallels can be drawn with data intensive science efforts such as high energy physics and astronomy.” These efforts show how data filtering approaches can be applied to reduce data storage and processing requirements well below the Yottabyte range.
The report suggested several research and development strategies for improving data management and analysis. The JASONs also proposed a series of “grand challenges” that would set ambitious technical goals and provide monetary rewards for their achievement.
The December 2008 JASON report was initially withheld from public access, but a copy was released in response to a Freedom of Information Act request from Secrecy News. See “Data Analysis Challenges”.
We sat down with biomedical research pioneer Lee Hood to talk moonshots, metascience in medicine, and the Human Phenome Initiative.
DNA synthesis and export controls remain the primary regulatory safeguards against de novo production of harmful biological agents, yet governance frameworks lack the situational awareness and enforcement capacity to keep pace with rapidly falling technical barriers.
Called today to speak on behalf of U.S. science and technology, Dr. Jedidah Isler, astrophysicist, educator, strategist, policy-maker, and science communicator, will provide constructive, nonpartisan feedback to the House Committee’s hearing “American Global Competitiveness at 250: Legislative Proposals to Secure U.S. Technology Leadership.”
“Federal data and access to it is not a partisan issue. It is a people issue. Our country cannot achieve greatness without access to the data that measure what we value, who we are, and where we’re heading.”