NASA strives to tame 'big data' flowing in from dozens of missions

NASA says new strategies will be needed to manage the ever-increasing flow of large and complex data streams from the agency's many space missions. Dozens of missions pour in data every day like rushing rivers -- data that need to be stored, indexed and processed so spacecraft engineers, scientists and people across the globe can use the data to understand Earth and the universe beyond, the agency said. For NASA missions, hundreds of terabytes -- one terabyte is equivalent to the information printed on 50,000 trees worth of paper -- are gathered every hour, creating what the technology community dubs "big data." "Scientists use big data for everything from predicting weather on Earth to monitoring ice caps on Mars to searching for distant galaxies," Eric De Jong of NASA's Jet Propulsion Laboratory in Pasadena, Calif., said. De Jong is the principal investigator for NASA's Solar System Visualization project, which converts NASA mission science into visualization products that researchers can use. "We are the keepers of the data, and the users are the astronomers and scientists who need images, mosaics, maps and movies to find patterns and verify theories," he said. Scientists face three challenges in dealing with the huge amounts of data from space missions, he said -- storage, processing and access. Rather than build more hardware for storage, engineers are developing creative software tools to better store the information, such as "cloud computing" techniques and automated programs for extracting data. For processing, JPL has been increasingly turning to open-source software, creating improved data processing tools for space missions. "We don't need to reinvent the wheel," said Chris Mattmann, a principal investigator for JPL's big-data initiative. "We can modify open-source computer codes to create faster, cheaper solutions." Huge amounts of data, stored and processed, are still of little use if it can't be easily accessed, the researchers said. "If you have a giant bookcase of books, you still have to know how to find the book you're looking for," said Steve Groom, manager of NASA's Infrared Processing and Analysis Center at the California Institute of Technology in Pasadena. NASA's "big data" work is intended to make it easy for users to grab what they need from the giant data archives, JPL said.