Infrastructure Continues to Be a Barrier for Big Data

The federal government generates and collects massive amounts of data. As the volume of data continues to grow, climbing from hundreds and thousands of petabytes to exabytes of data, the task of managing this data becomes more and more challenging. Beyond collecting, managing, and storing the data, agencies are looking to leverage this data to create actionable intelligence, inform decision-making, mitigate risks, increase efficiencies, and lower costs. Tapping into the power of big data is going to require investment in infrastructure and expertise.

In 2012 the Obama administration flagged $200 million dollars for big data research and development programs. Efforts continue to address development needs and potential applications for analytics, in the meantime, are making the most of the available resources and integrating new capabilities as they become available. For example, some agencies are tackling their data piecemeal. The National Aeronautics and Space Administration (NASA) creates, collects, and handles so much data, the idea of establishing a new, data-focused mission area is being explored. NASA’s Center for Climate Simulations generates petabytes of data for modeling communities, but accessing the data continues to be a hurdle. At a recent big data event, the center’s lead for high performance computing shared that NASA scientists often download data in 5 to 10 terabyte chunks onto their desktops. After running their analytics, they’ll delete that portion in order to download another 5 to 10 terabytes and repeat. The approach means the process can take weeks or months to complete.

This points to a critical issue for organizations looking to “do big data.” Not only do they need to collect the data, they need to be able to make it usable. The Center intends to improve data access through enhancements to storage and the analytics performed with in the server. These steps would provide greater efficiency for the experts within NASA as well as external users. Other agencies exploring big data are expressing concerns about the limitations of IT resources to pursue projects. In a recent survey of federal government, seventy-three percent of federal government respondents noted potential big data hurdles around storage, computer, and networking infrastructure. Almost as many (seventy percent) were skeptical of their agencies ability to analyze data once it had been collected.

The consensus around impact of big data projects is likely to serve as motivation to address shortcomings around resources and staffing. According to survey responses from federal executives with big data projects underway, “90 percent say [big data] has improved the quality and speed of decision-making, while nearly 90 percent agree that it has improved the ability to predict trends, boosted internal process efficiency, and resulted in better risk quantification.” Agencies are likely to pursue these improvements through several channels: investing in network infrastructure and security, developing in-house expertise by hiring data scientists and architects, engaging consultants to address implementation and management of solutions, and outsourcing to third-party providers.