By now everyone has probably read about the recent $45 million sole source award that the Defense Information Systems Agency (DISA) recently made to the Alliance Technology Group for Large Data Object Storage (LDOS). In case you missed the news, the Justification and Approval (J&A) notice states that ATG will provide DISA with a scalable storage solution the development of an intelligence, surveillance, reconnaissance (ISR) cloud. The resources ATG will provide can store hundreds of billions of objects for ISR uses across DoD networks, including “Wide-Area Motion Imagery (WAMI), Standard and High-Definition (HD) Full-Motion Video (FMV), HyperSpectral, Laser Imaging Detection and Ranging (LIDAR), Electro-Optical/Infra-Red (EO/IR) and Synthetic Aperture Radar (SAR) data formats.” The breadth of data objects to be stored is interesting, as is the fact that DISA is building an ISR cloud, but to me the real importance of this notice lies in what it says about the challenges the DoD faces in trying to handle big data. Many of these challenges are themes that have appeared in FIA’s blog posts and reports for the last year.
The Strain of Big Data
In a moment of candor, DISA admits in the notice that it “cannot provide the Storage Cloud in its Defense Enterprise Computing Centers (DECCs) due to the physical size of the necessary hardware” required. Similarly, DISA states that “it does not have the funding … to purchase the required hardware or storage facility.” DISA also admits in the notice that the new ISR cloud requires increased bandwidth that the agency cannot provide: “Alliance Technology Group is the only contractor with the ability to provide the ISR Cloud Solution with bandwidth at a secure and accessible location.”
Here is the crux of the challenge in three short sentences. DISA lacks the physical space it needs for a large investment in hardware, it lacks the money to buy the hardware, and it lacks the bandwidth capacity required for ISR data analysis. In this blog post from October 2012, I made the case that big data is a game changer in the federal IT market, not because of the technologies that will be used to exploit it, but because it acknowledges that the exponentially growing demands of data management have outstripped the limited resources agencies have to handle it.
Visualize if you will all of the data that the DoD accumulates as a large sea. The level of the water is rising. Then picture the resources the DoD has to handle that data as a system of dikes used to hold back the sea. Occasionally the dikes are opened to relieve the pressure. Nevertheless, the sea level beyond continues to grow. This is the big data challenge facing the DoD and other federal agencies and the timing could not be worse. The challenge is rising at precisely the moment when the fiscal resources required are not available. The challenge of big data is not an “efficiency” problem, it is an overwhelming volume, variety, and complexity problem that requires smart governance and, more importantly, increased investment in infrastructure (commercial or government), analytical capabilities, and trained personnel.
Turning to the Cloud
Having recognized the challenge, DISA is doing the only thing that it can – it is turning to commercial cloud providers to provide the capacity it requires. In this case the capacity is storage and bandwidth. The J&A makes clear that DISA anticipates the LDOS ISR Cloud will exceed 1 Exabyte within one year and may exceed 3-4 Exabytes in three to four years. DISA is being optimistic here. Neither the DoD nor the Intelligence Community have any intention of limiting the amount of data taken in. Go to any DoD event on big data and you will hear speakers say that they want to keep every bit and byte because they never know what will be important in the future.
Takeaways
All of this means the following. Vendors need to offer secure cloud storage solutions, big data analytics (preferably as a cloud service), and related cloud service solutions that meet the DoD’s security requirements. A recent memo issued by Navy CIO Terry Halvorsen makes this latter point explicitly. This J&A award to Alliance Technology Group is the tip of the iceberg. There is a tsunami of contract dollars building to address the DoD’s big data needs. These contract dollars will flow into modernized and optimized infrastructure – like the new DISN Optical Backbone that DISA intends to build – as well as new database software called out in the FY 2013 National Defense Authorization Act (NDAA), new processing capacity, new storage capacity, and the personnel services required to make all of this go. The only thing holding back the big data spending tsunami is the fiscal crisis. This is causing procurement to dribble out in small awards here and there. However, even with imposed fiscal restraint the path ahead is clear. The DoD and all federal agencies eventually will be forced by necessity to contract out the big data services they require to cloud providers. The call has gone out in this DISA J&A. Can you hear it?