Big Data technology is coming faster than federal employees and technology infrastructures are prepared to handle. Could agencies that are ‘cloud ready’ first be early adopters of the advanced data analytics and storage capabilities that are required to leverage Big Data?
Three years ago the emerging technology market in the forefront of the hype-cycle was cloud computing. Today the market in that spotlight is “Big Data,” a term used to describe the exploding volume of data that is burying federal agencies on a daily basis. Big Data is symptomatic of the success of information technology. IT was supposed to make tasks simpler, transactions easier, and communication more convenient. It did all of those things, but in doing so it also left behind artifacts - the records of those transactions, interactions, and messages. Commercial enterprises have discovered value in stepping back and looking at the collections of the accumulated data stored on their servers. This data tells them what their customers have bought, what they looked at, where their preferences lie, and what they might buy again, should a targeted marketing campaign reach them at an opportune time.
Federal agencies are also finding uses for the data they accumulate. This is helping them understand how better to serve citizens and how to carry out their mission goals more effectively. Agencies already awash in data see the benefit of analyzing it. What they have greater difficulty understanding is how to go about analyzing it efficiently and how to use that analysis to support better decision making. Serendipitously, perhaps, the answer to the data analysis efficiency conundrum may lay in cloud computing. The extent to which an agency is ‘cloud ready’ will likely bear directly on the degree to which that agency will be an early adopter of the advanced data analytics and storage capabilities that are required to leverage Big Data. A ‘cloud ready’ agency possesses an integrated IT infrastructure based on a standardized, service-oriented architecture that resides in a highly virtualized enterprise data center. The data center can be either proprietary or vendor-hosted. In turn, this data center contains significant storage space (preferably based on the latest available technology) that can be expanded as required and against which analytical queries can be made with relative ease.
What I’ve just described is simple enough to write about. Creating all of the pieces and fitting them together into a coherent environment is the problem that all agencies are currently struggling with. Those that achieve success earliest are the agencies that will be best prepared to consume the Big Data solutions vendors are lining up to sell them. Those that lag behind will find themselves unable to use Big Data solutions efficiently without making a significant financial investment.
To some extent, it was probably an awareness of the inability of agencies and departments to use Big Data solutions efficiently that informed the “Big Data Initiative” announced by the White House Office of Science and Technology Policy (OSTP) at the end of March. The OSTP recognizes correctly, I believe, that it is easiest in the near-term to leverage Big Data solutions for research purposes. Not only has the technology framework been created (e.g., supercomputing), the network of ready users exists to analyze the results of the data. Here again we see a parallel with cloud computing, which was also adopted earliest by the federal government for scientific purposes.
The OSTP is clearly front-running the adoption curve of this technology. In doing so they are trying to bring it into government data centers as rapidly as possible. However, given the government’s current ‘preoccupation’ (i.e., struggle) with cloud computing, the Big Data Initiative might be coming faster than federal employees and technology infrastructures are prepared to handle. I suspect, therefore, that the growth of the Big Data market in the federal government will be quite slow for the next 5-10 years. We should expect to see a considerable amount of market research concerning Big Data as government learns about the technology and its application, but I’d be willing to consider that the overall value of the federal Big Data market does not approach $500 million before 2020. The transition to cloud computing is a muddled, helter-skelter affair so far, filled with multiple adoption paths, service delivery models, and deployment types. Until 2015-2017 most agencies will be consumed with data center consolidation and with creating their preferred cloud environment. It is only after this that they will be able to approach Big Data with any semblance of confidence.
Readers curious about which of the largest agencies is approaching ‘cloud readiness’ the fastest might want to take a look at FIA’s recently published Federal Cloud Computing Services Outlook, 2012-2017 for a comparative analysis. I suspect those agencies closest to the ‘advanced’ side of the cloud readiness spectrum will be the organizations that can leverage Big Data solutions the earliest over the next 5-10 years.