Estimating Federal Big Data Spending
Published: September 25, 2013
Like cloud computing, much of the press on big data in the federal marketplace suffers from an overabundance of anecdotes and dearth of hard numbers. FIA will seek to remedy this situation in a forthcoming report that provides data clarifying federal spending on big data goods and services. This week’s blog post offers a look at some preliminary numbers based on our research done to date.
Here at Deltek GovWin, the Federal Industry Analysis (FIA) team is currently hard at work producing report updates on a number of subjects, including the state of the federal cloud computing market, data center consolidation, and cyber security. We are also delving deeper into the nebulous subject of big data. Like cloud computing, much of the press on big data in the federal marketplace suffers from an overabundance of anecdotes and dearth of hard numbers. FIA will seek to remedy this situation in our forthcoming report by providing data that clarifies federal spending on big data goods and services. This week’s blog post offers a look at some preliminary numbers based on our research done to date. The figures presented here will be updated in the final report when it is published at the end of October.
The spending numbers presented belowinclude data on efforts that federal agencies themselves define as “big data” projects. Also included is spending on goods and services that are commonly related to big data. For example, we looked at spending on Hadoop and Palantir software, analytical support for federal fusion centers, distributed computing, data warehouse projects, massively parallel computing projects, etc. Admittedly, this is a limited dataset. The big data ecosystem, however, includes an incredibly wide array of potential solutions, tools, and services. There is simply no way that all elements of every project could be captured. In order to include as much of the data into as possible in the charts, it has been divided into calendar years instead of federal fiscal years. Sometimes the level of detail required to determine a specific award date isn’t available. Rather than leave data out of the set because a specific date can’t be found, it is easier to present it by calendar year.
Big Data Spending
Moving to the data itself, Table 1 below shows that spending on goods and services related to big data processing rose from $47 million in 2009 to nearly $1.3 billion in 2013. Spending in 2012 experienced a slight dip, but remained high at $752 million.
This rise represents a compound annual growth rate (CAGR) of 93.5% over the five year period from 2009 to 2013. The CAGR slows to a more modest 15.24% in the three year period from 2011 to 2013. One thing to keep in mind is that some of this spending is non-addressable, meaning it was spent via grants and cooperative agreements. Exactly how much is non-addressable is unclear at this point. What we do know is that agencies are spending an as yet undetermined percentage of these dollars on grants for big data research and development. The numbers published at the end of October will separate addressable from non-addressable spending.
Big Data Projects by Function
Moving now to the number of big data projects separated by function. Table 2 includes the top 5 functions as defined based on the title of the project and any available description. Projects related to science, space, and technology development (i.e., the development of new computing paradigms), continue to employ big data tools and solutions most frequently. National security, intelligence, and cyber security uses are next, followed by homeland security and law enforcement related projects.