Big Data Spending Trends: What Agencies are Spending on Software Solutions

Published: October 15, 2014

Big DataForecasts and SpendingInnovationSequestration

Finding hard data on the federal big data market is difficult. This week’s post takes a look at federal spending on big data related software solutions over the period from FY 2010-2014. The data presented is derived from a collection put together by Deltek GovWin’s Federal Industry Analysis team for a forthcoming report on trends in federal cloud computing, big data, data center consolidation, and mobility adoption.

In October 2013 Deltek GovWin’s Federal Industry Analysis team began publishing an annual report that delves into the progress federal agencies are making toward reaching a specific set of information technology goals. The areas examined are cloud computing, data center consolidation, big data, and mobility. Cloud and big data are my areas of responsibility and expertise so in this week’s post I’ll provide a small sample of the data that we analyze in the report. The data details spending related to big data investment in both the civilian and defense sectors of the federal government. Collection of the data is based on a set of 69 keywords relevant to big data investments. These keywords range from the names of specific solutions (e.g., Sqrrl, Splunk, and Hadoop) to types of products and services (e.g., fusion centers, high performance computing, and predictive analytics). The resulting dataset is therefore discreet, constituting a narrowed-down picture of federal big data investment. It doesn’t capture all investment as that would be impossible given a lack of granularity in the government data, security clearance issues, and all of the spending that takes place as part of large-scale IT services efforts. What the data does provide, however, is a baseline by which to analyze and understand agency investment trends.

Federal Big Data Investment in Software
The chart below shows that federal investment in big data-related software nearly tripled in the 3 fiscal years from 2010 to 2012, until it hit the brick wall known as sequestration in fiscal 2013. 
No segment of federal government technology spending escaped sequestration related cuts – even emerging areas like big data where there are clear benefits for federal customers using solutions for data analysis and visualization. In the case of big data related software the “correction” in spending amounted to a 27% drop from $61 million in fiscal 2012 to $48 million in fiscal 2013. The good news is that spending recovered somewhat in fiscal 2014 thanks to some stability in the federal budget. Not all of the data for Q4 of fiscal 2014 had been reported as of this post’s publication so we can expect that the final number for FY 2014 will come in higher than fiscal 2013.
Spending by Market Sector
A closer look at spending by market sector reveals interesting details that show the picture of a fiscal 2014 recovery in spending is not rosy across the board.
As it turns out civilian sector spending on big data related software continued to rise in spite of sequestration. In contrast, spending in the defense sector fell precipitously after it had risen from fiscal 2010 to fiscal 2012. To note again, Q4 defense spending data for FY 2014 hasn’t been reported yet. Even so, the fiscal 2014 total is unlikely to surpass spending in fiscal 2013.
The brief analysis above shows that sequestration had a decisively negative impact on big data related software spending at the Defense Department. This news is troubling given reports that commonly cite defense organizations as being among those most in need of big data solutions. If the data is any indication, vendors seeking business opportunity in this technology area are best off focusing their efforts on civilian agencies as these customers are increasing their spending.
The analysis presented here is a small sample of the kind of valuable insight that readers will find in the next version of FIA’s Progress Update report, due to be published on 31 October.