Do you know big data's top 9 challenges?

Those challenges are also your opportunity

Challenges equal opportunities in today's market, and when it comes to big data, there are many of both.

Earlier this week, at the Big Data Technology Leadership Series 2013, presented by GTSI Corp. and FCW, Simon Szykman, chief information officer of the Commerce Department, outlined his top nine big data challenges:

  • Data acquisition
  • Storage
  • Processing
  • Data transport and dissemination
  • Data management and curation
  • Archiving
  • Security
  • Workforce with specialized skills
  • Cost of all of the above

Data acquisition is the question of how we get data, Szykman said. It leads into the next two challenges, data storage and processing, which also are linked to each other.

Storage is especially challenging because there are many different kinds of data that needs to be stored, Szykman said.

“You have long-term storage, and then there’s intermediate-term storage,” he said.

Then data processing comes in because “you also need to worry about the volatile storage: how you keep this information in RAM when you’re doing data processing, in ways that make your data processing more efficient,” he said.

Data transport and dissemination is an area where “networking technologies are pushing some of the advances that we need,” Szykman said.

It is the challenge of getting the data from the place where it is analyzed, to the place where it is used by the people who need it, and in cases when real-time data is needed, this is critical, Szykman said.

Data management, he said, is a short-term issue, but also a long-term issue, since some data sets will be used and reused at other points in the future, he said.

As for archiving, Szykman said that while this is similar to data storage, there are approaches to doing data storage for archival purposes that are more cost-effective than the every-day types of storage,” he said.

“This makes big differences in operational costs,” he added.

Security is an obvious challenge, but it is not just that you want your data to remain confidential; “very often, one of the key issues in many of these mission areas is more one of integrity—you just want to make sure your data is not being compromised, and that you know that it remains correct and accurate,” Szykman said.

As these challenges begin to be hammered out, big data technologies will become more sophisticated, which means there will be a need for “a more specialized and highly skilled workforce to help us deal with big data,” Szykman said.

Finally, as these challenges are solved, the key will be finding a solution that is cost-effective.

About the Author

Mark Hoover is a senior staff writer with Washington Technology. You can contact him at, or connect with him on Twitter at @mhooverWT.

Reader Comments

Tue, Mar 5, 2013 CyberH

Mark, good insight on Big Data's challenges. In my opinion, companies are faced with data challenges in three different areas. First, you know the type of results you want from your data but it’s computationally difficult to obtain. Second, you know the questions to ask but struggle with the answers and need to do data mining to help find those answers. And third is in the area of data exploration where you need to reveal the unknowns and look through the data for patterns and hidden relationships. The open source HPCC Systems big data processing platform can help companies with these challenges by deriving insights from massive data sets quick and simple. Designed by data scientists, it is a complete integrated solution from data ingestion and data processing to data delivery. More info at

Mon, Mar 4, 2013 TASC Public Affairs Chantilly, VA

Government technology execs can access government-owned and open source big-data solutions that allow them to address their data-analysis gaps, extract actionable information from their data, and share data across agencies seamlessly and securely. Learn more from a recent presentation given by big-data experts at TexelTek, a TASC company - you can download their remarks or a summary from our website:

Mon, Mar 4, 2013 Jeff Leston Illinois

All good, but technically-oreinted. First things to worry about are accuracy/validity of data and cost of acquisition

Fri, Mar 1, 2013 sunitha

Nice article, Big data is the future.

Please post your comments here. Comments are moderated, so they may not appear immediately after submitting. We will not post comments that we consider abusive or off-topic.

Please type the letters/numbers you see above.


WT Daily

Sign up for our newsletter.

Terms and Privacy Policy consent

I agree to this site's Privacy Policy.