Do you know big data's top 9 challenges?

Challenges equal opportunities in today's market, and when it comes to big data, there are many of both.

Earlier this week, at the Big Data Technology Leadership Series 2013, presented by GTSI Corp. and FCW, Simon Szykman, chief information officer of the Commerce Department, outlined his top nine big data challenges:

• Data acquisition
• Storage
• Processing
• Data transport and dissemination
• Data management and curation
• Archiving
• Security
• Workforce with specialized skills
• Cost of all of the above

Data acquisition is the question of how we get data, Szykman said. It leads into the next two challenges, data storage and processing, which also are linked to each other.

Storage is especially challenging because there are many different kinds of data that needs to be stored, Szykman said.

“You have long-term storage, and then there’s intermediate-term storage,” he said.

Then data processing comes in because “you also need to worry about the volatile storage: how you keep this information in RAM when you’re doing data processing, in ways that make your data processing more efficient,” he said.

Data transport and dissemination is an area where “networking technologies are pushing some of the advances that we need,” Szykman said.

It is the challenge of getting the data from the place where it is analyzed, to the place where it is used by the people who need it, and in cases when real-time data is needed, this is critical, Szykman said.

Data management, he said, is a short-term issue, but also a long-term issue, since some data sets will be used and reused at other points in the future, he said.

As for archiving, Szykman said that while this is similar to data storage, there are approaches to doing data storage for archival purposes that are more cost-effective than the every-day types of storage,” he said.

“This makes big differences in operational costs,” he added.

Security is an obvious challenge, but it is not just that you want your data to remain confidential; “very often, one of the key issues in many of these mission areas is more one of integrity—you just want to make sure your data is not being compromised, and that you know that it remains correct and accurate,” Szykman said.

As these challenges begin to be hammered out, big data technologies will become more sophisticated, which means there will be a need for “a more specialized and highly skilled workforce to help us deal with big data,” Szykman said.

Finally, as these challenges are solved, the key will be finding a solution that is cost-effective.