BIG DATA

Do you know big data's top 9 challenges?

Those challenges are also your opportunity

Challenges equal opportunities in today's market, and when it comes to big data, there are many of both.

Earlier this week, at the Big Data Technology Leadership Series 2013, presented by GTSI Corp. and FCW, Simon Szykman, chief information officer of the Commerce Department, outlined his top nine big data challenges:

  • Data acquisition
  • Storage
  • Processing
  • Data transport and dissemination
  • Data management and curation
  • Archiving
  • Security
  • Workforce with specialized skills
  • Cost of all of the above

Data acquisition is the question of how we get data, Szykman said. It leads into the next two challenges, data storage and processing, which also are linked to each other.

Storage is especially challenging because there are many different kinds of data that needs to be stored, Szykman said.

“You have long-term storage, and then there’s intermediate-term storage,” he said.

Then data processing comes in because “you also need to worry about the volatile storage: how you keep this information in RAM when you’re doing data processing, in ways that make your data processing more efficient,” he said.

Data transport and dissemination is an area where “networking technologies are pushing some of the advances that we need,” Szykman said.

It is the challenge of getting the data from the place where it is analyzed, to the place where it is used by the people who need it, and in cases when real-time data is needed, this is critical, Szykman said.

Data management, he said, is a short-term issue, but also a long-term issue, since some data sets will be used and reused at other points in the future, he said.

As for archiving, Szykman said that while this is similar to data storage, there are approaches to doing data storage for archival purposes that are more cost-effective than the every-day types of storage,” he said.

“This makes big differences in operational costs,” he added.

Security is an obvious challenge, but it is not just that you want your data to remain confidential; “very often, one of the key issues in many of these mission areas is more one of integrity—you just want to make sure your data is not being compromised, and that you know that it remains correct and accurate,” Szykman said.

As these challenges begin to be hammered out, big data technologies will become more sophisticated, which means there will be a need for “a more specialized and highly skilled workforce to help us deal with big data,” Szykman said.

Finally, as these challenges are solved, the key will be finding a solution that is cost-effective.

About the Author

Mark Hoover is a staff writer with Washington Technology. You can contact him at mhoover@washingtontechnology.com, or connect with him on Twitter at @mhooverWT.

Reader Comments

Tue, Mar 5, 2013 CyberH

Mark, good insight on Big Data's challenges. In my opinion, companies are faced with data challenges in three different areas. First, you know the type of results you want from your data but it’s computationally difficult to obtain. Second, you know the questions to ask but struggle with the answers and need to do data mining to help find those answers. And third is in the area of data exploration where you need to reveal the unknowns and look through the data for patterns and hidden relationships. The open source HPCC Systems big data processing platform can help companies with these challenges by deriving insights from massive data sets quick and simple. Designed by data scientists, it is a complete integrated solution from data ingestion and data processing to data delivery. More info at http://hpccsystems.com

Mon, Mar 4, 2013 TASC Public Affairs Chantilly, VA

Government technology execs can access government-owned and open source big-data solutions that allow them to address their data-analysis gaps, extract actionable information from their data, and share data across agencies seamlessly and securely. Learn more from a recent presentation given by big-data experts at TexelTek, a TASC company - you can download their remarks or a summary from our website: http://www.tasc.com/media-center/white-papers/Got-Data-Now-Use-It/

Mon, Mar 4, 2013 Jeff Leston Illinois

All good, but technically-oreinted. First things to worry about are accuracy/validity of data and cost of acquisition

Fri, Mar 1, 2013 sunitha

Nice article, Big data is the future.http://globalbigdataconference.com/news.php

Please post your comments here. Comments are moderated, so they may not appear immediately after submitting. We will not post comments that we consider abusive or off-topic.

Please type the letters/numbers you see above

What is your e-mail address?

My e-mail address is:

Do you have a password?

Forgot your password? Click here
close
SEARCH
contracts DB

Trending

  • Dive into our Contract Award database

    In an exclusive for WT Insider members, we are collecting all of the contract awards we cover into a database that you can sort by contractor, agency, value and other parameters. You can also download it into a spreadsheet. Read More

  • Is SBA MIA on contractor fraud? Nick Wakeman

    Editor Nick Wakeman explores the puzzle of why SBA has been so silent on the latest contractor fraud scandal when it has been so quick to act in other cases. Read More

Webcasts

  • How Do You Support the Project Lifecycle?

    How do best-in-class project-based companies create and actively mature successful organizations? They find the right mix of people, processes and tools that enable them to effectively manage the project lifecycle. REGISTER for this webinar to hear how properly managing the cycle of capture, bid, accounting, execution, IPM and analysis will allow you to better manage your programs to stay on scope, schedule and budget. Learn More!

  • Sequestration, LPTA and the Top 100

    Join Washington Technology’s Editor-in-Chief Nick Wakeman as he analyzes the annual Top 100 list and reveals critical insights into how market trends have impacted its composition. You'll learn what movements of individual companies means and how the market overall is being impacted by the current budget environment, how the Top 100 rankings reflect the major trends in the market today and how the biggest companies in the market are adapting to today’s competitive environment. Learn More!