Data Warehousing: Putting Information to Work

The clamor for governments to provide better services to businesses and citizens is proving a boon to data warehousing and business intelligence suppliers that help agencies obtain valuable data and boost efficiency.

Agencies have plenty of data, but not enough "actionable information" that organizations can use to improve decision-making processes, said Jim Waldron, director of federal sales for MicroStrategy Inc. of Vienna, Va., a provider of business intelligence software.

Federal agency managers must routinely defend their appropriations before lawmakers and justify their actions on various programs. At the same time, government initiatives are requiring managers to meet more rigorous performance standards.

"We are seeing the need for agencies to constantly defend themselves, and they need better and better information to do that," said Bill Smithson, group vice president for information technology at Matcom Inc., an IT, engineering and technical solutions company in Alexandria, Va.

If the data cannot be gleaned in a meaningful way, "it's tough for officials to make those assessments and show results," said Smithson, whose company focuses on the federal sector and is working on data warehousing projects as a subcontractor.

Data warehousing systems and business intelligence tools enable agencies to extract valuable information from their databases and deliver more useful services to citizens and other users.

A data warehouse is a repository organized in a format suitable for querying, reporting and analysis. Data warehouses are built from operational databases; after the operational data is cleaned, it is put in a format that makes it more responsive to retrieval and analysis.

"A big focus at the Pentagon today is information superiority," said Waldron, whose company has done work at the Defense Department for Army, Air Force and Navy clients. Defense planners want to provide better information to warfighters for strategic and tactical purposes. "They need to get the right information to the right place at the right time," he said.

A recent study from Deloitte Research of New York backs up claims of the growing business potential of data warehousing and the related area of decision support, Waldron said. He cited a survey asking government IT officials to rank the most important technologies, data warehousing moved to No. 2 in 2000-01 from No. 5 in 1998-99, while decision support vaulted from eight to four.

"That says not only is the money increasing for electronic government, but that data warehousing is the No. 2 priority, with No. 1 the Internet, and No. 3 e-commerce," Waldron said.

MicroStrategy's business intelligence customers include the departments of Education and Housing and Urban Development, the U.S. Postal Service and a defense client base that Waldron expects to augment soon.

The company's business intelligence platform offers organizations solutions to their query, reporting and advanced analytical needs and distributes insight via the Web as well as wireless and voice technologies. Its chief competitors in the public-sector market are France's Business Objects S.A. and Cognos Inc. of Ottawa.

Database market leaders in the data warehousing space include IBM Corp., Armonk, N.Y.; Microsoft Corp., Redmond, Wash.; and Oracle Corp., Redwood Shores, Calif. Other players are Accenture, Reston, Va.; NCR Corp., Dayton, Ohio; SAS Institute Inc., Cary, N.C.; and Sybase Inc., Emeryville, Calif.

"Warehousing today is about managing the information supply chain from end to end," according to the Data Warehousing Institute, a Seattle organization that offers education and training in the data warehousing and business intelligence industry. Warehousing "now extends over the front ends, delivery vehicles and exploratory methods to encompass the entire range of information exchange, ending only at the point where information facilitates action," the institute said in an article on its Web site.

While many agencies are turning to traditional relational database tools, or middleware, some industry officials said they sometimes fail to size the technology to their projects or consider fully what they are using the data warehouses to accomplish.

In Smithson's view, much of the middleware being used today does not have the horsepower to get the job done. He said organizations must look carefully at what they hope to accomplish and explore fully how data warehouses can be useful.

Today, data warehouses are being used to detect fraud and abuse in health care and welfare programs, to measure the quality of education programs and to monitor compliance with tax and revenue laws.

And there are myriad other ways for agencies to tame the data tiger. The New Mexico Highway and Transportation Department chose a solution from Avnet Enterprise Solutions, a business partner of IBM, when it sought a more efficient way to manage all the data that goes into building a highway.

Avnet Enterprise Solutions of Tempe, Ariz., implemented an end-to-end data protection solution that works with IBM's Tivoli Storage Manager to provide centralized management, administration and protection of mission-critical data.

Avnet Enterprise Solutions is a division of $2.3 billion Avnet Computer Marketing, an operating group of Avnet Inc. The latter is a computer products distributor with sales exceeding $11.7 billion.

"We lacked enterprise storage and certainly a way to archive data used for the design, construction and maintenance of the highways," said Frank Wood, a retired chief information officer of the New Mexico Highway and Transportation Department.

The solution has allowed the agency to improve productivity and efficiency. In the past, the agency used a time-consuming process that required employees to manually change tapes on more than 30 servers whenever it needed to perform any file backups or disaster recoveries.

"I believe in utilizing partnerships in the public sector and we had a great relationship and partnership with IBM and Avnet," said Wood, who spent 44 years in government, the last three involving statewide planning efforts for New Mexico.

In regions of the United States like the Southwest that are experiencing explosive growth, highways often must undergo redesigns because of capacity constraints, Wood said. The system implemented by Avnet gave officials the ability to archive design data in an intelligent way, he said, helping slash costs and shorten the redesign cycle.

In the end, the highway department found it was doing a better job of backing up data quicker and more securely and in a way that allowed it to be prepared for recovery in the event of a disaster, Wood said. With one building previously hit by lightning, "you learn to be prepared, and we had never been able to find mechanisms to archive data that were truly recoverable from a data mining aspect."

The highway department's experience reinforced some old lessons, he said, including the need for highly trained staff to do advance planning for a system of such complexity and "to enlighten management about the increasing role IT plays in meeting the missions, goals and objectives of the agency."

John Mercier, storage director at Avnet Enterprise Solutions, said Avnet has learned it is important to do as much preconfiguration as possible. His unit specializes in data storage and protection and network integration and management solutions. He said they are seeing increased interest in such systems across the board from private industry as well as state and local governments.

"What's driving the interest is not only the sheer growth of data but also what the Internet has done to the modern IT environment," Mercier said.