Data Warehousing Market Surges on all Fronts

By Edmund DeJesusThe market for data warehousing products and services in government is exploding, fueled by a growing need by government and citizens to access and analyze data for a variety of purposes, according to industry analysts. The expanding market includes both federal agencies and state and local governments. Federal government spending on data warehouse-related projects is expected to rise from $579 million in 1999 to $911 million in 2004, a cumulative average growth rate of 9 percent, according to Input, the Vienna, Va.-based IT market research firm.State and local government spending is rising even faster, growing from $550 million in 1999 to about $1.1 billion by 2004, said Rishi Sood, principal analyst for Dataquest, a research arm of the GartnerGroup in Stamford, Conn.Data warehousing makes a copy of other sources of data, and then makes that copy available for querying, reporting and analysis. The source of the data often is legacy or transaction data that otherwise would be difficult to access, although data warehouses also can combine data from disparate sources ? from different departments, for example ? to enable simpler access.The burgeoning demand for data warehousing is emanating from several sources. First, years of work to remedy the Y2K problem finally have concluded."Agencies that had been devoting resources to Y2K for the past year or more now are able to address more productive projects, often using data warehousing," said Jeff Babcock, vice president of public-sector sales and marketing for SAS Institute in Cary, N.C.Second, many agencies have heaps of transaction data, often on legacy systems, with no easy means to access that data. Third, new government directives on reporting and oversight require accessing this data, if only in summary form."A prime service of data warehousing is to get information out of legacy and stovepipe systems for mandated reporting," said Steven Lough, development manager with the business solutions group in Microsoft Corp.'s Federal Group in Washington.Fourth, government agencies are trying to satisfy their constituents by providing new services and access to information. And finally, using the World Wide Web for e-commerce or other transactions requires improved access to data. Data warehousing addresses all these issues, because it stages otherwise-inaccessible data in a structure that allows for simpler access and analysis.All the big names in the IT arena provide data warehousing solutions, including system integrators such as Andersen Consulting, IBM Corp. and Unisys Corp., major database vendors such as IBM, Informix Corp., Microsoft, Oracle Corp. and Sybase Inc., and other players like France's Bull and NCR Corp., said Dataquest's Sood.The size of a data-warehousing project can vary tremendously. Typical midsize projects could cost from $3 million to $5 million for products and services over three to five years, according to the SAS Institute. The prospect also exists for expanding existing projects into other related departments or applying the same solution to similar agencies.Sood suggested that some typical uses for data warehousing in government include:• Decreasing costs and increasing income by exposing patterns of fraud and waste, such as in taxation, health care and welfare programs;• Understanding the impact of programs and planning new programs, such as in health care, education and law enforcement;• Reporting on how agencies use resources, such as personnel and finances. All of these uses require the data flow, filtering and staging tasks that data warehousing solutions provide. "There is both a strategic and tactical view of data warehousing capabilities: to implement both long-term planning and short-term reporting," said Jon Wall, a senior technology specialist with Microsoft.Several key elements are required for data warehousing projects to succeed, according to Thomas Davies, senior vice president for Current Analysis of Sterling, Va., a market research company."Funding must be in place for the project," he said. "In addition, solutions must exist for the type of problem under consideration: You don't want to start from scratch. Finally, there must be some kind of immediate benefit, whether identifying waste and fraud or producing needed reports."Health care and taxation are common areas that can yield immediate financial savings on all levels of government, according to analysts. Indeed, some vendors operate under a shared-risk/shared-reward system in which the vendor is paid out of the savings obtained. This may be a flat fee or a percentage of the realized results. Naturally, this requires considerable upfront analysis to ensure the success of the program and to estimate the potential return on investment.In the eyes of some industry officials, government agencies are trailing the business world in adopting data warehousing. It is true that business does face the challenge of needing rapid returns on its data warehousing projects. However, government agencies were among the first users of data warehousing, a natural fit considering the large amounts of data that government deals with. Unfortunately, those early solutions were expensive and did not work out well, leaving those pioneering agencies wary."Now, there are fairly standard solutions for the major categories of problems," Wall said. "Customization is still crucial to fit any legacy systems, integrate with existing applications and support specific goals."In addition, certain development strategies produce better results. "We use an iterative-delivery approach," said Jeff Mudd, business relationship manager with SAS Institute. "A short pilot phase, on the order of five days, allows us to get a basic data warehouse up and running. We can adjust the solution rapidly to meet the agency's needs in a series of small steps. Government managers find they reach their goals more successfully and with reduced risk, in this way."Such a strategy is finding many adherents. Typical returns on investment ranging from more than 600 percent on the positive side to a negative 400 percent in struggling projects, according to The Data Warehousing Institute. Consequently, it makes sense to move slowly.Typical data warehousing projects have very complex final goals and no guarantee of success. Building a simple data warehouse first and trying it out permits changing directions without having to spend a lot of money.Because data warehousing requires several interacting components, such as collecting, moving, cleansing and storing data, selecting partners with the proper expertise is essential. "Government agencies prefer working with partners, as it reduces the risk in the project," said Jay Sloane, director of business operations for SAS Institute. And analysts said it is important to select partners whose expertise is complementary, preferably with a track record of successful data warehousing implementations.The opportunities for data warehousing are growing. As agencies continue to collaborate on various projects, such as law enforcement activities that cross agency lines, the need for consolidated data increases, according to Arnold Jackson, chief executive officer of James Martin Government Intelligence, a Fairfax, Va.-based company that provides management consulting services to federal and state government agencies."Data warehousing can serve as the foundation for intranet-based administrative and executive processes, and as a critical component in merging knowledge management initiatives," Jackson said.XXXSPLITXXX-By Edmund DeJesusData warehousing in the government arena includes a variety of projects.? The Army Reserve is consolidating finance, force structure, logistics, personnel and training data from multiple sources to improve manpower planning and accounting processes using solutions from American Management Systems Inc.? The Internal Revenue Service's Compliance Data Warehouse identifies areas where taxpayers may not be complying voluntarily, saving an estimated $250 million annually on an investment of $2 million with a Sybase database.? Missouri's 32 government agencies are leveraging their legacy systems into a real-time administrative system for financial management, budget preparation and purchasing with American Management Systems.? NASA is using engineering data from the Hubble Space Telescope to maintain the satellite with a Red Brick Warehouse database system.? A National Water Quality Assessment by the U.S. Geological Survey is being used to collect and analyze biological, chemical and physical water quality across the nation using an Oracle Corp. database.? The Naval Surface Warfare Center in Dahlgren, Va., is consolidating business information such as financial, logistics and personnel data using an Oracle database.? The Office of Personnel Management is combining sources of personnel information using SAS Institute's HR Vision to perform work force planning and retain personnel.? The Treasury Department is staging internal financial information with SAS' CFO Vision in order to run business intelligence applications.XXXSPLITXXX-By Edmund DeJesusWhen collecting data at any time, security is a top concern. Industry experts offer the following tips:? If the data is government classified, you will need a database that meets government security standards. Most of the major databases have a C2 rating (the minimum required by most agencies), including versions of DB2, Informix, Oracle and Sybase. Certain versions of Informix, Oracle and Sybase have the more rigorous B1 rating.? Web security is a concern if any of the data might be exposed to the Internet. Indeed, one of the reasons for using a data warehouse is to shield transaction systems from any Internet contact. Carefully consider what data will be accessible via the Internet and ensure that visitors cannot use the surface data as a stepping stone to gain deeper access.? Confidentiality is a prime concern, especially when dealing with health care or personnel information. In most cases, safeguards must be turned on to protect such information from all but those workers cleared to view it.? Finally, when combining data from various sources, it is good to screen one set of data from another contributor, according to Microsoft's Jon Wall. For example, if agencies A and B both contribute personnel data to a data warehouse on a higher administrative level, A and B should not be able to view each other's data. Failure to give strict attention to this common situation can have serious repercussions in situations where agencies may be in competition for funding or resources.