Not Your Father's Old Flat File

P> LOS ANGELES -- Data warehousing is a lot like those old flat-file databases that got everyone agog in the '80s -- except bigger, much bigger, and relational, very relational. If databases of the late 1980s were Lake Tahoe, the data warehouses of the late 1990s are the Pacific Ocean. Already, data warehousing is pooling the medical details of generations of Americans, from hang nails to heart attacks, and relating even those two.

These new gigabyte- and even terabyte-capacity data warehouses are also faster, much faster. The search engines on Tahoe were pretty basic outboards that would get a user across and back. Back then, finding the right data could be a lot like fishing -- takes awhile, even if you know where to drop your line. Now, for governments and companies with the money to buy into data warehousing, the churning digital ocean of data, ever-fed by fresh rivers of new data, is something to be grasped for practical use, not awed at in gee-whiz bull sessions.

The transition is huge. Take fishing. In data warehousing, the database operator sits in a stealthy jet zipping low over the chop. Inside the cabin, the operator can find the fish by intricate high-tech radar that half has a mind of its own for digitizing the location and projected movements of fish. In seconds, he knows where the big schools are going. Seconds later, he has a report that means something and is hard to argue with. Mission accomplished, the analyst zips back to LAX airport. As soon as the jet comes to a standstill, he can fire up his cellular fax/modem and broadcast a mass fax detailing trend analysis of fish school patterns up and down the West Coast to anglers willing to pay for the edge that data warehousing brings to analyzing everything from fish patterns to chances of being snagged yourself in an insurance scam.

The short story is that fishing for meaningful data -- once a half instinct, half high-tech endeavor -- has gone cold high-tech. In the brave new world of data warehousing, patterns are ascertained from on high -- or at least, from the glow of a computer terminal. Where the fuddy old outboard guys from the flat-file database times saw fish, the low-flying, stealthy geeks of the late 1990s see schools of fish and can trend the patterns where they're headed.

That analogy aside, the essential point is that those who know how to work an accurate data warehouse now have both the goods -- the data -- and the fast search engines to turn an ocean of intelligence goop into hard currency and high power like never before.

The bottom line: Data warehousing -- expected to rise to a $2 billion market this year -- is if anything a tenacious and never-going-away power resource for those who analyze and manipulate data. Those companies in a position to marshal data will make a killing "warehousing" it for high-paying subscribers.

This raises a larger question (see main story): Does the fact of massive data warehousing shift practical control of how to use intelligence from the craftsmen in the field -- doctors, lawyers, accountants, for example -- to some insurance or in-house factotum with an ocean of data at his fingertips and a 15-inch monitor in his face?