You've got problems: E-mail overloads archives

E-mail has generated a whole new category of electronic records. The messages have enormously variable sizes and difficult-to-classify subject matter, and can carry attachments, nonstandard formats and viruses. There may be uncertainty regarding their true origins and true intended recipients. And they exist in a paradigm in which the precise sequence and time stamps of messages could be critical to placing their content in proper context.All of this has created a new category of e-records management.E-records management includes workflow management, content management, version control, archiving, access control, backup, recovery and more.It covers photographs and document images, video, sound recordings, e-mail, documents captured through optical character recognition, text files, spreadsheets, instant messages, database records and even the contents of personal digital assistants. E-records can even be indexes and links to hard-copy locations, such as bar-coded paper file folders or document storage boxes. But only records stored in traditional databases have been around long enough to have powerful legacy management systems.Every other category of e-record has received hit-or-miss management that varies from agency to agency. Systems within a single office can change with management policies.E-mail in particular has often been managed ? or mismanaged ? by users or administrators who view it as a nuisance to be answered, deleted or saved when and if they get around to it.But regulatory changes and the implications of court decisions are driving a revolution in e-records management.When high-speed and high-volume document scanners were introduced, so were management tools, but most were tightly integrated with the hardware. What's new is the emergence of more tools intended to work with an existing IM infrastructure and deal with all forms of e-records.Because agencies have different retention needs, every IT department faces a unique and growing records management nightmare, especially when it comes to e-mail and IM.Rules regarding retention and even access to e-mail and IM are changing faster than other e-record policies. And the tools used to manage other e-records are often inadequate for dealing with the surging volume and diversity of e-mail records. E-mail could be, by far, the biggest e-record management challenge facing many offices.The chart includes some general e-record management tools, but the emphasis is on e-mail and IM. Before you even start to write specifications for e-record management software, you need to answer some basic questions with an eye on a changing regulatory and legal environment.Government agencies must first decide whether they can delete any electronic messages at all. Should you keep all legitimate messages or only the "meaningful" ones? Who should have access to e-mail records? Should you keep spam? At first glance, this seems simple: just delete the junk. Obviously, you want to filter it out so users don't have to see it. But dealing with spam costs a lot of money, and laws regarding spam are changing. If you don't maintain a historical record of spam, how can you support litigation to stop spammers, gain reimbursement for costs or help prosecute illegal operations involving child pornography or scams?Should spam be filtered out before it reaches the mail server? That would conserve resources, but then you can't record it for possible litigation or even internal action if the spam or pornography was invited. Will agencies someday be held legally responsible for failing to retain records of criminal wrongdoing on the Internet? How about retaining only legitimate and "meaningful" messages? Just which ones are they? Is the simple "OK" you send acknowledging receipt of a memo meaningful or meaningless? How about attachments or text included from the previous message? If there is ever a dispute over whether you got the message, then it's certainly meaningful.E-mail and IM have expanded water-cooler chat from the local office to entire agencies, and these messages can be essential to building interpersonal relationships that can improve the work environment. They often also include some mention of business.The only sure way to save storage space is to eliminate duplicate attachments by replacing them with links, but you still need a way to determine if they are truly identical before deciding if a particular copy is meaningful.Enough examples. E-mail and IM management is incredibly complex. With some government organizations, such as Congress, receiving millions of messages every day, managing electronic messages is rapidly becoming the biggest challenge facing many IT departments.It might seem less expensive ? and more legally defensible ? to save every message rather than decide which ones should be kept and to tweak software to properly categorize and delete unnecessary messages.A cynic might even foresee the day when, as Edgar Allen Poe suggested in "The Purloined Letter," embarrassing information might be more easily and effectively hidden by placing it in "plain sight," so to speak, among billions or even trillions of meaningless junk-mail messages rather than by classifying it.But as many corporations learn when presented with a subpoena to produce archived e-mail and other e-records, the costs of deleting spam and otherwise managing e-mail can pale in comparison with the cost of restoring, searching and printing out hundreds of millions of records.Managers in high-profile agencies know they will have many retrieval demands; archive management could be the largest single cost associated with e-mail, rather than the ongoing cost of maintaining servers, sorting messages, distributing them to the correct department or person, and providing security. But even obscure departments handling what seems like mundane information could someday be faced with demands to search their archives. Consequently, every IT department must consider the possibility and prepare for it. The first thing you should do is establish detailed retention policies. Given fast processor speed, massive data pipes and unlimited storage, you can afford to be slipshod about creating an e-mail management policy as long as you keep more and delete less. Settle on a policy that can carry your organization into the next decade.Government agencies face special problems that could require them to keep every e-mail message that hits the gateway. In addition to legal retention requirements, agencies might need to keep records so they can track e-mail-based attacks, employee misconduct, employee performance and abuse of message policies.Because e-mail volume likely will never decrease, merely seeing a demonstration of software working with 10, 50 or 100 messages won't really tell you anything. At minimum, try to see a demonstration of the e-record management software operating at two or three times the present traffic volume.Unless something meaningful is done to stop spam, the volume of e-mail hitting government servers could continue to double every year.Other things to consider: