Practice, practice, practice

With the fury of Hurricane Katrina approaching the Gulf Coast, employees at the National Finance Center began backing up their computer systems.They had just finished running the payroll for more than 500,000 federal government employees. They made three copies of the back-up. One went into the vault at the facility in Slidell, La. Two other copies went into two vehicles, which took different routes to the center's backup site in Philadelphia.At 2 a.m. Sunday, Aug. 28, the Agriculture Department facility shut down.It was all according to plan, said David Combs, Agriculture CIO. The department runs the National Finance Center, which provides payroll and personnel services for 125 government departments and agencies.Combs recounted the saga of the National Finance Center, which still hasn't returned to its Louisiana home, as part of a Feb. 15 Washington Technology breakfast on continuity of operations and disaster recovery.Practicing disaster recovery and continuity plans is critical, Combs said."Our plans were geared to a total loss of the facility," he said.The center has held drills twice annually since 1990 to rehearse its disaster recovery plan. The center also had uninterruptible power supply backup. Other steps included the formation of disaster response teams and the contracting of back-up facilities in Philadelphia.The plans worked so well, that the center was able to take on two previously scheduled new customers, the Transportation Security Administration and the Coast Guard, which brought 60,000 employees, Combs said."This is a good news story," he added.Disaster planning should comprise six steps, each one feeding into the next, said Federal Emergency Management Agency CIO Barry West, who also spoke at the Washington Technology breakfast.The first step is the initial response to the disaster, followed by recovery and mitigation. Mitigation leads to step four ? risk reduction ? and then to prevention. The last step is preparedness.Disasters are going to happen ? in 2005, the president declared 43 official disasters, West said.The two biggest spenders of the 2007 proposed federal budget's $6.5 billion for emergency preparedness and response are the Health and Human Services Department at $2.4 billion and the Homeland Security Department at about $2.1 billion.The inclusion of HHS might seem surprising until you consider its role in supporting state and local governments, hospitals and other health care facilities, said Kevin Plexico, executive vice president of Reston, Va., market research firm Input Inc.In its civilian agency budget proposal, the Bush administration is asking $154 million for disaster management, $87 million for continuity of operations planning and $306 million for key asset and critical infrastructure protection.The figures appear small but may be an anomaly for 2007 as larger programs move into the maintenance phase, Plexico said.No large programs may be proposed in the budget, but there are still opportunities ? particularly for small businesses ? to provide business continuity planning, communications, geographic information systems, IT systems that collect and disseminate information, and logistics support, he said."A lot of work is going through task order contracts," he said.But even with well-laid plans and well-planned systems, a lot of things go wrong when a disaster strikes, Combs said.With Katrina, the finance center's disaster recovery plan was knocked off course with the realization that the recovery would take months rather than weeks, he said.The center's backup site in Philadelphia was a shared facility run by SunGard Data Systems Inc. of Wayne, Pa.For a longer recovery, shared resources just didn't work, so the center established a data center just for its needs, Combs said.Center employees also were scattered, with some working in trailers in Louisiana, while others were in Agriculture Department sites in Kansas City, Alexandria or Rosslyn, Va., or Washington. Still others were placed at customer sites in Birmingham, Ala., and Fairfax, Va.Managing a dispersed workforce also was a challenge. "We found that some organizations need to be next to each other to work well," Combs said.Communications was a major part of the center's disaster recovery plan. They had set up a toll free call-in number for employees, which proved to be a big help, Combs said.But the center needed a better process for tracking down employees once the hurricane hit, he said."Your plan needs to ask what will you do when the infrastructure is all gone," Combs said.As the center moves forward, work on a revised disaster recover plan is under way, and the center is looking at changing several business processes, he said.One change will be that the data center computers will not return to the old facility in Louisiana. Instead, the computers will be housed at another site and will be managed remotely, he said.A review of the assumptions underlying a disaster recovery plan also needs to be done annually, Combs said."It's no problem getting people's attention now when I say we need to practice our plan," he said.Editor Nick Wakeman can be reached at .