Telecoms take on the elements

With Hurricane Charlie bearing down on Florida two years ago, southbound highway lanes were switched to northbound for the evacuation. As the population headed north, emergency responders headed south along the shoulders and side roads to respond to the Category 4 storm.Alongside the medical personnel and rescuers were trucks carrying AT&T Inc.'s disaster recovery teams. Restoring data and voice networks is so essential that the Department of Homeland Security certifies crews of telecommunications companies as first responders, said Robin Bienfait, senior vice president of AT&T Global Network Operations."That's us driving the reverse traffic to the affected area," Bienfait said.Whether it's a hurricane, forest fire, terrorist attack or other disaster, telecom companies have assembled the hardware, plans and people to get communications networks back online within days or hours.The equipment includes vans with satellite communications links, switching stations housed in semi-trailers, massive generators and more. The programs have been built up over the years to respond regularly to disasters and near misses. The ultimate goal is to get customers ? government, business and residential ? back online as quickly as possible."We have warehouses stocked with backup equipment that is sitting in trailers, ready to go," said Dick Price, director of business continuance emergency management at Verizon Business. "My team coordinates with various groups within the company to ensure that the equipment is constantly tested, maintained and ready. At time of deployment, we evaluate the specific need coming from our forward command positions."Verizon Business is focused on enterprise customers and large government customers for Verizon Communications Inc.Once on the scene, Verizon workers use an incident command system patterned after the National Response Plan used by the Homeland Security Department. Using a similar system makes it easier for Verizon to coordinate with government officials."The neat thing about using the incident management process is that it's scalable, so we can respond to anything, from a local switch going out to Hurricane Katrina," Price said.In the case of an impending hurricane, Verizon begins a process that evaluates the weather conditions about 96 hours out from landfall. When officials have an idea of where a storm will hit, the technical staff performs pre-emptive duties such as assessing generators, topping off fuel, and testing automatic transfer switches so systems will move from commercial power to generator power in case of an outage.Monitoring power is one of the team's most important duties."We have a really neat Web-based system that automatically reports into an internal Web page when a site has gone off commercial power to generator power, or off generator power to battery power," Price said.To test the readiness of its disaster readiness, AT&T recently conducted an exercise at FedEx Field in Landover, Md."We do four exercises or drills every year to practice putting together all of the equipment for specific kinds of offices," said Jeff Hauk, group operation manager for AT&T network disaster recovery. "We train our employees on all new processes, equipment and technologies we've implemented. Every time we do an exercise or we respond to a disaster or crisis, we learn something new."The company has invested more than $300 million in its disaster recovery program over the last decade. The investment includes managers, engineers, technicians and more than 150 equipment trailers and support vehicles."We're talking about supporting all of AT&T's telephone and data network systems. Whether you're a citizen, a business or a government office, most of that runs off communication networks, whether you're making phone calls or transferring data," Hauk said. AT&T staffs its disaster recovery efforts with people from outside the area so that local employees can deal with their own families and homes if affected by the event.That plan proved to be a problem in New York after the Sept. 11, 2001, terrorist attacks."We usually fly people in and, of course, air transport was down," Bienfait said. "So we had to scramble to get people there. It was the first time we had to say, 'How do we build out people assets?' "Other lessons learned have been applied over the years. AT&T has put together small restoration trailers to handle some flooding situations. The trailer has generators, power tools, pumps, vacuums and other equipment to dry out an office.Over the years, AT&T officials also discovered that many times when disasters strike, home stores either are closed or sold out of certain things."I remember the earthquake in Sherman Oaks, Calif., many years ago, we found the all the stores were either closed or damaged," Hauk said.A tool and supply trailer now ensures responders have all the equipment they need. The trailer has generators, compressor, welding equipment, shovels, hand tools and other supplies.To deal with areas where hazardous substances such as chemicals and gas have been released, AT&T has a special-operation team trained to work in that environment. The team has full protective suits for working in such areas.Verizon established a hazmat team in 1993 after officials noticed that train derailments often affected nearby fiber-optic lines."Our issue was not only for our right aways; many of our technical facilities are located in industrial areas," Price said. "If an incident occurs that we have no control over, we still need access to our facilities 24 hours a day, seven days a week. Now our team can respond and take care of our network."Competitors such as AT&T and Verizon have a history of collaborating in response efforts, Bienfait said, but have not done much coordinating beforehand. A recently signed waiver by the Federal Communication Commission should allow the carriers to collaborate unencumbered, she said."I would like to start planning with the other carriers so when we get into an effected area, we leverage each other's capabilities to repair service and prioritize things," she said. "That's not something we've ever done before."Staff Writer Doug Beizer can be reached at dbeizer@postnewsweektech.com.