Telecom infrastructure was no match for Katrina

A recent Federal Communications Commission meeting with executives from Verizon Wireless of Bedminster, N.J., and BellSouth Corp. of Atlanta related to Hurricane Katrina outages revealed that physical infrastructure remains the Achilles' heel of networks in disaster situations.

During the hurricane, getting enough power was a major issue for the Gulf Coast telecom providers, as was keeping the basic infrastructure running and providing physical security for workers and equipment.

Equipment failure during natural disasters is nothing new, but the extreme damage that Katrina inflicted offered insights into increasing network reliability, executives from Verizon Wireless and BellSouth told the FCC's Technological Advisory Council. The council provides technical advice to the FCC for issues and questions faced by the agency.

Katrina "was probably the most severely impacted situation that we've experienced," said Anthony Melone, vice president of network operations support for Verizon Wireless. "There were a lot of unique learning experiences."

Officials from both firms explained why their telephone service failed in the days following the hurricane's arrival.

Verizon Wireless' cellular phone coverage for Alabama, Louisiana and Mississippi had dipped to less than 50 percent of its full coverage, according to Melone. About 6 percent of BellSouth's customer base?about 1.2 million users?lost landline telephone usage.

For Melone, loss of physical infrastructure was the biggest culprit of the loss in coverage. "[Physical-layer] transport is the biggest factor right now," Melone added.

Verizon Wireless serves the Gulf Coast with two major switching centers in Baton Rouge, La., and Covington, La., north of New Orleans. These centers are the link between cell phone antennas scattered throughout the region and the rest of the global telephone network.

While the stations themselves remained operational, the Covington facility lost connectivity with the cell phone towers due to two breaks in a connecting fiber-optic ring run by BellSouth.

Normally, a fiber-optic link provides redundancy: if one link is cut, information can still travel along the other route. In this case, however, Katrina had knocked out both links?an extremely rare occurrence, according to Melone.

The connecting ring, which wound around New Orleans connecting the cell phone towers, was taken down by physical failures. In one case, the cable was destroyed when a bridge the cable ran over was wiped out.

"It is just not possible to have alternatives in every single situation. You just can't plan for it," Melone said. "No matter what we do, the physical-layer transport is still a point of vulnerability."

In addition to the breaches in the main line, at least 20 cell towers were downed due to either loss in power or flooding. The company installs backup generators at each cell site, should zoning allow, though many older sites the company inherited through acquisitions remain without generators.

Refueling the remote generators also proved problematic. Most of the generators had enough fuel to last 48 to 72 hours, with multiple suppliers contracted to refuel during emergencies.

During Katrina, however, the company had a number of portable generators stolen. A few fuel suppliers were hijacked and, in one case, the Mississippi State Police redirected a fuel tanker to a nearby hospital.

Getting enough power was a major headache for BellSouth as well, according to Gary Ludgood, vice president for integrated network planning and implementation for the company. Although BellSouth has extensive disaster planning experience, Katrina pushed the limits of the company's ability to operate.

"Fuel is a big deal. Refueling was extremely problematic, especially considering the security issues," Ludgood said.

On Sept. 1, 112 BellSouth central offices were running on emergency generators, an additional 17 were completely out of service, and another 32 had no connectivity to the backbone network. And while the main switching stations had fuel, many of the remote sites could not be refueled.

BellSouth also had security issues. In the days after the flooding, individuals in the company's Emergency Operations Center, located near the New Orleans Superdome, were evacuated. Company officials worried about the center being targeted by unruly individuals.

"We chose to evacuate our employees before anything happened," Ludgood said. The company hired armed escorts to accompany the workers out of the city.

Beyond the power issue, BellSouth also dealt with the limitations of physical infrastructure. "What seems to work well to mitigate wind damage doesn't work well for flood[ing]," Ludgood said, noting that cables hoisted onto above-ground towers are susceptible to wind, while underground cables can be damaged by floods.

Joab Jackson is an associate writer for Washington Technology's sister publication, Government Computer News.