NMCI: Cleaning up the Navy's act

By Patience WaitWhen Rick Rosenburg was tapped by Electronic Data Systems Corp. to lead the massive Navy-Marine Corps Intranet project, he expected a few bureaucratic snafus and environmental challenges along the way. So he took in stride a seven-week delay last April while experts determined whether Indiana brown bats, an endangered species, would be threatened by construction at the Naval Surface Warfare Center division in Crane, Ind.The same held true when suspected Indian bones and artifacts held up work for six weeks while archeologists evaluated the findings ? which turned out to be red grout and pieces of old tile ? at Naval Air Station Fallon in Nevada. What Rosenburg didn't anticipate was the huge number of legacy applications residing on the Navy's desktop computer infrastructure. When the project began in October 2000, Navy officials said they had identified less than 1,000 legacy applications in their inventory of software during the year 2000 crisis. After 15 months on the job, however, EDS officials have found tens of thousands of commercial and custom applications, and no one knows for sure just how many more they will find. "I did not realize we could even buy that many applications or have that many working," Rosenburg said.The applications, such as the WordStar word processing program first released in the 1970s, are installed on hard drives and servers, exist in multiple versions and releases and are scattered across Navy bases throughout the United States and its territories.Reducing the number of legacy applications is one of the biggest challenges facing EDS in the NMCI program, the federal government's largest-ever information technology outsourcing project. Under the eight-year, $6.9 billion effort, EDS and its hundreds of subcontractors will connect 360,000 desktops to an integrated network, with standardized software, hardware and telecommunications and capable of withstanding the most determined attack on its information security. But another challenge, not under EDS' control, also looms: The Navy doesn't have a program executive with overall responsibility for NMCI. Seven different naval offices are responsible for elements of NMCI. Consequently, decision-making can be slow, and EDS often finds itself fighting the same battles over software and processes at each location where it installs new computers and networks for the program.Both Navy and EDS officials acknowledge that such battles make it difficult for EDS and its subcontractors to meet plans to have all the seats and systems in place by the end of 2003.The Navy "needs one person with overall responsibility for the program to implement and make decisions for the enterprise, not every different organization being able to make their own decisions and set thresholds for what we do and what we don't do," Rosenburg said.Much is at stake for EDS and the Navy in the NMCI program. For the Navy, NMCI offers the opportunity to fundamentally redesign and modernize its day-to-day operations by replacing an unplanned hodgepodge of standalone PCs and multiple local area networks that grew up over decades and do not communicate with each other.For Plano, Texas-based EDS, the project represents a large chunk of business, perhaps worth $600 million to $700 million this year, according to analyst Bill Loomis at Legg Mason Wood Walker Inc. in Baltimore.NMCI also provides EDS with a high-profile platform to demonstrate its capabilities to other military and civilian agencies contemplating similar seat management projects. Indeed, many agencies are said to be gauging their own plans based on the success or failure of EDS and the Navy."NMCI is providing a great testing ground for outsourcing's business case," said Ray Bjorklund, vice president of consulting at Federal Sources Inc., a McLean, Va., consulting and research company.XXXSPLITXXX-By Patience WaitThe Navy's fragmented IT universe came about because the service has traditionally operated with a relatively decentralized command structure. Historically, Navy ship captains have been given great latitude in decision-making because they could not communicate in anything approaching real time when faced with a crisis, said Capt. Jim Newman, director of the operations division for the Chief of Naval Operations staff. "In the Army, if it's not in the rule book, don't do it," Newman said. "In the Navy, if it's not in the rules, go ahead and try it."The service's organizational structure did not suit emerging information technology, he said. Base commanders usually just tapped those who seemed interested in computers, telling them to set up a PC here, a network there. Individual users made their own decisions about which software to buy, and the programs were inexpensive enough that each base could cover the costs without requiring approval higher up the ladder.As a result, the Navy accumulated legacy applications at a rate that Rosenburg calls "truly staggering." Ron Turner, the Navy's deputy chief information officer for plans, policy, performance, infrastructure, systems and technology, said the Navy is using the need to move processes to NMCI as a "forcing function" to reduce duplication of legacy applications now in use."The situation gets better with time," Turner said, "but right now this keeps people awake at night."EDS' goal is to reduce the number of legacy applications to between 12,000 and 18,000 ? still a hefty number considering that 1,000 applications would be a lot for a Fortune 500 company, Rosenburg said. The Navy is responsible for deciding which applications will be transferred to the NMCI environment. Capt. Chris Christopher, the Navy's deputy program executive officer for IT, who also serves as director of NMCI services, said it is up to base commanders to decide what applications they need to fulfill their mission requirements."We're not telling you what you need to do your job, we're telling you that you don't need 11 label-making software applications. Pick one," Christopher said. "If you come back with seven of them, it'll go up the ladder to pick one."Once the initial screening is done, the legacy applications to be carried over are sent to the NMCI certification lab to make sure they meet certain standards. There are four certification levels: the software is "good to go;" it needs a little work to fit the new environment; it needs a lot of work; or it's rejected. A legacy application can be rejected outright for one of two reasons: either it won't work in a Microsoft Windows 2000 environment, or it doesn't meet Defense Department software application security requirements, Christopher said."The legacy application rationalization and certification process remains one of the biggest short-term hurdles" to the NMCI schedule, Turner said. Not surprisingly, many individuals at each base are practically and emotionally invested in their current software, no matter how old, cumbersome or limited. Still, progress is being made."We have some commanding officers that are taking true leadership positions and saying no [to continuing some legacy applications], and we have others for whom it's a smorgasbord," Rosenburg said.XXXSPLITXXX-By Patience WaitThe difficulty in reducing legacy applications reinforces the need for a single Navy official to be in charge of implementing NMCI.The seven naval offices overseeing NMCI are wide ranging in their responsibilities. One takes the lead in Navy IT acquisition, another handles policy, standards and architecture for the program, and a third handles those elements for the Navy as a whole. A fourth runs NMCI's operational component and sea-shore rotation staff, and a fifth is in charge of financial aspects of the program. The sixth is the acquisition group for the Navy for command, control, communications, computers and intelligence, while the seventh handles those responsibilities for the Marine Corps.The base commanders also have influence on NMCI seats, what legacy software is carried over, the level of clearance assigned to each position and so on. "The situation right now is that everybody believes they have a vote on a decision," Rosenburg said.Recognizing the problems that can arise under the Navy's decentralized management, Navy Secretary Gordon England has said NMCI would make more progress if one person was designated as the Navy's team leader."I do not believe you can take a program [such as] NMCI and meet all your objectives and all your ... schedules without one single person being in a decision-responsible position, day after day on the program," England said at an industry gathering. "So we're struggling with NMCI, and I think it's a systemic-type problem we have."The 2002 Defense Authorization Act, in fact, requires England to name an NMCI manager whose sole responsibility is to oversee and direct the program. A Navy official said the secretary is committed to creating a leadership position for NMCI and finding the right person to take on the job, though there is no timetable for the decision. NMCI was begun within the service's existing organizational model, but that model is "evolving" as experience is gained, the official said.XXXSPLITXXX-By Patience WaitThe first deployment of NMCI seats out of the Navy's initial order of 60,000 took place Sept. 7 at Naval Air Facility Washington at Andrews Air Force Base. The Sept. 11 attacks did not seriously affect plans; in fact, the program served as the contract vehicle for EDS to build a temporary Navy communications center and budget office for 700 people displaced by the Pentagon attack.The rollout of seats has continued just about every week since. Rosenburg said that as of the middle of January, about 1,000 users have been moved over to the NMCI environment, including 650 at Naval Air Facility Washington, more than 200 at Naval Air Station Lemoore, Lemoore, Calif., and a few at the newest location, the Patuxent River Naval Air Station in Maryland.EDS is performing extensive testing and evaluation of NMCI at the three bases, to be completed in March. At the request of John Stenbit, the Defense Department CIO, the Navy has also hired an independent review agent to report to his office.The tests represent a crucial milestone for NMCI. Based on the test results, the Defense Department will decide when to release a second order for 100,000 more seats.If personal experience counts toward the evaluation, Lt. j.g. Roger Morris has become a big fan of NMCI. Morris is stationed at Naval Station Norfolk, Va., home to one of two network operations centers for the program. (The other is in San Diego.) He is the officer in charge and the day-to-day manager of the Navy's liaison with the Norfolk operation center."One of the initial fears that I heard about NMCI was that we're buying a service, and we're going to lose control," Morris said, admitting to some misgivings of his own when first assigned to the center.But after about two months on the job, he has been won over: "The word that I would use for NMCI is amazing."Morris said he has been impressed by the contractor's speed in tracking and fixing outages and disruptions and in upgrading systems already in place. EDS found and stopped a virus attack within hours before anyone could notice there was a problem, he said.Eventually, Morris' experience will be shared by hundreds of thousands of Navy personnel. By the end of 2003, NMCI is slated to be in place at 300 locations in the United States, as well as in Puerto Rico, Okinawa, Japan, and Keflavik, Iceland.In the meantime, EDS has assumed responsibility for some 56,000 "as-is" desktops at about 50 sites across the country, Rosenburg said."You know the EDS commercial about building the airplane in flight? That's exactly what we're doing," Christopher said.XXXSPLITXXX-By Patience WaitNMCI's ambitious scope and price tag make it one of the most closely watched government programs. Congress wants to ensure taxpayer money is wisely spent, and other federal agencies are poised to begin similar modernization efforts."Within the Department of Defense, NMCI is very different from what the other services and [the Defense Information Systems Agency] are doing with their infrastructures," said the Navy's Turner. "They want to be assured what we are doing will not harm joint interoperability. So, we've put a lot of interoperability tests into the contract to help erase these fears."Other armed forces branches have been keeping an eye on NMCI, while federal agencies and even representatives from foreign allies have been briefed on the program. Rosenburg said he spent about three hours discussing NMCI with representatives of Richard Clarke, the president's special adviser on cybersecurity, who has solicited ideas from industry for GovNet, a proposal for a closed, secure federal network.Rosenburg is keeping a close eye on NMCI in another way: to see whether it lives up to the promise of reshaping how the Navy and Marine Corps do business. Rosenburg said he is confident NMCI will improve the Navy's infrastructure in terms of speed and create a single, rational system where one did not exist before. But it will be up to the Navy to re-engineer its business or operating processes to take advantage of NMCI's potential."What we're essentially doing is turning buggy paths into superhighways. And if we continue to run horse and buggy down those superhighways, we have improved nothing except maybe to take a few bumps out of the road," Rosenburg said.XXXSPLITXXX-By Patience Wait: Navy briefs industry on NMCI: Request for information released: RFP released: Revised solicitation released: Congress decides to withhold money for at least two months after the Navy justifies the project to the Hill: Proposals submitted by EDS, CSC, IBM and General Dynamics: NMCI report to Congress: Questions from Congress postpones award until Sept. 1: Award delayed again for more questions: Award postponed again: EDS wins contract: EDS takes responsibility for 28,250 seats: An additional 13,985 seats added to the contract, giving EDS responsibility for 42,235 at 26 Navy facilities: First network center in Norfolk opens; Sen. John Warner, R-Va., questions commercial testing of NMCI: House Armed Services Committee proposes Marines not be part of NMCI. Proposal later dropped: Second network center in San Diego opens: Navy and Defense Department settle dispute over how to test NMCI: First sailor logs on: Terrorists attack World Trade Center and Pentagon: EDS NMCI team called in to aid recovery efforts: More than 700 Navy personnel affected by terrorist attack are back online: Contract modification lowers fiscal 2002 payment to EDS to $600 million from $728 million; Congress requests more monitoring: 310 of 3,100 NMCI contract employees laid off by EDS because of slow rollout of the system: Naval Reserve Air Facility-Washington with 400 seats becomes first facility to exclusively use NMCI: Rollout begins for 3,500 seats at the Naval Air Station in Lemoore, Calif., and for 1,000 seats at the Patuxent River Naval Air Station in Maryland: Phase 3 testing and evaluation begins: Navy begins search for NMCI leader: Testing expected to be completed, triggering order for 100,000 additional seats First Marine Corps facilities expected to join: Base period of contract ends. Three option years begin: Contract expiresXXXSPLITXXX-By Patience WaitTwo locations serve as nerve centers for the Navy-Marine Corps Intranet program: the Global Network Operations Center at Naval Station Norfolk, Va., and its twin at Naval Air Station North Island in San Diego. These facilities are where Electronic Data Systems employees and subcontractors work alongside Navy personnel to monitor, maintain, repair and protect the network that comprises NMCI. Norfolk is the primary operations center; the San Diego facility also monitors the systems and is there for backup in case anything happens, no matter how major or minor.At each facility there is a room ? physically the heart of the center ? where technicians monitor the vital signs of the systems at work. Overhead screens use traffic-light images to let everyone know the status of services by location, while individual monitors track each component in more detail.Server farms and major backup systems are based at the operations centers. At the Norfolk facility, for instance, there stands a large box in the server area that resembles a room within a room. A single, small window shows rows upon rows of tapes; it looks like a gigantic jukebox filled with old-fashioned eight-track tapes. It is a StorageTek data backup system ? 240 terabytes worth ? which serves just the Norfolk center. By comparison, Verity Inc. of Sunnyvale, Calif., an Internet portal infrastructure software company, estimates the size of the World Wide Web is about 30 terabytes, or 1.5 billion pages.The help desk is one aspect of the operations centers of which NMCI users are most likely to be aware. The two centers have some 50 to 60 help-desk agents with people on duty around the clock. All the agents are provided by subcontractors, according to Annette Rogers, an EDS employee and the NMCI enterprise help-desk manager.The staff will expand as NMCI seats are established throughout the country, Rogers said. By the time NMCI reaches 300,000 seats, the two sites will have 500 to 600 help-desk agents, with call volumes projected to reach up to 20,000 per day, she said.XXXSPLITXXX-By Nick WakemanThe Army and other military and civilian agencies are unlikely to install a clone of the Navy-Marine Corps Intranet, but that doesn't mean the massive undertaking won't have a major impact on IT outsourcing contracts going forward."All of us are following it," said Kevin Carroll, program executive officer of the Army's Enterprise Information Systems.Carroll's group, for example, has been holding meetings with Navy IT officials such as Chief Information Officer Joseph Cipriano."We've been talking with their people to make sure we understand what is going on," Carroll said.The Army is evaluating its approach to managing its IT infrastructure through the Army Enterprise Infostructure Transformation project, known as AEIT. A request for information went out in late 2001, and Carroll's group is evaluating the 65 responses. A statement of the Army's objectives should be out in May, he said."Our instinct right now is not to outsource the whole deal," Carroll said, "though there are people in the Army who are not opposed to that."Fear of being tied to one prime contract and losing the benefits of competition is a factor that makes the Army hesitant to follow the NMCI model, he said.Another deterrent to outsourcing all of the IT infrastructure is the Army's belief that information technology is a core business function for the service, Carroll said.As Carroll tracks NMCI, he is following issues such as how the Navy and the EDS team are controlling the configuration of systems across the Navy, how network and applications management is structured, how well NMCI scales up and communications issues."We want to take advantage of their successes and not repeat their mistakes," he said.The Army wants to accomplish several goals through its AEIT program, Carroll said. First, the Army would like to create a common infrastructure, "so we don't have the haves and have nots," he said. "We know we need some level of enterprise IT infrastructure to guarantee support."Another goal is improving service and the value that network users receive. A third goal is saving money.Controlling applications is another Army priority."We'd like to get into the application side and create enterprise software solutions," Carroll said.The Army has 925 logistics systems and 330 personnel systems, Carroll said, citing areas that would benefit from an enterprisewide approach.Carroll's group has three teams ? requirements, architecture and acquisition strategy ? that are reviewing the comments from RFI and collecting other information on the Army's IT infrastructure needs."By May, we should have something to come out for discussion, maybe a draft RFP," he said.The Army likely will outsource portions of its IT infrastructure, although there is a chance the Army will not come out with any new contracts, he said."I'm getting all these vendors calling me and wanting to form teams," Carroll said. "They are all very excited, but we don't even have a contract yet."