New or old, computer systems don’t always behave

To upgrade or not to upgrade? That is the question facing many state governments and their computer systems.

It’s a dilemma that hasn’t been made any easier by two recent examples on opposite sides of the country.

After a week when a data storage failure in a relatively new statewide system created havoc for Virginia state agencies, the final three agencies – including the 74 branches of the Department of Motor Vehicles – are expected to resume services today.

"The crash – still baffling to state officials – exposes the vulnerability of modern, massively complex interconnected computer networks,” writes the Washington Post today.

State Chief Information Officer Sam Nixon told the newspaper that the problem began Aug. 25 when a pair of 3-year-old memory cards crashed.

One was supposed to back up the other, and both were considered highly reliable.

When both failed, 485 of the state’s 4,800 data servers went down.

“The thing that is never supposed to happen, happened,” Nixon said.

The weeklong failure is just the latest problem for the problem-plagued $2.4 billion contract that Richmond awarded the Northrop Grumman Corp. in 2003 to build and maintain a new system after a legislative review found the existing system out of date and increasingly expensive to maintain.

But missed deadlines, poor service and installation issues have hampered implementation of the new system.

According to the Post, a state audit in October 2009 “found that the computer system had caused problems at almost every state agency that uses computers.”

This past spring the state renegotiated the 10-year contract with Northrop, extending it for an additional three years and agreeing to pay another $100 million, but tacking on new penalties for future failures.

This summer, when California Gov. Arnold Schwarzenegger wanted to temporarily reduce state workers’ salaries to the federal minimum level during California’s budget stalemate, he was expecting resistance but not from the computer system.

But state Controller John Chiang warned that “the state’s Vietnam-era payroll system couldn’t process the temporary pay cut,” writes Steve Towns in Governing magazine.

The California Employment Development Department experienced the problem firsthand when unemployment caseloads skyrocketed during the current recession.

Congress’s seven extensions of unemployment benefits was a nightmare for the EDD, Towns writes.

“Every extension required changes to several hundred interconnected computer programs in the EDD’s eligibility system,” he said. “Those programs are written in common business oriented language, an ancient programming language, and modifications must be hand-performed by increasingly rare – and expensive – Cobol experts.”

“It typically takes two to three weeks to implement changes, depending on how complex the federal legislation is," said Dale Jablonsky, who until August was CIO of the agency. "Sometimes the legislation is so complex it takes five to six weeks to implement."

The EDD system, which handles 7 million transactions daily, dates back to the 1970s, according to the article.

“Besides its spaghetti-like web of Cobol programs, adding or changing information in the eligibility database is a nightmare because pieces of data must be rearranged manually,” Towns writes.

“There were close to 110,000 claimants who were entitled to benefits, but because of our problems in getting those changes in, they were delayed several weeks,” Jablonsky told the magazine.

“Imagine a spreadsheet with millions of columns,” he said, “but none of the columns shift automatically to accommodate a new piece of information. Each bit of existing data must be moved by hand to make room.”

The EDD is updating its eligibility system with more modern technology thanks to an infusion of federal funds.

Once it's completed, the system will react more quickly to legislative requirements, improve defenses against identity theft and make it simpler to serve citizens online, officials say.

But modernizing the system won’t be easy, cautioned Jablonsky, who likens the task to remodeling an old house.

“It’s an undertaking filled with expensive and unanticipated complications," he said. "The EDD’s IT staff is unraveling multiple layers of system interconnections and 40 years of programming shortcuts.”

Northrop Grumman Corp., of Los Angeles, ranks No. 2 on Washington Technology’s 2010 Top 100 list of the largest federal government contractors.