Lessons from VA's ups and downs

EDITOR'S NOTE: This article originally appeared in Government Computer News.

As large government organizations go, the Veterans Affairs Department has been an example of what to do and what not to do for large enterprise technology deployments. Although it has endured its share of missteps and scandals, in recent years, VA has become a model for how federal agencies can streamline and modernize their network architectures.

Much of this success can be credited to VA’s CIO. Nearly two years into the job, Roger Baker has overseen the consolidation of the department’s IT infrastructure, begun moving applications to the cloud, and pressed ahead with efforts to more productively share medical data with the Defense Department.

The move to consolidate VA’s disorganized IT systems began and was largely complete before Baker took office. But the centralization has served as a platform to enable most of the department’s ongoing IT deployments. VA now has a single infrastructure operating one standard e-mail system, which provides standardized configurations for desktop and laptop computers.

“We’re in a largely good news situation,” said Baker, who is also VA's assistant secretary for information and technology.

Every VA facility has a CIO reporting directly to Baker. Autonomy at the facility level is encouraged for customer service and satisfaction. But the primary enterprise services — desktops, laptops, networks, servers and applications — are all centrally managed and supported.

“We’re trying to make certain that we’ve got the appropriate balance between centrally done things, where standardization makes a lot of sense, and locally done things, where they’re really tailored around customer satisfaction for the facility,” he said.

This foundation puts VA in a good place for projects such as encrypting information on non-medical laptops. Yet even with a fully consolidated IT organization, “pretty good isn’t enough,” Baker said. He said he would like to see a single, standard virtual desktop and, to the extent possible, virtualized servers for all VA applications.

Given its complexity, IT architecture is a challenge for VA. Federal IT systems have grown in a largely ad hoc fashion, Baker said, resulting in most agencies having pasted-together architectures, rather than centrally planned ones. This legacy is reflected in VA’s staff of 2,500 software developers. Although they are necessary for such a large organization, Baker said, keeping personnel numbers under control while retaining expertise is a key goal.

VA cloud layers

Instituting cloud-based services is another primary goal of VA’s architecture and standardization efforts. VA’s cloud has three levels, Baker said. The first consists of internally managed data centers, which pull information from the department’s medical facilities. Baker calls this cloud VistA, a virtualized version of the Veterans Health Information Systems and Technology Architecture used to manage the department’s medical records. Most VA facilities do not need to know or care what data center their VistA applications are running in, as long as they're available when needed, he said.

VA’s second cloud layer supports outsourcing applications to remote data centers through which applications are treated and purchased as a service.

The last level of cloud services, which Baker called a work in progress, includes popular cloud-hosted services such as Facebook, Google Docs or Yahoo Calendaring. Although VA launched a major cloud-based e-mail service in March using Microsoft Exchange operating from contractor-owned data centers, many other applications have not yet been deployed, primarily because the security of these systems has not been thoroughly cleared.

“We’ve got the first two [levels] nailed down,” Baker said. “The thing we’re dealing with right now is how do we do that third one in a secure way, with the vendors that are providing it so that our users get an authentic version of the cloud application but we get the level of authentication and information security controls that we need while our users use those services.”

Protecting veterans’ personal information remains VA’s major IT challenge. Baker said his job as CIO is to make sure that staff can access applications in a way that protects veterans’ personal and private information. “That is a huge challenge I’m dealing with right now, relative to real cloud offerings," he said. "Our clinicians want them. I want to provide them. We’re just working through how we do that in a way that is secure enough to protect veterans’ information."

Data center caveat

Besides improving its network architecture and its ability to provide services to its medical staff, VA has also been consolidating data centers to cut costs and improve efficiency. The department is about halfway through this process, with the goal of initially reducing its IT facilities to 15 large data centers and then following up with further reductions in a second phase.

However, there is a significant constraint to consolidation, Baker said. VA medical facilities are very large, and they operate life-critical systems that cannot be interrupted. For example, VA is considering moving phone system management out of its medical facilities and into centralized data centers. This is possible, but Baker said it makes the clinics more dependent on the reliability and speed of the network link between the data center and the facility.

By the same token, medical imaging systems and archival technologies require immense amounts of bandwidth because radiology and magnetic resonance images are very detailed. “You have to make a pretty sane choice between where you run that [imaging and storage capability] and the reliability of the facility you run it in," Baker said.

Some of these choices depend on how a data center is defined. Baker noted that VA will continue to run raised floor and air conditioned facilities in almost every VA hospital for the foreseeable future. But VA is also pulling more enterprise systems out of its medical facilities and into centrally managed, high-availability data centers, he added.

Open-source role

Although VA has made strides in creating a model electronic health records system, Baker said, the private sector is moving faster than government. “We have been strongly urged by Congress, and we absolutely agree, that we need to do a better job of incorporating commercial off-the-shelf packages into what we’re doing,” he said.

That’s the primary rationale for VA’s exploration of open-source technologies to modernize VistA. Baker said VistA’s open-source roots have allowed private-sector developers to integrate their applications into the EHR. One of the main reasons that VA has not integrated more commercial products into VistA, he said, is that VA does the integration internally, and “it takes us years."

Baker said he believes that an open-source approach allows the private sector to have an important say in the future of VA medical record systems, especially VistA’s role in a future joint EHR under development by the VA and DOD. “We see [open-source application development] as compatible with where the DOD is likely to go,” he said.

DOD and VA already exchange some medical data through existing systems, Baker said.

“When you talk about a joint common platform, the implication there is that if I have a patient’s record in one system, that entire patient’s record can be read by the other system with no loss of information," he said. "That is the goal that we’re looking at."