Successful IT projects: Good news travels slowly
Steve Kelman finds a GAO report documenting successful IT acquisitions that apparently did not get much attention.
Good news travels slowly, so perhaps it is not surprising that I only recently found out about an October 2011 GAO report called Critical Factors Underlying Successful Major Acquisitions, which examines seven recent government IT systems acquisitions -- ranging in dollar value from $35 million to $2 billion -- that have met their schedule, cost, and performance targets. (I don't recall seeing anything even in FCW about this report when it came out, and a search of the fcw.com website with the key words GAO and the title of the report came up dry. FCW, tell me it isn't so!)
The projects ranges from a logistics support system fielded by the Defense Information Systems Agency to a Department of Homeland Security for high-volume Mexican and Canadian border crossings that uses license plate identification and other technologies to allow checking background information about visitors without slowing down border crossings too much. Agencies were asked to identify projects, and GAO vetted project performance. They then interviewed people involved in the program with open-ended questions about what factors in their view contributed the most to program success. GAO coded and tabulated replies, and the report presents information about the most-common success factors the interviews revealed.
The single most common success factor -- mentioned in all 7 of the programs -- was that "program officials were actively engaged with stakeholders." GAO noted that the stakeholders were internal (including top management) and external (such as oversight bodies and non-government customers). Internal stakeholders were involved in ongoing meetings with the contractors, assessing progress and issues, and in reviewing contractor deliverables. In all but two of the cases, end users and other stakeholders were involved in the development of requirements and in informal testing prior to formal acceptance testing.
The next most common factor, present in six of the seven programs, was "program staff had the necessary knowledge and skills" and "senior department and agency executives supported the programs."
In the successful projects, program managers were knowledgeable about procurement, contract management, organizational change, and/or earned value management. As for senior leadership, they were credited with interventions that would have been more difficult at the program manager level, such as reaching agreements with senior leaders in other departments about necessary cooperation or assuring end user involvement.
Perhaps surprisingly, sufficient funding was named as a success factor in only three of the cases.
It was also interesting that features of the technology approach used generally didn't make it on the list at all, with the exception of agile development, which some successful projects mentioned. Nobody mentioned commercial off-the-shelf products or cloud, or much else about the technology itself as success factors.
The message seems to be: it’s the management, stupid.
Researchers (including me) are often skeptical of research that tries to draw conclusions only from successful cases, because even if you see that all the successes did x, you don't know whether maybe the failures did x as well unless you compare successes and failures. In this report, however, I feel more confident in the results because past studies of failures have developed long lists of mistakes that IT systems that run into problems make. This report draws from that long list to identify mistakes avoided in the successful projects.
All in all, good job GAO.
And aside from the helpfulness of the lessons learned reported in this GAO study, this report should be getting wide distribution to give GAO credit for writing a report that tells us what we might learn from success instead of just dumping on failure. If the silence is deafening, it will hardly encourage more such reports from GAO.
NEXT STORY: CSC promotes exec to oversee customer solutions