Evaluation mistakes sunk NSA's first try at $10B cloud award

There is a lot to digest in the 34-page Government Accountability Office decision backing up Microsoft’s protest of a $10 billion National Security Agency cloud computing contract to Amazon Web Services.

We reported in October that GAO had sustained Microsoft’s protest and recommended the NSA rethink the award to AWS. Now GAO has released a public version of the decision that provides greater detail into how NSA awarded the highly-classified WildAndStormy contract.

NSA is attempting to hire a single cloud provider for all aspects -- infrastructure as a service, platform as a service and software as a service. NSA wants to be able to perform its mission at any time from any location, according to the GAO decision.

The contract would have a five-year base period and a five-year option period.

In their proposals, bidders were required to respond to a series of representative task orders: one covering top secret cloud services, a second covering unclassified cloud services, and then a third for program management.

Phase one saw bidders give oral presentations to demonstrate successful deployment, central monitoring, and certain benchmarking using an NSA tool.

After passing phase one, bidders had to submit written proposals that would be evaluated on seven factors: technical, technical acceptability, management, management acceptability, past performance, facilities acceptability, and price.

All the non-price factors were more important than price. Technical and management when combined were more important than the other factors.

Of the four initial bidders, only Microsoft and AWS passed phase one and submitted written proposals in phase two.

GAO's breakdown of the evaluation of the written proposals offers some insights into how NSA viewed the two bids and where Microsoft focused its protest.

AWS’ price for the task orders was higher than Microsoft’s. AWS proposed $482.3 million, compared to Microsoft’s $422.5 million.

But NSA liked AWS’s technical proposal better, scoring it Outstanding to Microsoft’s Acceptable. Both companies were scored the same for all the other factors. The past performance for the two companies was rated Very Relevant/Substantial Confidence.

Microsoft's protest went after the technical and management evaluations as improper. The company also argued that the price evaluation was unreasonable.

There are several pages exploring a dedicated cloud, where only NSA would use the data center; and a multi-tenant cloud where multiple customers would be in the data center.

Microsoft argued that NSA never announced that it preferred dedicated versus multi-tenant cloud services. But GAO denied this part of Microsoft’s protest.

But where Microsoft did get traction was the challenge to how its technical proposal was evaluated. Microsoft was dinged for the process it uses to get new features approved for defense and intelligence agencies.

NSA claimed that Microsoft needed approval from the Defense Information Systems Agency as its “authorizing” agent under a different contract, unrelated to WildAndStormy. NSA identified that as a “significant weakness” that introduced “significant performance and schedule risk.”

One problem there: what NSA told Microsoft isn't true. GAO found that there is no contract with DISA that requires it to approve new features of Microsoft's Azure offering.

Upon being challenged on that conclusion, NSA said they “assumed” there was a contract. But NSA didn’t describe it as an assumption when they picked AWS over Microsoft; they described it as a fact.

GAO wrote:

Rather, the evaluators erroneously reported--in no uncertain terms--that the existence of such a contract between DISA and Microsoft required DISA to be the authorizing agent for all new service offerings, classified and unclassified, by Microsoft to DOD agencies, including NSA. As a result, the SSA, erroneously concluded that DISA was contractually required to be the “approving authority gateway for WILDANDSTORMY Top Secret and Unclassified services.”

GAO concluded that was prejudicial against Microsoft because the NSA saw this erroneous assumption as a key differentiator between the two bids.

NSA also said it found fault with Microsoft’s use of FedRAMP accreditation, but only added this factor after the proposal was filed. There was nothing in the record to indicate NSA considered this when evaluating proposal, GAO said.

Microsoft also challenged how the NSA evaluated the latency of its cloud services. NSA gave a better score to AWS’ latency, but AWS excluded network equipment delays and Microsoft included those delays in how it measured latency.

“Microsoft reported actual, realistic latency values…while AWS reported estimated, theoretical latency values,” GAO wrote.

NSA compared the latency data as if the same methodology were used, which made Microsoft look worse.

“Microsoft was prejudiced because the agency then unreasonably relied on this comparison when making its award decision,” GAO wrote.

GAO denied Microsoft’s challenge of how the management proposals were evaluated and dismissed as untimely the company's argument over how pricing was evaluated.

But the challenges around the technical evaluations were enough to trigger GAO’s recommendation that the NSA re-evaluate proposals.

That re-evaluation is apparently ongoing and given the classified nature of WildAndStormy, we may not know when a new award happens until a protest is filed again.