Evaluation mistakes sunk NSA's first try at $10B cloud award

Getty.com/Hiroshi Watanabe

The decision to back Microsoft's protest of a $10 billion National Security Agency cloud computing award to Amazon reveals a series of mistakes in the NSA's evaluation of proposals.

There is a lot to digest in the 34-page Government Accountability Office decision backing up Microsoft’s protest of a $10 billion National Security Agency cloud computing contract to Amazon Web Services.

We reported in October that GAO had sustained Microsoft’s protest and recommended the NSA rethink the award to AWS. Now GAO has released a public version of the decision that provides greater detail into how NSA awarded the highly-classified WildAndStormy contract.

NSA is attempting to hire a single cloud provider for all aspects -- infrastructure as a service, platform as a service and software as a service. NSA wants to be able to perform its mission at any time from any location, according to the GAO decision.

The contract would have a five-year base period and a five-year option period.

In their proposals, bidders were required to respond to a series of representative task orders: one covering top secret cloud services, a second covering unclassified cloud services, and then a third for program management.

Phase one saw bidders give oral presentations to demonstrate successful deployment, central monitoring, and certain benchmarking using an NSA tool.

After passing phase one, bidders had to submit written proposals that would be evaluated on seven factors: technical, technical acceptability, management, management acceptability, past performance, facilities acceptability, and price.

All the non-price factors were more important than price. Technical and management when combined were more important than the other factors.

Of the four initial bidders, only Microsoft and AWS passed phase one and submitted written proposals in phase two.

GAO's breakdown of the evaluation of the written proposals offers some insights into how NSA viewed the two bids and where Microsoft focused its protest.

AWS’ price for the task orders was higher than Microsoft’s. AWS proposed $482.3 million, compared to Microsoft’s $422.5 million.

But NSA liked AWS’s technical proposal better, scoring it Outstanding to Microsoft’s Acceptable. Both companies were scored the same for all the other factors. The past performance for the two companies was rated Very Relevant/Substantial Confidence.

Microsoft's protest went after the technical and management evaluations as improper. The company also argued that the price evaluation was unreasonable.

There are several pages exploring a dedicated cloud, where only NSA would use the data center; and a multi-tenant cloud where multiple customers would be in the data center.

Microsoft argued that NSA never announced that it preferred dedicated versus multi-tenant cloud services. But GAO denied this part of Microsoft’s protest.

But where Microsoft did get traction was the challenge to how its technical proposal was evaluated. Microsoft was dinged for the process it uses to get new features approved for defense and intelligence agencies.

NSA claimed that Microsoft needed approval from the Defense Information Systems Agency as its “authorizing” agent under a different contract, unrelated to WildAndStormy. NSA identified that as a “significant weakness” that introduced “significant performance and schedule risk.”

One problem there: what NSA told Microsoft isn't true. GAO found that there is no contract with DISA that requires it to approve new features of Microsoft's Azure offering.

Upon being challenged on that conclusion, NSA said they “assumed” there was a contract. But NSA didn’t describe it as an assumption when they picked AWS over Microsoft; they described it as a fact.

GAO wrote:

Rather, the evaluators erroneously reported--in no uncertain terms--that the existence of such a contract between DISA and Microsoft required DISA to be the authorizing agent for all new service offerings, classified and unclassified, by Microsoft to DOD agencies, including NSA. As a result, the SSA, erroneously concluded that DISA was contractually required to be the “approving authority gateway for WILDANDSTORMY Top Secret and Unclassified services.”

GAO concluded that was prejudicial against Microsoft because the NSA saw this erroneous assumption as a key differentiator between the two bids.

NSA also said it found fault with Microsoft’s use of FedRAMP accreditation, but only added this factor after the proposal was filed. There was nothing in the record to indicate NSA considered this when evaluating proposal, GAO said.

Microsoft also challenged how the NSA evaluated the latency of its cloud services. NSA gave a better score to AWS’ latency, but AWS excluded network equipment delays and Microsoft included those delays in how it measured latency.

“Microsoft reported actual, realistic latency values…while AWS reported estimated, theoretical latency values,” GAO wrote.

NSA compared the latency data as if the same methodology were used, which made Microsoft look worse.

“Microsoft was prejudiced because the agency then unreasonably relied on this comparison when making its award decision,” GAO wrote.

GAO denied Microsoft’s challenge of how the management proposals were evaluated and dismissed as untimely the company's argument over how pricing was evaluated.

But the challenges around the technical evaluations were enough to trigger GAO’s recommendation that the NSA re-evaluate proposals.

That re-evaluation is apparently ongoing and given the classified nature of WildAndStormy, we may not know when a new award happens until a protest is filed again.

X
This website uses cookies to enhance user experience and to analyze performance and traffic on our website. We also share information about your use of our site with our social media, advertising and analytics partners. Learn More / Do Not Sell My Personal Information
Accept Cookies
X
Cookie Preferences Cookie List

Do Not Sell My Personal Information

When you visit our website, we store cookies on your browser to collect information. The information collected might relate to you, your preferences or your device, and is mostly used to make the site work as you expect it to and to provide a more personalized web experience. However, you can choose not to allow certain types of cookies, which may impact your experience of the site and the services we are able to offer. Click on the different category headings to find out more and change our default settings according to your preference. You cannot opt-out of our First Party Strictly Necessary Cookies as they are deployed in order to ensure the proper functioning of our website (such as prompting the cookie banner and remembering your settings, to log into your account, to redirect you when you log out, etc.). For more information about the First and Third Party Cookies used please follow this link.

Allow All Cookies

Manage Consent Preferences

Strictly Necessary Cookies - Always Active

We do not allow you to opt-out of our certain cookies, as they are necessary to ensure the proper functioning of our website (such as prompting our cookie banner and remembering your privacy choices) and/or to monitor site performance. These cookies are not used in a way that constitutes a “sale” of your data under the CCPA. You can set your browser to block or alert you about these cookies, but some parts of the site will not work as intended if you do so. You can usually find these settings in the Options or Preferences menu of your browser. Visit www.allaboutcookies.org to learn more.

Sale of Personal Data, Targeting & Social Media Cookies

Under the California Consumer Privacy Act, you have the right to opt-out of the sale of your personal information to third parties. These cookies collect information for analytics and to personalize your experience with targeted ads. You may exercise your right to opt out of the sale of personal information by using this toggle switch. If you opt out we will not be able to offer you personalised ads and will not hand over your personal information to any third parties. Additionally, you may contact our legal department for further clarification about your rights as a California consumer by using this Exercise My Rights link

If you have enabled privacy controls on your browser (such as a plugin), we have to take that as a valid request to opt-out. Therefore we would not be able to track your activity through the web. This may affect our ability to personalize ads according to your preferences.

Targeting cookies may be set through our site by our advertising partners. They may be used by those companies to build a profile of your interests and show you relevant adverts on other sites. They do not store directly personal information, but are based on uniquely identifying your browser and internet device. If you do not allow these cookies, you will experience less targeted advertising.

Social media cookies are set by a range of social media services that we have added to the site to enable you to share our content with your friends and networks. They are capable of tracking your browser across other sites and building up a profile of your interests. This may impact the content and messages you see on other websites you visit. If you do not allow these cookies you may not be able to use or see these sharing tools.

If you want to opt out of all of our lead reports and lists, please submit a privacy request at our Do Not Sell page.

Save Settings
Cookie Preferences Cookie List

Cookie List

A cookie is a small piece of data (text file) that a website – when visited by a user – asks your browser to store on your device in order to remember information about you, such as your language preference or login information. Those cookies are set by us and called first-party cookies. We also use third-party cookies – which are cookies from a domain different than the domain of the website you are visiting – for our advertising and marketing efforts. More specifically, we use cookies and other tracking technologies for the following purposes:

Strictly Necessary Cookies

We do not allow you to opt-out of our certain cookies, as they are necessary to ensure the proper functioning of our website (such as prompting our cookie banner and remembering your privacy choices) and/or to monitor site performance. These cookies are not used in a way that constitutes a “sale” of your data under the CCPA. You can set your browser to block or alert you about these cookies, but some parts of the site will not work as intended if you do so. You can usually find these settings in the Options or Preferences menu of your browser. Visit www.allaboutcookies.org to learn more.

Functional Cookies

We do not allow you to opt-out of our certain cookies, as they are necessary to ensure the proper functioning of our website (such as prompting our cookie banner and remembering your privacy choices) and/or to monitor site performance. These cookies are not used in a way that constitutes a “sale” of your data under the CCPA. You can set your browser to block or alert you about these cookies, but some parts of the site will not work as intended if you do so. You can usually find these settings in the Options or Preferences menu of your browser. Visit www.allaboutcookies.org to learn more.

Performance Cookies

We do not allow you to opt-out of our certain cookies, as they are necessary to ensure the proper functioning of our website (such as prompting our cookie banner and remembering your privacy choices) and/or to monitor site performance. These cookies are not used in a way that constitutes a “sale” of your data under the CCPA. You can set your browser to block or alert you about these cookies, but some parts of the site will not work as intended if you do so. You can usually find these settings in the Options or Preferences menu of your browser. Visit www.allaboutcookies.org to learn more.

Sale of Personal Data

We also use cookies to personalize your experience on our websites, including by determining the most relevant content and advertisements to show you, and to monitor site traffic and performance, so that we may improve our websites and your experience. You may opt out of our use of such cookies (and the associated “sale” of your Personal Information) by using this toggle switch. You will still see some advertising, regardless of your selection. Because we do not track you across different devices, browsers and GEMG properties, your selection will take effect only on this browser, this device and this website.

Social Media Cookies

We also use cookies to personalize your experience on our websites, including by determining the most relevant content and advertisements to show you, and to monitor site traffic and performance, so that we may improve our websites and your experience. You may opt out of our use of such cookies (and the associated “sale” of your Personal Information) by using this toggle switch. You will still see some advertising, regardless of your selection. Because we do not track you across different devices, browsers and GEMG properties, your selection will take effect only on this browser, this device and this website.

Targeting Cookies

We also use cookies to personalize your experience on our websites, including by determining the most relevant content and advertisements to show you, and to monitor site traffic and performance, so that we may improve our websites and your experience. You may opt out of our use of such cookies (and the associated “sale” of your Personal Information) by using this toggle switch. You will still see some advertising, regardless of your selection. Because we do not track you across different devices, browsers and GEMG properties, your selection will take effect only on this browser, this device and this website.