Overview of GAO Protests Analytics for 2017 and the Historical RAND Bid Protest Study

By: Christine V. Williams on 01/16/2018

Overview of GAO Protests Analytics for 2017 and the Historical RAND Bid Protest Study

By: Christine V. Williams

The RAND Bid Protest Study Performed pursuant to the National Defense Authorization Act (NDAA) of 2017 came out recently and is approximately 125 pages.  Within that report it had some interesting and unexpected results.  To put it into context, the GAO report on bid protests for 2017 will be quickly overviewed.  To that end, this update will summarize GAO protests for Fiscal Year 2017 and the RAND Bid Protest Report to Congress.  For the RAND Report, a summary of the observations and recommendations, with some emphasis on the small business arena, will be provided.  Notably, when taking a historical analysis approach, the time trend clearly shows that protest activity was much higher in the late 1980s and early 1990s than it is today. Interestingly, both the number of contracts and contract spending declined from FY 2008 to FY 2016. This is counter to the trend for Department of Defense specific bid protests, as demonstrated below.  Many of the RAND analytical charts are reproduced for ease of summation.  Here is the link to the RAND Report: RAND Report to Congress

The GAO Report-Protests in 2017-Summary

GAO Received 2596 Cases

  • 2433 Protests
  • 77 Cost Claims
  • 86 Requests for Reconsideration

GAO Closed 2672 Cases

  • 2471 Protests
  • 107 Cost Claims
  • 94 Requests for Reconsideration

Of the 2,672 cases closed, 256 were attributable to GAO’s bid protest jurisdiction over task orders.

Most Prevalent Grounds for Sustaining Protests-17 Percent Sustained

  • Unreasonable Technical Evaluation
  • Unreasonable Past Performance Evaluation
  • Unreasonable Cost or Price Evaluation
  • Inadequate Documentation of the Record
  • Flawed Selection Decision

Of note, agencies made agency correction before a significant number of protests were resolved. Agencies need not, and do not, report any of the myriad reasons they decide to take voluntary corrective action.

Bid Protest Statistics for Fiscal Years 2013-2017 

FY2017 FY2016 FY2015 FY2014 FY 2013
Cases Filed1 2596

(down 7%2)

2789

(up 6%)

2639

(up 3%)

2561

(up 5%)

2429

(down 2%)

Cases Closed3 2672 2734 2647  

2458

2538
Merit (Sustain + Deny) Decisions 581 616 587 556 509
Number of Sustains 99 139 68 72 87
Sustain Rate 17% 23% 12% 13% 17%
Effectiveness Rate4 47% 46% 45% 43% 43%
ADR5 (cases used) 81 69 103 96 145
ADR Success Rate6 90% 84% 70% 83% 86%
Hearings7 1.70% (17 cases) 2.51% (27 cases) 3.10% (31 cases) 4.70% (42 cases) 3.36% (31 cases)

 

The Rand Report Ordered by NDAA 2017-Summary of 125 Pages

The RAND National Defense Research Institute (RAND) a federally funded research and development study was tasked by Congress in NDAA 2017 to study bid protests and used 14 criteria for doing so.  RAND reported that 10 of the 14 criteria were viable for study and produced a report 125 pages long.  RAND studied both the GAO and the Court of Federal Claims (COFC) and came to some interesting observations and recommendations, which are summarized below.  The link to the study is found at the bottom of this post.

Notable Point for Small Businesses

Overall protest activity is increasing at both the COFC and the GAO and that small-business protests represent the majority of protest at both venues. (Page xvi).   These protests, overall, were less successful than other protests.

GAO-Specific Observations

  • The stability of the bid protest effectiveness rate over time—despite the increase in protest numbers—suggests that firms are not likely to protest without merit.
  • Small-business protests are less likely to be effective and more likely to be dismissed for legal insufficiency.
  • Protest filing peaks at the end of the fiscal year.
  • Task-order protests have a slightly higher effectiveness rate than other types of protests.
  • There are measurable differences between the services and defense agencies, but overall the DoD (services and agencies) has a slightly lower effectiveness rate than non-DoD agencies.
  • The largest DoD contractors have slightly higher sustained and effectiveness rates, but these differences are diminishing with time.
  • Cases in which legal counsel is required (i.e., a protective order was issued by GAO) have higher effectiveness and sustained rates.
  • DoD uses stay overrides infrequently.

The number of protesters and protest actions tends to grow with a contract’s value.

Court of Federal Claims Observations

  • The sustained rate at COFC is declining with time as the number of cases increases. These trends suggest that firms may be more willing to file protests with COFC.
  • There are no differences in sustained rates between DoD components and agencies or between small and larger businesses.
  • The appeals rate is declining over time.

Data and discussions suggest that the number of COFC cases that previously appeared at GAO may be increasing, but this potential trend needs further research.

 Common Both to GAO and COFC

  • While the statistical modeling indicates differences between types of cases (categorization of cases), it is not possible to predict the outcome of any case based on its general characteristics. Each case is different and its details affect the outcome.
  • The overall level (numbers) of bid protest activity (DoD and non-DoD) has been increasing at both GAO and COFC since 2008.
  • Bid protests by plaintiffs that identify themselves as small-business represent the majority of protests at both venues.
  • At both venues, in a nontrivial number of cases (approximately 4–8 percent), the contract value is less than $0.1 million (then-year dollars, as reported by the protester).
  • There are differences between the services and DoD agencies in terms of the number of cases filed. Specifically, the Army has the highest number of cases, year-on-year, at both venues. This is partly explained by its share of contract expenditures.

Trends differ between GAO and COFC, suggesting that any changes to the protest system should be tailored to the venue. For example, COFC’s sustained rate is declining whereas at GAO it is holding steady (and potentially increasing).

RAND Report Recommendations

  • Enhance the quality of post-award debriefings. The Army and Air Force have initiatives to improve the quality of the debriefings, which might serve as models. Section 818 of the FY 2018 NDAA has provisions for improving debriefings as well.
  • Be careful in considering any potential reduction of the GAO decision timeline. While 70 percent of cases at GAO are resolved in less than 60 days, it may be challenging to shorten the GAO decision timeline for all cases given that (1) protests are more frequently filed at the end of the fiscal year and (2) complex cases that go to decision usually take 90-100 days.
  • Be careful in considering any restrictions on task-order bid protests at GAO. Task-order protests have a slightly higher effectiveness rate than the rest of the protest population. This higher rate suggests that there may be more challenges with these awards and that task-order protests fill an important role in improving the fairness of DoD procurements.
  • Consider implementing an expedited process for adjudicating bid protests for procurement contracts with values under $0.1 million. One possible option is a process analogous to how traffic tickets are adjudicated in traffic court or how cases are adjudicated in small-claims court. A different approach would likely be needed for each venue. For example, COFC could “rule from the bench” on such smaller-value protests and not be required to generate written decisions. (This would limit the protester’s ability to appeal, however.) Another option is to require alternative dispute resolution for such small-value protests at GAO. Some discussion with each venue would be necessary to develop the most appropriate approach. Another but perhaps less desirable approach from a fairness perspective would be to restrict such low-value procurement protests to the agency level. Our recommendation is to come up with a quick way to resolve these cases commensurate with their value while preserving the right to an independent protest.
  • Consider approaches to reduce and improve protests from small businesses, such as improving debriefings, requiring protests to be filed by legal counsel, or providing legal assistance in filing.
  • Consider collecting additional data and making other changes to bid protest records to facilitate future research and decision making. Some examples include tracking cases that appear at COFC with a prior history at GAO, recording companies’ DUNS numbers, tracking corrective action at COFC, collecting and summarizing the reasons for corrective action, and generating annual reports of agency-level protest activity.

These recommendations are intended to inform future changes to the bid protest system. There is likely value in using the same or similar approaches across other departments and agencies of the U.S. government. In implementing these recommendations, there should be some consideration of costs and benefits, as some changes will require additional time or resources to implement.

Interesting Statistics-Some Demonstrated Through RAND’s Various Charts

Bid Protest Actions at GAO, FYs 2008–2016

There has been a significant upward trend in protest activity at GAO between FY 2008 and FY 2016, a period in which activity for both DoD and non-DoD agencies has approximately doubled. Protest actions associated with DoD agencies accounted for roughly 60 percent of the total protest actions over this period. Even excluding task-order protests (which were added to GAO’s jurisdiction in FY 2008), the upward trend is still statistically significant.

To put this trend in context, one needs to go back further in time. To do this comparison, RAND supplemented GAO’s data by adding prior years based on a 2009 GAO report on DoD protest activity. Figure 4.2 shows the number of procurements protested (a sum of primary B numbers) between FY 1989 and FY 2016. The blue bars show data from the 2009 GAO report; the red bars are based on the data set that GAO provided to RAND. There is only one year of overlap (FY 2008) between the two data sets. The values do not match exactly but are within 4 percent. Nonetheless, the time trend clearly shows that protest activity was much higher in the late 1980s and early 1990s than it is today.

 

The previous time trends are indifferent to changes in DoD spending and contracting. To better understand whether spending and contracting changes are driving the trend, RAND obtained data on DoD contracts (numbers of contracts and contract dollars) from FPDS-NG. Both the number of contracts and contract spending declined from FY 2008 to FY 2016. This is counter to the trend for DoD bid protests.

Table 4.1. Proportion of Protests, Spending, and Contracts by DoD Agency at GAO,  FYs 2008–2016

Agency % of DoD Protest Actions % of DoD Protested Contracts % DoD Contract $ % DoD Contracts
Army 43% 41% 34% 25%
Air Force 18% 18% 19% 9%
Navy 19% 19% 28% 19%
DLA 12% 16% 11% 44%
Other DoD 8% 7% 9% 3%

SOURCE: RAND analysis of GAO and FPDS-NG data.  NOTE: Columns may not sum to 100% due to rounding.

Table 4.2. DoD Bid Protest Characteristics at GAO, FYs 2008–2016

Characteristic All Actions Weighted by Procurement
Observations 11,459 7,368
Average number of protesters 1 1.2
From small businesses* 53.1% 58.1%
Value under $0.1 million* 7.9% 10.5%
Task order protests 10.6% 9.3%
Stay override issued 1.4% 1.3%
Alternative dispute resolution used 5.0% 3.6%
Protective order issued 48.9% 39.2%
Pre-award protest 26.9% 29.1%

SOURCE: RAND analysis based on GAO data.  * Values were self-reported by the protester.

Table 4.3. DoD Bid Protest Outcome Measures at GAO, FYs 2008–2016

Outcome Measures All Actions Weighted by Procurement
Observations 11,459 7,368
Sustained rate 2.6% 1.4%
Merit casesa 21.2% NAb
Sustained rate for merit cases 12.2% NAb
Corrective action rate 38.4% 38.6%
Effectiveness rate 41.0% 40.0%
Sustained rate (excluding reconsideration) 2.7% 1.5%
Effectiveness rate (excluding reconsideration) 42.4% 40.8%

SOURCE: RAND analysis of GAO data.

  • A merit case is a bid protest action that goes to decision (GAO either sustains or denies the protest and a written decision is issued.) We report the sustained rate for merit cases to be consistent with GAO’s approach.
  • Weighting is inappropriate for a subset of the data.

Conclusion

The RAND report rebutted some popular misconceptions in the bid protest arena.  It also gave valuable insight into statistics, resolutions, and recommendations to make the procurement process more effective.