Auto‐Quarantined Tests

How it works

We're making use of the Quarantine gem. When a test fails, Quarantine will initiate Rspec::Retry to give it another X chances to pass.

Here are the possible outcomes of tests:

Test Result	Outcome	Fails Suite?
passes the first time	consecutive passes count is increased; if it is quarantined and the consecutive passes number reaches the specified limit (see Settings below), then it is removed from the quarantine	NO
fails the first time but passes in one of the retries	quarantined as a flaky test	NO
fails the first time and all retries	considered a failing test and not quarantined	YES

An example:

Settings

All settings are changed in rails_helper.rb.

To set the number of retries, replace the value of X below:

config.around(:each) do |example|
  example.run_with_retry(retry: X)
end

To set the number of consecutive passes to remove a test from quarantine, replace the value of X below:

config.quarantine_release_at_consecutive_passes = X

To change when to record tests, change the logic here:

config.quarantine_record_tests = ENV["CI"]

Setup

Google Cloud

Enable APIs for a New Project

Create a new project. For that project, enable the following APIs:

1. Google Drive API
2. Google Sheets API

Add a Service Account

Go to APIs & Services --> Credentials, and click +Create Credentials. Choose Service Account. Add an appropriate name and skip the optional settings.

Once created, click the new account's Keys tab. Click Add Key --> Create new key and choose the JSON key type. The key will automatically be downloaded to your computer with the name service-account.json.

Copy the service account's email address.

Google Drive

Create a new Google sheet that will serve as the "database" for quarantined tests. Give the sheet an appropriate name.

Share edit permissions on the spreadsheet with the service account by pasting the email address into the share modal.

Set the name of the worksheet (tab) to test_statuses.

Copy the following titles into the first row of the worksheet (see above).

A	B	C	D	E	F	G
id	full_description	updated_at	last_status	location	extra_attributes	consecutive_passes

Copy the id of the Google sheet (the id of a Google sheet is the alpha-numeric hash between spreadsheets/d/ and /edit in the url). This goes in rails_helper.rb in the project:

config.quarantine_database = {
  type: :google_sheets,
  authorization: {type: :service_account_key, file: Rails.root.join("config", "service-account.json")},
  spreadsheet: {
    type: :by_key,
    key: "KEY_GOES_HERE"},
}