Auto‐Quarantined Tests

How it works

We're making use of the Quarantine gem. When a test fails, Quarantine will initiate Rspec::Retry to give it another X chances to pass.

Here are the possible outcomes of tests:

Test Result	Outcome	Fails Suite?
passes the first time	consecutive passes count is increased; if it is quarantined and the consecutive passes number reaches the specified limit (see Settings below), then it is removed from the quarantine	NO
fails the first time but passes in one of the retries	quarantined as a flaky test	NO
fails the first time and all retries	considered a failing test and not quarantined	YES

An example:

Settings

All settings are changed in rails_helper.rb.

To set the number of retries, replace the value of X below:

config.around(:each) do |example|
  example.run_with_retry(retry: X)
end

To set the number of consecutive passes to remove a test from quarantine, replace the value of X below:

config.quarantine_release_at_consecutive_passes = X

To change when to record tests, change the logic here:

config.quarantine_record_tests = ENV["CI"]

Setup

Google Cloud

In the Google Cloud Project console, do the following.

Enable APIs for a New Project

Create a new project. For that project, enable the following APIs:

1. Google Drive API
2. Google Sheets API

Add a Service Account

Go to APIs & Services --> Credentials, and click +Create Credentials. Choose Service Account. Add an appropriate name and skip the optional settings.

Once created, click the new account's Keys tab. Click Add Key --> Create new key and choose the JSON key type. The key will automatically be downloaded to your computer with the name service-account.json.

Copy the service account's email address.

Google Drive

Create a new Google sheet that will serve as the "database" for quarantined tests. Give the sheet an appropriate name.

Share edit permissions on the spreadsheet with the service account by pasting the email address into the share modal.

Set the name of the worksheet (tab) to test_statuses.

Copy the following titles into the first row of the worksheet (see above).

A	B	C	D	E	F	G
id	full_description	updated_at	last_status	location	extra_attributes	consecutive_passes

Copy the id of the Google sheet (the id of a Google sheet is the alpha-numeric hash between spreadsheets/d/ and /edit in the url).

Github

Create Sheet ID Variable

Click the Settings tab for the project. In the sidebar, click Security --> Secrets and variables --> Actions. Click the Variables tab. Click New repository variable. Set the Name as QUARANTINE_SHEET_ID and paste the sheet's ID into the value. Click Add variable.

Create Service Account Secret

Github recommends that secrets be single line, so we're going to minify the service-account.json file into a single line. You can use jq for this.

To install jq on a Mac, use Homebrew:

brew install jq

Once installed, navigate to the directory where the service-account.json file was downloaded, and run the following. Copy the output.

jq -c . < service-account.json

In the same section (Security --> Secrets and variables --> Actions), click the Secrets tab. Click the New repository secret button. Set the Name as SERVICE_ACCOUNT_JSON and paste the one-line JSON you just created above. Click Add secret.