Running important tests for a code change first & rest later

Launchable is a test recommendation engine that uses machine learning to speed up CI pipelines by selecting the right tests to run at the right stage of your development workflow. This is a form of Test Impact Analysis. Depending on your situation, you can add Launchable to your CI pipeline to shift tests left and right.

Shifting tests left and right

Shift-left is an approach where you run a subset of a long running test suite earlier in the development lifecycle. You might already run some suites less frequently, such as end-to-end UI tests, system tests, or nightly regression tests. These suites take too long to run on every git push or after every pull request is merged. Launchable lets you shift-left these tests by selecting a dynamic subset of the full test suite for each change to run earlier in the lifecycle.

On the other end of the spectrum, shift-right is an approach to optimize CI tests that already run on every git push. They are already shifted-left as far as possible (at least as far as CI is concerned), so now the task is to shift the less important tests to the right and run them less often. For example, you might already run unit and integration tests every time a developer pushes to their branch. Over time, these test suites may grow to run in 30 or 45 minutes. You can use Launchable to "shift right" the less risky tests and run them less frequently, keeping an intelligently selected subset of tests. 

In contrast with shifting left, shifting right involves splitting risky tests out from the less risky ones: we want to run the most risky ones more often and save the less risky ones for later. To enable this practice, we’ve added a feature to our CLI that we like to call The Split.

The Split: run risky tests first, then less risky ones later

When you adopt Launchable to shift your tests right, you can easily split your existing suite into two parts using the Launchable CLI:

1. An intelligently selected subset of tests – to provide faster feedback to developers, and

2. The rest of the test suite

The --rest option in the CLI makes this easy. For example, if you’re using Ruby and Minitest. To retrieve a subset of tests, first pass the full list of test candidates to Launchable subset. For example:

launchable subset \
  --build <BUILD NAME> \
  --target 20%> \
  --rest launchable-remainder.txt \
  minitest test/**/*.rb > launchable-subset.txt

This creates two files, launchable-subset.txt and launchable-remainder.txt, that you can pass into your command to run tests in two stages. For example, for Rails you would use:

# Stage 1
bundle exec rails test $(cat launchable-subset.txt)
# Stage 2
bundle exec rails test $(cat launchable-remainder.txt)

This approach lets you compare the results of the two runs and immediately see Launchable’s effectiveness. Launchable will place the more risky tests in the subset, pushing more failures into the subset and fewer in the remainder. This helps you dial in your subset target, too.

Once you’ve dialed in your subset target, you can then you can remove the second part to get fast feedback on every run. You’ll still want to run all of the tests at some point in your cycle, such as after every merge (you might already be doing this), but the very few test failures should appear in this phase since the subset makes sure to run the risky tests earlier.

This approach is covered in our documentation for each test runner. Just look for “Shift-right” on each page and go from there.