January 2023 Launchable Product Updates
The new year means new updates from Launchable! Learn more about our new test insights, filters and improved data resolution for Confidence curves.
Key Takeaways
New year, faster feedback, enhanced test suite insights, and amplified data resolution.
We expanded Unhealthy Tests Insights to include most failed tests and longest tests so you can you can find the tests that contribute the most to your test session duration and that are failing frequently.
You can now filter out flaky tests above a certain threshold from your subset runs, for early feedback on whether your change isn't likely to break the build.
We’re a sucker for specifics - that’s why we’ve increased Confidence Curve resolution so you can set targets that much more detailed.
New year, new Launchable features! 🎉 With approximately 26.9 million software developers in the world, we recognize the importance of a positive developer experience for each and every one of them. The team has been working hard to bring you updates to the platform that will improve developer velocity with faster, smarter testing.
To start the new year right, we've introduced additional insights on the Unhealthy Tests page, new filtering options for flaky tests, and improved data resolution for Confidence curves. Let's take a deep dive into the details:
More Insights on the Unhealthy Tests page: Longest Tests and Most Failed Tests
Following on from our previous announcement of the new Unhealthy Tests page, we've added two new Insight tables to the page: Longest Tests and Most Failed Tests!
The Longest Tests Insight table shows you the tests that consume the most testing time. You should review this list periodically to keep your overall test suite from getting too long. Similarly, if your test session duration has crept up over time (which you can see on the "Trends" page!), you can use this list to find the tests that contribute the most to your test session duration.
The Most Failed Tests Insight table shows you the tests that failed the most in the time period. This is useful because if a test is always failing, it may need to be updated or fixed. Tests that always fail interrupt your developers' CI cycles.
We hope these new Insights help you keep your test suite healthy!
New: Keep flaky tests out of your test runs
If you use Predictive Test Selection to intelligently shorten your developers' test sessions and provide faster feedback but flaky tests are slowing them down, you're in luck!
You can now filter out flaky tests above a certain threshold from your subset runs. This is great for early feedback where you want to quickly find out that your change isn't likely to break the build. You may want to run flaky tests later in your software development lifecycle, but if you're just looking for fast feedback, then it's great to get them out of the way.
You can use the new --ignore-flaky-tests-above option on launchable subset to do this. First, check your tests' flakiness scores on the Unhealthy Tests page, then set a threshold when you run a Launchable subset in your pipeline by adding the new option.
Here's a diagram explaining how it works:
Higher Resolution Confidence Curves
We recently increased the data resolution of Confidence curves in the Launchable webapp.
This makes it easier to choose the ideal optimization target for your Predictive Test Selection subset configuration.
