Starting today, the --variant option should be used to upload test results when the same tests are run on different environments, also known as matrix builds.
If you use a matrix strategy to run the same CI tests on different architectures or environments, test flakiness could be unique to an environment. Examples include testing a mobile app on iOS and Android, running tests for your CLI tool on macOS and Linux, or performing the same e2e tests on different browsers. Tests run on separate environments must be treated as unique to guarantee accurate flake detection.
Using variants, Flaky Tests will now treat the same test run on different environments as unique. This means flakes can be detected and quarantined on a per-environment basis.
To specify a variant during upload, use the --variant option:
The test details page in the Flaky Tests dashboard is getting a major UX overhaul.
User interviews and feedback from beta users helped us remove metrics that were not useful and update visualizations to better show a testās health over time. The tabbed views are also gone, which means you can get all the information about a testās health in a single, unified dashboard, including:Ā
A testās current status.
When a test was marked as flaky and quarantined.
Test flake and failure rates on mainline and PR branches.
Overall test health history.
Grouped unique test failure reasons.
These new charts break down how often a test is successful, failing, or quarantined for CI jobs for your mainline branch and PRs and show test health changes over time.
A more eventful test history timeline
The test history timeline has been moved to the main page, and updates, including when a test was marked as flaky, quarantined, or fixed, are always visible. Teams using the Jira integration to track test flakes will also see any updates to a flaky testās ticket reflected in the timeline.
You can still override the repo's default quarantine setting and always quarantine or never quarantine a test, but now you can leave a comment for other devs that shows up in the timeline.
For example, you might want to let your team know that the āTest is important - flakes must be fixed!ā when you override the repo default to never quarantine a test:
The table shows all the unique reasons the test fails in CI.Ā Trunk uses AI to auto-group unique failures so multiple runs will be grouped under a single failure reason. Individual runs, including stack traces, branch, commit, and job information, are available when a row in the table is expanded. This means all the information you need to debug and fix flaky or broken tests is available in a single place.
A new Quarantined Tests API is now available to all Flaky Tests users. This API fetches a list of currently quarantined tests for a given repo, allowing organizations to implement custom workflows to review and fix quarantined tests.
Trunk now supports single sign-on (SSO) for enterprise plan users. SSO allows your team to access Trunk with their work email account through your organization's identity and access management solutions, making it easier to manage your Trunk organization.
SSO is only available for enterprise plan users. If you're interested in configuring SSO for your team, contact us by email or Slack. We will reach out to help you configure SSO for your organization.
We have an exciting update for teams using the Flaky Test Jira integration to create and track their tickets.
Previously, the Jira integration only tracked ticket creation in each test's timeline. In this update, new events are added to the timeline for ticket assignments and status changes.
As your team continues to integrate Trunk into your workflow, Jira integration events combined with detection events in the timeline help tell a more complete story about the status of a flaky test. You can more easily understand if a flaky test is being tackled by an engineer and if their fix was effective during reviews.
Note: There can be a delay of up to 15 minutes between changes being made in Jira and the events being reflected in the timeline.
If your stable branch is not main, you can now set a custom stable branch for your repositories to improve flaky test detection.
Previously, Flaky Tests assumed that main was a repository's stable branch. This would lead to subpar flaky test detection if a different branch, master or develop for example, is defined as the default stable branch for your workflows.
Now you can override these stable branch defaults to better match your organization's custom CI workflows.
We're excited to introduce a new way to help your team stay on top of your repository's test health through weekly email reports.
With automatic detection and quarantining, Trunk can do a great job of mitigating the impact of test flakiness on your team's velocity. But we don't want you to forget about your flaky tests. Even if they're no longer causing friction, they should still be fixed eventually.
To help you stay on top of your repo's test health, we created a weekly digest for your flaky tests. Here's what you can expect to find in each report.
Key Metrics
At the top of each report, you'll see an overview of important metrics on flaky tests and their impact on your organization. You'll see the number of broken and flaky tests, how many PRs they block, and an estimate of engineering hours lost due to these flaky tests.
We also include the changes week to week. We believe it's important to keep track of week-to-week changes to understand if the flaky tests problem is controlled or if it's spreading. This helps you decide whether more resources should be devoted to fixing and controlling flaky tests. It can also alert against larger issues where a bug in code or infrastructure can cause a spike in test flakiness.
Per-Repo Breakdown
If your org is tracking flaky tests across many repos, the weekly report will break down the key metrics for each repo. This helps you see which repos are doing well and which might need extra love.
Frequently Failing Tests
Each report has a dedicated section for frequently failing tests. This alerts against high-impact broken and flaky tests. You can use this list to catch newly introduced high-impact flaky tests or get an overview of the flaky tests you should be investigating next with your team.
Important Action Items
Toolrot can be a real issue with the sheer scale of infrastructure required for modern CI. The weekly email reports will remind you if your CI is misconfigured or if there's a new version of the analytics-uploader tool you should upgrade to. All of these help ensure your detection remains accurate and effective.
Who Gets These Reports?
All org admins will get the emails by default as long as you've uploaded test results to us in the past week. The emails are delivered each Monday, and we go over the stats over the last week. You should see these emails in your mailbox starting next week.
If you've got any questions about these email reports or would like to suggest some new additions, please feel free to reach out to us on Slack.
We're excited to introduce webhook integrations for Slack, Microsoft Teams, GitHub Issues, and Linear. Webhook integrations make it easier to start automating notifications and ticket creation by providing tighter integrations and example transformations.
You can use the provided transformation to set up working notification messages and tickets immediately and customize the transformation for each webhook as you go. This means they're quicker to boot without sacrificing customizability.
We're excited to share our new webhook integrations for Trunk Merge. You can now send notifications about events in the merge queue to your Microsoft Teams and Slack channels.
These webhooks allow you to receive notifications when a PR enters the merge queue, begins testing, and succeeds or fails in the queue. These webhook come with a template transformation that works out of the box and can be customized using a JavaScript-based transformation code block.
You can create an integration under Organization > Webhooks by clicking Add Endpoint. Under Webhook, you can select the type of integration for your messaging service. Instructions are provided in each of the provided integrations.
The Trunk Web App now supports signing in with Microsoft. This makes it easier for teams who are primarily integrated with Microsoft's ecosystem (Outlook, Microsoft Teams) to get started with Trunk.
Weāre thrilled to announce support for CODEOWNERS in our Flaky Tests product, which is available for both GitHub and GitLab repositories. With this integration, we now process your CODEOWNERS file to identify and annotate ownership for tests automatically. This unlocks powerful capabilities for test ownership and management, including:
Test Details View: Instantly see who owns each test right from the test details page.
Webhook Integration: Retrieve ownership information in your webhooks for automated workflows.
Jira Integration: Automatically populate owner details in Jira tickets to streamline task assignments.
Why It Matters
CODEOWNERS is a key tool for maintaining code ownership within repositories. With this new integration, you can:
Notify Owners: Automatically inform test owners in messaging applications like Slack or Teams when their tests exhibit flakiness.
Streamline Workflows: Assign flakiness issues directly to owners using project management tools like Jira or Linear.
How to Get Started
Itās simple! Place your CODEOWNERS file in one of the standard locations supported by your Git provider and follow their conventions: GitHub and GitLab.
Hereās an example CODEOWNERS file:
1# In this example, @agraebe owns any file in the `/tests` directoryĀ Ā
2
3# at the root of your repository and its subdirectories.Ā Ā
Trunk Flaky Tests now supports uploading test results by parsing Bazel Build Event Protocol (BEP) files. Supporting BEP files gives Trunk more context when parsing Bazel test results for more accurate detection results. Parsing BEP files lets Trunk accurately track tests run by understanding which targets were actually built and which were cached and skipped. In the future, this will also allow Trunk to work better with Bazel's built-in retries.
Trunk expects a JSON serialization of the build event protocol which you can export by running the bazel test command with the options --nobuild_event_json_file_path_conversion and --build_event_json_file=build_events.json.
You can switch to uploading BEP files by running the upload command with the --bazel-bep-path build_events.json option.
Weāre excited to announce improved flaky test detection for merge queue users. In a merge queue, a single flaky failure will force every enqueued PR behind it to be retested, affecting every engineer involved and wasting tons of CI resources. If you run a merge queue like Trunk Merge to test your changes before they're merged, tests run on the merge branches are an important signal about your tests' health. Flaky Tests can now detect flaky tests using signals on your PR branches, merge branches, and protected branches.
Tests are expected to behave differently on protected branches such as main or master compared to tests run on PR branches. Trunk adjusts for this inherent volatility of PR branches by applying a slightly different set of rules to them:
Inconsistent test results on the same commit hash, where results are different on identical code is considered flaky
Inconsistent test results on different commit hashes, but across a large number of PRs is considered flaky
Merge branches created by merge queues, on the other hand, are treated similarly to protected branches. We make the assumption that before PRs are submitted to a merge queue, tests have already been run once on the PR and are passing.
If you'd like to learn more about how Trunk detects flaky tests, see the Detection page in docs.
Trunk Flaky Test supports detection on merge branch for Trunk Merge, GitLab Merge Trains, Graphite Merge Queues, and GitHub merge queue. If you're using an unsupported merge queue and would like to apply detection on merge branches, contact us on Slack.
Weāre excited to announce the addition of webhooks for flaky tests, designed to help you automate your workflows for better handling of flaky tests. The new test_case.status_changed event triggers whenever a test's status changes (healthy ā flaky ā broken). This makes it easier to track and respond to flaky tests as they show up. View the docs and example usage here.
With this update, you can automate actions to react to flaky tests in a way that suits you best. For instance, you can create tickets in project management tools (e.g. Jira, GitHub, Linear) and ensure they are automatically assigned to the right project and team. You can also automatically send notifications (e.g. in Slack, Discord, and Microsoft Teams) to the right engineers with the most relevant information, such as failure reasons and trace logs. This helps teams respond quickly, triage issues, and resolve flaky tests efficiently. Explore our webhook guide to get started.
Hi everyone, weāve introduced a new onboarding flow to make it easier to integrate your test framework and CI provider with Trunk.Ā
This flow can also be used to add more test frameworks and CI jobs to existing Trunk Flaky Tests projects. You can access this flow by clicking the Add Test Framework button on the top right of your dashboard.
The onboarding flow has guides for all of the test frameworks and CI provider combinations from the Trunk documentation. The docs will guide you through the setup process and help you validate your first upload.
The commands provided will have the API token and Trunk org slug automatically populated, so you can set up your repo without circling back to your settings.
If you donāt see the test framework or CI provider you see in the onboarding process, feel free to reach out to us on Slack.
Try the new add test framework flow in the Trunk Web App.
