If your stable branch is not main, you can now set a custom stable branch for your repositories to improve flaky test detection.
Previously, Flaky Tests assumed that main was a repository's stable branch. This would lead to subpar flaky test detection if a different branch, master or develop for example, is defined as the default stable branch for your workflows.
Now you can override these stable branch defaults to better match your organization's custom CI workflows.
We're excited to introduce a new way to help your team stay on top of your repository's test health through weekly email reports.
With automatic detection and quarantining, Trunk can do a great job of mitigating the impact of test flakiness on your team's velocity. But we don't want you to forget about your flaky tests. Even if they're no longer causing friction, they should still be fixed eventually.
To help you stay on top of your repo's test health, we created a weekly digest for your flaky tests. Here's what you can expect to find in each report.
Key Metrics
At the top of each report, you'll see an overview of important metrics on flaky tests and their impact on your organization. You'll see the number of broken and flaky tests, how many PRs they block, and an estimate of engineering hours lost due to these flaky tests.
We also include the changes week to week. We believe it's important to keep track of week-to-week changes to understand if the flaky tests problem is controlled or if it's spreading. This helps you decide whether more resources should be devoted to fixing and controlling flaky tests. It can also alert against larger issues where a bug in code or infrastructure can cause a spike in test flakiness.
Per-Repo Breakdown
If your org is tracking flaky tests across many repos, the weekly report will break down the key metrics for each repo. This helps you see which repos are doing well and which might need extra love.
Frequently Failing Tests
Each report has a dedicated section for frequently failing tests. This alerts against high-impact broken and flaky tests. You can use this list to catch newly introduced high-impact flaky tests or get an overview of the flaky tests you should be investigating next with your team.
Important Action Items
Toolrot can be a real issue with the sheer scale of infrastructure required for modern CI. The weekly email reports will remind you if your CI is misconfigured or if there's a new version of the analytics-uploader tool you should upgrade to. All of these help ensure your detection remains accurate and effective.
Who Gets These Reports?
All org admins will get the emails by default as long as you've uploaded test results to us in the past week. The emails are delivered each Monday, and we go over the stats over the last week. You should see these emails in your mailbox starting next week.
If you've got any questions about these email reports or would like to suggest some new additions, please feel free to reach out to us on Slack.
We're excited to introduce webhook integrations for Slack, Microsoft Teams, GitHub Issues, and Linear. Webhook integrations make it easier to start automating notifications and ticket creation by providing tighter integrations and example transformations.
You can use the provided transformation to set up working notification messages and tickets immediately and customize the transformation for each webhook as you go. This means they're quicker to boot without sacrificing customizability.
We're excited to share our new webhook integrations for Trunk Merge. You can now send notifications about events in the merge queue to your Microsoft Teams and Slack channels.
These webhooks allow you to receive notifications when a PR enters the merge queue, begins testing, and succeeds or fails in the queue. These webhook come with a template transformation that works out of the box and can be customized using a JavaScript-based transformation code block.
You can create an integration under Organization > Webhooks by clicking Add Endpoint. Under Webhook, you can select the type of integration for your messaging service. Instructions are provided in each of the provided integrations.
The Trunk Web App now supports signing in with Microsoft. This makes it easier for teams who are primarily integrated with Microsoft's ecosystem (Outlook, Microsoft Teams) to get started with Trunk.
We’re thrilled to announce support for CODEOWNERS in our Flaky Tests product, which is available for both GitHub and GitLab repositories. With this integration, we now process your CODEOWNERS file to identify and annotate ownership for tests automatically. This unlocks powerful capabilities for test ownership and management, including:
Test Details View: Instantly see who owns each test right from the test details page.
Webhook Integration: Retrieve ownership information in your webhooks for automated workflows.
Jira Integration: Automatically populate owner details in Jira tickets to streamline task assignments.
Why It Matters
CODEOWNERS is a key tool for maintaining code ownership within repositories. With this new integration, you can:
Notify Owners: Automatically inform test owners in messaging applications like Slack or Teams when their tests exhibit flakiness.
Streamline Workflows: Assign flakiness issues directly to owners using project management tools like Jira or Linear.
How to Get Started
It’s simple! Place your CODEOWNERS file in one of the standard locations supported by your Git provider and follow their conventions: GitHub and GitLab.
Here’s an example CODEOWNERS file:
1# In this example, @agraebe owns any file in the `/tests` directory
2
3# at the root of your repository and its subdirectories.
Trunk Flaky Tests now supports uploading test results by parsing Bazel Build Event Protocol (BEP) files. Supporting BEP files gives Trunk more context when parsing Bazel test results for more accurate detection results. Parsing BEP files lets Trunk accurately track tests run by understanding which targets were actually built and which were cached and skipped. In the future, this will also allow Trunk to work better with Bazel's built-in retries.
Trunk expects a JSON serialization of the build event protocol which you can export by running the bazel test command with the options --nobuild_event_json_file_path_conversion and --build_event_json_file=build_events.json.
You can switch to uploading BEP files by running the upload command with the --bazel-bep-path build_events.json option.
We’re excited to announce improved flaky test detection for merge queue users. In a merge queue, a single flaky failure will force every enqueued PR behind it to be retested, affecting every engineer involved and wasting tons of CI resources. If you run a merge queue like Trunk Merge to test your changes before they're merged, tests run on the merge branches are an important signal about your tests' health. Flaky Tests can now detect flaky tests using signals on your PR branches, merge branches, and protected branches.
Tests are expected to behave differently on protected branches such as main or master compared to tests run on PR branches. Trunk adjusts for this inherent volatility of PR branches by applying a slightly different set of rules to them:
Inconsistent test results on the same commit hash, where results are different on identical code is considered flaky
Inconsistent test results on different commit hashes, but across a large number of PRs is considered flaky
Merge branches created by merge queues, on the other hand, are treated similarly to protected branches. We make the assumption that before PRs are submitted to a merge queue, tests have already been run once on the PR and are passing.
If you'd like to learn more about how Trunk detects flaky tests, see the Detection page in docs.
Trunk Flaky Test supports detection on merge branch for Trunk Merge, GitLab Merge Trains, Graphite Merge Queues, and GitHub merge queue. If you're using an unsupported merge queue and would like to apply detection on merge branches, contact us on Slack.
We’re excited to announce the addition of webhooks for flaky tests, designed to help you automate your workflows for better handling of flaky tests. The new test_case.status_changed event triggers whenever a test's status changes (healthy ↔ flaky ↔ broken). This makes it easier to track and respond to flaky tests as they show up. View the docs and example usage here.
With this update, you can automate actions to react to flaky tests in a way that suits you best. For instance, you can create tickets in project management tools (e.g. Jira, GitHub, Linear) and ensure they are automatically assigned to the right project and team. You can also automatically send notifications (e.g. in Slack, Discord, and Microsoft Teams) to the right engineers with the most relevant information, such as failure reasons and trace logs. This helps teams respond quickly, triage issues, and resolve flaky tests efficiently. Explore our webhook guide to get started.
Hi everyone, we’ve introduced a new onboarding flow to make it easier to integrate your test framework and CI provider with Trunk.
This flow can also be used to add more test frameworks and CI jobs to existing Trunk Flaky Tests projects. You can access this flow by clicking the Add Test Framework button on the top right of your dashboard.
The onboarding flow has guides for all of the test frameworks and CI provider combinations from the Trunk documentation. The docs will guide you through the setup process and help you validate your first upload.
The commands provided will have the API token and Trunk org slug automatically populated, so you can set up your repo without circling back to your settings.
If you don’t see the test framework or CI provider you see in the onboarding process, feel free to reach out to us on Slack.
Try the new add test framework flow in the Trunk Web App.
We’re continually refining our UX with the help of our beta users’ feedback and we’ve made some changes to how Flaky Tests displays key information on the dashboards. As more users are using Trunk to tackle flakiness in their test suite, we’ve received some valuable feedback on which pieces of information are missing and which metrics are more important.
We’ve revisited the main Flaky Tests dashboard and added a new trend chart to help you better understand the health trends of your tests at a glance.
We’ve also made improvements to what information is displayed in the test list to help you find high-impact flaky tests easier, especially when quarantining is enabled.
In the test details page, we revisited the metrics that customers care most about and placed them more prominently. We also cleaned up how quarantining and ticket creation UI elements are displayed for a cleaner and more intuitive appearance.
A common pain point during onboarding is the lack of transparency after the Trunk CLI uploads test results. To help you better understand if test results have been uploaded successfully and if it has been processed, we’ve added a Uploads tab to the Trunk Flaky Tests dashboard.
The Uploads tab shows you the status of each upload, which is identified by the source branch and the commit hash. You can click on the link under Job Run to view the CI job in your provider.
Flaky Tests now supports the XCResults format outputted by Swift projects using XCTests. You can now upload the .xcresults format to Trunk directly, without configuring a JUnit XML reporter. Supporting XCTests directly also means you can include test file paths from .xcresults when reporting to Trunk Flaky Tests. This is especially important for CODEOWNERS support planned for a future release.
You can upload XCResults using the Trunk CLI with the --xcresult-path argument.
We're excited to introduce a powerful new feature that will help you accelerate your PR iterations: PR Test Summaries!
Imagine this: You're diligently working on a new feature in your branch and ready to submit your Pull Request (PR) for review. You've run your tests locally, and everything looks good. But then you push your code, and—oh no! Some tests you haven’t even touched are failing. This is where our new PR Test Summaries come in!
With this feature, you can quickly identify failing tests and their reasons, as well as pinpoint flaky tests. This way, you’ll know whether the issue lies with your code changes or the tests themselves.
PR Test Summaries
This new feature includes the following:
GitHub PR Comment: Each new PR will receive a comment summarizing the test results, including:
Counts for test runs, categorized by result (failed, flaky, skipped, quarantined)
Collapsible lists for failed, flaky, and skipped tests
For failed and flaky tests: detailed failure reasons and direct links to trace logs for quick investigation
PR Summary View: This new view in our web app gives you access to a comprehensive listing of all test run details:
A table showing all test runs, filtered by outcome (passed, failed, or skipped) and whether or not they have been quarantined
Every test run includes an icon to indicate whether a test is known to be flaky, the failure reason, and more details to investigate
Test Run Detail Panel: You are most likely already familiar with this view; it will show you all run details include tracelogs, branch, job run link, duration, and more.
