Changelog

We’re thrilled to announce support for CODEOWNERS in our Flaky Tests product, which is available for both GitHub and GitLab repositories. With this integration, we now process your CODEOWNERS file to identify and annotate ownership for tests automatically. This unlocks powerful capabilities for test ownership and management, including:

Test Details View: Instantly see who owns each test right from the test details page.

Webhook Integration: Retrieve ownership information in your webhooks for automated workflows.

1{
2
3   "status_change": { ... },
4
5   "test_case": {
6
7       "codeowners": ["@agraebe", "code", "owners", "@dev-team"],
8
9       "id": "fc7a6bdd-0ebe-55d5-aa31-ff0adb411d0e",
10
11... }
12
13}
14
15

Jira Integration: Automatically populate owner details in Jira tickets to streamline task assignments.

Why It Matters

CODEOWNERS is a key tool for maintaining code ownership within repositories. With this new integration, you can:

• Notify Owners: Automatically inform test owners in messaging applications like Slack or Teams when their tests exhibit flakiness.

• Streamline Workflows: Assign flakiness issues directly to owners using project management tools like Jira or Linear.

How to Get Started

It’s simple! Place your CODEOWNERS file in one of the standard locations supported by your Git provider and follow their conventions: GitHub and GitLab.

Here’s an example CODEOWNERS file:

1# In this example, @agraebe owns any file in the `/tests` directory  
2
3# at the root of your repository and its subdirectories.  
4
5/tests/ @agraebe 

Trunk Flaky Tests now supports uploading test results by parsing Bazel Build Event Protocol (BEP) files. Supporting BEP files gives Trunk more context when parsing Bazel test results for more accurate detection results. Parsing BEP files lets Trunk accurately track tests run by understanding which targets were actually built and which were cached and skipped. In the future, this will also allow Trunk to work better with Bazel's built-in retries.

Trunk expects a JSON serialization of the build event protocol which you can export by running the bazel test command with the options --nobuild_event_json_file_path_conversion and --build_event_json_file=build_events.json.

You can switch to uploading BEP files by running the upload command with the --bazel-bep-path build_events.json option.

1./trunk flakytests upload --bazel-bep-path build_events.json \
2--org-url-slug <TRUNK_ORG_SLUG> \
3--token $TRUNK_TOKEN

You can learn more about Bazel in the docs or chat with us on Slack.

We’re excited to announce improved flaky test detection for merge queue users. In a merge queue, a single flaky failure will force every enqueued PR behind it to be retested, affecting every engineer involved and wasting tons of CI resources. If you run a merge queue like Trunk Merge to test your changes before they're merged, tests run on the merge branches are an important signal about your tests' health. Flaky Tests can now detect flaky tests using signals on your PR branches, merge branches, and protected branches.

Tests are expected to behave differently on protected branches such as main or master compared to tests run on PR branches. Trunk adjusts for this inherent volatility of PR branches by applying a slightly different set of rules to them:

• Inconsistent test results on the same commit hash, where results are different on identical code is considered flaky

• Inconsistent test results on different commit hashes, but across a large number of PRs is considered flaky

Merge branches created by merge queues, on the other hand, are treated similarly to protected branches. We make the assumption that before PRs are submitted to a merge queue, tests have already been run once on the PR and are passing.

If you'd like to learn more about how Trunk detects flaky tests, see the Detection page in docs.

Trunk Flaky Test supports detection on merge branch for Trunk Merge, GitLab Merge Trains, Graphite Merge Queues, and GitHub merge queue. If you're using an unsupported merge queue and would like to apply detection on merge branches, contact us on Slack.

We’re excited to announce the addition of webhooks for flaky tests, designed to help you automate your workflows for better handling of flaky tests. The new test_case.status_changed event triggers whenever a test's status changes (healthy ↔ flaky ↔ broken). This makes it easier to track and respond to flaky tests as they show up. View the docs and example usage here.

With this update, you can automate actions to react to flaky tests in a way that suits you best. For instance, you can create tickets in project management tools (e.g. Jira, GitHub, Linear) and ensure they are automatically assigned to the right project and team. You can also automatically send notifications (e.g. in Slack, Discord, and Microsoft Teams) to the right engineers with the most relevant information, such as failure reasons and trace logs. This helps teams respond quickly, triage issues, and resolve flaky tests efficiently. Explore our webhook guide to get started.

✨ Improvements

• Upgrade ruby-build to 20241105. Thanks @amkisko for the contribution!

• Upgrade jdk to 23

• Upgrade rust to 1.82.0

• Upgrade php tooling to support 8.4

• Upgrade node to 18.20.5

🔧 Fixes

• Use a hadolint version that works on the latest versions of macOS

• Fix parsing in new versions of ruff for source files with syntax errors

• Update links to custom linters and parsers documentation by @amkisko

Hi everyone, we’ve introduced a new onboarding flow to make it easier to integrate your test framework and CI provider with Trunk. 

This flow can also be used to add more test frameworks and CI jobs to existing Trunk Flaky Tests projects. You can access this flow by clicking the Add Test Framework button on the top right of your dashboard.

The onboarding flow has guides for all of the test frameworks and CI provider combinations from the Trunk documentation. The docs will guide you through the setup process and help you validate your first upload.

The commands provided will have the API token and Trunk org slug automatically populated, so you can set up your repo without circling back to your settings.

If you don’t see the test framework or CI provider you see in the onboarding process, feel free to reach out to us on Slack.

Try the new add test framework flow in the Trunk Web App.

The Trunk CLI now includes commands for uploading and validating test results for Trunk Flaky Tests.

You can install the CLI in your CI systems like this:

1curl -fsSLO --retry 3 https://trunk.io/releases/trunk && chmod +x trunk

The launcher will install the appropriate Trunk CLI binary for your system.

You can use the Trunk CLI for Flaky Tests using the trunk flakytests subcommand.

1./trunk flakytests upload --junit-paths "test_output.xml" \
2  --org-url-slug <TRUNK_ORG_SLUG> \
3  --token $TRUNK_API_TOKEN

You can also now use the Trunk CLI to validate JUnit XML files during debugging.

1./trunk flakytests validate

Learn more in our Uploader CLI Reference docs.

We’re continually refining our UX with the help of our beta users’ feedback and we’ve made some changes to how Flaky Tests displays key information on the dashboards. As more users are using Trunk to tackle flakiness in their test suite, we’ve received some valuable feedback on which pieces of information are missing and which metrics are more important.

We’ve revisited the main Flaky Tests dashboard and added a new trend chart to help you better understand the health trends of your tests at a glance.

We’ve also made improvements to what information is displayed in the test list to help you find high-impact flaky tests easier, especially when quarantining is enabled.

In the test details page, we revisited the metrics that customers care most about and placed them more prominently. We also cleaned up how quarantining and ticket creation UI elements are displayed for a cleaner and more intuitive appearance.

See these improvements in the Trunk Web App.

🔧 Fixes

• Fix a daemon connection error that could occur on Windows.

• Fix a daemon connection error that could happen on some machines with restricted TCP permissions.

• Check mypy output for crashes and report to the user.

A common pain point during onboarding is the lack of transparency after the Trunk CLI uploads test results. To help you better understand if test results have been uploaded successfully and if it has been processed, we’ve added a Uploads tab to the Trunk Flaky Tests dashboard.

The Uploads tab shows you the status of each upload, which is identified by the source branch and the commit hash. You can click on the link under Job Run to view the CI job in your provider.

View your recent uploads in the Trunk Web App.

Flaky Tests now supports the XCResults format outputted by Swift projects using XCTests. You can now upload the .xcresults format to Trunk directly, without configuring a JUnit XML reporter. Supporting XCTests directly also means you can include test file paths from .xcresults when reporting to Trunk Flaky Tests. This is especially important for CODEOWNERS support planned for a future release.

You can upload XCResults using the Trunk CLI with the --xcresult-path argument.

1./trunk flakytests upload --xcresult-path "/path/to/Test.xresult"

To learn more, visit our XCTests documentation and Uploader CLI references.

We're excited to introduce a powerful new feature that will help you accelerate your PR iterations: PR Test Summaries!

Imagine this: You're diligently working on a new feature in your branch and ready to submit your Pull Request (PR) for review. You've run your tests locally, and everything looks good. But then you push your code, and—oh no! Some tests you haven’t even touched are failing. This is where our new PR Test Summaries come in!

With this feature, you can quickly identify failing tests and their reasons, as well as pinpoint flaky tests. This way, you’ll know whether the issue lies with your code changes or the tests themselves.

PR Test Summaries

This new feature includes the following:

GitHub PR Comment: Each new PR will receive a comment summarizing the test results, including:

• Counts for test runs, categorized by result (failed, flaky, skipped, quarantined)

• Collapsible lists for failed, flaky, and skipped tests

• For failed and flaky tests: detailed failure reasons and direct links to trace logs for quick investigation

PR Summary View: This new view in our web app gives you access to a comprehensive listing of all test run details:

• A table showing all test runs, filtered by outcome (passed, failed, or skipped) and whether or not they have been quarantined

• Every test run includes an icon to indicate whether a test is known to be flaky, the failure reason, and more details to investigate

Test Run Detail Panel: You are most likely already familiar with this view; it will show you all run details include tracelogs, branch, job run link, duration, and more.

Learn more about PR Test Summaries in the docs

🎁 New

• New tool: grpcui. Can be used like Postman to explore gRPC servers with a web UI, learn about running tools here.

✨ Improvements

• Add download for python@3.11.9. Specify runtime versions in the runtime.enabled section of your .trunk/trunk.yaml

• Add support for psscriptanalyzer versions 1.23.0 and beyond!

We’re excited to provide a new way for you to mitigate the negative impact of flaky tests in your repo through quarantining. 

Quarantining lets you isolate failures for known flaky tests so they don't fail your CI jobs while still continuing to run them. It looks for known flaky tests and determines which tests should be isolated at runtime, avoiding the code changes usually required to disable flaky tests.

Running flaky tests without disabling them lets you validate your fixes by continuously monitoring their results. This is especially important because most attempted fixes for flaky tests don’t reduce the test’s flakiness.

Enabling Quarantining

We’re currently onboarding organizations to try quarantining manually. This is because quarantining can drastically impact the results of your CI jobs. We want to help you validate quarantining results and educate your team on its behavior during onboarding. To try Quarantining, reach out to us on Slack.

Quarantining Features

Key Impact Metrics: You can see the impact of quarantining on your repo. Understand how many CI jobs with flaky failures were rescued by quarantining and how many engineering hours were saved.

Keep track of quarantined tests: You can continue to monitor the results of quarantined flaky tests through the PR Tests Summaries generated by Trunk. You can filter failures by quarantined runs and see the failure reasons at a glance.

Per test overrides: You can override quarantining for individual tests in your repo to always quarantine or never quarantine. This helps you handle edge cases such as tests that should never break or tests that are flaky/broken due to reasons outside your team’s control, like external APIs being down.

Detailed audit trails: You can see the audit trail of quarantine setting changes in both the test’s status history and an overall audit trail in the settings.

Learn more in the Quarantining documentation.

🔧 Fixes

• Fix a rare deadlock that could cause a linter execution to hang indefinitely.

🎁 New

• New linter: squawk

• New linter: markdownlint-cli2

• New tool: dbt-cli

🔧 Fixes

• Disable incorrect caching for pyright

✨ Improvements

• Help text improvements.

• trunk fmt will now use configured verbs when presenting results to the user.

🔧 Fixes

• Fix a crash in the LSP server when giving diagnostics for empty files.

• Trunk no longer erroneously follows symlinks to directories when watching the filesystem on Linux, saving on resources.

We’re excited to announce the latest enhancement: Jira integration for managing flaky tests. This update builds on our MVP by streamlining the issue management process within Jira.

With this release, we’re introducing key features designed to simplify and enhance your workflow:

• Create Jira Issues with One Click: Generate Jira issues directly from the dashboard or test details view. After a quick setup, a single click creates the issue, saving you time and effort.

• Customize Issue Defaults: Set up default issue types and labels to match your workflow. This ensures that every issue you create aligns with your process.

• Track Issue Updates: Monitor the status of your Jira issues with ease. A new column on the dashboard shows the current issue status, while the test details page displays status changes in the timeline. These updates help confirm if an issue completion has resolved the flaky test.

• Review Work Progress: Use the new tracking features to review progress and identify trends. Filter by status to pinpoint flaky tests that need attention and assess their impact on your team.

Get Started Today

Start the process by clicking "Create Issue" on the dashboard and follow the prompts to link your Jira account with your Jira instance URL, username, and Personal Access Token.

✨ Improvements

• ruff@>=0.6.0 now runs on Jupyter notebooks by default

🔧 Fixes

• Fix definition for textproto filetype

• Fix trivy config files, and don't recommend by default