Changelog

We’re happy to announce that a new test_case.quarantining_setting_changed webhook is now available for all Flaky Tests users.

This webhook is fired when a test’s quarantine setting is manually changed between ALWAYS_QUARANTINE, NEVER_QUARANTINE, or UNSPECIFIED, where UNSPECIFIED means that the repo’s quarantine settings will be applied to the test case.

The webhook payload contains useful details like the new and previous settings, a user-inputted reason for the change, who made the change, and all the info you need for the affected test case:

1{
2 "quarantine_setting_changed": {
3   "actor": {
4     "email": "riley@trunk.io",
5     "full_name": "Riley Draward"
6   },
7   "previous_quarantining_setting": "UNSPECIFIED",
8   "reason": "Test is important, should not be quarantined!",
9   "timestamp": "2025-06-11T14:17:06.251Z",
10   "updated_quarantining_setting": "NEVER_QUARANTINE"
11 },
12 "test_case": {
13   "codeowners": [],
14   "failure_rate_last_7d": 0,
15   "html_url": "https://app.trunk.io/rileys-org/flaky-tests/test/cd7ef2fe-0ec1-5e40-a323-027029579f5b?repo=rdraward%2Fflaky-remix",
16   "id": "cd7ef2fe-0ec1-5e40-a323-027029579f5b",
17   "most_common_failures": [],
18   "name": "sums some nums > adds 1 + 1 to equal 2",
19   "pull_requests_impacted_last_7d": 0,
20   "quarantined": false,
21   "repository": {
22     "html_url": "https://github.com/rdraward/flaky-remix"
23   },
24   "status": {
25     "reason": "",
26     "timestamp": "2025-03-17T21:53:43.327Z",
27     "value": "flaky"
28   },
29   "test_suite": "app/utils/sum.test.ts",
30   "ticket": {
31     "html_url": "https://linear.app/riley-at-trunk/issue/RIL-10/[flaky-test]-apputilssumtestts-sums-some-nums-adds-1-1-to-equal-2"
32   },
33   "variant": ""
34 },
35 "type": "test_case.quarantining_setting_changed"
36}
37

This allows you to build automations and custom workflows when a test’s quarantine setting is changed, such as automatically creating Linear or Jira tickets to fix flaky or broken tests marked as ALWAYS_QUARANTINED, or sending a message to the team in Slack so everyone is kept up to date on status changes.

To subscribe to the new webhook:

1. Go to Settings in the Trunk app.

2. Click on Webhooks under Organization.

3. Click Add Endpoint and connect to an existing endpoint, Slack, MS Teams, Linear, or Jira.

4. Follow our guides for handling the webhook payload.

For more information on the webhook events, check out our Svix documentation.

✨ Improvements

• Python runtime environment now passes thru required environment variables for PIP and Request component to work correctly with proxies and custom certificates

• Updated dependencies in local package.json and node runtime bumped to 22 LTS

🔧 Fixes

• Fixed up some testing environment to better support some linter specifications

✨ Improvements

• Updated embedded CURL to v8.13 (enables support for running inside Open AI Codex)

• Formatting issues can now be downgraded to non-blocking using the threshold value like any other linter issue. To make use of this feature you need to set the unformatted_level for any formatters you want to control this way as such:

1lint:
2  definitions:
3   - name: foo_formatter
4     commands:
5      - name: format
6        ...
7        formatter: true
8        unformatted_level: low

🔧 Fixes

• Specifying a runtime with equality operator will use system version if available. (e.g. ruby@=3.4.2 will now use ruby found on PATH if version matches exactly)

We’ve updated the Merge Queue Failure tab so that only PRs that failed and have not been resubmitted to the queue are displayed.

Previously, the Failure tab contained a list of all historical PR failures, making it difficult to identify current failures requiring action. This change gives you a concise list of PR failures that can be fixed and resubmitted to the queue. Upon resubmission, the PR is removed from the page.

More changes have landed on the Flaky Tests dashboard: unique failure details on the test details page are now available in a single table view.

This table is helpful for developers tasked with debugging and fixing flakes. Trunk uses AI to automatically group related failure reasons for an individual test and presents them in a single table:

The test summary, charts that display quarantine or error status, count, and dates of failures help you determine where to start your investigation. Each unique failure reason can also be expanded to show all historical stack traces for that failure type:

This means that all the information you need to start debugging flaky tests is in a single place. Developers don’t have to manually dig through historical CI logs to unearth various stack traces and logs.

Have questions or feedback on the updates? Reach out on our community Slack.

The new Flaky Tests RSpec plugin is the best way to run RSpec tests and upload the results to Trunk.

The plugin provides a couple of different advantages over generating JUnit XML files when testing with RSpec, all of which contribute to more accurate flake detection in Ruby projects:

• By default, running RSpec tests and creating a report using rspec_junit_formatter will produce programmatic test names, which makes flake detection difficult. The plugin eliminates this issue by uploading consistent test names to Trunk.

• The plugin allows for a more direct capture of test run retries.

• Flaky tests can be quarantined as the tests run, not after the fact.

Get started with rspec_trunk_flaky_tests

To get started with the rspec_trunk_flaky_tests plugin:

• Add the plugin gem to your Gemfile:

1# Gemfile
2gem "rspec_trunk_flaky_tests"

• Install the plugin by running the install command in your terminal

1bundler install

• Load the plugin in spec_helper.rb:

1# Add to spec/spec_helper.rb
2require "trunk_spec_helper"

Don’t forget to disable automatic retries on failures.

That’s it, you are ready to run your RSpec tests and upload the results to Trunk. Start by uploading locally run test results:

1TRUNK_ORG_URL_SLUG=<TRUNK_ORG_URL_SLUG> \
2TRUNK_API_TOKEN=<TRUNK_ORG_TOKEN> \
3bundle exec rspec

Your TRUNK_ORG_URL_SLUG and TRUNK_API_TOKEN are found under Settings in the Trunk web app.

When you run this command locally, your RSpec tests will run, and results will automatically be uploaded to Trunk. Any uploaded results will be displayed in the Uploads tab.

The last step is integrating with your CI provider. After integrating, Trunk will automatically detect and quarantine your flaky tests every time the CI job runs.

See the RSpec docs for more info.

Trunk CI Analytics and Code Quality web issue browser are being shut down today. This is a strategic change as we reinvent and consolidate our tools for the future.

What’s changing?

• CI Analytics has been shut down as of today.

• Code Quality’s web issue browser has been shut down as of today.

• Code Quality workflows like Check Nightly and Check on PRs have been marked as deprecated and will be shut down on July 27th, 2025.

The core value of Code Quality comes from the CLI, IDE, and CI integrations, and these critical components will continue to work as they always have.

Code Quality migration

Checking for issues nightly and on pull requests has been marked as deprecated. You can check if you’re using these features in the Code Quality section of your repo settings. You can still run Code Quality in CI if you migrate your workflow following this guide.

Code Quality will no longer collect your linting issues, which means the --upload flag will no longer work. You will also stop receiving Slack notifications about new issues discovered by linters.

1# --upload will no longer work
2trunk check --upload --series=main

What's next?

AI has changed the way we work. It allows us to go beyond traditional dashboards to surface root causes of test and CI instability and develop AI-driven tools to address these problems. We have shut down these parts of Trunk to unify them as a single AI-powered DevEx platform.

We can’t wait to show you what’s next. You can follow our progress by signing up for our in-app waitlist. If you have questions or concerns, please reach out via email (support@trunk.io) or Slack.

Webhooks are now available for batched PR merges in Merge Queue. This allows you to build custom automations and respond to events when batching is enabled.

The pull_request_batch.merged webhook payload lists all the PRs included in the batch, as well as information about the repo, the target branch you are merging into, and the test (or feature) branch:

1 "action": "merged",
2  "pr_numbers": [
3    1,
4    2
5  ],
6  "repository": {
7    "host": "github.com",
8    "name": "trunk-io",
9    "owner": "mergequeue"
10  },
11  "status": "some string",
12  "target_branch": "main",
13  "test_branch": "trunk-merge/pr-1-2/abc123",
14  "test_branch_sha": "a23f8853080d3fe873c3419a735513898c2329f500ce6aa39cf2253213fd00e4",
15  "test_pr_number": 15
16}

If you aren’t using batching on your Merge Queue, the pull_request.merged webhook is still available for individual PR merges.

Read the webhook docs to learn how to subscribe to Merge Queue webhooks.

We now have a built-in Linear integration that creates tickets with relevant test and failure information from your flaky tests.

Detecting and quarantining flaky tests aren’t the only steps required to eliminate them from a repo. A follow-up process must also be established to collect, triage, and fix flaky tests.

Ticketing system integrations help with that follow-up. Trunk generates tickets automatically from detected flaky tests, enabling you to triage and address flakes. Flakes on important tests should be dealt with immediately, and other flakes can be added to a backlog or the tests can be marked for future removal.

To set up the Linear integration:

• Go to your repo settings: Settings > repo > Ticketing Integration.

• Select Linear as your system and enter a Linear API key.

• Select a team and connect to Linear.

You can also select a default project and assignee for newly created tickets. This enables you to designate a person on a team to triage, prioritize, update, and assign tickets for flaky tests when they are created.

Once the integration is set up, you can create a new ticket:

• In the options menu on the test cases overview dashboard.

• With the Create ticket button on the test details dashboard.

You will see a preview of the ticket title and description, and can click Create Linear Ticket to create the new ticket.

When a ticket is created, any changes to that ticket, such as status or person assigned, are also visible in the timeline on the test case details page in Flaky Tests.

Read the docs for more information on setting up the Linear integration.

Today, we’re introducing a new Link Ticket to Test Case API that allows you to link your existing Linear or Jira tickets to tests in the Flaky Tests dashboard.

Trunk has built-in ticketing integrations with Jira and Linear, allowing you to create detailed bug tickets for flaky tests with a single click. If you’re already tracking flaky test issues with existing tickets, you can now link those Jira or Linear tickets directly to your flaky tests in Trunk, keeping all relevant context in one place.

Calling the API requires the ID of your test case in Flaky Tests, which can be found on incoming webhook payloads or the URL of the test case details page, and the ID of the ticket in your ticketing system:

1curl -L \​
2  --request POST \​
3  --url 'https://api.trunk.io/v1/flaky-tests/link-ticket-to-test-case' \​
4  --header 'x-api-token: YOUR_API_KEY' \​
5  --header 'Content-Type: application/json' \​
6  --data '{
7    "test_case_id": "01234567-0123-0123-0123-0123456789ab",
8    "external_ticket_id": "KAN-123"

Once a ticket is linked to a test, ticketing information will be available in Flaky Tests webhooks. This makes it possible to build custom integrations and automations with the context of any linked ticket information. And when ticket information is not included in the payload, you know one has not been created for the flake.

You can only use the Link Ticket to Test Case API if you have already set up the Linear or Jira integration.

Read the reference docs for more information on the Link Ticket to Test Case API.

Starting today, the --variant option should be used to upload test results when the same tests are run on different environments, also known as matrix builds.

If you use a matrix strategy to run the same CI tests on different architectures or environments, test flakiness could be unique to an environment. Examples include testing a mobile app on iOS and Android, running tests for your CLI tool on macOS and Linux, or performing the same e2e tests on different browsers. Tests run on separate environments must be treated as unique to guarantee accurate flake detection.

Using variants, Flaky Tests will now treat the same test run on different environments as unique. This means flakes can be detected and quarantined on a per-environment basis.

To specify a variant during upload, use the --variant option:

1./trunk flakytests upload --junit-paths "test_output.xml" \
2--org-url-slug <TRUNK_ORG_SLUG> \
3--token $TRUNK_API_TOKEN \
4--variant linux_x86_64

Variant names are visible in brackets on your Flaky Tests dashboard:

See the reference docs for more information on uploading test reports with variants.

We’ve added a new updateQueue API to Trunk Merge Queue that enables you to update a Merge Queue’s state.

This means that you no longer need to manually go into the settings page in the web app to update the merge queue state. Instead, queue state can be managed with custom tooling or automations using updateQueue:

1curl -L \
2  --request POST \
3  --url 'https://api.trunk.io/v1/updateQueue' \
4  --header 'x-api-token: YOUR_API_KEY' \
5  --header 'Content-Type: application/json' \
6  --data '{
7    "repo": {
8      "host": "github.com",
9      "name": "mergequeue",
10      "owner": "trunk-io"
11    },
12    "targetBranch": "main",
13    "state": "DRAINING"
14  }'

As a refresher, the possible queue states are:

• RUNNING: PRs are being enqueued and are merged after tests pass. This is the default state for a merge queue.

• PAUSED: PRs in the queue are tested, but the queue will not merge successful PRs or begin testing new PRs. A queue can be paused as an immediate response to a CI incident, reducing noise from failures while preserving the order of PRs that have already been enqueued.

• DRAINING: PRs currently in the queue will be tested and merged, but new PRs will not begin testing. This state is helpful for use cases like a code freeze, so PRs in the queue can be shipped, but newly enqueued PRs are not tested or merged. 

See the Merge Queue API reference for more information.

🎁 New

• New linter: golangci-lint2. Users who wish to upgrade should replace golangci-lint with golangci-lint2

✨ Improvements

• Add in-repo docs for converting some native linter ignores to trunk-ignore

• Update kube-linter to 0.7.2. Thanks @janisz for the contribution!

• Update ruby build and latest stable ruby release to 3.4.2

🔧 Fixes

• Fix Dockerfile file pattern regex to match spec

The test details page in the Flaky Tests dashboard is getting a major UX overhaul.

User interviews and feedback from beta users helped us remove metrics that were not useful and update visualizations to better show a test’s health over time. The tabbed views are also gone, which means you can get all the information about a test’s health in a single, unified dashboard, including: 

• A test’s current status.

• When a test was marked as flaky and quarantined.

• Test flake and failure rates on mainline and PR branches.

• Overall test health history.

• Grouped unique test failure reasons.

These new charts break down how often a test is successful, failing, or quarantined for CI jobs for your mainline branch and PRs and show test health changes over time.

A more eventful test history timeline

The test history timeline has been moved to the main page, and updates, including when a test was marked as flaky, quarantined, or fixed, are always visible. Teams using the Jira integration to track test flakes will also see any updates to a flaky test’s ticket reflected in the timeline.

You can still override the repo's default quarantine setting and always quarantine or never quarantine a test, but now you can leave a comment for other devs that shows up in the timeline.

For example, you might want to let your team know that the “Test is important - flakes must be fixed!” when you override the repo default to never quarantine a test:

The table shows all the unique reasons the test fails in CI.  Trunk uses AI to auto-group unique failures so multiple runs will be grouped under a single failure reason. Individual runs, including stack traces, branch, commit, and job information, are available when a row in the table is expanded. This means all the information you need to debug and fix flaky or broken tests is available in a single place.

Have feedback for us? Reach out on our community Slack.

A new Quarantined Tests API is now available to all Flaky Tests users. This API fetches a list of currently quarantined tests for a given repo, allowing organizations to implement custom workflows to review and fix quarantined tests.

1// https://api.trunk.io/v1/flaky-tests/list-quarantined-tests
2// a sample response with a single quarantined flaky test
3{
4  "quarantined_tests": [
5    {
6      "name": "sums some nums > subtracts 1 from 2 to equal 1",
7      "parent": "app/utils/subtract.test.ts",
8      "file": null,
9      "classname": "app/utils/subtract.test.ts",
10      "status": "FLAKY",
11      "codeowners": [],
12      "quarantine_setting": "AUTO_QUARANTINE",
13      "status_last_updated_at": "2025-02-13T15:52:03.000Z",
14      "test_case_id": "48cd26cb-6333-528a-919f-ed597ee11715"
15     }
16   ],
17   "page": {
18     "total_rows": 1,
19     "total_pages": 1,
20     "next_page_token": "",
21     "prev_page_token": "nUj35JF5aNJFjUkIakxff89f1b...",
22     "last_page_token": "bVUNwekgkHadgZfSgwuac75640...",
23     "page_index": 0
24   }
25}

Dev teams are using the Quarantined Tests API to do things such as:

• Automatically skip quarantined tests when testing locally

• Check for any tests that should be manually un-quarantined

• Determine what teams, projects, or modules have the most quarantined tests

• Assign engineers to fix certain quarantined tests, based on recent code changes

• Send Slack or email notifications to let teams know when the number of quarantined tests reaches a threshold

See the API reference for more information on required headers and payload.

Trunk now supports single sign-on (SSO) for enterprise plan users. SSO allows your team to access Trunk with their work email account through your organization's identity and access management solutions, making it easier to manage your Trunk organization.

SSO is only available for enterprise plan users. If you're interested in configuring SSO for your team, contact us by email or Slack. We will reach out to help you configure SSO for your organization.

We have an exciting update for teams using the Flaky Test Jira integration to create and track their tickets.

Previously, the Jira integration only tracked ticket creation in each test's timeline. In this update, new events are added to the timeline for ticket assignments and status changes.

As your team continues to integrate Trunk into your workflow, Jira integration events combined with detection events in the timeline help tell a more complete story about the status of a flaky test. You can more easily understand if a flaky test is being tackled by an engineer and if their fix was effective during reviews.

Note: There can be a delay of up to 15 minutes between changes being made in Jira and the events being reflected in the timeline.

See the new timeline events in Trunk or reach out to us for discussions on Slack.

If your stable branch is not main, you can now set a custom stable branch for your repositories to improve flaky test detection.

Previously, Flaky Tests assumed that main was a repository's stable branch. This would lead to subpar flaky test detection if a different branch, master or develop for example, is defined as the default stable branch for your workflows.

Now you can override these stable branch defaults to better match your organization's custom CI workflows.

Learn how to customize your stable branch in the Trunk docs.

We're excited to introduce a new way to help your team stay on top of your repository's test health through weekly email reports.

With automatic detection and quarantining, Trunk can do a great job of mitigating the impact of test flakiness on your team's velocity. But we don't want you to forget about your flaky tests. Even if they're no longer causing friction, they should still be fixed eventually.

To help you stay on top of your repo's test health, we created a weekly digest for your flaky tests. Here's what you can expect to find in each report.

Key Metrics

At the top of each report, you'll see an overview of important metrics on flaky tests and their impact on your organization. You'll see the number of broken and flaky tests, how many PRs they block, and an estimate of engineering hours lost due to these flaky tests.

We also include the changes week to week. We believe it's important to keep track of week-to-week changes to understand if the flaky tests problem is controlled or if it's spreading. This helps you decide whether more resources should be devoted to fixing and controlling flaky tests. It can also alert against larger issues where a bug in code or infrastructure can cause a spike in test flakiness.

Per-Repo Breakdown

If your org is tracking flaky tests across many repos, the weekly report will break down the key metrics for each repo. This helps you see which repos are doing well and which might need extra love.

Frequently Failing Tests

Each report has a dedicated section for frequently failing tests. This alerts against high-impact broken and flaky tests. You can use this list to catch newly introduced high-impact flaky tests or get an overview of the flaky tests you should be investigating next with your team.

Important Action Items

Toolrot can be a real issue with the sheer scale of infrastructure required for modern CI. The weekly email reports will remind you if your CI is misconfigured or if there's a new version of the analytics-uploader tool you should upgrade to. All of these help ensure your detection remains accurate and effective.

Who Gets These Reports?

All org admins will get the emails by default as long as you've uploaded test results to us in the past week. The emails are delivered each Monday, and we go over the stats over the last week. You should see these emails in your mailbox starting next week.

If you've got any questions about these email reports or would like to suggest some new additions, please feel free to reach out to us on Slack.

We're excited to introduce webhook integrations for Slack, Microsoft Teams, GitHub Issues, and Linear. Webhook integrations make it easier to start automating notifications and ticket creation by providing tighter integrations and example transformations.

You can use the provided transformation to set up working notification messages and tickets immediately and customize the transformation for each webhook as you go. This means they're quicker to boot without sacrificing customizability.

Learn to set up webhooks and see examples in the Trunk docs