There has been a recent surge of interest in merge queues, partially thanks to GitHub’s announcement of the general availability of their merge queue product. This increased interest is a good thing! Many development teams could benefit from integrating merge queues with their CI pipeline.
If you’ve worked on a large project or monorepo, you’ve likely experienced some common CI pain points. Broken builds, logical merge conflicts, and outdated pull requests are terms you’re all too familiar with. A merge queue might help fix these issues and get you shipping faster.
In this article, we’ll cover everything you need to know about a merge queue. If you're less interested in the basics and want to jump straight into implementation, skip to the end of this article.
Why Do You Need a Merge Queue?
You probably don’t need a merge queue if you’re an individual developer in a simple codebase or a small team working on a single project.
However, merge queues are a great fit for companies that build multiple projects using a monorepo or have teams working on large codebases.
Deciding whether you need a merge queue depends on your unique situation. Organizations often reach for merge queues when:
There is a high volume of merges into
mainso PRs are out of date within a few hours.These out-of-date PRs are not tested on the latest code from
main, which results in conflicts between PRs not being caught, andmainis often broken.Failures on main block feature releases and bug fixes from being deployed, and branches created from this broken state must be manually fixed.
Developers need to rebase branches and existing PRs once
mainis fixed, which causes another delay in CI and takes engineering time.Manual rebasing becomes the norm and is a hassle for engineers to keep up with due to the high volume of changes.
Developer productivity can grind to a halt if you get caught in this cycle of manual rebases, and it can be increasingly difficult to deliver new features and bug fixes to end users.
A merge queue helps solve these problems by always running CI with the most up-to-date code from main and any additional PRs at the head of the queue. Developers no longer need to manually rebase their branches, and failures are kept out of main.
Production Example: Uber uses a Merge Queue for their Monorepo
Uber uses a merge queue to handle PR merging for their monorepo. They built an in-house merge queue to protect their main branch from failures while ensuring high CI throughput.
Uber’s monorepo could see thousands of commits per day, and it might take hours to perform all build steps. Keeping main green was imperative for Uber to be able to work and merge at scale. Prior to implementing their merge queue, their iOS mainline was only green for 52% of the week!
After switching to a merge queue, 92% Uber engineers responding to a survey said that keeping mainline green was a positive change, and 94% of respondents said the relative performance of the merge queue was on par or significantly better than prior experiences.
You can read more details about Uber’s merge queue in their paper: Keeping Master Green at Scale.
What is a Merge Queue?
A merge queue is a tool that automates PR merges into your repository's main branch. They are a best practice for trunk-based development in repos with 10 to +1000 active engineers. Merge queues guarantee that your main branch is never broken. This means that changes resulting in a failing CI pipeline run such as a service not functioning correctly, segfaults, compilation errors, failing tests, or anything else that could fail a build will not be merged into main.
There are two common ways your default branches "break":
A PR was branched off of on an old commit of main, and when merged into main it no longer functions with the latest state of the repo.
Two (or more) PRs recently merged and worked independently, but combining their changes breaks something.
The higher a repo’s PR velocity, the more issues that arise. These issues are called logical merge conflicts.
A merge queue handles this problem by testing PRs queued to be merged in combination with each other based on the latest main. This means developers don’t need to rebase branches on main to test with the latest changes because the merge queue handles this automatically.
For example:
PR1 updates function
foo()and is merged intomain.PR2 makes changes to function
foo()is opened without rebasing onmain.
A merge queue prevents any conflicts or failures from landing on main by testing PR2 with the most recent code from main and including the changes to foo() from PR1.
Watch how predictive testing works in a merge queue:
Merge queues quietly manage the chaos of code changes that often come up in large teams or monorepos. They're all about organizing the flood of contributions into a neat, orderly process that doesn't trip over itself.
Managing a Merge Queue with Trunk Merge Queue
Trunk Merge Queue is a sophisticated merge queue that prevents broken builds on your main branch without sacrificing developer velocity. It ensures that every PR is tested against the latest commit on your main branch so you’ll never have to worry about the freshness of your CI results. Merge Queue is both a GitHub bot and a web app, and works with any CI provider so long as your repo is hosted on GitHub.
Merge Queue uses a combination of predictive testing and optimistic merging to speed up your CI process and help deal with flaky tests. Predictive testing means that an enqueued PRs includes all of the changes from PRs ahead of it in the queue. This means you are always testing on the latest code. Optimistic merging builds on predictive testing. Merge Queue tests all PRs in a queue concurrently - if a PR at the back of the queue is ready to merge before PRs ahead of it are finished, all PRs will be marked as ready to merge.
For example:
PR1 is added to the merge queue.
PR2 is added to the merge queue behind PR1.
The CI job for PR2 is successful before PR1 is finished.
Both PR1 and PR2 are merged (optimistically!) because PR2 was tested against main + PR1 + PR2 thanks to predictive testing.
See how optimistic merging improves CI throughput on a merge queue:
Merge Queue can also be configured to use batching to test multiple enqueued PRs as a single unit. This helps reduce an organization’s total number of CI runs and CI throughput can be improved by up to 90%.
Batching can be tailored to suit your organization by using either a target batch size or a wait time to determine when to run a batched CI workflow. CI failures are handled automatically: the merge queue splits apart the batch and retests sub-batches until the failing PR is discovered.
For example:
PRs 1, 2, and 3 are all added to the merge queue within the 5-minute batch window
A single CI workflow is kicked off with changes from all of these PRs.
Once this workflow is successful, PRs 1, 2, and 3 will all be merged.
And here's a video that explains batching:
Batching also combines with optimistic merging so a batch further back in the queue that passes will result in batches ahead of it in the queue being merged.
Handling Flaky Tests in a Merge Queue
Flaky tests kill the productivity gains of merge queues. Without flaky test handling, every PR behind a failed PR must be re-tested after the failure is ejected from the queue, taking up time and delaying code from being merged.
Merge Queue uses a configurable pending failure depth value in an attempt to avoid these re-runs.
For example, a pending failure depth of 2 means that a failed PR will wait for the next 2 PRs in the queue to finish running before it is kicked out of the queue. If a test fails in a PR but then passes in one of the next 2 PRs, the failure is considered transient and all of these PRs will be successfully merged. These transient failures like test flakes or disconnected CI runners are mitigated without manual intervention.
Watch how pending failure depth helps avoid the negative impacts of flaky tests on merge queues:
If you have a bunch of flaky tests slowing down CI, you can also look into Flaky Tests. Flaky Tests will automatically detect and quarantine test flakes so developer time isn’t spent re-running CI and hunting down non-deterministic errors. Bonus: Flaky Tests works great alongside Merge Queue!
Speeding up Merge Queue with Parallel Queues
A common drawback to merge queues is that teams often sacrifice velocity as PRs wait in the queue. This isn’t an issue with Merge Queue as we have two queue modes you can choose from depending on how you’d like to prioritize velocity: Single and Parallel mode.
In Single mode, Merge Queue acts like a typical queue: first in, first out. All PRs are tested and merged on a single queue in the order they arrive. This is a good way to start using Merge Queue for your repo.
Parallel queues are a great way to scale monorepos to meet the needs of tens, hundreds, or thousands of developers. In Parallel mode, Trunk uses information from build systems like Bazel and Nx, or a custom glob file matching pattern, to understand which PRs contain related changes. This information helps Trunk define the impacted targets for each PR, which are then used to group PRs in separate queues that can run in parallel. PRs that impact one another will be run on the same queue, and unrelated changes can be run in parallel on a different queue.
For example:
I am working in an Nx monorepo with two projects, a backend and a frontend.
PR1 is enqueued and makes changes to the frontend.
PR2 is enqueued and makes changes to the backend.
PR3 is enqueued and also makes changes to the frontend.
PR1 and PR3 are added to the same merge queue, while PR2 can be put in a separate queue because the changes are independent of code changes in PRs 1 and 3.
Here is a video explaining how parallel queues work in a merge queue:
These Merge Queue optimizations work together out of the box and are configurable to suit your organization’s workflow.
Setting up a Merge Queue with Trunk Merge Queue
Setting up your first merge queue with Trunk Merge is relatively straightforward. Feel free to start using the guidelines below for setup or navigate to our docs.
Connect your Trunk Organization to GitHub
First, sign up at app.trunk.io and create a Trunk organization. Then connect your organization to your GitHub Repositories. Select the repository you would like to use and click Get Started.
Setup Trunk Merge Queue
Go to the Merge Queue tab. You will see the Set Up Trunk Merge Queue page. From there, you can specify the name of the branch that Merge Queue should help merge PRs into, for example, main.
That should be all you need to do to get started. Now you can open up a PR in GitHub to see your merge queue in action.
Submit Pull Requests
Try making a simple change on a branch and submit it as PR in GitHub.
Now trigger Merge Queue to process this PR by doing one of the following:
Adding a
/trunk-mergecomment on the PR in GitHubInteracting with the GitHub bot in a PR by checking the provided merge box
Using the Trunk CLI
trunk mergecommand
Read the branch protection docs if you have problems adding PRs to the merge queue.
Pull Request Processing
When a PR is submitted to the merge queue, it will be labeled as Not Ready until all required submission conditions are met. These conditions are configurable and often require the CI jobs to pass and approval from a reviewer on the PR.
Once these conditions are met, the merge queue will pick up the changes and run tests. Tests passing doesn’t mean this PR is merged right away, the PR may still need to wait for PRs upstream in the queue to finish testing.
After any upstream PRs have finished, the PR will merge and be removed from the merge queue. If a PR fails or is canceled it will go to the failed or canceled state and be evicted from the queue.
Configuring your Merge Queue
Once you have set up and tested your merge queue, you might want to configure the queue to better match your CI workflow and requirements.
This includes modifying settings such as the Single or Parallel mode, or modifying the concurrency for parallel merge queues, all of which can be found in your repository settings in Trunk.
Define Required Status Checks For Testing
Trunk needs to know which status checks must pass while testing pull requests in the queue before it can merge a PR into your branch. Merge can pick up this list of required statuses in one of two ways:
From the list of Require status checks to pass before merging specified in the GitHub branch protection rule for your merge branch. This is particularly useful if you want Merge to track the same status checks that must pass before GitHub can merge a PR.
From the
merge.required_statusessection of your.trunk/trunk.yaml. Choose this setup if you want the list of status checks that must pass before Merge can merge a PR to be different from the required status checks that must pass as a part of GitHub branch protection.
To specify using the .trunk/trunk.yaml file, set the merge.required_statuses to the name(s) of the GitHub status checks or jobs that must pass:
1version: 0.12cli:3 version: 1.16.04merge:5 required_statuses:6 - Trunk Check7 - Unit tests & test coverage8 # Add more required statuses here9
To use GitHub branch protection instead follow GitHub's instructions for requiring status checks.
Configure a CI workflow to run with Trunk Merge Queue
Merge Queue creates branches with the prefix trunk-merge/ to test PRs. To ensure the required status checks that gate Merge Queue get triggered when PRs are tested, your CI provider must be configured to run these checks whenever a branch with the `trunk-merge/ prefix is pushed to.
For GitHub Actions, that'll mean setting up a push-triggered workflow, filtered to trunk-merge/** branches, like so:
1name: Run Required Checks2run-name: PR Checks for ${{ github.ref_name }}34# Trigger jobs whenever Merge Queue tests a PR using a `trunk-merge/` branch5on:6 push:7 branches:8 - trunk-merge/**910jobs:11 trunk_check:12 runs-on: ubuntu-latest13 # "Trunk Check" is specified in merge.required_status above14 name: Trunk Check15 steps:16 - name: Checkout17 uses: actions/checkout@v31819 unit_tests:20 runs-on: ubuntu-latest21 # "Unit tests & test coverage" is specified in merge.required_status above22 name: Unit tests & test coverage23 steps:24 - name: Checkout25 uses: actions/checkout@v32627 # Add more steps here..
Now you are ready to submit your first PR.
Next Steps for your Merge Queue
Trunk Merge Queue is now set up with your repo. Whenever a PR is pushed to your merge branch it will be safely tested and automatically merged when all tests pass, regardless of the order PRs are pushed.
Schedule a demo with the Trunk team if you want to learn more about Merge Queue and how it could improve CI for your dev organization. Because Trunk Merge Queue does not affect normal merging, you can trial it with zero impact on your production CI system.
You can also read how Merge Queue helps the Faire engineering team, which is more than 350 people strong, avoid merge issues, saving over 18 engineering hours daily.
