Keeping CI Green with

Keeping CI Green with

Trunk Merge Queue

Faire

Faire is the leading online wholesale marketplace connecting local retailers with independent brands globally. Founded in 2017, they welcomed 26,000 new brands and 160,000 new retailers in 2021 alone.

Faire's main branch was failing 20% of the time. Not from bad code. From what their team calls "green-green conflicts."

Here's how it happens: Developer A merges a commit that uses library Alpha. Developer B merges a commit that deletes library Alpha. Git sees no conflict because neither touched the same files. Both PRs passed CI. Both merged. Main breaks.

With 350+ engineers shipping hundreds of changes daily across multiple monorepos, this wasn't an edge case. It was constant.

The blast radius extended far beyond the engineers who couldn't merge. Anyone pulling from main had a 23% chance of getting broken code. They wouldn't discover it until running tests locally, then waste time figuring out if the failure was their code or a false positive from upstream.

Slack channels lit up. Deployments stalled. Each incident took roughly 2 hours to resolve: investigate, revert or fix-forward, run CI, merge the fix.

• 350+ engineers across backend, iOS, Android, and frontend repos

• 20% failure rate on main from green-green conflicts

• ~20 conflicts caught per month after implementing merge queue

• 2 hours average resolution time per incident (before)

• 23% of the time, engineers were pulling broken main

The "green branch" system: A safety-net branch that only accepted changes after CI passed. Engineers would start work from green branch but PR against main.

It helped, but pain points remained:

• Main could still break when changes collided

• Deployment cadence slowed

• Developers couldn't build on the latest code

• Main didn't emit artifacts like dependencies for external workstreams

• Every break still required manual intervention

Building their own merge queue: The team considered it, but the maintenance math didn't work.

GitHub Merge Queue

Seemed easier to implement initially. Support required opening tickets that took days to resolve. Not viable for a critical workflow.

Trunk

Similar setup to GitHub's, but more stable. Direct Slack channel with real-time support.

Faire didn't just turn on the merge queue. They tuned it for their scale.

1. Streamlined CI validation Running the full PR test suite in the queue would've created massive backlogs. They selected a focused subset of critical validations to keep queue times fast while maintaining quality. Occasionally an issue slips through, but far less frequently than the problems before the merge queue.

2. Parallel processing In a monorepo where service changes are often independent, sequential queuing is wasteful. Faire configured parallel mode: PRs report which services they touch during CI, then only queue behind other PRs that affect the same code surface area.

Example: If commits A through G are queued and only some share code surface area, unrelated commits can merge without waiting for the full queue.

They couldn't find a tool that does this for gradle, so they built their own and open-sourced it.

3. Chrome extension for GitHub Direct merges to main are disabled. Instead of relying on "/trunk merge" comments or checkboxes, they added a Chrome extension that replaces GitHub's default merge button with "Merge with Trunk." Same muscle memory, new workflow.

18.8 hours of engineering time saved per day by catching conflicts before they land.

"Trunk has saved us 330 hours by preventing merge issues over the last 35 days. That's 9.4 hours of engineering productivity saved per day. It's been great." — George Jacob, Senior Software Engineer

The failure rate dropped from 20% to a level the team controls based on their own speed/quality tradeoffs.

Merge queues aren't free. Faire's been running Trunk for two years and notes these realities:

• A stuck queue hurts. If a CI build never starts due to an infrastructure issue, a merge queue that doesn't track how long a commit has been waiting will cause all downstream commits to stall.

• Queue length matters. Repos with high code delivery that don't use parallel processing or batching can have very long queues, slowing delivery to main.

• It's another thing to maintain. A separate CI pipeline for the queue adds cognitive load to your platform team.

• Merge times increase. Variable based on queue depth and CI duration.

Their take: "Choosing to implement a merge queue is about trading one set of problems for another. But in our experience, the problems that come with a merge queue are far more manageable than dealing with a constantly broken main branch."

George's rule of thumb: "If your main branch goes red even once a day, you should definitely consider a merge queue."

The rollout pitch to each team was simple: "When I said there would be no more logical conflict issues, people were sold. There weren't many questions after that."