Why Retool stopped trying to fix flaky tests in-house

Retool's DevEx team owns CI and the pager. The rule is simple: three consecutive failed builds on main and someone gets woken up.

For a while, the team was getting woken up a lot.

“We were getting paged two out of every three on-call days,” says Atsushi Yamamoto, Software Engineer on Retool's DevEx team. “Because our CI was filled with flaky tests.”

Flaky tests pass or fail at random, even when nothing in the code has changed. Most of those pages weren't real problems. The code was fine, but the tests were unreliable, and with 150,000 of them, three random failures in a row was only a matter of time.

When Atsushi started asking why the pages kept coming, he found that everyone already knew about the problem. In fact, the team had tried to solve it three separate times.

The first attempt was a Jest wrapper. Engineers could tag a test as flaky in the code, and an environment variable would decide whether it ran. Behind it sat an S3 file listing the known flaky tests. It sounded reasonable until Atsushi opened the file.

“The system was unmaintained and covered only about 1% of our tests.”

The second attempt automated detection with a Buildkite pipeline. When a test failed on main, the pipeline re-ran it ten times to confirm it was flaky, then posted it to a public Slack channel. Finding flaky tests worked, but getting them fixed didn't. DevEx still had to work out who owned the test and ask them to fix it.

“I noticed the same flaky tests were being pinged over and over across quarters.”

Even when an owner did respond, the fix was slow. Marking a test as flaky meant waiting through several CI runs. “That would take 40 minutes just to manage the CI errors.”

The third attempt was the most ambitious. A separate database tracked how every test performed. When a build failed, a Buildkite annotation pointed to the likely cause, something like: this test also failed in the last 25 builds, so it's probably flaky.

It was closer. But the signals weren't accurate enough to act on, and the team felt the design wasn't the right fit for the problem.

Each of the three systems solved a slice of the problem, but not all of it.

Looking back at all three attempts, Atsushi could describe the missing piece precisely. First, each test's flaky status needed to live in its own fast, dedicated data store, separate from the test code. Second, quarantine had to happen at the CLI layer in CI. It couldn't depend on annotations written into the tests themselves.

Building it was on the table. The team had capable coding agents, and a flaky test detector isn't hard to write. But Atsushi listed out what a real one would need: a CLI compatible with four different test runners, a data ingestion pipeline to collect tests from every build, a detection rule set to decide which tests are actually flaky, and a UI that engineers would actually use.

“We already had three failed attempts at managing this. I don't think we have the capacity to do this.”

The shortlist had three names on it. One was a large observability vendor with a flaky test feature, and it was out fast: no real Bazel support. The second was the test product from a CI vendor Retool already used, which made it the natural choice on paper because the integration would be native. But it also lacked Bazel support, the CLI ergonomics weren't there, and the pilot left the team unconvinced. The Trunk pilot went the other way. Getting test data flowing was a matter of pointing Trunk at the JUnit XML output, and issues raised in the shared Slack channel got fixed quickly.

If you ask Atsushi where flaky test management sits on his list of problems now, the answer is short.

“

I consider flaky test management a solved problem, thanks to Trunk.

Atsushi Yamamoto, Software Engineer, DevEx, Retool

The pager is quieter, CI is stable, and the 40-minute process of marking a test as flaky is gone. “Now I click a button, it's quarantined, and it can be triaged async.” The flaky test stops failing builds right away, and its owner can look into it later without holding anyone up.

Routing got fixed too, and DevEx no longer has to track down who owns each test. Trunk reports the GitHub code owner for every flaky test, and Retool maintains a mapping from each code owner to their Slack on-call handle. A small service built on the Trunk API then sends each new flake straight to the right on-call engineer.

“The tests are being triaged within the same day at least, if not within a few hours or minutes.”

Test history turned out to matter as much as quarantine. When Retool launched a new product, that team ran its end-to-end tests through Trunk from day one. A few of those tests looked flaky, and the team assumed it was a testing problem. It wasn't.

“They traced it to a race condition in the product.”

Before Trunk, a failure like that would have been re-run until it passed, and the bug would have stayed hidden. Retool has since added its staging end-to-end suite to Trunk as well. Trunk takes care of detection, quarantine, and test history, and Retool's engineers get to trust their tests again. When a test fails now, it's worth looking into, and sometimes, like with the race condition, it leads to a real fix in the product. Atsushi's team doesn't have to maintain more homegrown tooling, so they can spend their time making Retool better for its customers.