How a three-person QA team at TheyDo cut flaky failures 3X
Up to 40% of E2E runs were failing, and nobody could tell flakes from real bugs. With Trunk, 94% of failing CI jobs now go green, and real bugs still block the merge.
“A couple of steps in a single pull request, and you can see the results. No test refactoring, no custom framework.
A Gate Nobody Trusted
TheyDo deploys as soon as code merges, so the tests on a pull request are the last check before production. Oleksii Ivashchenko, Engineering Manager, Quality Platform, runs the three-person QA team that builds the testing tools TheyDo's product teams rely on. “Our quality gates have to be required, and they have to block,” he says.
Most of those gates are Playwright end-to-end tests, and Oleksii has been in QA long enough to know they will never be perfectly stable. “End-to-end tests are flaky by nature, whatever tool you use,” he says. A flaky test is one that passes or fails at random, even when the code hasn't changed. When TheyDo first talked to Trunk, up to 40% of its E2E runs were failing. “Some were real bugs, some were flaky tests, and it was hard to tell which.”
Every rerun added another 10 to 13 minutes to a pull request, but the time wasn't what worried him most. “Every flaky failure eroded our developers' trust,” he says. “Even when a failure was a real bug, people tend to rerun it a few times before taking it seriously.”
One PR for the Integration Is All It Took
After TheyDo's compliance team finished its security review, one QA automation engineer set Trunk up by adding a JUnit XML reporter for Playwright and an upload step in GitHub Actions. Some tools ask for a round of test refactoring or a custom framework before a proof of concept can even start, so the fact that Trunk fit into a single pull request mattered.
Within six weeks Trunk was quarantining about 135 flaky test runs a day, and by the end of the POC it had saved TheyDo roughly 180 hours of CI time that would otherwise have gone to reruns.
The clearest proof came by accident. Between the end of the POC and the signed contract, TheyDo briefly ran without Trunk, and every flaky test started failing builds again. “It gave us a clean before-and-after,” Oleksii says. “Going without Trunk showed us what it was worth.”
Results
Only about 2% of TheyDo's tests are flaky, but over the last 30 days those tests accounted for 91% of all failed test runs, and Trunk quarantined them. That changes what a red build means. In 94% of the CI jobs that hit a failure, the only failures came from tests Trunk already knew were flaky, so those jobs now pass. A failure in any other test still blocks the merge.
“Usually Trunk picks it up: same commit, different results, quarantine,” Oleksii says. By his count, flaky failures are down 3X.
How the Team Runs It
“Trunk is a painkiller,” Oleksii says. “It takes the pain away, but someone still has to fix the tests.”
His QA team owns that part. When a quarantined test needs attention, they open a Linear ticket through Trunk's integration with the failure data attached, and if a test needs fixing a second time, they either rewrite it from scratch or move the check down to a faster unit or integration test.
“We're happy with it, and it's saving us time.
