If you’d like to try out our new flaky tests solution, sign up here.
Software Tests
Here at Trunk we love software tests. Sometimes we’ll even write the tests before the code! What we don’t love are flaky tests. If a test passes, great, we know the code is good. If a test fails, great, we know where to fix the code. However, flaky tests are the worst of both worlds. We don’t know where to fix it, and we don’t even know if the bug is real or a result of something else in the build system breaking. Maybe it was just a network issue? Maybe Chrome needs too much RAM.
Heisenbugs
Flaky tests are tests that sometimes pass and sometimes fail, even with no code changes. They are the non-deterministic Heisenbugs. They are evil tests that will wake you up in the middle of the night when the build breaks and your fellow engineer says: but I didn’t even change anything near that code. That’s when you know you are being haunted by flaky tests.
The insidious part of flaky tests is that they waste build time as well as developer time. Even if it’s just a few minutes a day, this adds up and begs for automated solutions. In the cases where the flaky tests represents an actual intermittent failure, it’s hard to distinguish between these and the more common flaky tests, so your developer get alert fatigue and end up leaving nasty bugs in your product.
Big companies (at least the ones who care about code quality), have built internal systems to deal with Flaky Tests. These systems run the tests multiple times to see if it is a persistent failure or not, and can quarantine the tests (make them not required for merging), until a human can look at the problem. Some systems even automatically file tickets in the bug tracker to tell an engineer to fix it.
DIY systems for handling flaky tests can be great, but not every company has the luxury of dedicating a team to just build and maintain them. What should you do if you are a medium sized company that can’t afford to have a dedicated tools team? Or worse yet, a scrappy startup that can’t afford to have development slowdown due to testing problems. In that case we go to Plan-B: The build police.
The Build Police
You’ve probably been deputized to join the build police before. You’re on call when the build breaks. You have to dig through logs to find the offending PR. If this is a test that has failed before, you’re most likely going to bang it on the side real hard and stick it back in the queue. And you know what? This works! For a while anyway. After all, that’s why we all do it. But it doesn’t scale. We need a better solution.
A Better Way
We’ve all faced this problem, so that’s why Trunk is building a tool to conquer flaky tests once and for all. You’ll get all of the features of the big guys internal systems, without all the headache of managing it, and without the cost of a dedicated team. You’ll be able to:
Autodetect the flaky tests in your build system.
See them in a dashboard across all your repos.
Quarantine tests with one click, or automatically.
Get detailed stats to target the root cause of the problem.
Get reports weekly, nightly, or instantly sent right to email and slack.
Intelligently file tickets to the right engineer.
Most importantly, it will work with the testing and ticketing systems you are already using; giving you and your team all of the details you need to go in and actually fix the damn things, then get back to real work. We are working hard building our new Flaky Tests solution with our beta testers.
If you’d like to get in early, sign up here.
