Trunk vs Gradle
Trunk’s Flaky Tests is a comprehensive platform designed to detect, analyze, and manage flaky tests within your CI/CD pipeline.
Tired of clicking re-run? They were too.
Features
Notifications (GitHub, Slack, Email, etc.)
Test Framework and CI Provider Agnostic
Test Quarantining
Auto Detection
Comprehensive Dashboard
Integrated Ticketing
Detailed Failure Analysis
"Trunk's Flaky Test solution is so far the best one we've worked with, and we look forward to continuing to work with it."
"I primarily focused on the flaky tests tab and found all the information I sought. The app provided an excellent summary of our E2E pain points."
Security Overview
Your code is your IP, that’s why security and privacy are core to our design. We minimize data collection, storage, and access whenever possible. We operate using the principle of least privilege at all levels of our product and processes.
FAQs
How does Gradle handle flaky tests?
Gradle primarily handles flaky test detection through its Develocity product. The platform monitors test executions across builds and identifies tests that produce inconsistent results. When a test passes in some runs but fails in others without code changes, Develocity flags it as potentially flaky. While Develocity provides good visualization of flaky test patterns, Trunk Flaky Tests offers more robust flaky test detection with additional integration capabilities across various CI providers and testing frameworks, not limited to the Gradle ecosystem.
What tools does Develocity offer for flaky test detection?
Develocity (formerly Gradle Enterprise) provides several tools for flaky test detection, including build scans that visualize test execution patterns and a dashboard to track flaky tests over time. Their system uses historical test data to identify inconsistent test behavior and offers different selection profiles to balance between test thoroughness and speed. Trunk Flaky Tests takes a different approach by focusing on real-time detection and automatic test quarantining solutions that prevent flaky tests from blocking CI pipelines while teams work on resolving the underlying issues.
How can I mitigate flaky tests using Gradle?
With Gradle and Develocity, you can mitigate flaky tests by using their test retry mechanism, which automatically reruns failed tests to determine if the failure is consistent. You can also leverage their detailed test analytics to identify patterns in test failures. For more comprehensive mitigation, Trunk Flaky Tests offers additional capabilities like automated quarantining workflows and direct integration with issue tracking systems through Jira integration for flaky tests, allowing teams to track and prioritize fixes more efficiently.
What plugins are available for dealing with flaky tests in Gradle?
Gradle offers the test retry plugin, which allows tests to be rerun multiple times if they fail. There's also the build scan plugin that helps track test execution history. For teams using JUnit, plugins like junit-retry or testng-retry can be configured within the Gradle environment. Trunk Flaky Tests doesn't require specific plugins since it works at the CI level rather than the build tool level, making it framework-agnostic. It connects directly to your existing test infrastructure through the flaky test uploader, supporting various testing frameworks regardless of build system.
Are there best practices for managing flaky tests in Gradle?
Yes, some best practices for managing flaky tests in Gradle include regularly analyzing build scans to identify patterns, implementing automatic retries for potentially flaky tests, and prioritizing fixes based on impact. It's also recommended to tag known flaky tests to prevent them from blocking critical workflows. Trunk expands on these practices by recommending automated notification systems through Slack integration via webhooks and implementing quarantine mechanisms that isolate flaky tests while maintaining visibility into their status.
How does Trunk Flaky Tests differ from Develocity's flaky test management?
Trunk Flaky Tests differentiates itself by offering platform-agnostic flaky test detection that works across multiple CI systems and testing frameworks, not just Gradle-based ones. While Develocity excels at deep Gradle integration and build analytics, Trunk focuses on seamless workflow integration with development tools like GitHub pull request comments and automated issue creation. Trunk also provides more extensive automation options for quarantining and tracking flaky tests, making it particularly valuable for teams that prioritize maintaining CI pipeline efficiency.
What metrics should I track to understand the impact of flaky tests?
The most important metrics to track include flakiness rate (how often a test flips between pass and fail), the number of blocked pull requests due to flaky tests, and time wasted on false failures. You should also monitor the mean time to resolution for flaky tests and the distribution of flakiness across test suites. Trunk Flaky Tests automatically captures these metrics and provides visualization tools that help teams understand the real cost of flakiness, while also enabling proactive alerts through integrations like GitHub issues integration when critical patterns emerge.
What is Gradle's approach to predictive test selection?
Develocity implements predictive test selection to reduce CI times by intelligently choosing which tests to run based on code changes. Their algorithm analyzes historical data to predict which tests are most likely to fail for a given code change. While effective for reducing test execution time, this approach differs from Trunk's focus on flaky test management. Trunk prioritizes reliable test execution by identifying and isolating flaky tests, ensuring that CI pipelines aren't blocked by non-deterministic failures rather than skipping potentially relevant tests.
How do I set up automated notifications for flaky tests in Gradle vs. Trunk?
In Gradle with Develocity, you can set up notifications by configuring their event API to trigger alerts based on test failure patterns. However, this typically requires custom implementation work. Trunk Flaky Tests offers more out-of-the-box notification options through direct integrations with communication tools like Slack integration via webhooks and development platforms like GitHub. Trunk's approach reduces the configuration burden while providing more contextual information in the notifications.
What is test quarantining and how do Gradle and Trunk implement it?
Test quarantining is the practice of isolating identified flaky tests to prevent them from disrupting development workflows while they await fixes. Gradle with Develocity supports manual flagging of flaky tests, but lacks comprehensive quarantine automation. Trunk Flaky Tests offers more sophisticated test quarantining solutions with automatic detection, isolation, and tracking features. Trunk's quarantine system maintains visibility into quarantined tests while preventing them from causing false failures in CI pipelines.
How do I interpret test failure patterns to identify flakiness?
To identify flaky tests, look for tests that fail intermittently without code changes, tests that pass when run individually but fail in a suite, or tests with timing-related failures. Develocity helps visualize these patterns through build scans, while Trunk Flaky Tests automates this analysis with machine learning algorithms that identify statistical patterns of flakiness. Trunk's approach reduces the manual analysis burden by automatically flagging tests with suspicious failure patterns and providing detailed context for each flagged test.
What strategies do Develocity and Trunk recommend for fixing flaky tests?
Both Develocity and Trunk recommend similar strategies for fixing flaky tests: isolate environmental dependencies, improve asynchronous handling, eliminate test interdependencies, and add appropriate waits for UI tests. The difference lies in how they help implement these strategies. Develocity provides visibility into the issues, while Trunk Flaky Tests adds workflow automation through GitHub pull request comments and issue-tracking integration that streamlines the process of assigning and tracking fixes for flaky tests.
