Built with:Built with: Glydways
Accelerating Development at Glydways: How Trunk Merge Queue Streamlined Their Workflow
Accelerating Development at Glydways: How Trunk Merge Queue Streamlined Their Workflow
Long wait times for merging changes into the main branch.
High CI costs and delays caused by outdated branches needing retesting.
Developer productivity bottlenecks due to limited CI resources.
Implemented Trunk Merge Queue for faster and automated merging into the main branch.
Improved CI resource allocation and utilization.
Introduced priority merging for critical changes, streamlining workflows.
Increased daily commit volume from 24 to 32 during busy periods.
Reduced queue time to an average of 20 minutes.
Enhanced developer satisfaction with noticeable improvements in merge speed.
"I heard some comments... a week after we enabled it, and they basically told me, 'Man, you got that just in time. We had this big push to meet this milestone. A lot of people were trying to get their changes in. A lot of people were frustrated about this."
Glydways is reimagining public transportation with a network of small, autonomous electric vehicles. This innovative system, capable of moving 10,000 people per hour in lanes just two meters wide, presents complex challenges spanning embedded systems, infrastructure, and software development. Juan Ortega, leading the developer platform team at Glydways, shares how adopting Trunk Merge Queue has transformed their workflow and boosted productivity, helping bring this ambitious vision to life.
Total employees
PRs merged per week
To understand the impact of Trunk Merge Queue, let's first look at Glydways' technical setup:
Team: About 100 employees, including 60 software engineers
Tech stack: Backend, frontend, embedded systems, OS-level code, and infrastructure
Velocity: 160 PRs merged into main per week
CI speed: Highly parallelized to achieve 15min average job time with a 20min timeout
This diverse and complex environment underscores the need for efficient development processes, especially as Glydways aims to deploy worldwide. Here's how Trunk Merge Queue helped them level up their game.
Before implementing Trunk Merge Queue, Glydways faced significant challenges with their merge process:
Long wait times for changes to get into the main branch
High CI costs and delays due to developers constantly updating their branches to test against the latest main before merging
Developers prematurely enabling auto-merge on unapproved PRs without up to date testing
Decreased developer productivity from waiting for CI machine resources
Juan describes the resource allocation issues they faced: "As you have two parallel things running, as soon as one gets merged, the other one's like, 'Oh great, let me get up to latest main and rerun all my CI again.'" This led to a spiraling effect of resource usage: "You end up using more resources. And then we put a cap on the number of resources we're using, like the number of instances and stuff like that. So then that starts bottlenecking and it just becomes this kind of spiral effect."
Juan recalls, "I heard some comments... a week after we enabled it, and they basically told me, 'Man, you got that just in time. We had this big push to meet this milestone. A lot of people were trying to get their changes in. A lot of people were frustrated about this.'"
Glydways adopted Trunk Merge Queue to address these issues. The implementation was smooth, with Juan noting, "I basically had a choice to either wait... or take a risk and essentially rip it up and use Trunk Merge Queue to get people to merge faster. Luckily it was a smooth transition."
Key benefits of the implementation included:
1. Faster merges: "People saw a noticeable difference in their speed to be able to merge into main."
2. Improved CI resource utilization
3. Streamlined workflow for developers
After implementing Trunk Merge Queue, observed the following metrics:
Daily commit volume: About 24 commits per day, increasing to 30-33 during busy periods
Queue time: Average time in queue around 20 minutes
Weekly throughput: About 160 commits per week
Juan described their current merge patterns: "We were getting like 24 or so commits... I guess this is every day. Suddenly, we jumped up to like, you know, 30 or so, 33 because we had a big milestone that we just hit a few days ago. And so we're trying to merge a bunch of things in right before that."
The transition to Trunk Merge Queue was well-received by the development team. Juan mentions, "I got that feedback about a week after we implemented it that people saw a noticeable difference in their speed to be able to merge into main."
The team particularly appreciated features like:
Priority merging for critical changes
Integration with Slack for notifications
Easy-to-use UI
Juan highlighted the usefulness of the priority feature: "We were doing some release testing and there was some high priority thing that needed to be merged. I advised them to use the priority zero comment as part of the bot, and they were able to get it in, kicking out anything else that was in front of it. It was good in that respect."
"Overall, my experience has been a pretty positive one... The integration was really easy to do also. So in terms of overall experience, it's been positive."
With the success of Trunk Merge Queue, Glydways is now exploring further optimizations to support their rapidly evolving technology:
1. Implementing parallel merge queues: This could further speed up their build process, critical for a system combining hardware and software components.
2. Exploring Trunk's flaky test detection capabilities: Essential for maintaining reliability in a complex, safety-critical system.
3. Considering CI analytics for deeper insights into build performance: This could help Glydways continually refine their development process as they scale.
Juan is excited about these possibilities, stating, "I'm planning on trying to get some better analysis of affected targets and get parallel merge queues going. That's the dream, right? It could really accelerate our ability to iterate on both our software and hardware designs."
When asked if he would recommend Trunk Merge Queue to other companies, Juan's response was unequivocal: "Yeah, absolutely."
He summarizes his experience: "Overall, my experience has been a pretty positive one... The cost part was great. The integration was really easy to do also. So in terms of overall experience, it's been positive."
For teams looking to streamline their development process and boost productivity, Glydways' experience with Trunk Merge Queue offers valuable insights into the potential benefits and ease of implementation.