Using GitHub's pull request merge queue in your pipelines

🗣️ Update: GitHub has announced that Merge Queue is now generally available.

There's been an audible buzz of excitement about pull request merge queues lately. The merge queue pattern increases the efficiency and stability of software delivery workflows by automatically handling pull request merges into your busiest target branches. Merge queues ensure that pull requests get merged in the correct order, and significantly reduce CI workloads, and potential delivery blockers.

Prior to merge queue, merging pull requests into main involved:

• Merging main into the pull request branch, so changes wouldn’t break main when merged
• Kicking off a build in CI (for these newly pulled and merged changes)
• Discovering another PR was merged before the build finished
• Rinse and repeat 🌀😵‍💫
• Merging the PR into main

I’d often be rushing to get my PR merged before someone else snuck theirs through, or I’d forget to pull the latest changes from main into my feature branch before pushing to main. Often, my PR merge would fail, because it wasn’t in sync with the main branch. I saw red–quite literally.

Merge queues automatically build pull request branches with the most recent change (or PR) to be merged ahead of it in the queue. 🙌

The merge queue isn’t new. Uber and Shopify have been using them for close to five years in their pipelines in order maintain high throughput and low commit turnaround time on their busy main branches (think thousands of merges to main per day!). If you have lots of contributors, and high volume of pull requests going to main and if you have frequent flaky merges blocking subsequent commits and creating a backlog of builds, you might find pull request merge queue to be a useful tool,as it minimizes merges tripping over each other on the way to main. GitHub’s pull request merge queues are available in any public repository owned by an organization, or in private repositories owned by organizations using GitHub Enterprise Cloud.)

GitHub merge queue step-by-step:

• The merge queue creates temporary branches to validate pull request changes prefixed with main/pr-1, main/pr-2 etc.
• Once required checks specified in the branch protection rules are satisfied, a pull request is added to the merge queue.
• Changes are grouped into a merge_group with the latest base_branch version, as well as changes from any other pull requests ahead of it in the queue.
• Once the base_branch required checks pass, the merge_group changes are merged into the base_branch

Getting started with GitHub merge queue in your pipelines

Include the gh-readonly-queue/{base_branch}/* filter pattern in your pipeline's Pipeline Settings → GitHub → Branch Limiting. This means builds will automatically run for merge queues.

In your GitHub repository’s Settings → Webhooks configuration check the merge_group checkbox.

You can optionally "skip builds with existing commits" by selecting this in your pipeline’s Pipeline Settings → GitHub → GitHub Settings. It's worth noting that scheduled builds with the same commit message will no longer build with this option selected.

The efficiency merge queue offers is exciting, though you should keep in mind that configuring branch protection and required checks don’t currently work in Buildkite and this work is on our radar.

Check out the merge queue docs, and if you run into issues let us know at support@buildkite.com or post in the Buildkite community forum.