diff --git a/Public/.gitkeep b/Public/.gitkeep deleted file mode 100644 index e69de29..0000000 diff --git a/README.md b/README.md index dd4caeb..f0f7838 100644 --- a/README.md +++ b/README.md @@ -2,13 +2,13 @@ A bot that monitors and manages your pull requests by ensuring they are merged when they're ready and don't stack up in your repository πŸ€“ -### Motivation +## Motivation Pull Requests are conceptually asynchronous and they go through a series of iterations until they are finally ready to be merged which not always happens when we are expecting, we can be waiting for CI to test it, waiting for a review, ... That can lead to the pull request staying in the repository for longer than it needs to be and potentially stacking up with other pull requests making the integrations more time consuming and challenging. -### The notion of ready +## The notion of ready Pull Requests should meet a specific set of criteria before being merged. @@ -18,22 +18,125 @@ Pull Requests should meet a specific set of criteria before being merged. Depending on the workflow of each team some of them may be disabled to suit their needs. -### How? +## How the bot works -πŸ‘·β€β™€οΈ **WIP, come back later** πŸ‘·β€β™‚οΈ +Wall-E is a bot written in Swift using Vapor. -![](https://media.giphy.com/media/26ybvJNaZZKpPONEc/giphy.gif) +It works by listening to the GitHub webhook to know when labels are being added/removed and when Status Checks are updated on PRs -### Client app +### Detection of the Merge Label + +When a specific label – configurable via the `MERGE_LABEL` environment variable – is added to your Pull Request, Wall-E will: + + - Add that PR to a queue dedicated to the branch this PR is targeting + - Post a comment in the PR to let you know that the PR has been queued and its position in the queue + +(†) Wall-E handles one serial merge queue per target branch, to allow PRs targeting different branches to be merged in parallel. Which means that each PR targeting the same branch will be merged in order one after the other, but PRs targeting different branches can be merged independently in parallel. + +### Integration of a PR + +When the bot dequeues the top PR from a queue, it will start its integration, which consists of the following steps: + + - Merge the target branch back into the PR to ensure it's up-to-date if not + - Wait for its status checks to pass + - you can configure if you want the bot to only merge if _all_ statuses are green or only the _required_ ones via the `REQUIRES_ALL_STATUS_CHECKS` environment variable + - Once the PR is deemed ready (status checks pass, green on GitHub with the minimal number of approvals reached), it will merge the PR into the target branch (squash) + - Then it will go to the next PR in the queue + +If there is a failure at some point during the integration of the PR – e.g. one of the (required) status check fails: + + - the bot will post a comment with the error on the PR + - then it will remove the merge label + - and go to the next PR in the queue + +If you remove the merge label from a PR that was in the queue, that PR gets removed from the queue immediately and bot goes to the next PR. + +### Top Priority PRs + +The bot also supports "Top Priority Labels" (configurable via the `TOP_PRIORITY_LABELS` environment variable) + +When you add one of those "Top Priority" labels to your PR, the bot will ensure that this PR will be merged before any non-TP PRs targeting the same branch, by making that PR jump at the front of all the other non-top-priority PRs in the queue. + +For example, if your queue already contains PRs `A`,`B`,`C`,`D`,`E` with `A` and `B` already marked with one of the Top Priority label, then adding a Top Priority label to the PR `E` will make it jump in front of `C` and `D` but still after `A` and `B`, so the queue will become `A`,`B`,`E`,`C`,`D`. + +## Configuration + +The bot is mainly configured via environment variables. Here are the main ones that you are at least required or recommended to provide to be able to start using this bot: + +Env Var | Description +---|--- +`GITHUB_ORGANIZATION`
`GITHUB_REPOSITORY` | The GitHub organisation and repo name this bot will watch +`GITHUB_TOKEN` | The OAuth token to use for calls to the GitHub API +`GITHUB_WEBHOOK_SECRET` | The webhook secret to use to validate webhook payloads +`MERGE_LABEL` | The name of the label to use to add a PR to the queue +`TOP_PRIORITY_LABELS` | The name of the labels to use to mark as PR as top-priority – separate multiple label names by a comma +`REQUIRES_ALL_STATUS_CHECKS` | Defines if the bot should require _all_ status checks to be green before allowing to merge a PR, or only the ones configured as _required_ in GitHub settings (the default) + +Some other environment variables allow further configuration of the bot, like values for various timeouts; for the list of them all, see [`Sources/App/Extensions/EnvironmentProperties.swift`](https://github.com/babylonhealth/Wall-E/blob/master/Sources/App/Extensions/EnvironmentProperties.swift). + +## Implementation details + +The whole codebase is implemented in Swift using [Vapor](https://vapor.codes/). + +πŸ’‘ _You can use the `vapor xcode` command to generate an `xcodeproj` project and edit the code from there._ + +If you need to maintain/improve the code, here are some high-level implementation details that might help you navigate the codebase. + +### MergeService + +`MergeService` is a service class representing a single merge queue. + + - It handles the logic of the state machine for the various states and transitions to process each Pull Request in its queue in order + - It is implemented using [ReactiveFeedback](https://github.com/babylonhealth/ReactiveFeedback) + +Below are some diagrams to help you visualise the ReactiveFeedback state machine logic implemented in `MergeService`: + +
+πŸ”ƒ State Diagram + +![State Diagram](assets/mergebot_states.png) + +
+ +
+▢️ Action/Feedbacks Diagrams + +![Action/Feedbacks Diagrams](assets/mergebot_feedbacks.png) + +
+ +### DispatchService + +`DispatchService` is responsible for managing multiple `MergeService` instances, one per target queue. + + - The `DispatchService` single instance is the one receiving the events from the webhook, and will dispatch them to the right instance of `MergeService` associated with the target queue of the event's PR + - If such a `MergeService` instance doesn't exist yet for that target branch, it will instantiate one. + - Idle `MergeService` instances are cleaned up after a delay of inactivity – configurable via the `IDLE_BRANCH_QUEUE_CLEANUP_DELAY` environment variable – to free up the memory + +### Other + +The rest of the code is mainly API calls (in `API/`) and objects modelling the API data (`Models/`) + + +## Client app (Menu Icon) + +This repository also comes with a Client app that allows you to quickly check the state of the Merge Bot queue from the menu bar. To install the client app: - - build `WallEView.xcodeproj` in Xcode and copy the app from build products directory to your applications directory or download the app attached to the [latest GitHub release](https://github.com/babylonhealth/Wall-E/releases) - - run `defaults write com.babylonhealth.WallEView Host ` to set the url to the app - - launch the app and enjoy + + - Build `WallEView/WallEView.xcodeproj` in Xcode and copy the app from build products directory to your applications directory, or download the app attached to the [latest GitHub release](https://github.com/babylonhealth/Wall-E/releases) + - Run `defaults write com.babylonhealth.WallEView Host ` to set the url to the app + - Launch the app and enjoy. + + Once the app has been launched, a new icon should appear in your menubar. + + When opening the menu item by clicking on its icon, you can select a branch to see its associated merge queue. + + To kill the app and remove the menubar icon, right-click on the icon and select "Close". Iconography Β© https://dribbble.com/shots/2772860-WALL-E-Movie-Icons -### Debugging +## Debugging Using [the ngrok tool](https://dashboard.ngrok.com/get-started) you can run the app locally and still get all incoming events from GitHub webhooks. diff --git a/Sources/App/Extensions/EnvironmentProperties.swift b/Sources/App/Extensions/EnvironmentProperties.swift index ffb98e2..423e974 100644 --- a/Sources/App/Extensions/EnvironmentProperties.swift +++ b/Sources/App/Extensions/EnvironmentProperties.swift @@ -13,12 +13,12 @@ extension Environment { return try Environment.get("GITHUB_TOKEN") } - /// GitHub Organisation name (as seen in the github.com//* urls) + /// GitHub Organisation name (`` part of `github.com//` url) static func gitHubOrganization() throws -> String { return try Environment.get("GITHUB_ORGANIZATION") } - /// URL of GitHub Repository to run the MergeBot on + /// name of GitHub Repository to run the MergeBot on (`` part of `github.com//` url) static func gitHubRepository() throws -> String { return try Environment.get("GITHUB_REPOSITORY") } diff --git a/WallEView.xcodeproj/project.pbxproj b/WallEView/WallEView.xcodeproj/project.pbxproj similarity index 93% rename from WallEView.xcodeproj/project.pbxproj rename to WallEView/WallEView.xcodeproj/project.pbxproj index 41772d0..5fe14f0 100644 --- a/WallEView.xcodeproj/project.pbxproj +++ b/WallEView/WallEView.xcodeproj/project.pbxproj @@ -7,6 +7,7 @@ objects = { /* Begin PBXBuildFile section */ + 38291C5223E1D8C2005F572A /* CommitState.swift in Sources */ = {isa = PBXBuildFile; fileRef = 38291C5123E1D8C2005F572A /* CommitState.swift */; }; B5BCDAE5238D8E170010DE06 /* PullRequest.swift in Sources */ = {isa = PBXBuildFile; fileRef = B5BCDAE4238D8E170010DE06 /* PullRequest.swift */; }; B5CACCED238D8CFB000D3F14 /* AppDelegate.swift in Sources */ = {isa = PBXBuildFile; fileRef = B5CACCEC238D8CFB000D3F14 /* AppDelegate.swift */; }; B5CACCEF238D8CFB000D3F14 /* ViewController.swift in Sources */ = {isa = PBXBuildFile; fileRef = B5CACCEE238D8CFB000D3F14 /* ViewController.swift */; }; @@ -16,7 +17,8 @@ /* End PBXBuildFile section */ /* Begin PBXFileReference section */ - B5BCDAE4238D8E170010DE06 /* PullRequest.swift */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.swift; name = PullRequest.swift; path = ../Sources/Bot/Models/PullRequest.swift; sourceTree = ""; }; + 38291C5123E1D8C2005F572A /* CommitState.swift */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.swift; path = CommitState.swift; sourceTree = ""; }; + B5BCDAE4238D8E170010DE06 /* PullRequest.swift */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.swift; path = PullRequest.swift; sourceTree = ""; }; B5CACCE9238D8CFB000D3F14 /* WallEView.app */ = {isa = PBXFileReference; explicitFileType = wrapper.application; includeInIndex = 0; path = WallEView.app; sourceTree = BUILT_PRODUCTS_DIR; }; B5CACCEC238D8CFB000D3F14 /* AppDelegate.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = AppDelegate.swift; sourceTree = ""; }; B5CACCEE238D8CFB000D3F14 /* ViewController.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = ViewController.swift; sourceTree = ""; }; @@ -38,6 +40,16 @@ /* End PBXFrameworksBuildPhase section */ /* Begin PBXGroup section */ + 38291C5323E1D8D0005F572A /* Bot.Models */ = { + isa = PBXGroup; + children = ( + B5BCDAE4238D8E170010DE06 /* PullRequest.swift */, + 38291C5123E1D8C2005F572A /* CommitState.swift */, + ); + name = Bot.Models; + path = ../Sources/Bot/Models; + sourceTree = ""; + }; B5CACCE0238D8CFB000D3F14 = { isa = PBXGroup; children = ( @@ -60,13 +72,13 @@ B5CACCEC238D8CFB000D3F14 /* AppDelegate.swift */, B5CACCEE238D8CFB000D3F14 /* ViewController.swift */, B5CACCFC238D8DA8000D3F14 /* EventMonitor.swift */, - B5BCDAE4238D8E170010DE06 /* PullRequest.swift */, + 38291C5323E1D8D0005F572A /* Bot.Models */, B5CACCF0238D8CFB000D3F14 /* Assets.xcassets */, B5CACCF2238D8CFB000D3F14 /* Main.storyboard */, B5CACCF5238D8CFB000D3F14 /* Info.plist */, B5CACCF6238D8CFB000D3F14 /* WallEView.entitlements */, ); - path = WallEView; + name = WallEView; sourceTree = ""; }; /* End PBXGroup section */ @@ -142,6 +154,7 @@ B5CACCFD238D8DA8000D3F14 /* EventMonitor.swift in Sources */, B5CACCEF238D8CFB000D3F14 /* ViewController.swift in Sources */, B5BCDAE5238D8E170010DE06 /* PullRequest.swift in Sources */, + 38291C5223E1D8C2005F572A /* CommitState.swift in Sources */, B5CACCED238D8CFB000D3F14 /* AppDelegate.swift in Sources */, ); runOnlyForDeploymentPostprocessing = 0; @@ -277,12 +290,12 @@ isa = XCBuildConfiguration; buildSettings = { ASSETCATALOG_COMPILER_APPICON_NAME = AppIcon; - CODE_SIGN_ENTITLEMENTS = WallEView/WallEView.entitlements; + CODE_SIGN_ENTITLEMENTS = WallEView.entitlements; CODE_SIGN_STYLE = Manual; COMBINE_HIDPI_IMAGES = YES; DEVELOPMENT_TEAM = ""; ENABLE_HARDENED_RUNTIME = YES; - INFOPLIST_FILE = WallEView/Info.plist; + INFOPLIST_FILE = Info.plist; LD_RUNPATH_SEARCH_PATHS = ( "$(inherited)", "@executable_path/../Frameworks", @@ -298,12 +311,12 @@ isa = XCBuildConfiguration; buildSettings = { ASSETCATALOG_COMPILER_APPICON_NAME = AppIcon; - CODE_SIGN_ENTITLEMENTS = WallEView/WallEView.entitlements; + CODE_SIGN_ENTITLEMENTS = WallEView.entitlements; CODE_SIGN_STYLE = Manual; COMBINE_HIDPI_IMAGES = YES; DEVELOPMENT_TEAM = ""; ENABLE_HARDENED_RUNTIME = YES; - INFOPLIST_FILE = WallEView/Info.plist; + INFOPLIST_FILE = Info.plist; LD_RUNPATH_SEARCH_PATHS = ( "$(inherited)", "@executable_path/../Frameworks", diff --git a/WallEView.xcodeproj/project.xcworkspace/contents.xcworkspacedata b/WallEView/WallEView.xcodeproj/project.xcworkspace/contents.xcworkspacedata similarity index 100% rename from WallEView.xcodeproj/project.xcworkspace/contents.xcworkspacedata rename to WallEView/WallEView.xcodeproj/project.xcworkspace/contents.xcworkspacedata diff --git a/WallEView.xcodeproj/project.xcworkspace/xcshareddata/IDEWorkspaceChecks.plist b/WallEView/WallEView.xcodeproj/project.xcworkspace/xcshareddata/IDEWorkspaceChecks.plist similarity index 100% rename from WallEView.xcodeproj/project.xcworkspace/xcshareddata/IDEWorkspaceChecks.plist rename to WallEView/WallEView.xcodeproj/project.xcworkspace/xcshareddata/IDEWorkspaceChecks.plist diff --git a/WallEView.xcodeproj/xcshareddata/xcschemes/WallEView.xcscheme b/WallEView/WallEView.xcodeproj/xcshareddata/xcschemes/WallEView.xcscheme similarity index 100% rename from WallEView.xcodeproj/xcshareddata/xcschemes/WallEView.xcscheme rename to WallEView/WallEView.xcodeproj/xcshareddata/xcschemes/WallEView.xcscheme diff --git a/assets/mergebot_feedbacks.dot b/assets/mergebot_feedbacks.dot new file mode 100644 index 0000000..1470ae5 --- /dev/null +++ b/assets/mergebot_feedbacks.dot @@ -0,0 +1,114 @@ +# https://edotor.net/ + +digraph MergeBot { + pack=true + + subgraph states { + node [shape=oval][style=filled,fillcolor=lightgray] + starting [penwidth=3] + # idle + ready + integrating + runningStatusChecks + integrationFailed + "*" + } + + subgraph events { + node [shape=box][style=dotted] + ".pullRequestsLoaded" + ".pullRequestDidChange(.include)" + ".pullRequestDidChange(.exclude)" + ".noMorePullRequests" + ".integrationDidChangeStatus(.done)" + ".integrationDidChangeStatus(.failed)" + ".integrationDidChangeStatus(.updating)" + ".integrate" + ".statusChecksDidComplete(.passed)" + ".statusChecksDidComplete(.failed)" + ".statusChecksDidComplete(.timedOut)" + ".integrationFailureHandled" + ".retryIntegration" + } + + subgraph actions { + node [shape=rect] + dequeuePR + fetchPullRequest1 [label=fetchPullRequest] + fetchPullRequest2 [label=fetchPullRequest] + fetchPullRequest3 [label=fetchPullRequest] + mergePullRequest + updatePR # merge target branch back in PR + fetchCommitStatus + fetchAllStatusChecks + includePR + excludePR + postComment + removeLabel + } + + subgraph feedbacks { + color=blue + + subgraph cluster_whenStarting { + label="whenStarting" + starting -> ".pullRequestsLoaded" + } + + subgraph cluster_whenReady { + label="whenReady" + ready -> dequeuePR + dequeuePR -> ".noMorePullRequests" [label="nil"] + dequeuePR -> fetchPullRequest1 -> ".integrate" + } + + subgraph cluster_whenIntegrating { + label="whenIntegrating" + integrating -> ".integrationDidChangeStatus(.done)" [label="pr.isMerged"] + + # clean + integrating -> mergePullRequest [label="state=clean"] + mergePullRequest -> ".integrationDidChangeStatus(.done)" [label="ok"] + mergePullRequest -> ".integrationDidChangeStatus(.failed)" [label="error"] + + # behind + integrating -> updatePR [label="state=behind"] + updatePR -> ".integrationDidChangeStatus(.updating)" [label="success|upToDate"] + updatePR -> ".integrationDidChangeStatus(.failed)" [label="conflict"] + + # blocked|unstable + integrating -> fetchAllStatusChecks [label="state=blocked|unstable"] + fetchAllStatusChecks -> ".integrationDidChangeStatus(.updating)" [label="pending"] + fetchAllStatusChecks -> ".integrationDidChangeStatus(.failed)" [label="failure"] + fetchAllStatusChecks -> ".retryIntegration" [label="success"] + + # dirty + integrating -> ".integrationDidChangeStatus(.failed)" [label="state=dirty (conflicts)"] + + # unknown + integrating -> fetchPullRequest2 -> ".retryIntegration" [label="state=unknown"] + } + + subgraph cluster_whenRunningStatusChecks { + label="whenRunningStatusChecks" + runningStatusChecks -> fetchPullRequest3 [label="on statusCheckObserver change"] + fetchPullRequest3 -> fetchCommitStatus + fetchCommitStatus -> ".statusChecksDidComplete(.failed)" [label="failure"] + fetchCommitStatus -> ".statusChecksDidComplete(.passed)" [label="success"] + runningStatusChecks -> ".statusChecksDidComplete(.timedOut)" [label="timeout"] + } + + subgraph cluster_whenIntegrationFailed { + label="whenIntegrationFailed" + integrationFailed -> postComment -> removeLabel -> ".integrationFailureHandled" + } + } + + subgraph cluster_reduceDefault { + color=blue + label="reduceDefault()" + "*" -> ".pullRequestDidChange(.include)" -> includePR + "*" -> ".pullRequestDidChange(.exclude)" -> excludePR + } + +} diff --git a/assets/mergebot_feedbacks.png b/assets/mergebot_feedbacks.png new file mode 100644 index 0000000..060c2d9 Binary files /dev/null and b/assets/mergebot_feedbacks.png differ diff --git a/assets/mergebot_states.dot b/assets/mergebot_states.dot new file mode 100644 index 0000000..9e14950 --- /dev/null +++ b/assets/mergebot_states.dot @@ -0,0 +1,70 @@ +# https://edotor.net/ + +digraph MergeBot { + + subgraph states { + node [shape=oval][style=filled,fillcolor=lightgray] + starting [penwidth=3] + idle + ready + integrating + runningStatusChecks + integrationFailed + } + + subgraph events { + node [shape=box][style=dotted] + ".pullRequestsLoaded(empty)" + ".pullRequestsLoaded(!empty)" + ".pullRequestDidChange(.include)" + ".pullRequestDidChange(.exclude)" + ".noMorePullRequests" + ".integrationDidChangeStatus(.done)" + ".integrationDidChangeStatus(.failed)" + ".integrationDidChangeStatus(.updating)" + ".integrate" + ".statusChecksDidComplete(.passed)" + ".statusChecksDidComplete(.failed)" + ".statusChecksDidComplete(.timedOut)" + ".integrationFailureHandled" + ".retryIntegration" + } + + subgraph cluster_reduce { + color=blue + label="reduce()" + + subgraph reduceStarting { + starting -> ".pullRequestsLoaded(empty)" -> idle + starting -> ".pullRequestsLoaded(!empty)" -> ready + } + + subgraph reduceIdle { + idle -> ".pullRequestDidChange(.include)" -> ready + } + + subgraph reduceReady { + ready -> ".noMorePullRequests" -> idle + ready -> ".integrate" -> integrating + } + + subgraph reduceIntegrating { + integrating -> ".integrationDidChangeStatus(.done)" -> ready + integrating -> ".integrationDidChangeStatus(.failed)" -> integrationFailed + integrating -> ".integrationDidChangeStatus(.updating)" -> runningStatusChecks + integrating -> ".pullRequestDidChange(.exclude)" -> ready + integrating -> ".retryIntegration" -> integrating + } + + subgraph reduceRunningStatusChecks { + runningStatusChecks -> ".statusChecksDidComplete(.passed)" -> integrating + runningStatusChecks -> ".statusChecksDidComplete(.failed)" -> integrationFailed + runningStatusChecks -> ".statusChecksDidComplete(.timedOut)" -> integrationFailed + runningStatusChecks -> ".pullRequestDidChange(.exclude)" -> ready + } + + subgraph reduceIntegrationFailed { + integrationFailed -> ".integrationFailureHandled" -> ready + } + } +} diff --git a/assets/mergebot_states.png b/assets/mergebot_states.png new file mode 100644 index 0000000..908aa48 Binary files /dev/null and b/assets/mergebot_states.png differ