A GitHub App for VSTS Build

Over the last few months, I've been trying to take a more broad view of DevOps. I've been working on Git for a bunch of years, and other version control systems for many more, so that will always be my home. But lately, I've been thinking a lot about build and release pipelines. So last weekend I decided to work on a fun project: using Probot to build a GitHub app that integrates with the Visual Studio Team Services build pipelines.

Why?

Over on the libgit2 project, we've been moving over to Visual Studio Team Services for our continuous integration builds and pull request validation. I'm obviously a bit biased, as I work on the product, but I'm very happy to move us over to VSTS — previously, we used a mix of CI/CD providers for different platforms, but since VSTS provides hosted Linux, Windows and macOS build agents, we're able to consolidate. Plus we have an option to run build agents on our own hardware or VMs, so we can expand our supported platform matrix to include platforms like this cute little ARM.

One thing that VSTS hasn't fixed for us, though, is some occasionally flaky tests. We have tests that hit actual network services like Git hosting providers and HTTPS validation endpoints. And when you run nine builds for every pull request update, eventually one of those is bound to fail. So we need a way to rebuild PR builds when we hit one of these flaky tests.

Obviously, I can set everybody up with permissions to VSTS to be able to log in and rebuild things. But wouldn't it be easier if we could do that right from the pull request? I thought it would - plus it would give me an excuse to play with Probot, a simple framework to build GitHub Apps.

I was really impressed how easy it was to build a GitHub App to integrate with VSTS build pipelines using the VSTS Node API and how quickly I could set up an integration so that somebody can just type /rebuild in a pull request and have the VSTS build pipeline do its thing.

Getting Started

When you read Probot's getting started guide, you'll notice that there's a handy bootstrapping script that you can use to scaffold up a new GitHub App. And it will optionally do it with TypeScript:

npx create-probot-app --typescript my-first-app

So of course I included that flag. If I'm going to learn node.js, I might as well learn TypeScript, too. And I'm incredibly happy that I did.

Me: Hmm, maybe I should explore TypeScript...
30 minutes later me: WHY ISN'T EVERYTHING TYPESCRIPT?

— Edward Thomson 🚀 (@ethomson) September 1, 2018

Then, all I had to do was install the VSTS Node API.

npm install vso-node-api --save

And then gluing this all together is a pretty straightforward interaction between Probot, the GitHub API and the Visual Studio Team Services API.

How it Works

You can — of course — grab all this from the GitHub repository for probot-vsts-build, but here's a quick walk-through to explain how Probot, the GitHub API, and the VSTS API work and work together:

1. Probot: set up the event listener

   First, we set up an event handler to listen for when new comments on an issue are created. (This will fire for new comments on a pull request as well.)

   app.on(['issue_comment.created'], async (context: Context) => {
     var command = context.payload.comment.body.trim()
   
     if (command == "/rebuild") {
       context.log.trace("Command received: " + command)
   
       new RebuildCommand(context).run()
     }
   })
   

   This will create a new RebuildCommand and run it. I decided that I might want to expand this to do additional things in the future, even though the only thing it listens to today is the /rebuild command.

2. GitHub: query the issue to make sure it's a pull request

   Since we get these events for both issues and pull requests, we want to make sure that somebody didn't type /rebuild on an issue - if that were the case, there wouldn't be anything to do.

   Probot gives us a GitHub API context that we can use to query the pull request API and ensure that the it's really a PR. If it's not, we'll just exit as there's nothing to do:

   var pr = await this.probot.github.pullRequests.get({ owner: this.repo_owner, repo: this.repo_name, number: this.issue_number })
   
   if (!pr.data.base) {
     this.log.trace('Pull request ' + this.probot.payload.issue.number + ' has no base branch')
     return null
   }
   

3. GitHub: ensure the user requesting the rebuild has permission

   We want to limit the people who can request a rebuild to project collaborators. This prevents someone from (accidentally or intentionally) DoS'ing our build service. A misbehaving bot or a not-nice person could just post /rebuild over and over again in an issue and tie up our build queue, preventing PR builds from happening.

   Looking at project collaborators is, admittedly, a pretty arbitrary way to restrict things. It was pointed out that I could have also looked at write permission to the repository.

   Nice! This is awesome 👏 Is there a reason you chose Org membership instead of Repo permission?

   — Jason 👾🤖 (@JasonEtco) September 6, 2018

   It just turns out that this is the first way I thought to do it. 😀

   var response = await this.probot.github.repos.getCollaborators({ owner: this.repo_owner, repo: this.repo_name })
   var allowed = false
   
   this.log.debug('Ensuring that ' + this.user.login + ' is a collaborator')
   
   response.data.some((collaborator) => {
     if (collaborator.login == this.user.login) {
       allowed = true
       return true
     }
   
     return false
   })
   

4. VSTS: load all the Team Projects for the given VSTS account

   I want to keep configuration simple, so the only thing you need to use this app is a VSTS account (URL) and a personal access token to authenticate to VSTS. VSTS has the notion of a "Team Project" which is another layer you can use to subdivide your account.

   For my personal VSTS account, I have it split up into different projects, one for each of my open source projects, so that their build pipelines aren't all jumbled together.

   

   Since the build definitions for pipelines live in a Team Project, the first thing to do is look up all the projects unless the VSTS_PROJECTS environment variable is set. (This lets you skip this round-trip, at the expense of another bit of configuration.)

   if (process.env.VSTS_PROJECTS) {
     return process.env.VSTS_PROJECTS.split(',')
   }
   
   var coreApi = await this.connectToVSTS().getCoreApi()
   var projects = await coreApi.getProjects()
   var project_names: string[] = [ ]
   
   projects.forEach((p) => {
     project_names.push(p.name)
   })
   
   return project_names
   

5. VSTS: find the build definitions for pull requests for this GitHub repository

   Once we have the list of team projects, we want to look at all the build definitions within those team projects for a definition that is triggered for pull requests in the GitHub repository where we typed /rebuild.

   So we want to query all build definitions for this GitHub repository:

   var all_definitions = await vsts_build.getDefinitions(
       project,
       undefined,
       this.repo_fullname,
       "GitHub",
       undefined,
       undefined,
       undefined,
       undefined,
       undefined,
       undefined,
       undefined,
       undefined,
       true)
   

   Some of these definitions might be set up only for continuous integration — when something is pushed or merged into the master branch — and not for pull requests. So we want to iterate these definitions looking for the ones that have a pull request trigger configured.

   definition.triggers.some((t) => {
     if (t.triggerType.toString() == 'pullRequest') {
       var trigger = t as PullRequestTrigger
   
       if (!trigger.branchFilters) {
         return false
       }
         
       trigger.branchFilters.some((branch) => {
         if (branch == '+' + pull_request.base.ref) {
           this.log.trace('Build definition ' + definition.id + ' is a pull request build for ' + pull_request.base.ref)
           is_pr_definition = true
           return true
         }
   
         return false
       })
   
       if (is_pr_definition) {
         return true
       }
     }
   
     return false
   })
   

   (If there's one thing that I truly regret in this code, it's using a some here. It felt idiomatic at first, but a simple for loop would have been more sensible. I'll fix this up at some point.)

6. VSTS: see what builds were run for this pull request

   We want to requeue builds, not start new ones. This sounds like a subtle distinction, but it ensures that the pull request gets updated with the new build status.

   That means we need to query all the builds that have been performed for this pull request for the definitions that support PR builds:

   var builds_for_project = await vsts_build.getBuilds(
       definition_for_project.project,
       definition_for_project.build_definitions.map(({id}) => id),
       undefined,
       undefined,
       undefined,
       undefined,
       undefined,
       BuildReason.PullRequest,
       undefined,
       undefined,
       undefined,
       undefined,
       undefined,
       undefined,
       1,
       undefined,
       undefined,
       'refs/pull/' + this.issue_number + '/merge',
       undefined,
       this.repo_fullname,
       "GitHub")
   

   (Oops — here's another thing that I just realized — since build definitions are optional in this API, we could have skipped that last query, and just left the second argument undefined. Another thing to improve when I have a minute!)

7. VSTS: requeue those builds

   Now that we have the list of builds that were originally queued, we can requeue them.

   var queuedBuild = await vsts_build.requeueBuild(sourceBuild, sourceBuild.id, sourceBuild.project.id)
   

   But wait! You might notice that the VSTS API doesn't actually have a requeueBuild function. That's because it's a very new endpoint, but I noticed the "Rebuild" button in the VSTS UI:

   

   A quick peek at the network traffic showed that it was POSTing an empty body at the URL for the existing build endpoint for that build id. It's fortunate that the new method is against the same endpoint, I was able to look up the getBuild and deleteBuild APIs to understand to construct a URL for that same endpoint, using its GUID, and create a request.

   var routeValues: any = {
     project: project
   };
   
   let queryValues: any = {
     sourceBuildId: buildId
   }
   
   try {
     var verData: vsom.ClientVersioningData = await this.vsoClient.getVersioningData(
       "5.0-preview.4",
       "build",
       "0cd358e1-9217-4d94-8269-1c1ee6f93dcf",
       routeValues,
       queryValues)
   
     var url: string = verData.requestUrl!
     var options: restm.IRequestOptions = this.createRequestOptions(
       'application/json',
       verData.apiVersion)
   
     var res: restm.IRestResponse<Build>
     res = await this.rest.create<Build>(url, { }, options)
   
     var ret = this.formatResponse(res.result, TypeInfo.Build, false)
   
     resolve(ret)
   }
   catch (err) {
     reject(err)
   }
   

   And I can even create that as an extension method on the VSTS API:

   declare module 'vso-node-api/BuildApi' {
     interface IBuildApi {
       requeueBuild(build: Build, buildId: number, project?: string): Promise<Build>
     }
   
     interface BuildApi {
       requeueBuild(build: Build, buildId: number, project?: string): Promise<Build>
     }
   }
   

8. GitHub: tell the user that we did it

   Finally, all we need to do is tell the user that we succeeded, so we'll post something back in that issue thread:

   this.probot.github.issues.createComment(this.probot.issue({
     body: 'Okay, @' + this.user.login + ', I started to rebuild this pull request.'
   }))
   

And that's it! Once we configure and deploy our GitHub App, we'll now listen for /rebuild commands and queue new builds: