Multi-branch builds in Jenkins

One of the coolest features of Atlassian Bamboo is the ability to automatically detect multiple branches of a repository and run a plan with that branch.

Image

Unfortunately, Jenkins cannot do that, but with the help of a couple of plugins and some duct tape, we can emulate that same behavior. Disclaimer: I don’t know if this works for Hudson, because I haven’t tested it, but it simple enough that I don’t see why it shouldn’t.

I spent a few days trying to figure out how to do this without modifying the source code of the existing plugins or writing my own, and I was able to do it. I also think my solution is generic enough that it will fit your needs without major issues (other than the cons that I’ll discuss towards the end of the post).

This tutorial (if you can call it that) assumes you have a few plugins that you probably already have if your Jenkins plans need such a level of sophistication that you’re reading this. If you find that something that is explained here is not provided by Jenkins out of the box, a simple plugin search should give you the answer.

Let me explain what one of the main problems is: the Mercurial Plugin, makes an effort to minimize the disk usage footprint so it only attempts to clone or pull the minimum changeset graph that will lead to the specified changeset. Because of this, when using the Poll SCM feature, it won’t detect (or pull) changes in any other branch than the one specified in the SCM section of the job.

The way around this is to use the Repository Sharing feature.

Screen Shot 2013-06-28 at 3.38.21 PM

 

If you enable that option, then Jenkins will store a local cache of every HG repository so less requests are made to the remote server. You don’t need to know how the feature is implemented, for now, just knowing that the entire repository is there, and not just the branch you configured.

Ok, we’ve jumped one hurdle. Next step (assuming you have configured your job to poll the repo), how do you make Jenkins poll for all changes if it’s only polling one branch? Well, it’s not really just polling one branch. I know the SCM configuration has a field Branch, but it should really be ‘Revision’, because you can enter any arbitrary thing that would identify one changeset: branch name, SHA1, or tag name, including “tip” that in this case, will be your friend.

You know how the tag called “tip” is automatically moved to the latest changeset? If we have Jenkins poll for this tag, it will detect if any changesets have been added to the repository.

But we’re not out of the woods yet. Yes, Jenkins will detect changes but the Mercurial plugin will only pull the changes from whatever branch the “tip” tag would happen to be. This is where the gluing begins.

First of all we would need to have two jobs for each repository that you want to trigger. I called them <repository> and <repository>-all-branches. The <repository> job does the actual build/test/deployment, etc. that you want to do. The <repository>-all-branches does the polling and triggers the real job. Of course substitute <repository> with your repository name. I’ll get into more detail further down. As I was writing the last lines of this post, I realize that it’s possible that this can be done using one job, but I haven’t done it, I’ll probably write another post if I can make that work.

The triggering job will do the following steps:

  • Poll the repository for changes.
  • Figure out the branches that have changes and require a build.
  • Use the Jenkins REST API to call the real job.

Here’s another problem, Jenkins attempts to retrieve the incoming changes before you do anything else. So, the first plugin we need is pre-scm-buildstep, this allows us to execute build steps at the pre-scm phase. It is at this point that we execute this script, I’m deliberately not putting it here, to save space, but if you don’t want to read it, just know that it creates a file with the branches with incoming changes that require a build.

So, now we have that file and we let Mercurial do its thing. At this point it is irrelevant since we don’t need the actual code, we just use the repository to detect changes and to query for branches with changes.

After this we use this other script, this time as a main build phase step, to iterate over the branches captured in the pre-scm step and trigger an execution of the main job with each one of these branches. There are a few things that the main job needs to have before we can do this. Think about it as an Interface that the job has to implement to be able to do this (if you’re a OO programmer).

First, it has to at least take the branch name as a parameter.

Screen Shot 2013-06-28 at 4.42.17 PM

 

And plug it into the SCM configuration.

Screen Shot 2013-06-28 at 4.44.16 PM

 

And finally, allow it to be triggered remotely, and adding a token that other jobs will use to invoke it.

Screen Shot 2013-06-28 at 4.46.06 PM

 

With this, your target job is ready to be invoked remotely. Now let’s go back to the triggering job, one of the things that is needed for authentication purposes is an API token for the user that will be triggering the build. Since this is done in the context of a shell script, there is no knowledge of the user that’s executing the script, so it can’t be inherited, thus the need for authentication. The script referenced earlier (this one) does a curl command around line 6 (maybe a different line if it has been modified since I wrote this post). This is a simple command with three parameters.

  • -F $json -> It passes the json file created just above that line and submits the request as a Form POST.
  • -u $user:$APIToken -> The authentication credentials for the $user. These are parameters to the script and the APIToken should be the string that can be retrieved from http://your.jenkins.server/user/youruser/configure, under the Show API Token button. The $user parameter should be the one that you’re retrieving the token for.
  • $JENKINS_URL/job/$targetJob/build?token=$targetJobToken -> $JENKINS_URL is an environment variable and doesn’t need to be passed as a paremter, $targetJob and $targetJobToken are script parameters and should match the settings you configured in the main job earlier. Some documentation says that you should use /buildWithParameters instead of /build, but this worked for me.

Also, the last step in that script is to pull the repository so the job has a baseline for the next detected change.

This pretty much does the trick, but you can take a few extra steps to smooth the edges and make it maintainable.

If you use the Managed Script Plugin, you can have the script in one central location and invoke them as a build step (and also as a pre-build step, since you already have the pre-scm-buildstep plugin). This helps if you have a high number of jobs that you’re automating and you have to change the script. You only change it in one place and it works seamlessly across the board. Additionally, if you need to do some logic before you execute the managed scripts, you can use another feature of this plugin that allows you to put a copy of one of your managed scripts in the workspace and put the name of the script as a variable, so you can reference it easily.

You can also use the Mask Passwords Plugin to mask the API Token value from the console log and prevent others from seeing it and be able to authenticate as that user. The plugin also allows you to store the masked password (hidden as *’s) as a global variable so it’s available at run time.

Finally, because this job can potentially be executed often, I would check the option to discard old builds and limit it to a smaller number that you’re comfortable with.

One caveat of this is that you effectively have two clones of the same repo, in two different Jenkins workspaces, but if you’re not concerned with disk space, this shouldn’t be a showstopper. As I’m writing this, I realize a different approach that could reduce the complexity to one job, so this downside could be eliminated.

Also, note that the user you’re using to authenticate the curl call, should have permissions (either global or project-based) required to run the job.

One last note, but this should be obvious, the scripts are shell scripts so they won’t work in a Windows environment, although I don’t see why not someone can come with a good batch script in five minutes that does the same thing.

Did you notice how we did this without writing a line of code outside of the 5-6 lines in the shell scripts? I think it beats the hell out of writing your own plugin to implement this.

Wow, this was a long post, the longest I’ve written so far, but I hope it helped. If you have questions, my contact information is in the left of your screen (if I haven’t changed the layout), feel free to shoot me any questions you have.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: