Open-sourcing infrastructure

People contribute code to open source projects in many languages. Many of these projects are tools or modules that started out as internal development that became good enough (and generic enough) that it could be exposed as an open source project.

There are some tradeoffs to opening up your code to the world, such as getting free patches at the expense of having to spend time merging pull requests. But generally the number one benefit is ‘street cred’, both for the company and for the individual developers.

But the majority of the open source projects are either the code for a specific product or some other executable piece of software (libraries, frameworks, plug-ins, etc.). But how about other parts of a stack that can also be codified? With the increased popularity of tools such as Chef and Puppet a lot of companies are managing their systems through a set of scripts that use a DSL defined by the specific framework. This is what it’s known as Infrastructure as code.

This is an exploratory thought, but what if we combine both ideas? Open up our Chef or Puppet implementations of common infrastructure. Enough organizations use a Node.js installation, an Apache server or a Mongo database to benefit from a generic implementation that can be fine-tuned or managed as the needs change. Yes, all these can be installed easily, but not necessarily in a consistent manner across the board. Versions or configurations can vary from one instance to the other.

Github already has many repositories with puppet code, so this is not a new idea, but it just needs a name. I would call it open-sourcing infrastructure.

Commented out code is dead code

“I’m uncommenting and fixing this code that I commented out a few days ago”, said no programmer ever.

Commenting out code is a nasty habit. I try to avoid it as much as I can. I try to not even use it for comments. On numerous occasions, I’ve seen /*commented out code*/ that has been sitting there for months, judging from the annotation on source control.

What’s the number one reason for committing commented out code? “To keep it as reference in case we want to add it back in the future”. Ok, this is an argument I can rationalize and tolerate. But how about these two thoughts?

  1. If you’re adding back that logic/configuration within a few days, then it’s going to be fresh in your mind, no need to make your code your personal post-it; if after a couple of weeks, those lines are never added back, it probably means they’ll never be used.
  2. What is a VCS for if not for keeping history? You can safely delete any lines you want and be rest assured that you can recover them at any point in the future. In fact I would take this a step further and delete all unused code, whether commented out or not. Don’t keep anything you think you might use in the future. If you do use it in the future, you can dig it out.

But the reason that should not be acceptable for commenting out code is: “I’m commenting out these test cases, so the build is green; I’ll get back to them later”. You’re defeating the purpose of automated testing and continuous integration.

If the code is dead, bury it, if your tests are broken, fix them.

Overengineering

Engineering problems are hard. Not NP-complete hard (although sometimes they are), but rather technologically hard. For this same reason we need to solve them with an engineering approach. Code smells, low performance and poorly automated processes, among other problems, are good candidates for problems to be tackled with an engineering mindset.

There’s no denying that solving these problems is necessary and in some cases, critical. But how do we know when we’re overengineering them? Here’s a real life example.

ProblemWe want to have a unified deployment script, that works for all of our applications.
Thought process: We already have a script that does most of the work, but we want to use it for new application A. The script, however, needs to support deploying application A with curl instead of scp. We should augment the existing script to make it more powerful by supporting application A.
The result: A generic, catch-all script that effectively supports the new project, but looks kind of like this:

deployment.script -version=1.0 -user=admin -password=admin -url=http://url.com/target -file1=/path/to/file1 -file2=/path/to/file/2 -anotherPath=/path/to/another/path -yetAnotherPathThatIsNeededForApplicationAnly=/path/for/application/a -someRandomLegacyParameter=scxpr55 -webServer=apache -include=includeExpression -exclude=ExcludeExpression …

You get the idea.

Without having to look at the file, you already know that is going to be full of if/else statements or some other switching directive, which complicates matters. There’s even an urban legend of a big software company (I don’t know which one) that had to patch and recompile the Linux kernel to support more command line parameters than the shell allowed them to. This is mentioned in Jez Humble’s Continuous Delivery book.

What if we could modularize the script a little bit? We don’t necessarily have to be coding in an object oriented language, but if we can slice the logic into different sub-operations (configure, package, push) then clients can grab only the pieces they need and run with them. We can still share the common logic but bisect and decouple unrelated logic.

Proposed solutionHave multiple, smaller scripts that do well defined actions:

  • package.script –target=/target/path
  • test.script #no parameters
  • deploy.script –artifact-to-deploy=/path/to/file –target-server=target

Obviously, this a very simplistic example, but I hope the point is not lost. Just because we’re cleverly using parameters and flags doesn’t necessarily mean that we’re engineering the correct solution, it just means we’re trying to modify the minimum amount of code, which doesn’t always scale.

Actually it’s not about the flags and parameters (that apply to this example only) but it’s about not trying to solve everything with code. If the first thing you do when you see a problem is think about the lines of code that you need to modify, then you’re probably going to end up overengineering it. Step back, look at the full picture and carefully think about making a quick fix, specially if you’ve touched the same portion of the code several times to accommodate for new features.

 

Build once, deploy many

I’ve heard many times the phrase “We need a build to X environment” or “We’ll be doing a build to production on X date”, when in fact I think we should substitute build for push, deploy, install or some other verb that indicates that we have already generated the artifact that you’re going to go live with.

In most cases, this is accurate, as we’re compiling or otherwise generating the file that we’ll be then moving it to the specified target. But why are we building this artifact just before we’re going to push it to an environment? In software we certainly don’t enjoy the benefits of an on-demand, just-in-time philosophy that other industries do.

So, why do we insist on building and packaging every time? I think this often is a bad habit that teams acquire when their codebase is small. Compiling and/or packaging is relatively cheap (no more than a few seconds), so this cost is often neglected. And since it’s easier to re-generate the artifact than it is to create an infrastructure to persist these files and have the deployment scripts use it, teams usually skip this step in favor of short term gains. But as the codebase grows, this cost cannot be neglected and sometimes I’ve seen it take over 50% of the time of the overall deployment process. I’ve also seen (and worked with) a team in which the build was cloning their dependent repositories and building them from scratch, taking at least two hours. So much for continuous integration.

Build once, deploy many is a philosophy that has two advantages:

  1. Our deployment scripts become just a few instructions on how to move the package from one place to another. If your deployment scripts are doing something other than just deploying, that’s a big smell.
  2. We guarantee that the file that we’re deploying is going to be exactly the same across all environments.

The infrastructure for this could be as simple as copying the file to a shared drive and have the deployment scripts pick it up from there or as sophisticated as using tools like Artifactory or Nexus. These artifact repositories even have a REST API that allows you to perform operations like deploying the artifacts directly from the container, so your scripts don’t even have to download them. Your deployment scripts become even thinner!

Another advantage of these tools over simple file copy is that they add a layer of metadata on top of the files that makes file versioning easier.

So, you can have an independent process that generates the binaries to be deployed, versions them and pushes them to your artifact repository. Then your deployment scripts pick it up and just moves it across your different environments, making its way towards your production environment. As a former boss put it: “You wrap a gift and just hand it over to the next guy until it gets to the hands of the right person”.

And just for the record, I don’t like the term “production”; I’d rather use the term “live environment”, but that’s a different discussion.