Functional Abstractions for CI

Continuous Integration (CI) is the practice of automatically running builds and tests to ensure the integrity of a codebase. What does this mean? Whenever someone proposes a change, CI quickly checks whether the change is OK.

CI improves development velocity by giving developers the information they need to fix bugs early — without adding extra steps to their development workflow.

The benefits of CI are clear. How do you set it up? Typically, it looks like a small program that runs for each proposed change to a codebase. This program compiles the codebase into artifacts to test and publish. It’s often a simple pipeline, but not always. Today’s variety of CI languages reflects a broad range of complexity.

The key to a good CI language is creating abstractions that allow a CI pipeline to evolve over time. Let’s begin with a simple pipeline and explore how to add functionality without decreasing its maintainability.

Suppose the codebase is a Go library. Before allowing changes to be committed, we want to make sure the codebase will still compile successfully and pass all tests:

These two steps form a simple program that runs each time someone proposes a change to the codebase.

As our needs grow, the program can become more complex. For example, if we want to build and test using different compiler versions:

We added one simple concern, and the program became twice as complex. Do this a few more times, and the program isn’t manageable any more. Suppose we parameterized the original program and ran it twice, once for each version of Go:

Instead of four steps, there are two. We’ve accomplished this by hiding some complexity from ourselves — by putting a multi-step process behind the name “Run the pipeline with X”. This is called abstraction.

Because abstraction hides information, it can be used to document which details are important and which ones are not. In the example above, the abstraction says the number of steps in the pipeline doesn’t matter. What matters is that we’re running two identical pipelines, one for each Go version.

The strongest abstractions hide information so thoroughly that you can get away with completely ignoring what’s hidden. An example is “pure” functions, which don’t produce any observable output aside from their return values. If you know the return value, you never need to know how it was produced.

Another benefit of abstraction is that hiding details makes different things look the same. This makes it possible to use generic code. Imagine a generic mechanism for running multiple actions in parallel. We’ve already hidden the details of a build-and-test pipeline so it looks like a single “action”. We can get parallel execution for free:

We wouldn’t be able to use this mechanism if we couldn’t talk about each sub-pipeline as a single piece. “Run steps 1 and 2 in parallel” makes sense as a generic function. But there’s no built-in function for “Run steps 1 and 3 in parallel and steps 2 and 4 in parallel, and make sure step 2 starts when step 1 finishes and step 4 starts when step 3 finishes.”.

If “running actions in parallel” is itself an action, then we can “run in parallel” in parallel! Let’s parallelize part of each sub-pipeline:

Now we’re running parallel pipelines for Go 1.8 and 1.9 and parallelizing the tests within each pipeline. It’s two different levels of parallelism made possible by a single generic mechanism.

A simple toolbox of good abstractions is often more usable than a similarly powerful collection of specialized language features. With the right presentation, this simplicity can be easier to learn as well. CI in particular has a lot to gain from adding expressiveness without increasing complexity.

Related posts:

Game monetization and community are the two facets of the singular commercial relationship that underpins our industry. This article explores their relationship and imagines future studio structures.

We always think to make a recipe that boosts our immunity and is the need of the hour. Isn’t it? Try making RAJMA CHAWAl. Here is the best rajma recipe you can try to make at home with different…

While I can appreciate that parents care enough to restrict their children’s internet access; I think we can all agree it’s annoying. In my opinion, if you want to disable your child’s WiFi without…