Starting work on a freshly installed Mac: I clone our main repo, type "make id_minter-build", and it starts fetching dependencies.
Ten minutes later, with nothing else from me, I've got a complete build.
I really felt the benefits last Monday. Normally I commute to London on the train, but there was snow and ice at the weekend, and cold weather wreaks havoc on the railways. I decided to work from home, rather than sit on a train that would probably be delayed or cancelled.
Because I wasn’t expecting to work from home when I left on Friday, I’d left my work laptop in the office. And even worse, I recently had to reinstall my home computer, so I didn’t have any of my existing setup.
Not to worry: all our projects are on GitHub so I could easily get the code. (And they’re open source with an MIT license, so working on a personal computer doesn’t cause legal issues.) We run all our dev processes in Docker containers, so I only had to install Docker. Then we use Make to manage Docker itself: building the containers, keeping them up-to-date, and running the test commands.
So after I’d cloned the repo, I could run my tests with just one command:
$ make api-test
followed by ten minutes of dependencies compiling in the background, which I could spend catching up on email. When I came back, the code had compiled, the tests run, and I was ready to start work.
Other projects I’ve worked on took much longer to get set up. At a previous job, the instructions for your first build ran to two sides of A4 (!). Tweaking compiler flags, laying out submodules, getting my network settings ~just so – I’d been working for a fortnight before I saw “build: ok”.
If I had to go through something like that, I’d have wasted the whole day. Instead, I was back up-and-running in less than half an hour.
Let’s start without Make or Docker. Our API is a Scala application, with tests and builds managed by a tool called sbt. We also run mocks/stubs of a few AWS services in our tests, which are Docker containers orchestrated by Docker Compose.
So if you have the right versions of sbt, Scala and Docker/Compose installed, you run our tests with the command:
$ sbt "project api" ";dockerComposeUp;test;dockerComposeStop"
The first thing we can do is write a Makefile, and a Make rule that wraps this command:
# Makefile api-test: sbt "project api" ";dockerComposeUp;test;dockerComposeStop"
Now we can run
make api-test, and that has the same effect as running the sbt command, but we’ve hidden the details behind a human-friendly name.
You don’t have to remember the exact sbt command, just api-test. Tests for our other applications have a similar naming scheme: transformer-test, loris-test, ingestor-test, and so on — you don’t have to worry how to invoke sbt, or even if those tests use sbt at all! And if the test command ever changes, you only have to edit the Makefile once, and everybody picks up the change.
If you already know how Make works, you can skip to the next section. If you’re unfamiliar with Make, there’s a quick primer below.
target: [dependency dependency ...] [command 1] ... [command n]
A Makefile is a series of rules. Each rule starts with a dependency line, with a target before the colon, and a list of dependencies after the colon. In the first example, the target is
api-test, and there are no dependencies. Often the target is the name of the file you want to create, but a rule doesn’t have to create a file.
Each dependency line is followed by a series of tab-indented commands which will be run to build this target (and the indents have to be tabs, not spaces). In the first example, the rule has a single command: the sbt call that runs the API tests.
Here’s what happens when you run
Make looks to see if the target already exists.
If the target already exists, it looks at the modified dates: is the target newer than all the dependencies? If so, it’s already up-to-date, and there’s nothing to do. If not, the target is out-of-date, and needs to rebuilt.
If it doesn’t exist, it needs to be built from scratch.
If the target needs to be built, Make runs the commands in sequence. If any of the commands fail (have a non-zero exit code), the build has failed and Make will return an error.
We’ll see some more complicated examples in the rest of the post. Check you understand what they’re doing.
If I want to run these tests, I need to install Scala, sbt, and docker-compose on my local machine. I’d better hope they’re available in my package manager, and I can install them easily, and they don’t break something else I’ve installed… ick.
This is precisely the sort of problem that can be solved with Docker. Rather than installing dependencies on our main machine, we can install them in a Docker image, and run our tests there. Everything runs in a pristine, reproducible environment, and the installation process can be entirely automated.
Here’s a Dockerfile for installing our dependencies:
# sbt_wrapper.Dockerfile FROM pvansia/scala-sbt:0.13.13 RUN apk update && apk add docker py-pip RUN pip install docker-compose WORKDIR /repo ENTRYPOINT ["sbt"]
This file defines a new Docker image:
sbtwhen we run this container; any arguments we put on the end of
docker runwill be passed to sbt
The Dockerfile reference explains the syntax in much more detail.
We can build this image with
docker build, which we’ll wrap inside a Make rule:
build-sbt-docker-image: docker build --tag sbt_wrapper --file sbt_wrapper.Dockerfile .
And then we can modify our first Make rule to use the Docker image:
api-test: docker run \ --volume $$(pwd):/repo \ --volume /var/run/docker.sock:/var/run/docker.sock \ --net host \ sbt_wrapper "project api" ";dockerComposeUp;test;dockerComposeStop"
docker run as follows:
So now we have to build the image, then run the tests:
$ make build-sbt-docker-image $ make api-test
We’re improving: we only need to install Docker, dependency installation is automated, and we still have human-friendly ways to run our tasks. But we have to remember to build the Docker image first, or this doesn’t work. Can we automate that as well?
Remember that Make allows us to declare dependencies for a rule. If we add dependencies after the colon, Make will try to build those first before it builds the main target.
So we can tell Make that api-test depends on build-sbt-docker-image like so:
api-test: build-sbt-docker-image ...
And we’re back to running:
$ make api-test
When you run that command, Make will build a new Docker image, then run your tests inside the image. Hooray!
So we’re done… right?
If you try this out, you’ll find that Make rebuilds your Docker image every time you run your tests. Because Make never sees a file called build-sbt-docker-image, it assumes it has to rebuild it, and re-runs the rule. Docker caching means the rebuild is fairly fast on subsequent runs, but it still clutters up your console with logs. Could we do better?
We need Make to know when we’ve built the Docker image, so let’s modify the Make rule to drop a marker after it’s build the image. The next time Make runs, it’ll see the marker, know the image has already been built, and skip rebuilding it.
This is the pattern I typically use:
.docker/sbt_wrapper: sbt_wrapper.Dockerfile docker build --tag sbt_wrapper --file sbt_wrapper.Dockerfile . mkdir -p .docker touch .docker/sbt_wrapper api-test: .docker/sbt_wrapper ...
(Notice that I’ve also added a dependency on the Dockerfile, so if the Dockerfile does change, we’ll get a one-off rebuild.)
.docker directory is gitignored, and it serves as a collection of markers for Docker images.
This pattern makes a small difference to build times, but a big difference to console noise.
I’ve been using this system for about half a year, and it’s worked remarkably well. I have one-step rebuilds, human-friendly names for my build process, and a really mature and powerful dependency management system. You can see some examples in the platform repo’s Makefiles, or the Makefile for this blog.
It works particularly well for polyglot repos, where devs may have varying levels of familiarity with any given language. Somebody can get started without needing to learn the intricacies of the install process for a new language.
If you’re looking for a way to simplify or improve your build processes, I’d really encourage giving these technique a look.