gobench: a tool for comparing Go benchmark runs across branches

I’ve recently joined a new team that’s focused on performance, stability, and scalability – a change from my previous projects, where traffic was low enough that performance was rarely a concern.

This codebase has already gone through multiple rounds of optimisation, so there’s no low-hanging fruit left. Any proposed change is inherently complex, and it’s rarely obvious whether a tweak will actually speed things up or slow them down. To measure the impact of our work, we’re leaning on the benchmarking tools in Go’s standard library.

On one of my recent pull requests, I wrote a benchmark, and I pasted the output from two before/after runs. Paul improved this with a short shell script that ran multiple samples, and compared the results with the benchstat command.

$ for N in {1..6}; do for C in 613000c fc12d4c; do echo $N: $C; git checkout -q $C && ./tool/go test -run=^X -bench=BenchmarkPostureAnalyzer ./control/posture/analysis/ | tee -a bench-$C.txt; done; done
$ benchstat bench-fc12d4c.txt bench-613000c.txt

benchstat does statistical analysis on the output from multiple go test -bench=... runs, which is more reliable and easier to read than the raw output. In particular, it puts all the data in a single table, and shows the percentage change against the baseline.

I like this script, but it’s a little fiddly. I have to identify the two commits to compare, it leaves text files in my working directory, and it leaves my repo in a detached head state when it’s done. I wanted something more ergonomic.

I’ve written a new tool gobench to give me a smoother experience. I check out my feature branch, run gobench with the arguments I’d pass to go test, and it prints the final analysis from benchstat.

Under the hood, it creates a Git work tree for the base branch (leaving my working directory as-is), captures the benchmark output in memory (so there are no extra text files), and does an alternating sample of benchmarks.

The alternating samples is a feature I really like from Paul’s original script. I run benchmarks on my laptop rather than a dedicated workstation, and the benchmarks could be sensitive to thermal throttling, background processes, or other things happening on my computer. Alternating between the two branches reduces the effect of this noise.

Let’s look at an example. Suppose my main branch has a recursive implementation of Fibonacci numbers, while my feature branch has an iterative approach. Unsurprisingly, the iterative approach is faster, and gobench makes it easy to quantify the improvement.

$ go run ./cmd/gobench ./example/fib/ -bench=BenchmarkFib
Setting up temporary worktree for main (8f2351d)...
Running 6 alternating samples...
 100% |█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| (12/12, 42 it/min)

== Benchmark results ==
goos: darwin
goarch: arm64
pkg: alexwlchan.net/projects/gotools/example/fib
cpu: Apple M4 Pro
       │ main (8f2351d)  │        faster-fib (0969b03)        │
       │     sec/op      │   sec/op     vs base               │
Fib-12   14572.500n ± 6%   5.455n ± 2%  -99.96% (p=0.002 n=6)

My first version of gobench was an expansion of Paul’s script, but as I started adding features, it became clear that Bash was the wrong tool. I rewrote it as a Go tool that shells out to git, go test, and benchstat.

If you’d like to try gobench yourself, the code is available in my gotools repository, or you can install it with:

$ go install alexwlchan.net/projects/gotools/cmd/gobench@latest

gobench has already saved me from shipping code that looked fast on paper but regressed in practice. As I keep working on performance-critical Go code, having low-friction benchmarks will make it easier to test my ideas early and often.