Recently I’ve been looking for a new HTTP library, because my previous choice seems abandoned. I was using httpx, but the maintainer has closed issues on the GitHub repo, there’s only been one commit since January, and the last release was over a year ago. The easy choice would be switching to requests or urllib3, but I wondered: can I just use the standard library?

My usage is pretty basic – I have some manually-invoked scripts that make a handful of GET requests to public websites. I don’t have long-running processes; I’m not making thousands of requests at once; I’m not using proxies or authentication. There are plenty of features you can only get from third-party HTTP libraries – from connection pooling to HTTP/2 support – but I don’t need any of them.

I started experimenting, and what I realised is that I don’t miss the features, but I do miss the API.

Here’s how you make a basic GET request with httpx:

import httpx

resp = httpx.get(
    "https://example.com",
    params={"name": "pentagon", "sides": "5"},
    headers={"User-Agent": "Shape-Sorter/1.0"}
)
print(resp.content)

Here’s the same request with urllib.request:

import urllib.parse
import urllib.request

url = "https://example.com"
params = {"name": "pentagon", "sides": "5"}
headers = {"User-Agent": "Shape-Sorter/1.0"}

u = urllib.parse.urlsplit(url)
query = urllib.parse.urlencode(params)
url = urllib.parse.urlunsplit(
    (u.scheme, u.netloc, u.path, query, u.fragment)
)

req = urllib.request.Request(url, headers=headers)

resp = urllib.request.urlopen(req)
print(resp.read())

Verbose! I’ve wrapped it in a helper function in chives, my personal utility library. Here’s the same request a third time:

from chives.fetch import fetch_url

resp = fetch_url(
    "https://example.com",
    params={"name": "pentagon", "sides": "5"},
    headers={"User-Agent": "Shape-Sorter/1.0"}
)
print(resp)

Much cleaner!

The code in chives does have one dependency – certfi, a lightweight package that provides Mozilla’s collection of root certificates.

There are lots of good reasons to use a third-party HTTP library, but I can do everything I need with the standard library and my personal wrapper. Let’s go through how it works.

Building the urllib.request.Request object

The first step is building the Request object. Other HTTP libraries provide helper functions or hide this step for simple requests (notice the basic httpx.get call doesn’t mention an httpx.Request), but for urllib.request we have to do it ourselves. Here’s mine:

import urllib.parse
import urllib.request


QueryParams = dict[str, str] | list[tuple[str, str]]
Headers = dict[str, str]


def build_request(
    url: str,
    *,
    params: QueryParams | None = None,
    headers: Headers | None = None
) -> urllib.request.Request:
    """
    Build a urllib Request, appending query parameters and attaching headers.
    """
    if params is not None:
        params_list = list(params.items()) if isinstance(params, dict) else params

        u = urllib.parse.urlsplit(url)
        query = urllib.parse.parse_qsl(u.query) + params_list
        new_query = urllib.parse.urlencode(query)
        url = urllib.parse.urlunsplit(
            (u.scheme, u.netloc, u.path, new_query, u.fragment)
        )

    req = urllib.request.Request(url, headers=headers or {})

    return req

I can pass params as a dict or as a list of (key, value) tuples; I start by converting it to the list form. This means I can pass the same query parameter multiple times in a URL. That’s admittedly unusual, but I use it on a couple of my websites so I wanted to support it here.

I’m using the urllib.parse module to manipulate the URL and append the query parameters. I parse the initial URL with urlsplit, encode the query parameters, then reassemble the URL with urlunsplit. This preserves any existing query parameters and fragments, and returns a complete URL I can pass to the Request object.

(If, like me, you’d reach for the urlparse function, you’re showing your age – one thing I learnt during this project is that urlparse is now obsolete, and urlsplit is the replacement.)

This function only handles GET requests, which is all I need for my scripts – but it wouldn’t be difficult to extend it to handle POST requests or form data if the need arises.

This is a pure function, so it’s easy to test thoroughly.

Getting a web page or an API endpoint

In most cases, I just care about getting the response body from the remote server, not the headers or URL – for example, if I’m fetching a web page or an API endpoint. If I want something different in a single script, I’ll eschew my wrapper and use urllib.request directly.

Here’s my fetch_url wrapper:

import certifi
import ssl


def fetch_url(
    url: str,
    *,
    params: QueryParams | None = None,
    headers: Headers | None = None
) -> bytes:
    """
    Fetch the contents of a URL and return the body of the response.
    """
    req = build_request(url, params=params, headers=headers)
    
    ssl_context = ssl.create_default_context(cafile=certifi.where())

    with urllib.request.urlopen(req, context=ssl_context) as resp:
        data: bytes = resp.read()

    return data

The key function is urllib.request.urlopen, which is what actually makes the HTTP request. I’m passing it two parameters: a Request and an SSLContext.

We build the Request using the build_request function.

The SSLContext tells urllib.request which HTTPS certificates it can trust, in this case by pointing to a “cafile” (Certificate Authority file) file provided by the certifi library. This file contains a list of trusted root certificates, and all valid HTTPS certificates should eventually point back to an entry in this list.

The certifi library is a lightweight wrapper around Mozilla’s list of trusted Root Certificates. It’s not in the standard library because it’s important to stay up to date with changes to the list, and you don’t want those changes coupled to Python version releases. Although this exercise is about reducing dependencies, I’m okay with certifi because it’s tiny – you can read the whole thing in less than five minutes. I know what it’s doing.

The urlopen function looks for a 200 OK status code, and throws an HTTPError if it gets an error response from the server. I considered wrapping that in another type, but for now I’m just catching HTTPError.

This function doesn’t set a timeout on HTTP requests. That would be an issue in a lot of contexts, but I’m normally using this from a script I run manually. If something gets stuck, I can stop the script and debug manually.

This function doesn’t support streaming responses; it reads the whole thing into memory at once. That’s fine for web pages or API calls, but I wouldn’t use this to download large files or videos.

There’s a lot of stuff this function doesn’t do, but it works well in all of my scripts, it has a friendly API, and it only has one third-party dependency.

Downloading images with format-based file extensions

As I started using the fetch_url in my projects, I realised the one time I often care about response headers is when I’m downloading images. I want the filename to have the appropriate filename extension – .jpg for JPEGs, .png for PNGs, and so on. Sometimes I can guess the file format from the URL, but sometimes I need to inspect the Content-Type header.

I considered exposing the headers from fetch_url, but since I only need the headers for downloading images and that’s a pretty common operation, I decided to make a download_image helper instead.

First, I wrote a helper function that picks a filename extension based on the Content-Type header:

def choose_filename_extension(content_type: str | None) -> str:
    """
    Choose a filename extension for an image downloaded with the given
    Content-Type header.
    """
    if content_type is None:
        raise ValueError(
            "no Content-Type header, cannot determine image format"
        )

    content_type_mapping = {
        "image/jpeg": "jpg",
        "image/png": "png",
        "image/gif": "gif",
        "image/webp": "webp",
    }

    try:
        return content_type_mapping[content_type]
    except KeyError:
        raise ValueError(f"unrecognised Content-Type header: {content_type}")

The mapping contains the four image formats I encounter in practice; it’s easy for me to add more if I try to download a newer format someday.

Then I wrote a function that takes an image URL and an “out prefix” (an initial guess at the path), downloads the image and choose a new file extension, and returns the final path:

from pathlib import Path


def download_image(
    url: str,
    out_prefix: Path,
    *,
    params: QueryParams | None = None,
    headers: Headers | None = None,
) -> Path:
    """
    Download an image from the given URL to the target path, and return
    the path of the downloaded file.

    Add the appropriate file extension, based on the image's Content-Type.

    Throws a FileExistsError if you try to overwrite an existing file.
    """
    req = build_request(url, params=params, headers=headers)

    ssl_context = ssl.create_default_context(cafile=certifi.where())

    with urllib.request.urlopen(req, context=ssl_context) as resp:
        image_data: bytes = resp.read()

    image_format = choose_filename_extension(content_type=resp.headers["content-type"])

    out_path = out_prefix.with_suffix("." + image_format)
    out_path.parent.mkdir(exist_ok=True, parents=True)
    with open(out_path, "xb") as out_file:
        out_file.write(image_data)

    return out_path

The first half of this function is the same as fetch_url; the second half constructs the final path and writes the download image to disk. I like this approach because it allows the caller to specify a meaningful directory and filename without worrying about the filename extension (which is important but not meaningful).

The function creates the output directory if it doesn’t exist, for convenience. Nothing grinds my gears like getting a FileNotFoundError when trying to write to a file in a folder that doesn’t exist. My text editor is smart enough to auto-create missing folders; I want my code to do the same.

I open the file in xb mode to avoid overwriting existing files – if I try to write to an image I’ve already saved, I get a FileExistsError. I find that a useful safety check, and I use exclusive creation mode in a lot of my scripts now.

Packaging and testing

A few months ago, I created a personal utility library chives for dealing with tiny archives, and that was a good place to keep this code.

The HTTP code is in chives.fetch, and the accompanying tests are in test_fetch.py. I’m testing it using the vcrpy library, which knows how to record responses from urllib.request.

I now use this code across all my personal scripts, and it’s been rock-solid. There are lots of good reasons to use Python’s more advanced HTTP libraries, but they’re for use cases I don’t have.

[If the formatting of this post looks odd in your feed reader, visit the original article]

So far I’ve been unaffected, because the attacks have only involved libraries or packages I don’t use – but it would be foolish to imagine that will always be the case. I have a lot of local Python projects, and I’ve been thinking about how I’d react if a Python package I use was compromised.

The first step is detection: once I know a package version is malicious, how do I know if I’ve installed it? Because I use virtual environments, this turns out to be a non-trivial question.

What are virtual environments?

Virtual environments (or “virtualenvs”) are a tool to create isolated Python environments, each with its own set of installed packages. They allow you to have different dependencies for different projects. For example, if two projects depend on different versions of the same package, you can create per-project virtualenvs, each with the appropriate version.

A virtualenv is stored in a folder that includes symlinks to the global Python interpreter and the packages you’ve installed in the virtualenv. When you “activate” the virtualenv, commands like pip install install packages in the virtualenv folder rather than your global Python.

Here’s an example:

$ # `python3` points to my global interpreter
$ which python3
/Library/Frameworks/Python.framework/Versions/3.13/bin/python3

$ # Create the virtualenv
$ python3 -m venv .venv

$ # Activate the virtualenv, so now `python3` and `pip` commands will
$ # run inside the virtualenv
$ source .venv/bin/activate

$ # `python3` now points to the symlink in the virtualenv
$ which python3
/private/tmp/example/.venv/bin/python3

$ # Pillow will be installed inside the `.venv` folder
$ pip install Pillow

I create a new virtualenv for every Python project, so I have a lot of different virtualenvs on my personal Mac.

To check if I’d installed version X of package Y, I’d have to check each of my virtualenvs. Python itself doesn’t keep a running list of virtualenvs I’ve created, so I have to manage that list myself.

Getting a list of my virtualenvs

I’m very consistent about naming my virtualenvs: the folder is always named .venv. (I actually have a shell function for creating virtualenvs, which enforces that convention.)

This means I can find all the virtualenvs in my home directory with a one-line command:

$ find ~ -type d -name .venv
/Users/alexwlchan/repos/snippets/.venv
/Users/alexwlchan/repos/alexwlchan.net/.venv
/Users/alexwlchan/repos/colour-scheme/.venv
…

I can similarly search external drives and volumes where I have virtualenvs:

$ find /Volumes/Media/ -type d -name .venv
/Volumes/Media/Screenshots/.venv
/Volumes/Media/Social Media/.venv
/Volumes/Media/Bookmarks/.venv
…

These commands take about 30 seconds to run – just long enough to be annoying – so I’ve saved the results to a text file:

$ find ~ -type d -name .venv >> ~/.venv_registry
$ find /Volumes/Media/ -type d -name .venv >> ~/.venv_registry

I’ve also modified my shell function that creates virtualenvs to update this file whenever I create a new virtualenv. Now I have an up-to-date list of all my virtualenvs that I can use to search for vulnerable dependencies.

What about Python packages installed outside virtualenvs?

If you run pip install without activating a virtualenv, the packages will get installed in your global Python installation, and they wouldn’t be included in this list. This is generally a bad idea, because you’re back to the problem of different projects using incompatible dependencies.

You can tell pip that it should only use virtualenvs, either with an environment variable or a config file. Once you set up that config, pip will refuse to install packages outside a virtualenv.

Alternatively, if you use uv instead of pip, you can’t install packages outside a virtualenv unless you explicitly pass the --system flag to modify your system Python.

I set the PIP_REQUIRE_VIRTUALENV=true in my shell config file, and I use uv, so I don’t have any Python packages installed outside virtualenvs.

Searching my virtualenvs for package versions

Now I have a text file with a list of all my virtualenvs, I can write scripts that run commands in each of them.

For example, here’s a bash script that runs uv pip freeze in every virtualenv to print a list of installed dependencies:

#!/usr/bin/env bash

set -o errexit
set -o nounset

while read -r venv_dir; do
  if ! test -d "$venv_dir"; then
    echo "does not exist: $venv_dir" >&2
    continue
  fi
  
  echo "== $venv_dir =="
  uv pip freeze --python "$venv_dir/bin/python"
  echo ""
done < ~/.venv_registry

Within half a second, I have a complete list of every Python package installed in every virtualenv on my Mac. I dump the output to a text file, and then I can look for compromised package versions – or reassure myself that I don’t have a package installed, not even as an indirect dependency.

I skip missing virtualenvs because they’re probably temporary environments I have yet to clean up from my registry, or virtualenvs on external drives that are currently unmounted.

I like that this script doesn’t run the Python interpreter itself, so I won’t make things worse if I’ve already installed a malicious package. In particular, uv is a Rust tool that doesn’t run any Python code, it just knows how to understand Python installations.

For example, with the recent LiteLLM compromise, the attackers installed a .pth file which would run as soon as you started Python, even if you didn’t import LiteLLM. Even a basic python --version or pip freeze would compromise your machine. I could easily modify this script to look for the malicious .pth file in all of my Python environments, without ever running Python.

Other uses for a virtualenv registry

I originally wrote this to detect compromised packages, but I found other uses:

• I can find outdated versions of packages, and make sure all my virtualenvs are up-to-date.

• If I’m trying to stop using a package, I can find any places I’m stll using it and remove it. For example, I’m trying to replace some third-party HTTP libraries with the standard library, and these scripts help me find where I’m still using the third-party libraries.

• I can search all my Python code for places where I use specific functions or features, in a more efficient way than grepping my entire disk. For example, I have a couple of personal utility libraries, and I can see which functions I’m still using and which can be deleted. I do this by searching the parent directory of each .venv path, which is the root of each project.

I hope none of the libraries I use are ever compromised, but if they are, I’ll be ready – and in the meantime, this is a useful tool to have around.

[If the formatting of this post looks odd in your feed reader, visit the original article]

I found Filippo Valsorda’s post especially helpful in understanding the scale of the risk. We should assume that practical quantum computers are arriving imminently, and roll out quantum-resistant cryptography everywhere, lest we be caught unprepared and leave ourselves at risk.

Google have set a 2029 deadline for moving to quantum-resistant cryptography; Cloudflare have done similar. (Similar internal discussions are happening at my workplace but there aren’t any public announcements yet.)

Amidst all the concern, I was pleasantly surprised to discover that my website is already using quantum-resistant cryptography, and I didn’t even realise.

What’s the threat?

All “classical” public-key cryptography relies on hard mathematical problems – operations that are easy to compute in one direction, but incredibly difficult to do in reverse.

For example, it’s easy to multiply two prime numbers together and compute the result, but working out those two prime numbers if you only have the result is impossibly hard. Even for fairly small numbers, you could be working until the heat death of the universe and still not have an answer.

Quantum computers work differently to traditional computers, and a sufficiently powerful one can reverse these one-way computations. That would break all of our existing cryptography.

This is the cryptography that underpins almost everything we do online – protecting banks, governments, militaries, and pretty much everyone else. If somebody had a quantum computer that could crack it, all of that information would become readable to them. It would be disastrous.

Small-scale quantum computers already exist in labs, but nothing powerful enough to break public key cryptography – for now. Researchers have been trying to build bigger and better quantum computers, but they were a long way away from building anything this powerful. They’d likely get there eventually, but that was expected to be a long time away – late 2030s at the earliest.

Other researchers have been developing new cryptographic algorithms that rely on different maths problems, which can’t be easily broken by quantum computers. These new algorithms are the so-called “post-quantum cryptography (PQC)” or “quantum-resistant cryptography”. They’ve gradually been formalised as standards, and are starting to be used by our devices. For example, all the popular web browsers now support PQC for HTTPS certificates.

Previously, organisations like the NCSC or NIST recommended a 2035 deadline for migrating to PQC. The idea was to be fully migrated long before quantum computers became a practical threat. That recommendation wasn’t just an abundance of caution – it’s to eliminate the risk of Harvest Now, Decrypt Later (HNDL) attacks, where an adversary records data encrypted with classical cryptography, and waits until they have a quantum computer that can unlock it. The sooner we migrate to PQC, the more expensive and less valuable such an attack becomes.

Now, it appears we need more urgency.

The two recently published papers narrow the gap between the experimental machines we have today and a practical threat. They describe efficiency improvements that would allow quantum computers to reverse these mathematical operations with far less computing power. It’s become more plausible that somebody could build a “sufficiently powerful” machine within a few years. It’s also becoming a smarter bet to throw lots of money at building one right now, where previously the odds of success were so low as to make that an unwise bet.

This is why Google, Cloudflare, and others are moving forward their deadlines for migrating to post-quantum cryptography. The threat has gone from “late 2030s if we’re unlucky” to “early 2030s, maybe sooner”.

What about this blog?

While reading the recent news about this issue, I found Cloudflare’s post-quantum encryption radar, which tells you how many websites are protected using post-quantum cryptography. My website isn’t hosted on Cloudflare but I decided to try it anyway, and I was surprised by the result. I’m already protected!

I never set up post-quantum cryptography for this site, but it’s enabled anyway, because I’m using Caddy as my web server, and Caddy’s default TLS settings include PQC support. At some point I updated to a new version of Caddy, I got these new defaults, and my site started quietly serving traffic with quantum-resistant cryptography.

This is exactly what I wanted when I switched to Caddy. I’m not an expert on cryptography, or TLS, or securing servers, so I wanted a web server that would make sensible decisions for me. I’ve mostly been ignorant of post-quantum cryptography and developments in quantum computing, but Caddy was protecting me anyway.

There’s a lot more work to do to use quantum-resistant cryptography everywhere, and recent announcements have made it far more urgent – but we can all sleep easier knowing my little blog is safe from quantum computers.

[If the formatting of this post looks odd in your feed reader, visit the original article]

Every spring, my council publish a new bin collection calendar. These calendars are typically published as a single PDF to cover the entire region, with the information packed into a compact design. I imagine this design is for economy of printing – you can print one calendar in bulk, and post the same thing to everybody.

Here’s an example of this sort of compact diagram from South Cambridge, where breaks the county into four different regions:

For example, if your usual bin day is Thursday, your final collection of the year would be on Monday 22nd December.

This compact representation is a marvel of design, but it’s not that useful for me, a person who only lives in a single house. I only care about bin day on my street, not across the county.

For several years now, I’ve created a personalised calendar which shows when my bins will be collected, which gets printed and stuck it on my fridge. It’s a manual process, but a small amount of effort now pays off across the year.

I start by generating an HTML calendar using Python. There’s a built-in calendar module, which lets you output calendars in different formats. It doesn’t embed individual date information in the <td> cells, so I customise the HTMLCalendar class to write the date as an id attribute.

Here’s my script, which generates a calendar from April 2026 to March 2027:

from calendar import HTMLCalendar
from datetime import date


class PerDateCalendar(HTMLCalendar):
    """
    A customised HTML calendar that adds an `id` attribute to every day
    (for example, `d-2026-03-27`) and uses single-letter abbrevations for
    days of the week (M, Tu, W, …).
    """

    def formatday(self, day: int, weekday: int) -> str:
        """
        Returns a table cell representing a single day, or an empty cell
        if this is a blank space in the calendar.
        """
        if day == 0:
            return f'<td class="{self.cssclass_noday}">&nbsp;</td>'
        else:
            current_date = date(self.current_year, self.current_month, day)
            date_string = current_date.strftime("%Y-%m-%d")
            return f'<td id="d-{date_string}">{day}</td>'

    def formatmonth(self, year: int, month: int, withyear=True) -> str:
        """
        Returns a table representing a month's calendar.
        """
        # Store the current month/year so they're visible to formatday()
        self.current_year = year
        self.current_month = month

        return super().formatmonth(year, month, withyear)

    def formatweekday(self, day: int) -> str:
        """
        Returns a table header cell representing the name of a single weekday.
        """
        custom_names = ["M", "Tu", "W", "Th", "F", "Sa", "Su"]

        return f"<th>{custom_names[day]}</th>"


if __name__ == "__main__":
    cal = PerDateCalendar()

    start_year, start_month = 2026, 4
    end_year, end_month = 2027, 3

    full_calendar_html = (
        "<html>"
        '<head><link href="style.css" rel="stylesheet"></head>'
        '<body><div id="grid">'
    )

    current_year, current_month = start_year, start_month

    while (current_year < end_year) or (
        current_year == end_year and current_month <= end_month
    ):
        month_html = cal.formatmonth(current_year, current_month)
        full_calendar_html += month_html

        if current_month == 12:
            current_month = 1
            current_year += 1
        else:
            current_month += 1

    full_calendar_html += "</div></body></html>"

    with open("bin_calendar.html", "w") as f:
        f.write(full_calendar_html)

This writes a calendar to an HTML file, where each month is a table, and each day is an individually identifiable cell. Here’s a sample of the output:

<table border="0" cellpadding="0" cellspacing="0" class="month">
<tr>
  <th colspan="7" class="month">April 2026</th>
</tr>
<tr>
  <th>M</th>
  <th>Tu</th>
  <th>W</th>
  <th>Th</th>
  <th>F</th>
  <th>Sa</th>
  <th>Su</th>
</tr>
<tr>
  <td class="noday">&nbsp;</td>
  <td class="noday">&nbsp;</td>
  <td id="d-2026-04-01">1</td>
  <td id="d-2026-04-02">2</td>
  <td id="d-2026-04-03">3</td>
  <td id="d-2026-04-04">4</td>
  <td id="d-2026-04-05">5</td>
</tr>

The HTML references an external stylesheet style.css, which contains some basic styles that turn the calendar into a three-column view:

#grid {
  display: grid;
  grid-template-columns: repeat(3, 1fr);
  gap: 3em;
  width: 600px;
  margin: 0 auto;
  font-family: Helvetica;
}

th {
  padding-bottom: 5px;
}

td {
  font-size:   0.9em;
  line-height: 1.4em;
  text-align: center;
}

Then I can highlight the individual days for my bin collections, by targeting the <td> cells for each day using the id I created:

#d-2026-04-03,
#d-2026-04-24 {
  font-size: 1.1em;
  font-weight: bold;
  background: black;
  color: white;
  
  border-bottom: 1px solid white;
  border-top:    1px solid white;
}

#d-2026-04-10,
#d-2026-04-24 {
  font-size: 1.1em;
  font-weight: bold;
  background: green;
  color: white;
  
  border-bottom: 1px solid white;
  border-top:    1px solid white;
}

It takes less than five minutes for me to transcribe all my bin dates to the calendar by hand, and this is what the result looks like:

That fits nicely on a single sheet of paper, so I print it and stick it on my fridge. It’s easy to see when I have an off-cycle bin day, or when my next collection is going to be.

I often use this to know if I can skip a collection. I live on my own and I only generate a small amount of waste, so my bins are rarely more than half-full. I don’t think it’s worth putting out a half-empty bin, but I’ll do it anyway if I can see I’ll be away for the next few collections.

[If the formatting of this post looks odd in your feed reader, visit the original article]

This year’s theme was “Prepping Craft” – in a world where safety and normality are slipping away, how do we make our code, our communities, and our lives truly resilient? I loved the topic when I first heard it, and it’s only become more urgent by the day.

There was very little discussion of specific tools or technologies; instead, the event focused on the politics and human impact of the tech industry. Monki Gras has always been an interdisciplinary event, but this year it felt even more focused on humans over technology, and the politics were more pronounced than ever.

Despite the state of the world, the mood was constructive and upbeat. Everything is in a bad place, so how do we make it better?

Some particular highlights for me: Heidi Waterhouse and Kim Harrison describing how to create resilient, interdependent communities in Minneapolis and Oaxaca. Hazel Weakly gave a deeply personal, poetic talk about rebuilding a sense of self, and finding joy in your own existence. Adam Zimman made me cry when he talked about being a cis parent of a trans child, and what he learnt about supporting them. These are not talks you’d get at most tech events.

Three of this year’s talks were about trans experiences, at a time when the political winds are blowing in a transphobic direction. Some people would find that surprising, but I didn’t – Monki Gras has always felt like a safe place to be trans and to talk about being trans. I’ve always felt like I can be my full self there, and being trans at the event is a complete non-issue.

My one regret is that there’s never enough time to talk to all the cool and interesting people who attend. The community around this event is lovely, and I wish I had more time to chat. (Although after two days I was ready to collapse into bed; it’s as fun as it is intense.)

I was flattered by the number of people who expressed surprise that I wasn’t speaking this year. I did write a proposal, but James politely suggested that since I’ve spoken at the last three events in a row, I should step aside for new speakers. Months in advance, I knew it was a good call – preparing a talk is a lot of work, and it was nice to relax and just enjoy the event. Now the event is over, I’m even more glad I skipped this year.

Watching the talks made me realise that since I left social media, I’ve lost my nerve for talking about personal or political topics. I worry the talk I’d have given this year would have felt anodyne next to the other speakers, and I want to fix that. I’m inspired to be more personal and vulnerable in my public writing, and some of my talk ideas will get recycled as articles on this site. I’ve already published one (Dreaming of a ten-year computer) and there’s more to come.

Monki Gras was created by James Governor, and he and his team have done an excellent job of creating a thoughtful, inclusive event that explores the impact of tech on the world.

I’ve shared my notes and highlights below, but they’re no substitute for the real thing. If you’ve never been, I’d strongly recommend it for 2027.

Be prepared for more: data vs hype, by Laura Tacho

• Storytelling can be misused: Laura opened with the story of Percival Lowell, who successfully convinced many people that there were canals on Mars. Any criticism was deflected back to the critic – for example, if you can’t see them, it means your telescope isn’t good enough. The burden of proof was shifted from the sceptics to the claimant.

• “What could somebody with bad intentions get you to believe?”

• The average impact of AI shows a moderate improvement, but there’s increase variance and disparity. These tools may be enhancing individual productivity, but that isn’t leading to an impact on the P&L.

• What’s the value of innovation? NASA publishes Spinoff which describes the benefits of the space race; will AI be the same?

• There is no silver bullet that will help you solve human and org-level problems.

• Key takeaways:

  1. Be sceptical of extraordinary claims. “How does it benefit me to believe this claim?”
  2. Use good storytelling; narratives stick.
  3. Set goals and measure progress.
  4. Developer experience matters more than ever.
  5. Experiment by solving real customer problems.

Preparing to Win in The Age Of Robots, by Ana Hevesi

• “What if the next model makes us irrelevant?” – is this fear grounded in reality, or is it a leftover from the initial shock of ChatGPT?

• Focus on your impact on people.

• What can you contro;?

  1. How much you know about how you’re serving people – metrics, user research, and s on
  2. Interaction patterns
  3. Docs and acquisition of mastery
  4. Mastery and meaningful personal experience leads to shared culture and community

• Look for opportunities to do “cultural ratcheting”.

Trust Before Truth, by Ashley Rolfmore

• Numbers are scary because they are real.

  Their context gives them meaning. 400 means nothing on its own, but £400.14 is the maximum universal credit you can receive per week. That’s a real and scary number.

• Trust in institutions has been declining over past decades; the tech industry is part of that. (Both causing the decline in trust, and having trust decline in it.)

• Ashley talked about how double-entry bookkeeping was the foundational idea that eventually led to modern accounting.

• Show your working. Help people understand where your numbers come from.

• Be correctable, not just correct.

• “Incorrect numbers disrupt your shared reality with your user.”

• Railway time was a standardised time used on the railways to deal with the fact that different places in Britain had different local timezones. This is a precursor to the timezones we have today.

  GMT is centred in England. “This system had caught colonialism from its humans.”

  Imagine if you had to explain the history of railway time and timezones for every timestamp you shared; it would be impossible. Timekeeping standards on modern railways treat railway time as a historical footnote, not essential knowledge, and everything works fine.

  Key insight: “Cognitive debt is not failure if you keep the context window intentional.”

• Extend mental models, don’t rebuild them.

• Fail safe: minimise harm vs maximise accuracy. Consider what will happen when the system goes wrong, and design for safe failure. 100% accuracy is impossible; don’t forget about failures in the pursuit of perfection.

• “If your site URL ends with .ai, all website bugs are now AI bugs.”

Do your fingers remember how to code? by Sue Smith

• What skills should devs be focusing on? Don’t know – this talk is about strategies and techniques for discovering what developer skills are.

• Acquiring a skill is different to retaining it. Once you lose achieve a level of proficiency, it’s difficult to lose skill

• Skills are repeatable. This means generative AI in its current form is not a skill; it can’t be used predictably.

• Teaching means guessing where the learner is, and finding out where you were wrong.

• The act of saying something aloud can help you understand it. [Alex: this is the production effect.]

• Sue described a swimmer Terry Law (sp?), whose attitude was: “every time I get out of water, I’ll be a better swimmer”. He took copious notes after ever swim.

• Semantic waves of learning: from a high level of theory to practical implementation, and back again.

• “Where do you find value in the process and not the outcome?”

What to do when your passport is taken away, by Liz Fong-Jones

• Preparation means acting on incomplete information, and preserving resources to be able to do that. In Liz’s case, that meant driving several hours to renew her passport after the news broke that the Trump administration wouldn’t issue new passports for trans people – but she was able to renew an existing passport.

• Trans people are under greater threat than in a long time: existentially threatening politics in the US; bathroom bans in the UK.

• Liz is good at resilience engineering because it affects her personal life: runbooks for personal threats, plans for leaving her home and country, and so on. She showed us a complex risk spreadsheet she uses to track personal risk; I didn’t have time to read it in detail.

Prepping Hazel Weakly – being intentional in life and in software, by Hazel Weakly

• Hazel’s first memory is one of loneliness; she has about ten memories from before she was 26. She talked about dissociating as a child.

• Axioms of self:

  1. I choose to love humanity
  2. I know what I know, what I don’t know, the difference
  3. Extraordinary claims require extraordinary evidence, but nothing is absolute
  4. I Embrace my shortcomings (maybe too much)
  5. I choose my addictions so they don’t choose me

• Being deaf means questioning everything I hear, and never believing my own reality.

• She had to learn “normality” – a combination of being autistic, trans, and ADHD.

• Our past shapes us, but it does not have to guide us.

• “I have thoughts. Lots of thoughts. They never stop thinking. Never stop thunking.”

• Oxygen is vital to life, but it’s also poisonous in its pure form.

• Purpose:

  • Teach the world to build self-organising ecosystems
  • Find (?) the infinitesimal spirit of chaos
  • Dance in the rain

• Desires:

  • To help others feel understood and make them feel safe
  • To build a world where nobody experiences my childhood again

• Reception: what impression do I want to give? What do I want others to see in myself?

• Fashion and makeup: “Vanity is a gift to myself. It is permission to live in my body with joy.”

• “When I give myself permission to experience my existence with joy, then I give everybody else permission to do the same.”

• “I rebuilt myself into a completely different person who is far cuter, and she’s more (entertaining? everything?) and has a spark in her eye she didn’t feel possible.”

• “Build that community of belonging and you will be surprised at what life and the universe of chaos will bring to you in sparkles and fits of unmitigated joy.”

Atproto, federation, kindness, and resilience, by Daniel Roe

• Daniel started the npmx project, an alternative npm browser that’s built in part on atproto. The project grew quickly, and attracted a lot of contributors – to the point the core maintainers took a week-long break as it was gaining traction, to prevent burnout.

• There aren’t 10x developers, but there are 10x teams. They iterate together, and make each other better.

• atproto allows you to get something akin to federated JSON. [Alex: the best explanation I’ve seen for this so far is Dan Abramov’s article A Social Filesystem.]

• Standard.site is a schema for publishing blogs on atproto.

• atproto is not a USP, it’s an implementation detail.

• Running an app based on atproto means less responsibility, because you don’t have to run a backend – other people can look after their data.

• Enshittification comes from asymmetry of power; atproto can help to level that balance.

Fixing the Open Source Bus Number, by Holly Cummins and Sanne Grinovero

• This talk covered a lot of material I’m familiar with, so I didn’t take many notes.

• The problems facing open source maintainers: money, time, burnout, boredom, user hostility.

• “The value of people isn’t cranking out code; it’s being guardians of quality and performance.”

• Other bad things: losing keys, losing the vision and meaning of a brand.

Resilience needs conscious observability, by Adrian Cockcroft

• “Footprints on the sand of time”, from Henry Wadsworth Longfellow’s poem A Psalm of Life.

• “Why are you trying to make something resilient?”

• Career resilience:

  • Quality over quantity
  • Prioritise what you want to do
  • Make your work visible [Alex: see Julia Evans’ article Write a brag document]
  • APIs allowed developers to do the work of dedicated ops and infrastructure teams; AI is the same thing but for developers writing code

• Business resilience:

  • There’s a lot of FOMO around AI; speed wins here
  • Probe, sense, and respond
  • When you get a new manager, ask them for stuff. “Your predecessor didn’t approve this interesting thing, do this and it will make you look good.”
  • There are constraints that affect your choices

• Systems resilience:

  • What should users see when the system falls over?
  • Test that your mitigation mechaisms work; an untested backup is broken

• Risk = severity × probability × obscurity × delay

Preparing for unimaginable change, by Danilo Campos

• If we’re entering a post-scarcity future, we should be more excited than scared.

• “We prefer our world of toilets to open sewers. Think about this for a moment. This is a science fiction shit. You have a magic bowl where you press a button and whatever you want to get rid of is gone.”

• A brief history of civilisation [Alex: reminded me of How to Invent Everything].

• A heat pump and solar means that on hot days, air conditioning essentially free.

• “The fruits of abundance are often captured.”

• We can harness solar and improve energy infrastructure: it’s possible, but it needs work.

• “It’s not as grim as we’ve always been told.” [Alex: London’s air pollution sprung to mind; it was initially projected that it would take 193 years to bring NO2 pollution to within legal levels, but it was achieved in 9 years.]

• “Thought is fuel.”

• There’s a crappy remote for the air conditioner and heating controllers; replace it with microcontrollers and a vibe-coded app in Python.

• “Give me a context limit large enough and I can move the world.”

• Mario’s grandfather is Fairchild Semiconductor, originally developed for military applications. This led to an era of compounding automations on semiconductors: modems, video games, programming.

  LLMs and AI are the start of a new 50-year super cycle of compounding automation.

• The person who created Mario was “not a video game designer” because that job didn’t exist yet. What new careers will be created in the era of LLMs?

• We have abundant energy and cognition.

• Modern warfare is different to what it used to be – widespread use of drones and automonous vehicles. But warfare is how we got the microprocessor in the first place.

• We can have a positive vision for this abundance; we can counter the vision of darkness.

• “We can assert a positive vision of how these strange discoveries shape our future.”

Story of a prepper, by Chad Metcalfe

• When we hear the word “prepper”, we think of a doomsday-obsessed, right-wing obsessive with guns.

• Attitudes to war have changed. Many of Chad’s peers from Texas joined the military, and they can’t walk 5 minutes before somebody thanks them for their service. His father fought in Vietnam, and didn’t want to talk about it when he returned.

• “In a world that feels out of control, prepping is a mechanism to assert personal control.” This takes different forms and the meaning of “prepper” and “survivalist” has changed over time, but it’s the same underlying desire.

• Eric Diven:

  I no longer build software; I now make furniture out of wood. The hours are long, the pay sucks, and there’s always the opportunity to remove my finger with a table saw, but nobody asks me if I can add an RSS feed to a DBMS, so there’s that :-)

• Chad bought a 20-acre gold mine to turn into a self-sufficient space. He had to learn about water systems, power, sewage; he broke both wrists falling off a 12′ ladder; he had to navigate open shafts across the property (one over 150′ deep). [Alex: the pictures reminded me of Firewatch.]

• This sort of living forces you to update your threat model.

• Fire is scary! Half of the USA is protected by volunteer fire departments; they can’t be everywhere.

  [Alex: this reminded me of the Lemony Snicket series; the concept of a volunteer fire department was completely new to me as a British reader, but I guess it would make more sense to a US-ian reader?]

• After a lightning strike on the corner of his property: “A grass fire moves at 14mph. That lightning strike struck 800 feet away. That’s 39 seconds.”

• We’re all prepping for something, so how do you react?

  1. Put your head in the sand and ignore it
  2. Outsource it and become dependent (as we are with GPS and maps), or
  3. You can prepare

CIS Parent: Transguide, by Adam Zimman

• The talk was framed with art and typography from the game Silksong. It was a shared experience with their child; a way to connect with them.

• Graphic of Infinite Combinations, from Seeing Gender by Iris Gottlieb.

• Consider the character creator. There is no global “best”, there’s only “best for me”, the individual player. Some players even like “bad” combinations because it makes the game harder.

• Act I: How I defined you. The parent defines who the child is (name, gender, pronouns) and what the child does (clothing, toys, colours).

• The reality of parenting is very different to expectations, and you cannot prepare for it in theory – you need to figure it out on the fly.

• About 1.7% of people are intersex, a similar proportion to redheads (although many intersex individuals will not know it themselves).

• “Listen to your kid when they are young. Believe them when they tell you who they are. It can change at any time, but you should believe what they say to be their truth in the moment.”

• If a kid tells you something, it means they trust you – show them that you’ve earned it.

• Don’t assume that your lived experience is the same as theirs.

• Act II: How I defined you to who you are.

• Items you consider permanent are not. What do they want to do? Things like makeup, haircuts, friends and peers.

• Picking a name. When his child first mentioned changing their name, he dropped a clanger: “Your name is the first gift I ever gave you”.

  Their child’s reply is a model of grace: “You picked a name for me before you knew me. This is a chance for both of us to pick a name together now that I know and you know who I am.”

• “When somebody is willing to share with you their true self, they are extending an aspect of trust to you.”

• You will fail a lot. The important thing is that you keep trying.

• Act III: Who I am to who I want to become.

• How do I want to be seen?

• “I didn’t feel it was fair to have everything that my kids were experiencing have them be the ones to explain it to me” – Adam went and found trans and neurodivergent communities to do his own learning, so he could be informed in conversations with his children (but not opinionated!).

• “How can you support a human so that they feel comfortable being themselves and sharing themselves with everyone else?”

• Who am I legally? Passports and paperworks.

• Support vs dangers: being yourself vs being safe. For example, the family missed a bar mitzvah during the Trump administration because both children had X markers in their passports, and it was unclear if they could safely interact with TSA during that time. They now have passports that match their sex assigned at birth to make travel easier.

• Knowing vs asking; failure vs trying. “Get good.”

• “There are no shortcuts.”

Resilience in Communities, by Heidi Waterhouse & Kim Harrison

• Case study of two communities: South Minneapolis and Oaxaca.

• Change is good if it comes at the pace you’re expecting; right now change is not coming at that rate.

• We’re protected by the systems around us. What is it like to participate in the creation and maintenance of those systems?

• What are the properties of a resilient system?

  • tested
  • flexible
  • responsive to shocks
  • compassionate
  • sustainable

• Oaxaca: mutual aid tied to local resources. Kim explained the idea of [tequio][wiki-tequio] and several other practices whose names I didn’t catch; it’s collective work for the benefit of the community, not paid work.

• “If all you have is a wheelbarrow or shovel, you show up because that’s one more than nothing.”

• You can be kicked out of a community, lose voting rights and similar if don’t participate.

• How do you build resilient systms? Decide on shared priorities.

• Wendigo spirit means taking more resources than you need, and depriving others of them (aka capitalism). This can lead to people getting kicked out of communities.

• We’re bad at understanding our needs and gifts; we’re not all the same, and we need/provide different things.

• Minneapolis is under ICE occupation; how do you ensure people can still fulfill their basic needs?

• There are two strands of work involved:

  foot patrol	driving
  resistance	food aid
  “community”	child care
  observation	delivery
  demonstration	bus patrol

  It’s not safe to cross sides; these need to be picked up by different people.

• Mutual interdependence and support. They didn’t start from zero; they ratched up, learning from previous communities, preparing to help whose next. Previous examples: George Floyd, Portland, Chicago.

• What do we owe to each other? “We exist in ecosystems […] we cannot all be rugged individual martyrs.”

• “Mr Rogers stold children then when scary things happen, look for the helpers.” When scary things happen, adults look for the helpers, and then go help.

• “Reaching out a hand and saying ‘are you okay’ is the foundation of interdependent community” – the last words of Alex Pretti before he was murdered by ICE.

Planning FOR Uncertainty, by Matt LeMay

• In March 2020, companies were reluctant to change plans already in motion – which seems laughable now.

• Strategic planning only happens once a year, even though it’s meant to align you with customers and markets.

  Customers don’t give a shit about your “planning season”.

• “Why are you making a five-year plan when the world doesn’t look the same as it did five years ago?”

• How do you stop companies from treating planning as a (multi)-year planning exercise?

  You can’t, but you can add a quarterly review… and a monthly review… and a fortnightly review. (Flashbacks to Hazel’s makeup tutorial the previous night.)

• What if more planning makes us more adaptable?

• Following a plan can help us respond to change, if we plan to change. If responding to change is part of plan, we’re more likely to do it.

• It’s the difference between change as threat to the plan, and change as the plan. We can frame plans as resilient, iterative, and the way we change course.

  [Alex: at this point I was reminded of some career advice from a previous manager: it doesn’t matter how correct you are, if you’re annoying, you’ll be ignored.]

• The way you communicate a plan
  is
  [as important as] the plan.

• The way you communicate an idea is the idea.

• “Planning is communication. Communication is craft. Let’s craft plans that make us less resistant to change.”

[If the formatting of this post looks odd in your feed reader, visit the original article]

When people talk about why libraries should exist, it’s usually framing them as a social good. Libraries provide low-cost access to books and information, they’re a hub for digital literacy and access to technology, and they’re one of the few remaining third places where people are treated as citizens rather than consumers. Their role has grown from being a lender of books to being a vital community resource.

Those are noble reasons for libraries to exist, but not necessarily reasons to use them. I sometimes hear them said with a patronising tone, from people who think libraries are only for other people. To them, a library is a charity for the poor and illiterate, not a destination for the well-read.

I think this is a failure of imagination. Even though I can afford to buy my own books and you could argue I don’t “need” a library, using them has made me a happier reader.

A risk-free way to find my next favourite author

Libraries are a cheap and safe way to me to try lots of different books, including books I’m not sure if I’ll like. Sometimes, those experiments become new favourites.

If I’m buying books in a bookshop, I lean towards the familiar, towards books like the ones I already enjoy. It’s unusual for me to get anything radically different, because I don’t want to gamble my money on a complete unknown.

Borrowing a book from the library is free, so it’s easier to try a new author or genre. I can read two chapters and return a book guilt-free if it’s not my cup of tea. But sometimes, I try something very different, and I discover a whole new collection of books to enjoy.

I found some of my favourite books and authors through library books I probably wouldn’t have picked off a bookshop shelf:

• Alexandra Bellefleur: Listening to the library audiobook of Written in the Stars introduced me to sapphic romances, not something I’d read before, now one of my favourite genres.
• Jodie Chapman: I’m not religious, so I’d probably have skipped her debut novel Another Life, which has strong religious themes. I borrowed it from the library instead, absolutely adored it, and I still think about it four years later.
• Ravena Guron: Murder mystery is a crowded space, and the library helped me find an author who is now on my “read anything she writes” list.

Most library books are just “fine” – I enjoy them and return them to the shelf. I wouldn’t want every book to be a massive revelation, but those discoveries happen more often because the library reduces the cost of being curious.

A release valve for my crowded shelves

I love living in a house with books, but I only have so much shelf space. Libraries allow me to read lots of books without cluttering up my home.

If I buy a book, I’m also buying a future decision: when I’m done, do I keep it, gift it, or donate it? It’s not a difficult decision, but it’s just another thing to think about. It’s easy for a book to get “stuck” in my home for years, even when I don’t actually want to keep it.

When I finish a library book, there is no decision to make: I know I have to return it to the library. When I’m done, I drop it in the returns bin and forget about it. It’s an easy, safe default that keeps my home clutter-free.

I especially love using libraries for books that I know I’m only going to read once, like romance novels and murder mysteries. Once I know the ending, I’m unlikely to revisit those books unless they’re really exceptional.

I still buy books, and if I really like a library book I’ll buy my own copy – but for everything else, the library has helped refine my shelves into the set of books I really love, not just a record of everything I’ve ever touched.

A built-in deadline to keep me moving

As I prepare to move house later this year, I’ve been uncovering stashes of unread books. Some have followed me across multiple moves; some I’ve owned for over a decade. I used to tell myself that these books were “maturing” on my shelf, but really they were just stagnating. I’ve donated many of them to my local charity shop, because I’ve finally admitted I’ll never actually read them.

I don’t have this problem with library books, because the return period triggers a “use it or lose it” response in my brain. I have to read a book before it’s gone, or decide not to and return it. This works even though I know it’s a completely artificial deadline – if I run out of time, I can always renew or re-borrow a book – I still feel that sense of urgency.

Return periods are some form of literary placebo: the ticking clock tricks me into prioritising a book, not letting it blend into the furniture.

My “To Be Read” list is longer than ever, but most of it is now in the library’s digital catalogue rather than physical piles in my home. When I get a book, I read it quickly or not at all – and either way, it doesn’t sit around untouched for fifteen years.

An all-you-can-read buffet

Public libraries will never be my sole source of books – I have to go elsewhere for niche, specialist, and academic texts – but using them has helped me read more and find new favourites. They’re a vital social good, but I don’t use them out of a sense of civic duty. I use them because they make me a happier, more adventurous, and more prolific reader.

[If the formatting of this post looks odd in your feed reader, visit the original article]

I use my computer for the same fundamental tasks I did ten years ago: browsing the web, writing, editing photos, running scripts, and building small websites. Today’s computers can do all that and have power to spare. You can still push their limits with high-end tasks like video editing, 3D modelling, or gaming – but I don’t do any of those things.

I don’t need the latest and greatest, and I haven’t for a long time. Add in the expense, the hassle of upgrading, and the environmental impact of new hardware, and you can see why I’m keen to use my computers for as long as possible.

This won’t be easy. My needs might not change, but the world around me will. I won’t get software updates forever, the web is a bloated mess that becomes more resource-hungry every day, and AI may introduce unforeseen demands on my computer. I’ve had to set up my computer carefully to give it the best chance of lasting the decade.

In my first job, we sold telecoms hardware that sat in data centres for years, unmodified. We had to write software updates that would run on the machine as-is, because hardware upgrades were impossible. If a new feature needed more resources, we had to find a way to make the existing code more efficient to compensate. It was a stark contrast to cloud computing, where a more powerful machine is just a few clicks in a console. We had to be in the habit of thinking about efficiency, because there was no other option.

That habit has stuck. I try to be efficient my personal devices, and I’m very conservative about what I install – what apps, dependencies, and processes I allow to run.

• I limit what runs in the background. Currently, I have just four apps that are always running: Alfred, Backblaze, Tailscale, and a small static web server.
• I browse the web on my own terms. I disable JavaScript by default. It breaks a lot of websites so I can’t recommend it for everybody, but it makes my computer run faster and cooler. It’s hard for bloated websites to slow me down when they can only serve HTML and CSS.
• I’m comfortable ignoring AI. I don’t do much with AI, and right now I have little interest in exploring it further. Even if I do, the most powerful models run in cloud data centres – so if my local machine starts to fall behind, I won’t be missing out.
• I use lots of static websites. These are very lightweight, and they provide easy access to my media collections – my bookmarks, movies, TV shows, and so on. This is ’90s era web tech that still works brilliantly today.

I also write a lot of my own tools. If something feels slow or sluggish, I don’t have to buy a faster machine; I can look for a way to improve my code.

This might look like a process that requires discipline, but at this point it’s just my standard routine. I’ve always tried to use my computer efficiently and it’s meant my computers last a long time; it’s only with my latest purchase that I’ve made it an explicit goal.

Why now?

I bought this computer as I was wrapping up my career in digital preservation, and that’s why I approached it with such a long-term mindset. In that job, I’d been designing collections to survive over decades and centuries; what seems like an eternity in tech is a heartbeat in heritage. With that mindset, trying to keep a computer for a decade didn’t seem so ridiculous – especially when I remembered that I almost did it already, with an eight-year-old iMac that was running perfectly until the desk underneath it collapsed, a calamity that would kill any computer.

Global politics is another factor; I’m keen to avoid needing to buy a new computer in the near future, because I’m not sure how easy it will be. Right now I can just walk into a high street store, but that relies on a fragile and complex supply chain that’s showing cracks.

I bought my computer in November 2024, just after Trump was re-elected as US president, and his campaign threatened heavy tariffs and trade wars. A year later, that trade uncertainty has become the status quo; his war with Iran threatens global energy markets; computer prices are rising as parts are diverted to AI data centres; and the majority of the world’s microprocessors are still built in Taiwan, under the constant shadow of a Chinese invasion. And the background to all of this is climate change, which won’t make manufacturing computers any easier.

I hope I’m wrong, and that buying a new computer continues to be as simple as it is today. But if I’m right, and they become scarce or expensive, I’ll be glad to have a device that I’m ready to use for years more, rather than be stuck with something too slow that I can’t afford to upgrade.

I’ll have to replace it eventually, but hopefully I can be patient and outlast any short-term disruptions to the supply chain. And if there are long-term disruptions, I’ll have more time to plan my next purchase.

What computer did I buy?

I have a home office with a fixed setup, and my main computer is a desktop, which makes this easier – I don’t know if I could make a laptop last ten years. A desktop never moves, so it’s less vulnerable to dings and drops (assuming the desk stays standing), and there’s no internal battery to degrade or swell. I also don’t eat or drink at my desk, so there’s minimal risk of liquid damage.

I use Macs, and Apple offers three Mac desktops: the Mac mini, the Mac Studio, and the Mac Pro. The Studio and Pro are overpowered for my needs, and while that extra power would give me headroom, it would be a lot of extra expense for marginal gain. Instead, I looked at the Mac mini.

When I was buying, Apple offered two stock configurations of the Mac mini: an M4 chip with 16GB of RAM and 256GB of storage for £599, or an M4 Pro chip with 24GB of RAM and 512GB of storage for £1399. Both models got favourable reviews and seemed like good value, because they avoid Apple’s egregiously-priced upgrades.

I bought the M4 Pro rather than the base M4 – I think I’d been fine with the base M4 for now, but 16GB of RAM might become tight as macOS gets more memory hungry. I do want some headroom, I just don’t want to pay Mac Studio prices for it.

I’ve expanded the storage with a 4TB external SSD which is permanently plugged in. It was much cheaper than Apple’s upgrades, and it means I won’t run out of space any time soon. It also reduces wear on the internal SSD, which feels like the most likely component to fail.

The big question mark is software support, and I’m keeping my fingers crossed. Macs are typically supported by the latest version of macOS for six to eight years, and they get security updates for another two years after that. That should take me close to a decade, if not all the way.

For comparison, the M1 MacBook Air was released in November 2020, and I expect it will still be supported in this year’s macOS 27 release. Apple have already announced that this release will drop support for Intel Macs; it would be aggressive to drop M1 support at the same time, especially as an M1 MacBook Air was on sale at Walmart until a few weeks ago. If so, the M1 will get macOS updates until at least autumn 2027, and security updates until 2029 – a nine year span. Suddenly, running my M4 Mac mini for ten years doesn’t feel so ridiculous.

Apple’s hardware is in fantastic shape, and I absolutely believe their Mac minis can run for a decade without failing. (Maybe their hardware chief should be in charge of more things?) There’s always a risk of buying a lemon which has a manufacturing defect, but I’ve had mine for over a year and nothing has failed yet. I’m confident this machine can go the distance.

One year down, nine years to go

So far, it’s great – my Mac mini is a fantastic machine. It never feels slow; it’s never crashed; it takes up a tiny space on my desk; and I have enough storage that I never need to worry about cleaning up files. It’s just what I want a computer to be – an appliance I never have to think about.

You’d expect it to feel easy right now, because I’m still in the usual lifetime of this product. This will get harder over time, and the first year will be easier than the final year, but it’s still an encouraging start.

I hope that I’ve bought a decade of not having to think about hardware. Modern computers are ridiculously capable, and short of a catastrophic failure, it’s hard to imagine a reason to upgrade.

See you again in 2034!

[If the formatting of this post looks odd in your feed reader, visit the original article]

I turned off every light in my office, cranked up my monitor to max brightness, then I changed the colour on the screen to turn my room red or green or pink. Despite the terrible image quality, I enjoyed looking at myself in the webcam as my outfits took on a vivid new hue.

Here are three pictures with my current office lit up in different colours, each with a distinct vibe:

For a while I was using Keynote to change my screen colour, and Photo Booth to use the webcam. It worked, but juggling two apps was clunky, and a bunch of the screen was taken up with toolbars or UI that diluted the colour.

To make it easier, I built a tiny web app that helps me take these silly pictures. It’s mostly a solid colour background, with a small preview from the webcam, and buttons to take a picture or change the background colour. It’s a fun little toy, and it’s lived on my desktop ever since.

Here’s a screenshot:

If you want to play with it yourself, turn out the lights, crank up the screen brightness, and visit alexwlchan.net/fun-stuff/gumdrop.html. All the camera processing runs locally, so the webcam feed is completely private – your pictures are never sent to me or my server.

The picture quality on my webcam is atrocious, even more so in a poorly-lit room, but that’s all part of the fun. One thing I discovered is that I prefer this with my desktop webcam rather than my iPhone – the iPhone is a better camera, but it does more aggressive colour correction. That makes the pictures less goofy, which defeats the purpose!

I’m not going to explain how the code works – most of it comes from an MDN tutorial which explains how to use a webcam from an HTML page, so I’d recommend reading that.

I don’t play dress up as much as I used to, but on occasion I’ll still break it out and amuse myself by seeing what I look like in deep blue, or vivid green, or hot pink. It’s also how I took one of my favourite pictures of myself, a witchy vibe I’d love to capture more often:

Computers can be used for serious work, but they can do silly stuff as well.

[If the formatting of this post looks odd in your feed reader, visit the original article]

For a long time, I used GitHub as my sync layer – it’s free, convenient, and I was already using it – but recently I’ve been looking at alternatives. I’m trying to reduce my dependency on cloud services, especially those based in the USA, and I don’t need most of GitHub’s features. I made these repos public, in case somebody else might find them useful, but in practice I think very few people ever looked at them.

There are plenty of GitHub-lookalikes, which are variously self-hosted or hosted outside the USA, like GitLab, Gitea, or Codeberg – but like GitHub, they all have more features than I need. I just care about keeping my files in sync. Maybe I could avoid introducing another service?

As I thought about how Git works, I thought of a much simpler way – and I’m almost embarrassed by how long it took me to figure this out.

A Git repo is just a collection of files

In Git repos, there’s a .git folder which holds the complete state of the repo. It includes the branches, the commits, and the contents of every file. If you copy that .git folder to a new location, you’d get another copy of the repo. You could copy a repo with basic utilities like cp or rsync – at least, as a one-off. I wouldn’t recommend using them for regular syncing; it would be easy to lose data, because they don’t know how to merge changes from different devices.

Git’s built-in push and pull commands are smarter: they can synchronise this state between locations, compare the history of different copies, and stitch the changes together safely. Within a repo, you can create a remote location, a pointer to another copy of the repo that lives somewhere else. When you push or pull, your local .git folder gets synchronised with that other copy.

We’ve become used to the idea that the remote location is a cloud service – but it can just as easily be a folder on your local disk – and that gives me everything I want.

Bare and non-bare repositories

Before I explain the steps, I need to explain the difference between bare and non-bare repositories.

In our day-to-day work, we use non-bare repositories. They have a “working directory” – the files you can see and edit. The .git folder lives under this directory, and stores the entire history of the repo. The working directory is a view into a particular point in that history.

By contrast, a bare repository is just the .git folder without the working directory. It’s the history without the view.

You can’t push changes to a non-bare repo – if you try, Git will reject your push. This is to avoid confusing situations where the working directory and the .git folder get out of sync. Imagine if you had the repo open in a text editor, and somebody else pushed new code to the repo – suddenly your files would no longer match the Git history.

Whenever we push, we’re normally pushing to a bare repository. Because nobody can “work” inside a bare repo, it’s always safe to receive pushes from other locations – there’s no working directory to get out of sync.

My new setup

I have a home desktop which is always running, and it’s connected to a large external drive. For each repo, there’s a bare repository on the external drive, and then all my devices have a checked-out copy that points to the path on that external drive as their remote location. The desktop connects to the drive directly; the other devices connect over SSH.

This only takes a few commands to set up:

1. Create a bare repository on the external drive.

   $ cd /Volumes/Media/bare-repos
   $ git init --bare dotfiles

2. Set the bare repository as a remote location.

   On the home desktop, which mounts the external drive directly:

   $ cd ~/repos/dotfiles
   $ git remote add origin /Volumes/Media/bare-repos/dotfiles

   On a machine, which can access the drive over SSH:

   $ cd ~/repos/dotfiles
   $ git remote add origin alexwlchan@desktop:/Volumes/Media/bare-repos/dotfiles

   This allows me to run git push and git pull commands as normal, which will copy my history to the bare repository.

3. Clone the bare repository to a new location.

   When I set up a new computer:

   $ git clone /Volumes/Media/bare-repos/dotfiles ~/repos/dotfiles

This approach is very flexible, and you can store your bare repository in any location that’s accessible on your local filesystem or SSH. You could use an external drive, a web server, a NAS, whatever. I’m using Tailscale to get SSH access to my repos from other devices, but any mechanism for connecting devices over SSH will do. (Disclaimer: I work at Tailscale.)

Of course, this is missing many features of GitHub and the like – there’s no web interface, no issue tracking, no collaboration – but for my small, personal repos, that’s fine. There’s also no third-party hosting, no risk of outages, no services to manage. I’m just moving files about over the filesystem. It feels like the Git equivalent of static websites, in a good way.

Reflections

I used to throw every scrap of code onto GitHub in the vague hope of “sharing knowledge”, but most of it was digital clutter.

Nobody was reading my personal repos in the hope of learning something. They’re a grab bag of assorted snippets, with only a loose definition or purpose – it’s unlikely another person would know what they could find, or spend the time to go looking. Sharing knowledge requires more than just publishing code somewhere; you need to make it possible for somebody to find.

Extracting my ideas into standalone, searchable snippets makes them dramatically more useful and discoverable. There are single blog posts that have done more good than my entire corpus of code on GitHub – and I have hundreds of blog posts.

I still have plenty of public repos, but it’s specific libraries or tools with a clear purpose. It’s more obvious whether you might want to read it, and better documented if you do. It’s an intentional selection, not a random set of things I want to keep in sync.

For years, I’ve been using a social media site as a glorified file-syncing service, but I don’t need pull requests, an issue tracker, or a CI/CD pipeline to move a few macros between my machines – just a place to put my code. As with so many digital things, files and folders are all I need.

[If the formatting of this post looks odd in your feed reader, visit the original article]

My current web server is Caddy, so I define redirects in my Caddyfile with the redir directive. Here’s an example that creates permanent redirects for three URLs:

alexwlchan.net {
  redir /videos/crossness_flywheel.mp4  /files/2017/crossness_flywheel.mp4 permanent
  redir /2021/12/2021-in-reading/       /2021/2021-in-reading/ permanent
  redir /2022/12/print-sbt/             /til/2022/print-sbt/ permanent
}

This syntax is easy to write by hand, but it’s annoying if I want to define lots of redirects – and when I’m doing a big restructure, I do. In particular, it’s tricky to write scripts to modify this file.

This is a good use case for Cog, made by Ned Batchelder.

How I automate this with Cog

Cog is a tool for running snippets of Python inside text files, allowing you to generate content without external templates or additional files. When you process a file with Cog, it finds those snippets of Python, executes them, then inserts the output back into the original file.

Here’s an example:

alexwlchan.net {
  #[[[cog
  # import cog
  # 
  # redirects = [
  #     {"old_url": "/videos/crossness_flywheel.mp4", "new_url": "/files/2017/crossness_flywheel.mp4"},
  #     {"old_url": "/2021/12/2021-in-reading/", "new_url": "/2021/2021-in-reading/"},
  #     {"old_url": "/2022/12/print-sbt/", "new_url": "/til/2022/print-sbt/"},
  # ]
  # 
  # for r in redirects:
  #     cog.outl(f"redir {r['old_url']} {r['new_url']} permanent")
  #]]]
  #[[[end]]]
}

All the Python code that Cog runs is inside a comment, so it will be ignored by Caddy. The [[[cog …]]] and [[[end]]] markers tell Cog where to find the code, and it’s smart enough to remove the leading whitespace and comment markers.

When I process this file with Cog (pip install cogapp; cog Caddyfile), it runs the Python snippet, and anything passed to cog.outl() is written between the markers. This is the output, which gets printed to stdout:

alexwlchan.net {
  #[[[cog
  # import cog
  # 
  # redirects = [
  #     {"old_url": "/videos/crossness_flywheel.mp4", "new_url": "/files/2017/crossness_flywheel.mp4"},
  #     {"old_url": "/2021/12/2021-in-reading/", "new_url": "/2021/2021-in-reading/"},
  #     {"old_url": "/2022/12/print-sbt/", "new_url": "/til/2022/print-sbt/"},
  # ]
  # 
  # for r in redirects:
  #     cog.outl(f"redir {r['old_url']} {r['new_url']} permanent")
  #]]]
  redir /videos/crossness_flywheel.mp4 /files/2017/crossness_flywheel.mp4 permanent
  redir /2021/12/2021-in-reading/ /2021/2021-in-reading/ permanent
  redir /2022/12/print-sbt/ /til/2022/print-sbt/ permanent
  #[[[end]]]
}

If I want to write the output back to the file, I run Cog with the -r flag (cog -r Caddyfile). All the original Cog code is preserved, so I can run it again and again to regenerate the file. This means that if I want to add a new redirect, I can edit the list and run Cog again.

Cog is running a full version of Python, so I can rewrite the snippet to read the list of redirects from an external file. Here’s another example:

alexwlchan.net {
  #[[[cog
  # import cog
  # import json
  #
  # with open("redirects.json") as in_file:
  #     redirects = json.load(in_file)
  # 
  # for r in redirects:
  #     cog.outl(f"redir {r['old_url']} {r['new_url']} permanent")
  #]]]
  redir /videos/crossness_flywheel.mp4 /files/2017/crossness_flywheel.mp4 permanent
  redir /2021/12/2021-in-reading/ /2021/2021-in-reading/ permanent
  redir /2022/12/print-sbt/ /til/2022/print-sbt/ permanent
  #[[[end]]]
}

This is a powerful change – unlike the original Caddyfile, it’s easy to write scripts that insert entries in this external JSON file, and now I can programatically update this file.

My scripts that are rearranging my URLs can populate redirects.json, then I only need to re-run Cog and I have a complete set of redirects in my Caddyfile.

I usually run Cog with two flags:

• -r writes the output back to the original file, and
• -c adds a checksum to the end marker, like [[[end]]] (sum: Rwh4n2CfQD). This checksum allows Cog to detect if the output has been manually edited since it last processed the file – and if so, it will refuse to overwrite those changes. You have to revert the manual edits or remove the checksum.

You can also run Cog with a --check flag, which checks if a file is up-to-date. I run this as a continuous integration task, to make sure I’ve updated my files properly.

Why I like Cog

What separates Cog from traditional templating engines like Jinja2 or Liquid is that it operates entirely in-place on the original file. Usually, you have a source template file and a build step which produce a separate output file, but with Cog, the source and the result are stored in the same document. Storing templates in separate files is useful for larger projects, but it’s overkill for something like my Caddyfiles.

Having everything in a single file makes it easy to resume working on a file managed with Cog. I don’t need to remember where I saved the build script or the template; I can operate directly on that single text file. If I come back to this project in six months, the instructions for how the file is generated are right in front of me.

The design also means that I’m not locked into using Cog. At any point, I could delete the Cog comments and still have a fully functional file.

Cog isn’t a replacement for a full-blown templating language, and it’s not the right tool for larger projects – but it’s indispensable for small amounts of automation. If you’ve never used it, I recommend giving it a look – it’s a handy tool to know.

[If the formatting of this post looks odd in your feed reader, visit the original article]

The tool I changed was my static site generator. I write blog posts in text files using Markdown, and then my static site generator converts those text files into HTML pages. I upload those HTML pages to my web server, and they become available as my website.

I’ve been using a Ruby-based static site generator called Jekyll since late 2017, and I’ve replaced it with a Python-based static site generator called Mosaic. It’s a new tool I wrote specifically to build this website, so I know exactly how it works. I’m getting rid of a Ruby tool I only half-understand, in favour of a Python tool I understand well.

Nothing is changing for readers (yet). I tried hard to avoid breaking anything – URLs haven’t changed, pictures look identical, the RSS feed should be the same as before. Please let me know if you spot something broken!

You’ll see more changes soon, because I have lots of ideas to try this year. I want to make this website into more of a “digital garden”, getting even further away from a single list of chronologically ordered posts. I don’t want to build that with Jekyll – or to be precise, I don’t want to build it with Ruby.

It’s not Ru(by), it’s me

I don’t want to sound dismissive of Jekyll. It’s an impressive project that powers thousands of sites, and I used it happily for over eight years. I pushed it to build a lot of custom and bespoke pages, and it handled it with ease.

Jekyll’s superpower is its theming and plugin system, which allow you to customise its behaviour. Want something that Jekyll can’t do out of the box? Create your own template or plugin. But those plugins have to be written in Ruby, the same language as Jekyll itself – and I only write Ruby to make blog plugins. I can do it, but I’m slow, I’m unsure, and writing Ruby has never felt familiar.

You can build a digital garden with Jekyll and Ruby – plenty of people already have – but I know I’d find it a difficult and frustrating experience. My lack of Ruby experience would slow me down.

While my Ruby knowledge has sat still, I’ve become a much better Python programmer. Since I set up Jekyll in 2017, I’ve worked on big Python projects with extensive tests, thorough data validation, and an explicit goal of longevity. I tried writing a Python static site generator in 2016 and I got stuck; a decade later and I’m ready for another attempt.

This isn’t just general Python expertise – I’ve written about how I’m using static websites for tiny archives, and all the surrounding tools are written in Python. Porting this website to Python means I can reuse a lot of that code.

I hacked together an experimental Python static site generator over Christmas, and I wrote it properly over the last few weeks. I named it “Mosaic” after the square-filled headers on every page, and I really like it. I already feel faster when I’m working on the site, writing a language I know properly.

How does Mosaic work?

Mosaic works like other static site generators: it reads a folder full of Markdown files, converts them to HTML, and writes the HTML into a new folder. And just like Jekyll and similar tools, I’m building on powerful open-source libraries.

Here’s a comparison of the key dependencies:

Here are some thoughts on each.

Templates with Jinja

Jinja is the templating engine used by Flask, a framework I’ve used to build dozens of small web apps, so I was very familiar with the basic syntax. It’s similar to Liquid – both use {% … %} for operators and {{ … }} to insert values – so I could reuse my templates with only small changes.

The tricky part was replicating my custom tags, which I’d previously implemented using Jekyll plugins. I had to write my own Jinja extensions, which are harder than writing Jekyll tags. In Jinja, I have to interact directly with the lexer and parser, whereas a Jekyll plugin is a simple render function.

Markdown with Mistune

Mistune is a Markdown library I discovered while working on this project. I used Python-Markdown previously, but Mistune is faster and easier to extend. In particular, it provides a friendly way to customise the HTML output by overriding named methods. For example, I can add an id attribute to my headings by overriding the header(text, level) method.

The tricky part about changing Markdown renderer is all the subtle differences in the places where Markdown isn’t defined clearly. Mistune and kramdown return the same output in 95% of cases, but there’s a lot of variation and broken HTML in the remaining 5%.

One particular difficulty was all my inline HTML. This is one of my favourite Markdown features – you can include arbitrary HTML and it gets passed through as-is – and I make heavy use of it in this blog. But kramdown and Mistune disagree about where inline HTML starts and ends, and Mistune was wrapping <p> tags around HTML that kramdown left unchanged. I had to adjust my templates and whitespace to help Mistune distinguish Markdown and HTML.

Image generation with Pillow

I generate multiple sizes and formats for every image, so they get served in a fast and efficient way. I use Pillow to generate each of those derivatives.

Pillow is easier to install and supports a wider range of image formats than any of the Ruby gems I tried; it’s a highlight of the Python ecosystem.

The picture handling code has always been the thorniest bit of the website, and I hope that building it atop a nicer library will give me the space to simplify that code.

Syntax highlighting with Pygments

Rouge and Pygments are both capable libraries, and they return compatible HTML which made it easy to switch – I could reuse my CSS and my syntax highlighting tweaks.

I think Pygments theoretically supports highlighting a wider variety of languages, but I never found Rouge lacking so it’s not a meaningful improvement.

Data validation with Pydantic

Every Markdown file in my site has YAML “front matter” for storing metadata, for example:

---
layout: article
title: Swapping gems for tiles
---

Jekyll treats this as arbitrary data and doesn’t do any validation on it, which made it harder to change and keep consistent as the site evolved. I built a rudimentary validation layer using json-schema, but it was always an add-on.

In Mosaic, this front matter is parsed straight into a Pydantic model, so it’s type-checked throughout my code. This means I can write stricter validation checks, and catch more issues and inconsistencies before they break the website.

Linting HTML with HTML-Proofer

I’ve been using the HTMLProofer gem to check my HTML since 2019. It checks my HTML for errors like broken links or missing images, so I’m less likely to publish a broken page. It’s caught so many mistakes.

There’s no obvious Python equivalent, so for now I’m still running it as a separate step after I generate my HTML. It has a much lower overhead than running Jekyll so I’m not in a hurry to remove it – although eventually I’d like to reimplement the checks I care about with BeautifulSoup, so I can fully expunge Ruby.

I’m also considering using Playwright for some static site testing, but that’s a larger piece of work.

It’s not named after a museum in Georgia

The name isn’t so important, because I’m the only person who will ever use this tool – but I discovered a fun nugget that’s too juicy not to share.

I named my tool “Mosaic” after the tiled headers that appear at the top of every page. Those headers are a design element I added in 2016, and I’m so fond of them now I can’t imagine getting rid of them. I later remembered that Mosaic is also the name of a discontinued web browser, and I like the “old web” vibes of that name. One of the best compliments I’ve ever received about this site was “it looks like something from the 1990s” – fast, clean, and not junked up with ads.

One of the bizarre things I discovered while writing this post is that it’s not the first time the names “Mosaic” and “Jekyll” have appeared alongside each other.

There’s a small historical island off the coast of Georgia (the USA one) called Jekyll Island. It includes bike trails, golf courses, a beach that’s been in several films… and a history museum called Mosaic. What are the chances?

I know nothing about Jekyll Island or the history of Georgia, but if I ever feel safe enough to return to the US, I’d love to visit.

Growing the garden

I’ve been using Mosaic for several weeks and I’m really enjoying it. I wouldn’t recommend using it for anything else – it’s only designed to build this exact site – but all the source code is public, if you’d like to read it and understand how it works.

Switching to Mosaic has allowed me to start working on three improvements to the site:

1. Replace my “today I learned” (TIL) posts with “notes”. I really like how the TIL section has allowed me to write more frequent, smaller posts, but they’re still point-in-time snapshots. I want to replace them with notes that aren’t tied to a particular date, and instead can be living documents I update as I learn more.

2. Make the list of topics more useful. My current tags page is a wall of text, a list of 241 keywords with minimal context or explanation. Nobody is wading through that to find something interesting – I want to add some hierarchy to make it easier to read, and give a better overview of the site.

3. Fold my book reviews into my main site. My book reviews currently live on a separate site, which is only half-maintained. I’d like to merge them into the main site, let them benefit from the design improvements here, and start writing reviews of other entertainment.

I’ve had these ideas for months, and I’m excited to finally ship them, and bring this site closer to my idea of a “digital garden”

[If the formatting of this post looks odd in your feed reader, visit the original article]

In this post, I’ll explain how I created this collection, and why I spent so much time on it.

Glossary of the Galaxy: what do the titles mean?

Each title is a reference to a concept in CSS or web development:

Apollo 13px

Pixels (px) are a unit of length in CSS. They’re a common way to define fixed sizes for text, borders, and spacing.

Breakpoint at Tiffany’s

A breakpoint is the screen width at which a website’s layout changes – for example, when it switches from a single column on a phone to a multi-column grid on a desktop.

The Color #9D00FF

This is a hexadecimal colour code for a shade of purple. Hex codes are a common way to define colours in CSS.

Chungking Flexpress

Flexbox is a layout model that allows elements to “flex” – growing to fill extra space, or shrinking to fit into small spaces.

The Devil Wears Padding

The padding is the space inside an element, between its content and its border. In this list, the padding is the gap between the grey border and the text.

The Empire Strikes Block

A block-level element is one that starts on a new line and reserves the full width available, like a heading or a paragraph.

Git Out

Git is a version control tool used to track changes in source code. It’s the industry standard for managing web development projects.

Gridiator

CSS Grid is a layout system for arranging elements in rows and columns. Unlike Flexbox, which is one-dimensional, Grid is designed for two-dimensional layouts.

Hidden <Figure>

The <figure> element is used to show an image with a caption, while the hidden attribute tells browsers not to render a specific element on a page.

Interstyler

The <style> element is used to embed CSS rules directly in an HTML page. These rules are colloquially referred to as “styles”.

The Margin

The margin is the space outside an element, the gap between it and its neighbours. In this list, the margin is the gap between the grey border and the text above it.

vh for Vendetta

The viewport is the visible area of a web page in your browser. The vh unit stands for viewport height, where 1vh is equal to 1% of the screen's height.

I’m pretty happy with this list, and with the amount of variety and wordplay I managed to fit into a dozen titles.

Top Pun: choosing the movie titles

The trick to writing good puns is to write lots of puns, then throw away the bad ones. I only needed a dozen movies, but I had over thirty other titles that I didn’t use.

If the puns aren’t coming immediately, I write two lists: the phrases or words I want to parody, and the words I’m trying to shoehorn in. In this case, the first list had phrases like X-Men or Mission Impossible, and the second had words like pixel, margin, and flex.

This is where I reach for search engines – I won’t find anybody else making the exact puns I want, but I can find pre-existing lists of these building blocks. In this case, I looked at lists of famous and iconic films, and I read web development tutorials and glossaries. I leant toward popular films so more people would get the reference; a pun on an obscure film would likely be missed.

As I build the two lists, I start to spot connections, like the fact that X-Men could become Flex-Men. I write down all my ideas, even the bad ones – often a bad idea is the jumping off point for a good one. For example, an early idea was Block to the Future, which isn’t very good, but later I realised I could use Back/Block for The Empire Strikes Block instead, which is much better.

If this was a purely text-based exercise, the titles would be enough – but I also needed posters.

Blurhemian Rhapsody: making the posters with Primitive

I needed some posters to go with the titles, but what to use?

I wanted to use the movie posters because many films have iconic posters, and that would help people recognise the pun – but I didn’t want to use the real movie posters, because they often show the title. That would contradict my text, not help it.

But I do have an image editor, and while I lack the Photoshop skills to replace the title in a convincing way, I can make text that looks okay if you squint – and that gave me an idea.

Several years ago, I used Michael Fogleman’s Primitive tool to create some wallpapers. Primitive redraws images with a simple geometric shapes, adding one shape at a time, trying to get closer and closer to the original image.

Here’s an example, in which my face has been redrawn as several hundred triangles:

This gives a recognisable version of the image, but it’s a distinct style and you won’t mistake it for the real thing.

For each movie I was considering, I downloaded a poster from The Movie Databaase, and I used Primitive to blur it. Sometimes the original title would appear through the blur, in which case I used an image editor to replace the title and re-blurred it. The blurring meant I could get away with a rough edit – for example, I didn’t need the exact font – because any imperfections would be blurred away by Primitive.

This added a new dimension to my search for puns – I wanted movie posters that would still be recognisable after this blurring. This ruled out posters that are very busy, because it’s difficult to distinguish individual elements after the blurring. I looked at lists of iconic movie posters, which often have clear, distinct shapes that hold up well when converted into triangles.

One of the best examples of an iconic poster is The Devil Wears Prada. I know nothing about the film, but I remember the poster with the big red heel. When you blur the poster with Primitive, it becomes recognisable almost immediately. This is what it looks like with 5, 25, and 50 triangles:

I had a year where all my desktop wallpapers were photos that I’d blurred using Primitive, and I’ve been waiting for a chance to use it in a bigger project. I’m really pleased with the result – it lets me lean into the titles I’ve created, and it gives the whole collection a coherent appearance.

Widening the Lens: choosing a more diverse selection

I picked a dozen movies and started writing the article. But as I was taking screenshots of the movie grid, I noticed that my initial selection wasn’t very representative. Ten of the twelve films had all-or-mostly men in the main roles, and all of the lead characters were white.

I was tempted to ignore this problem, because this is just a fake collection for a blog post and does it really matter? But that was disingenuous – I cared enough to put in all this effort, so it must be a meaningful selection to me. I wanted a more diverse and interesting selection.

I looked for lists of famous movies which centre women and non-white characters, and added several to my made-up collection. Ideally I’d also have some movies that centre queer or disabled characters, but I couldn’t find any with an iconic poster or a pun-worthy title.

Blooper Reel: the movies I didn’t use

I made a lot of puns and posters, including a couple of personal favourites that I cut from the post:

Fifty Shades of Grey became Fifty Shades of #999 and was the first movie where I considered replacing a colour with a hex code. I swapped this out for The Colour Purple when I was trying to create a more diverse list, and replacing a mostly-grey poster with a pop of colour helped too.

X-Men became Flex-men, and I’m really sad I couldn’t use that pun. This was let down by the poster – the original X-Men branding is very prominent and would be hard to change, and all of the colourful X-Men posters are very busy with lots of characters.

Home Alone became Home Align, which is a weaker pun but another easily-recognisable poster.

I had good reasons to cut all of them, and the selection is better off without them – but maybe they’ll reappear in a future post.

Why would you do this?

This is a lot of effort for placeholder data in a single blog post. I did it because it was fun, and it helped me enjoy writing the rest of the post. Every time I thought of another title or saw a poster in a screenshot, it made me smile. That’s enough of a reason.

This sort of fun detail is why I like having a personal blog which isn’t a business or an income stream. I write because I enjoy it, and I can make decisions that don’t make commercial sense because it’s not a commercial website. This side quest had terrible return on investment if you only care about time and money – but it was fantastic for joy.

[If the formatting of this post looks odd in your feed reader, visit the original article]

This scrolling grid of posters is something I’d like to reuse for other media collections – books, comics, and TV shows.

I wrote an initial implementation with CSS grid layout, but over time I found rough edges and bugs. I kept adding rules and properties to “fix” the layout, but these piecemeal changes introduced new bugs and conflicts, and eventually I no longer understood the page as a whole. This gradual degradation often happens when I write CSS, and when I no longer understand how the page works, it’s time to reset and start again.

To help me understand how this layout works, I’m going to step through it and explain how I built the new version of the page.

Step 1: Write the unstyled HTML

This is a list of movies, so I use an unordered list <ul>. Each list item is pretty basic, with just an image and a title. I wrap them both in a <figure> element – I don’t think that’s strictly necessary, but it feels semantically correct to group the image and title together.

<ul id="movies">
  <li>
    <a href="#">
      <figure>
        <img src="apollo-13px.png">
        <figcaption>Apollo 13px</figcaption>
      </figure>
    </a>
  </li>
  <li>
    <a href="#">
      <figure>
        <img src="breakpoint-at-tiffanys.png">
        <figcaption>Breakpoint at Tiffany’s</figcaption>
      </figure>
    </a>
  </li>
  ...
</ul>

I did wonder if this should be an ordered list, because the list is ordered alphabetically, but I decided against it because the numbering isn’t important.

Having a particular item be #1 is meaningful in a ranked list (the 100 best movies) or a sequence of steps (a cooking recipe), but there’s less significance to #1 in an alphabetical list. If I get a new movie that goes at the top of the list, it doesn’t matter that the previous #1 has moved to #2.

This is an unstyled HTML page, so it looks pretty rough:

Step 2: Add a CSS grid layout

Next, let’s get the items arranged in a grid. This is a textbook use case for CSS grid layout.

I start by resetting some default styles: removing the bullet point and whitespace from the list, and the whitespace around the figure.

#movies {
  list-style-type: "";
  padding: 0;
  margin:  0;
  
  figure {
    margin: 0;
  }
}

Then I create a grid that creates columns which are 200px wide, as many columns as will fit on the screen. The column width was an arbitrary choice and caused some layout issues – I’ll explain how to choose this properly in the next step.

#movies {
  display: grid;
  grid-template-columns: repeat(auto-fill, 200px);
  column-gap: 1em;
  row-gap:    2em;
}

By default, browsers show images at their original size, which means they overlap each other. For now, clamp the width of the images to the columns, so they don’t overlap:

#movies {
  img {
    width: 100%;
  }
}

With these styles, the grid fills up from the left and stops as soon as it runs out of room for a full 200px column. It looks a bit like an unfinished game of Tetris – there’s an awkward gap on the right-hand side of the window that makes the page feel off-balance.

We can space the columns more evenly by adding a justify-content property which tells the browser to create equal spacing between each of them, including on the left and right-hand side:

#movies {
  justify-content: space-evenly;
}

With just ten CSS properties, the page looks a lot closer to the desired result:

After this step, what stands out here is the inconsistent heights, especially the text beneath the posters. The mismatched height of The Empire Strikes Block is obvious, but the posters for The Devil Wears Padding and vh for Vendetta are also slightly shorter than their neighbours. Let’s fix that next.

Step 3: Choosing the correct column size

Although movie posters are always portrait orientation, the aspect ratio can vary. Because my first grid fixes the width, some posters will be a different height to others.

I prefer to have the posters be fixed height and allow varied widths, so all the text is on the same level. Let’s replace the width rule on images:

#movies {
  img {
    height: 300px;
  }
}

This causes an issue with my columns, because now some of the posters are wider than 200px, and overflow into their neighbour. I need to pick a column size which is wide enough to allow all of my posters at this fixed height. I can calculate the displayed width of a single poster:

Then I pick the largest display width in my collection, so even the widest poster has enough room to breathe without overlapping its neighbour.

In my case, the largest poster is 225px wide when it’s shown at 300px tall, so I change my column rule to match:

#movies {
  grid-template-columns: repeat(auto-fill, 225px);
}

If I ever change the height of the posters or get a wider poster, I’ll need to adjust this widths. If I was adding movies too fast for that to be sustainable, I’d look at using something like object-fit: cover to clip anything that was extra wide. I’ve skipped that here because I don’t need it, and I like seeing the whole poster.

If you have big columns or small devices, you need some extra CSS to make columns and images shrink when they’re wider than the device, but I can ignore that here. A 225px column is narrower than my iPhone, which is the smallest device I’ll use this for. (I did try writing that CSS, and I quickly got stuck. I’ll come back to it if it’s ever an issue, but I don’t need it today.)

Now the posters which are narrower than the column are flush left with the edge of the column, whereas I’d really like them to be centred inside the column. I cam fix this with one more rule:

#movies {
  li {
    text-align: center;
  }
}

This is a more subtle transformation from the previous step – nothing’s radically different, but all the posters line up neatly in a way they didn’t before.

Swapping fixed width for fixed height means there’s now an inconsistent amount of horizontal space between posters – but I find that less noticeable. You can’t get a fixed space in both directions unless all your posters have the same aspect ratio, which would mean clipping or stretching. I’d rather have the slightly inconsistent gaps.

The white background and blue underlined text are still giving “unstyled HTML page” vibes, so let’s tidy up the colours.

Step 4: Invert the colours with a dark background

The next set of rules change the page to white text on a dark background. I use a dark grey, so I can distinguish the posters which often use black:

body {
  background: #222;
  font-family: -apple-system, sans-serif;
}

#movies {
  a {
    color: white;
    text-decoration: none;
  }
}

Let’s also make the text bigger, and add a bit of spacing between it and the image. And when the title and image are more spaced apart, let’s increase the row spacing even more, so it’s always clear which title goes with which poster:

#movies {
  grid-row-gap: 3em;
  
  figcaption {
    font-size:  1.5em;
    margin-top: 0.4em;
  }
}

The movie title is a good opportunity to use text-wrap: balance. This tells the browser to balance the length of each line, which can make the text look a bit nicer. You’ll get several lines of roughly the same length, rather than one or more long lines and a short line. For example, it changes “The Empire Strikes // Block” to the more balanced “The Empire // Strikes Block”.

#movies {  
  figcaption {
    text-wrap: balance;
  }
}

Here’s what the page looks like now, which is pretty close to the final result:

What’s left is a couple of dynamic elements – hover states for individual posters, and placeholders while images are loading.

Step 5: Add a border/underline on hover

As I’m mousing around the grid, I like to add a hover style that shows me which movie is currently selected – a coloured border around the poster, and a text underline on the title.

First, I use my dominant_colours tool to get a suitable tint colour for use with this background:

$ dominant_colours gridiator.png --best-against-bg '#222'
▇ #ecd3ab

Then I add this to my markup as a CSS variable:

<ul id="movies">
  ...
  <li style="--tint-colour: #ecd3ab">
    <a href="#">
      <figure>
        <img src="gridiator.png">
        <figcaption>Gridiator</figcaption>
      </figure>
    </a>
  </li>
  ...
</ul>

Finally, I can add some hover styles that use this new variable:

#movies {
  a:hover {
    figcaption {
      text-decoration-line: underline;
      text-decoration-thickness: 3px;
    }
  
    img {
      outline: 3px solid var(--tint-colour);
    }
  }
}

I’ve added the text-decoration styles directly on the figcaption rather than the a, because browsers are inconsistent about whether those properties are inherited from parent elements.

I used outline instead of border so the 3px width doesn’t move the image when the style is applied.

Here’s what the page looks like when I hover over Breakpoint at Tiffany’s:

We’re almost there!

Step 6: Add placeholder colours

As my movie collection grows, I want to lazy load my images so I don’t try to load them all immediately, especially posters that aren’t scrolled into view. But then if I scroll and I’m on a slow connection, it can take a few seconds for the image to load, and until then the page has a hole. I like having solid colour placeholders which get replaced by the image when it loads.

First I have to insert a wrapper <div> which I’m going to colour, and a CSS variable with the aspect ratio of the poster so I can size it correctly:

<ul id="movies">
  ...
  <li style="--tint-colour: #ecd3ab; --aspect-ratio: 510 / 768">
    <a href="#">
      <figure>
        <div class="wrapper">
          <img src="gridiator.png" loading="lazy">
        </div>
        <figcaption>Gridiator</figcaption>
      </figure>
    </a>
  </li>
  ...
</ul>

We can add a coloured background to this wrapper and make it the right size:

#movies {
  img, .wrapper {
    height: 300px;
    aspect-ratio: var(--aspect-ratio);
  }
  
  .wrapper {
    background: var(--tint-colour);
  }
}

But a <div> is a block element by default, so it isn’t centred properly – it sticks to the left-hand side of the column, and doesn’t line up with the text. We could add margin: 0 auto; to move it to the middle, but that duplicates the text-align: center; property we wrote earlier. Instead, I prefer to make the wrapper an inline-block, so it follows the existing text alignment rule:

#movies {
  .wrapper {
    display: inline-block;
  }
}

Here’s what the page looks like when some of the images have yet to load:

And we’re done!

The final page

There’s a demo page where you can try this design and see how it works in practice.

Here’s what the HTML markup looks like:

<ul id="movies">
  <li style="--tint-colour: #dbdfde; --aspect-ratio: 510 / 768">
    <a href="#">
      <figure>
        <div class="wrapper">
          <img src="apollo-13px.png" loading="lazy">
        </div>
        <figcaption>Apollo 13px</figcaption>
      </figure>
    </a>
  </li>
  ...
</ul>

and here’s the complete CSS:

body {
  background: #222;
  font-family: -apple-system, sans-serif;
}

#movies {
  list-style-type: "";
  padding: 0;
  margin:  0;
  
  display: grid;
  grid-template-columns: repeat(auto-fill, 225px);
  column-gap: 1em;
  row-gap:    3em;

  justify-content: space-evenly;

  figure {
    margin: 0;
  }
  
  li {
    text-align: center;
  }
  
  a {
    color: white;
    text-decoration: none;
  }
  
  figcaption {
    font-size:  1.5em;
    margin-top: 0.4em;
    text-wrap: balance;
  }
  
  a:hover, a#tiffanys {
    figcaption {
      text-decoration-line: underline;
      text-decoration-thickness: 3px;
    }
    
    img {
      outline: 3px solid var(--tint-colour);
    }
  }
  
  img, .wrapper {
    height: 300px;
    aspect-ratio: var(--aspect-ratio);
  }

  .wrapper {
    background: var(--tint-colour);
    display: inline-block;
  }
}

I’m really happy with the result – not just the final page, but how well I understand it. CSS can be tricky to reason about, and writing this step-by-step guide has solidified my mental model.

I learnt a few new details while checking references, like the outline property for hover states, the way text-decoration isn’t meant to inherit, and the fact that column-gap and row-gap have replaced the older grid- prefixed versions.

This layout is working well enough for now, but more importantly, I’m confident I could tweak it if I want to make changes later.

[If the formatting of this post looks odd in your feed reader, visit the original article]

The implementation is pretty straightforward: I have one function that generates the coordinates of each square, and another that generates varying shades of the tint colour. Put those together, and it draws the header image.

I recently improved the way I choose the shades of the tint colour, which makes the headers look more coherent, especially in dark mode. The change is subtle, but a definite improvement.

The old approach: varying the HSL lightness

Before, this is how I generated the shades:

1. Map to HSL. Convert the tint colour to the hue-saturation-lightness (HSL) colour space.
2. Define the bounds. I chose 7/8 and 8/7 of the original lightness, because it looked good in the first few colours I tried.
3. Jitter lightness. Pick a random lightness value in this range.
4. Recombine and convert. Pair this new lightness with the original hue and saturation, and convert back to sRGB.

I was trying to create colours which looked similar and varied only in lightness, so you’d see lighter or darker shades of the tint colour. My headers are PNG images, which are usually saved as sRGB, which is I why I convert back in the final step.

Here’s what the old code looked like:

require 'color'

# Given a hex colour as a string (e.g. '#123456') generate
# an infinite sequence of colours which vary only in brightness.
def get_colours_like(hex)
  seeded_random = Random.new(hex[1..].to_i(16))

  hsl = Color::RGB.by_hex(hex).to_hsl
  
  min_luminosity = hsl.luminosity * 7 / 8
  max_luminosity = hsl.luminosity * 8 / 7
  luminosity_diff = max_luminosity - min_luminosity
  
  Enumerator.new do |enum|
    loop do
      new_hsl = Color::HSL.from_values(
        hsl.hue,
        hsl.saturation,
        min_luminosity + (seeded_random.rand * luminosity_diff)
      )
      enum.yield new_hsl.to_rgb
    end
  end
end

I seeded the random generator so it always returned the same colours – this meant my local dev environment and web server would always generate identical header images. Note that it’s seeded based on the colour, so different tint colours will have light/dark squares in different places.

All the colour calculations are done by Austin Ziegler’s excellent color gem, which saved me from implementing colour conversions myself.

This approach is simple, but it has problems. Varying the lightness by proportion means the range varied from colour to colour – headers for dark colours didn’t have enough contrast, while light colours had too much contrast.

Here are three examples – notice how the dark header is almost solid colour, while the light header has enough contrast to become distracting:

This heuristic worked for the first colour I tried (#d01c11, the site’s original tint colour) but it breaks down as I’ve added more colours, especially in dark mode.

I could replace the percentages with fixed offsets – for example, plus or minus 25% lightness – but this wouldn’t fix the problem. Humans aren’t machines; we don’t perceive colours as linear numerical values. The human eye is more sensitive to some colours than others, so the same numerical jump in HSL doesn’t feel like the same visual difference.

Let’s look at another example, where I’ll fix the hue and saturation, and step the lightness by 25%. These differences don’t feel the same:

There are alternative colour spaces like OKLCH and CIELAB which try to capture the nuances of human biology and how we interpret colours, and that’s where I looked at for a replacement.

The CIELAB colour space

The CIELAB colour space is based on opponent process theory, which suggests that we perceive colour as a battle of three opposing pairs: black vs. white, red vs. green, and blue vs. yellow. Think about how you never see a reddish-green or a blueish-yellow – these colours are opposites.

These three pairs give us the three coordinates in CIELAB space:

• L* is the perceptual lightness (black vs. white)
• a* is the red-green axis
• b* is the blue-yellow axis

(The other three letters stand for Commission internationale de l’éclairage, the standards body who developed CIELAB in 1976.)

Within this colour space, we can calculate the perceptual difference between two colours. Ideally, that numerical distance should match our human perception of the change. The goal is perceptually uniformity: if you move a fixed numerical distance anywhere in the space, the “amount” of change should feel the same to a human observer.

That’s much easier said than done: the measurement formulas (like Delta E) have been refined over decades, and deficiences have been found in CIELAB, especially for shades of blue. Newer spaces like OKLAB try to capture the nuances of human biology even more accurately. But for the purpose of my header images, CIELAB is good enough, and a big improvement over HSL.

One place I already use CIELAB is in my tool for extracting dominant colours. I’m using k‑means clustering to group colours that are “close” together, and it makes sense to measure closeness using perceptual distance.

The Ruby gem I’m using supports CIELAB but not OKLAB, which also informed my decision. Colour maths is complicated, and I’d rather use an existing implementation than write it all myself.

My new approach: varying the CIELAB perceptual lightness

Here’s my new heuristic:

1. Map to CIELAB. Convert the tint colour to CIELAB space.
2. Define the bounds. Choose a fixed distance, and find how much you need to increase/decrease the perceptual brightness L* to reach that distance.
3. Jitter lightness. Pick a random L* value in this range.
4. Recombine and convert. Pair this new lightness with the original a* and b* components, and convert back to sRGB.

To find the bounds, I do a binary search on the possible lightness values to find the perceptual lightness which gets me closest to the target distance.

If I’m looking for the lighter shade, I search the range $(L*,100)$.

If I’m looking for the darker shade, I search the range $(0,L*)$.

Here’s the code:

require 'color'

# Find the perceptual lightness of a CIELAB colour that's a specific
# perceptual difference (target_distance) from the original colour, while
# maintaining the original hue and colourfulness.
def lightness_at_distance(original_lab, direction, target_distance)
  # 1. Define the search range for L*
  if direction == 'lighter'
    low_l = original_lab.l
    high_l = 100
  else
    low_l = 0
    high_l = original_lab.l
  end

  # 2. Run a binary search on L*
  best_lab = original_lab
  best_delta = 0

  15.times do
    mid_l = (low_l + high_l) / 2.0

    candidate_lab = Color::CIELAB.from_values(mid_l, original_lab.a, original_lab.b)
    candidate_delta = original_lab.delta_e2000(candidate_lab)

    # Are we closer than the current best colour? If so, replace it.
    if (candidate_delta - target_distance).abs < (best_delta - target_distance).abs
      best_lab = candidate_lab
      best_delta = candidate_delta
    end

    if candidate_delta < target_distance
      # We need more distance, move away from the original L*
      direction == 'lighter' ? (low_l = mid_l) : (high_l = mid_l)
    else
      # We've gone too far, move back toward the original L*
      direction == 'lighter' ? (high_l = mid_l) : (low_l = mid_l)
    end
  end

  best_lab.l
end

Then I can write a very similar function to what I wrote for HSL:

# Given a hex colour as a string (e.g. '#123456') generate
# an infinite sequence of colours which vary only in lightness.
def get_colours_like(hex)
  seeded_random = Random.new(hex[1..].to_i(16))
  
  lab = Color::RGB.by_hex(hex).to_lab

  min_lightness = lightness_at_distance(lab, 'darker',  6)
  max_lightness = lightness_at_distance(lab, 'lighter', 6)
  lightness_diff = max_lightness - min_lightness

  Enumerator.new do |enum|
    loop do
      new_lab = Color::CIELAB.from_values(
        min_lightness + (seeded_random.rand * lightness_diff),
        lab.a,
        lab.b
      )
      
      # Discard colours which don't map cleanly from CIELAB to sRGB
      if new_lab.delta_e2000(new_lab.to_rgb.to_lab) > 1
        next
      end
      
      enum.yield new_lab.to_rgb
    end
  end
end

One gotcha is that CIELAB is a wider range than sRGB, so CIELAB colours don’t always map cleanly into sRGB. For example, certain bright colours like neon green may lose their vibrancy when converted from CIELAB to sRGB.

When it does the conversion, the color gem automatically clamps colours to fit into the sRGB space, but this creates some unusually dark or bright squares. I check if this clipping has occurred by converting back to CIELAB and looking at the distance – if there’s too much drift, I discard the colour and pick another. This is another subtle difference, but I think it improves the overall vibe.

Let’s look at the results, which compare the HSL heuristic (top), the original tint colour (middle), and the CIELAB heuristic (bottom):

The dark squares have a bit more variety, while the light squares have much less and avoid the bright and noticeable shades. It’s a particular improvement in dark mode, where I always use light tint colours. There’s almost no difference for the middle colour, which makes sense because it was how I designed the original heuristic. It already looked pretty good.

The new colours are closer to what I want: a bit of subtle texture, not loud enough to draw attention. I switched to them a fortnight ago, and nobody noticed. It’s small refinement, not a radical change.

[If the formatting of this post looks odd in your feed reader, visit the original article]

In practice, I have a handful passwords that I think worth memorising. It’s still a short list, but I’m not convinced that a single password is either sensible or feasible. I generally trust my password manager, but I don’t want it to be a single point of failure for my entire digital life.

What passwords do I try to remember?

1. The login password for my computer. I configure my Mac to sleep after a few minutes of inactivity, then ask for my login password when I try to resume. Although I often use Touch ID to log in, I remember this password because I still have to enter it multiple times a day.

2. The master password for my password manager. This unlocks all of my other passwords.

3. The login passcode for my phone. I use an alphanumeric password, and I remember it because I have to enter it multiple times a day.

4. My email password. My email account is the gateway to all my other digital accounts. If I lost access to my password manager but could still receive email, I could reset my passwords and regain access to everything.

5. My remote backup password. I have offsite backups of all of my computers with Backblaze. If all of my devices were destroyed at once (for example, in a house fire), this would allow me to retrieve files from my backups, even without access to my password manager.

6. The encryption password for my multi-factor authentication (MFA) recovery codes. I have an MFA app on my phone, protected by Face ID or my passcode – but in an emergency, I have single-use recovery codes I can use instead. These are stored in an encrypted disk image.

7. My Apple Account password. I’m heavily enmeshed in the Apple ecosystem, and this account has powerful access to my devices, including remote wiping and backups.

8. The “memorable word” for my online banking. When I log in to my bank account, my password manager autofills a password, and then I have to fill in three characters of a longer “memorable word”. For example, I might be asked to enter the 1st, 5th, and 8th characters.

   I memorise this both for security and convenience. If somebody compromises my password manager, my bank account is safe – and even if this was in my password manager, it can’t fill in single characters this way.

All of these passwords are long, alphanumeric, and unique.

I have a regular calendar reminder to review them, and make sure I still remember them correctly. This would be a useful feature in a password manager – periodic tests on whether you still remember important passwords.

Where are these passwords stored?

Although I’ve memorised all eight passwords, there are some copies elsewhere.

Five of them are stored in my password manager, because it’s convenient: my computer’s login password, my phone’s login passcode, my email password, my remote backup password, and my Apple account password. (My email and Apple account are protected by multi-factor authentication, and the codes aren’t in my password manager.)

Two of them aren’t written down anywhere, but they might be soon: the master password for my password manager, and the encryption password for my MFA recovery codes. At some point I’d like to change this, probably with a paper copy in a fire safe or similar. This would allow my family to retrieve those passwords in an emergency.

The “memorable word” for my online banking isn’t written down anywhere, and I doubt it ever will be. If I lose access to my bank account and I’m really stuck, I can visit a physical branch.

How would I regain access to my accounts?

Here’s how I’d get back into my key accounts:

• Remote backups. My Backblaze account is only protected with a password, not MFA. I have this memorised, so I could download files from my remote backups on any device.

• MFA recovery codes. These are in an encrypted disk image in my remote backups. I’ve memorised the disk image password, so I could retrieve my MFA codes once I get to my remote backups.

• Email inbox. This is protected by a password and MFA. I’ve memorised the password, and I could use an MFA recovery code to regain access to the account.

• Password manager. I use 1Password. Logging in on a new device needs two secrets: my Master Password and Secret Key.

  I remember the former, but the latter is a random UUID I don’t type in or see regularly. Instead, I have a 1Password Emergency Kit which includes my Secret Key (but not my Master Password). I have a printed copy of this kit in my folder of important papers, and a digital copy in my disk image of MFA recovery codes.

  If I can get a copy of the Emergency Kit, I can regain access to my password manager.

What passwords don’t I remember?

There are a couple of important passwords you might expect me to memorise, but I don’t:

1. The email password for my work email. This password is stored in the password manager I use at work, and if I unexpectedly lost access, I’d contact the IT team for help. I don’t need self-service recovery for this account.

2. The master password for my password manager at work. For similar reasons to work email, I’d rely on the IT team to regain access in an emergency.

3. My banking app username or password password. Logging into my bank requires three values: my username, the full-length password, and the “memorable information”. I’ve memorised the memorable information, but not the username or password. If I need emergency access to my bank account, I can visit a high street branch.

What scenario am I trying to prevent?

Imagine you lost all of your devices. Could you regain access to your digital life? That’s my worst-case scenario that I’m trying to avoid, and these memorised passwords should be enough to bootstrap everything.

I can retrieve my MFA recovery codes from my remote backups, and then I can either log into my password manager and retrieve the current passwords, or log into my email inbox and reset all my passwords. Either way, I’m back into my accounts.

This doesn’t cover the scenario where I lose access to both my email inbox and my password manager, but that would be a catastrophic digital disaster.

It also doesn’t cover the scenario where I’m incapacitated and a family member needs emergency access to my digital accounts. That’s something I’m planning to fix this year. My plan is to purchase a fire safe that somebody else can open, in which I’d place printed instructions for access my password manager. Inside my password manager, I’ll have a note that explains what the key accounts are, which I can update regularly without reopening the safe.

I hope I can continue to rely on my password manager, and I never encounter one of these emergency scenarios – but I feel better knowing I’ve tried to prepare.

[If the formatting of this post looks odd in your feed reader, visit the original article]

1. Create unique passwords for each account, and store them in a password manager.
2. Enable multi-factor authentication (MFA), and use an authenticator app or hardware token as your second factor.

But enabling MFA isn’t everything – what if you lose access to that second factor? For example, I store my MFA codes in an app on my phone. What happens if my phone is broken or stolen?

Most services that support MFA give me a set of recovery codes I can use in an emergency to regain access to my account, but don’t explain what to do with them. I’m advised to “store them securely”, but what does that mean in practice?

I don’t want to store my recovery codes in my password manager, because that compresses multiple authentication factors back into one. Somebody who compromised my password manager would have access to everything. (That’s the same reason I don’t store my MFA codes in there.)

Instead, I have an encrypted disk image on my Mac, which I created using Disk Utility. The password is a long, unique password that I only use for this purpose, and I only keep the disk image mounted when I’m editing or using a recovery code.

This disk image contains two files:

• My 1Password Emergency Kit, a PDF document that contains the details for my 1Password account – including a secret key that I don’t see or type on a day-to-day basis
• An HTML file I write by hand, which has all my MFA recovery codes and notes on when I created them

Here’s what the HTML file looks like (with fake data, obviously):

You can download the HTML file I use as a template.

When I need to save some new recovery codes, I mount the disk image, edit the HTML file, then eject the disk image. When I need to use a recovery code, I mount the disk image, copy a code out of the HTML file, make a note that I’ve used it, then eject the disk image. This is a plain text file that’s not dependent on proprietary software or cloud services.

I have a second disk image on my work laptop, with a similar file, where I store recovery codes related to my work accounts.

A malicious program could theoretically read the HTML file while the disk image is mounted, so I try to keep it mounted as little as possible. But if a malicious program had long-running access to arbitrary files on my Mac, it can do more damage than just reading my recovery codes.

This setup assumes I’ll always have access to this disk image, which is why I have offsite backups that include it (in encrypted form, of course). I’ve memorised the password for my offsite backups, which gives me a clear recovery path if I lose my phone and all my home devices:

1. Log into my offsite backup
2. Download the encrypted disk image
3. Mount the disk image
4. Retrieve my 1Password Secret Key and MFA recovery codes

From there, I can get access to everything in my digital life.

Later this year, I plan to get a fire safe where I can store important documents, and a paper copy of these codes will definitely be in there. That’s why I include dates in the notes, and the current date at the top of the file – that way, I can see if the printed version is up-to-date.

I’m fortunate that I’ve never had to use this system in a real emergency – and helpful as it could be, let’s hope I never have to.

[If the formatting of this post looks odd in your feed reader, visit the original article]

When I’m reading unfamiliar code, I like to use print debugging to follow what’s happening. I print what branches I’m in, the value of different variables, which functions are being called, and so on. Some people like debuggers or similar tools, but when you’re learning a new language they’re another thing to learn – whereas printing “hello world” is the first step in every language tutorial.

The built-in way to do print debugging in Go is fmt.Printf or log.Printf. That’s fine, but my debug messages get interspersed with the existing logs so they’re harder to find, and it’s easy for those debug statements to slip through code review.

Instead, I’ve taken inspiration from Ping Yee’s Python module “q”. If you’re unfamiliar with it, I recommend his lightning talk, where he explains the frustration of trying to find a single variable in a sea of logs. His module provides a function q.q(), which logs any expressions to a standalone file. It’s quick and easy to type, and the output is separate from all your other logging.

I created something similar for Go: a module which exports a single function Q(), and logs anything it receives to /tmp/q.txt. Here’s an example:

package main

import (
	"github.com/alexwlchan/q"
	"os"
)

func printShapeInfo(name string, sides int) {
	q.Q("a %s has %d sides", name, sides)
}

func main() {
	q.Q("hello world")

	q.Q(2 + 2)

	_, err := os.Stat("does_not_exist.txt")
	q.Q(err)

	printShapeInfo("triangle", 3)
}

The logged output in /tmp/q.txt includes the name of the function and the expression that was passed to Q():

main: "hello world"

main: 2 + 2 = 4

main: err = stat does_not_exist.txt: no such file or directory

printShapeInfo: a triangle has 3 sides

I usually open a terminal window running tail -f /tmp/q.txt to watch what gets logged by q.

The module is only 120 lines of Go, and available on GitHub. You can copy it into your project, or it’s simple enough that you could write your own version. It has two interesting ideas that might have broader use.

Getting context with the runtime package

When you call Q(), it receives the final value – for example, if you call Q(2 + 2), it receives 4 – but I wanted to log the original expression and function name. This is a feature from Ping’s Python package, and it’s what makes q so pleasant to use. This gives context for the log messages, and saves you typing that context yourself.

I get this information from Go’s runtime package, in particular the runtime.Caller function, which gives you information about the currently-running function.

I call runtime.Caller(1) to step up the callstack by 1, to the actual line in my code where I typed Q(). It tells me the “program counter”, the filename, and the line number. I can resolve the program counter to a function name with runtime.FuncForPC, and I can just open the file and look up that line to read the expression. (This assumes the source code hasn’t changed since compilation, which is always true when I’m doing local debugging.)

Not affecting my coworkers with a local gitignore

To use this file, I copy q.go into my work repos and add it to my .git/info/exclude. The latter is a local-only ignore file, unlike the .gitignore file which is checked into the repo. This means I won’t accidentally check in q.go or push it to GitHub.

It also means I can’t forget to remove my debugging code, because if I do, the tests in CI will fail when they can’t find q.go.

This avoids other approaches that would be more disruptive or annoying, like making it a project dependency or adding it to the shared .gitignore file.

[If the formatting of this post looks odd in your feed reader, visit the original article]

I’m a bit disappointed in the books I read this year; compared to previous years, there were only a few books that I feel compelled to recommend. I’m not sure if it was bad luck or sticking too close to familiar favouites – but I can’t help notice that all of this year’s favourites are from new authors. That feels like a sign to look further afield in 2026.

What saved the reading year was community and connection. My a book club just passed its third anniversary, and the discussions are always a highlight of my month. In particularly enjoy the conversation if it’s a book we all liked – it’s more fun to celebrate what works than to tear a book to shreds. Two of my top picks below come from the book club list.

I also found some unexpected serendipity: Lizzie Huxley-Jones’s festive romance Make You Mine This Christmas has a meetcute in a bookshop in St Pancras station. I use the station regularly so I know the shop well, and it’s where my partner and I took our first photo together as a couple, at the beginning of our second date.

I track the books I read at books.alexwlchan.net, and writing the annual round-up post has become a fun tradition. You can see how my tastes have changed in 2024, 2023, 2022, and 2021.

Here are my favourite books from 2025, in the order I read them.

Service Model

by Adrian Tchaikovsky (2024)

read 8 January 2025

What if the robot apocalypse happened, but nobody told the robots?

We follow Charles, a robot butler who finds himself unexpectedly unemployed, and he travels through an apocalyptic wasteland to find a new purpose. In a world mostly devoid of humans, he struggles to find another household to serve. It’s a dark and absurd journey which I very much enjoyed, and the style reminds me of Douglas Adams. This world isn’t tragic, but absurd.

Beneath the lyrical and humorous style are messages about automation, class division, and our attitude towards work. The world is full of robots who are doing things because that’s Their Purpose, with no thought for who the automation is serving or whether it’s still necessary.

If you enjoy this, you should follow up by reading Human Resources, the prequel short story about a “human resources” department that only exists to fire all the humans.

I’d also recommend The Incomparable podcast’s Book Club episodes; I read Service Model on the strength of Jason Snell’s recommendation.

How You Get the Girl

by Anita Kelly (2024)

read 24 June 2025

A charming sapphic romance between a basketball teacher and a professional player.

Elle is a famous basketball player who’s proud and confident about being queer, but she struggles to be a foster parent to her niece, Vanessa. Julie is a capable high school coach who bonds well with her team, but feels unsure and uncertain about her queer identity. They both look up to the other, and are looked up to in return.

It felt like a very balanced romance, and I enjoyed our discussion of it at Ace Book Club. It hits all the classic sapphic tropes, and it has a feel-good ending.

My particular reading was enhanced by the annotations – my partner read this book first, and she highlighted passages with comments like “this seems familiar” or “remind you of anyone?”. Sadly that’s not a transferable experience, but I can tell you that I enjoyed it.

Surprisingly, I didn’t enjoy Anita Kelly’s other books. This book is the third in the trilogy, and I tried to read the other two – one of them was so-so, and the other I gave up on.

The End Crowns All

by Bea Fitzgerald (2024)

read 9 September 2025

A sapphic retelling of the Trojan War, in which Cassandra’s curse is cast by a petty Apollo who just wants sex, and an enemies-to-lovers romance between Cassandra and Helen.

I really enjoyed this. It’s a well-written story and I enjoyed the first-person perspective of the two protagonists. It builds well towards its conclusion – a lot of stuff that becomes relevant later is established early and builds towards the end. Apollo’s curses on Cassandra, the gods forcing a narrative on Troy, how the story eventually deviates from the conventional myth.

The book has modern sensibilities, but retrofits them in a thoughtful way. It discusses consent, rape culture, and asexuality – in particular, Cassandra is implied to be ace – but never uses those words explicitly. These themes fit into the narrative, and don’t stand out as twenty-first century terminology or ideas shoved into Greek myth.

This was another book club pick, and I’m planning to read Bea Fitzgerald’s other books next year.

Finding Hester

by Erin Edwards, Greg Callus, Rose Crossgrove, and others (2025)

read 17 November 2025

This is the true story of Hester Leggatt, a woman who wrote fake love letters for Operation Mincemeat during World War II, then became a character in a hit musical.

Unlike the men in the story, almost nothing was known of Hester when SpitLip wrote the musical version of Operation Mincemeat, except that she wrote the fake love letters. Her character has the most emotional song in the show, but the fictional version had to be invented from scratch.

A group of fans were dissatisfied with this gap in history, and tracked down the real Hester. They traced the initial mistake to an interview that misspelt her name as Leggett instead of Leggatt. Once they knew the correct surname, they went through paper archives, old school records, even contacted MI5 – all to reconstruct her life story.

The book weaves this discovered history into a narrative, which is organised it into a coherent and readable story of Hester’s life – her place of birth, her career before and after the war, her love life, and even coincidental similarities to her fictional depiction.

I’m biased because several of those fans are dear friends, and I enjoyed watching their work from the sidelines – but I enjoyed reading the details even more so.

[If the formatting of this post looks odd in your feed reader, visit the original article]

One of the simplest Truchet tiles is a square made of two colours:

AAA

These can be arranged in a regular pattern, but they also look nice when arranged randomly:

The tiles that really caught my eye were Christopher Carlson’s. He created a collection of “winged tiles” that can be arranged with multiple sizes in the same grid. A tile can be overlaid with four smaller tiles with inverted colours and extra wings, and the pattern still looks seamless.

He defined fifteen tiles, which are seven distinct patterns and then various rotations:

The important thing to notice here is that every tile only really “owns” the red square in the middle. When laid down, you add the “wings” that extend outside the tile – this is what allows smaller tiles to seamlessly flow into the larger pattern.

Here’s an example of a Carlson Truchet tiling:

Conceptually, we’re giving the computer a bag of tiles, letting it pull tiles out at random, and watching what happens when it places them on the page.

In this post, I’ll explain how to do this: filling the bag of tiles with parametric SVGs, then placing them randomly at different sizes. I’m assuming you’re familiar with SVG and JavaScript, but I’ll explain the geometry as we go.

Filling the bag of tiles

Although Carlson’s set has fifteen different tiles, they’re made of just four primitives, which I call the base, the slash, the wedge, and the bar.

The first step is to write SVG definitions for each of these primitives that we can reuse.

Whenever I’m doing this sort of generative art, I like to define it parametrically – writing a template that takes inputs I can change, so I can always see the relationship between the inputs and the result, and I can tweak the settings later. There are lots of templating tools; I’m going to write pseudo-code rather than focus on one in particular.

For these primitives, there are two variables, which I call the inner radius and outer radius. The outer radius is the radius of the larger wings on the corner of the tile, while the inner radius is the radius of the foreground components on the middle of each edge. For the slash, the wedge, and the bar, the inner radius is half the width of the shape where it meets the edge of the tile.

This diagram shows the two variables, plus two variables I compute in the template:

Here’s the template for these primitives:

<!-- What's the length of one side of the tile, in the red dashed area?
     tileSize = (innerR + outerR) * 2 -->

<!-- How far is the tile offset from the edge of the symbol/path?
     padding = max(innerR, outerR) -->

<symbol id="base">
  <!--
    For the background, draw a square that fills the whole tile, then
    four circles on each of the corners.
    -->
  <g class="background">
    <rect x="{{ padding }}" y="{{ padding }}" width="{{ tileSize }}" height="{{ tileSize }}"/>
    <circle cx="{{ padding }}"            cy="{{ padding }}"            r="{{ outerR }}"/>
    <circle cx="{{ padding + tileSize }}" cy="{{ padding }}"            r="{{ outerR }}"/>
    <circle cx="{{ padding }}"            cy="{{ padding + tileSize }}" r="{{ outerR }}"/>
    <circle cx="{{ padding + tileSize }}" cy="{{ padding + tileSize }}" r="{{ outerR }}"/>
  </g>
  <!--
    For the foreground, draw four circles on the middle of each tile edge.
    -->
  <g class="foreground">
    <circle cx="{{ padding }}"            cy="{{ tileSize / 2 }}"       r="{{ innerR }}"/>
    <circle cx="{{ padding + tileSize }}" cy="{{ tileSize / 2 }}"       r="{{ innerR }}"/>
    <circle cx="{{ tileSize / 2 }}"       cy="{{ padding }}"            r="{{ innerR }}"/>
    <circle cx="{{ tileSize / 2 }}"       cy="{{ padding + tileSize }}" r="{{ innerR }}"/>
  </g>
</symbol>

<!--
  Slash:
    - Move to the top edge, left-hand vertex of the slash
    - Line to the top edge, right-hand vertex
    - Smaller arc to left egde, upper vertex
    - Line down to left edge, lower vertex
    - Larger arc back to the start
-->
<path
  id="slash"
  d="M {{ padding + outerR }} {{ padding }}
     l {{ 2 * innerR }} 0
     a {{ outerR }} {{ outerR }} 0 0 0 {{ outerR }} {{ outerR }}
     l 0 {{ 2 * innerR }}
     a {{ innerR*2 + outerR }} {{ innerR*2 + outerR }} 0 0 1 {{ -innerR*2 - outerR }} {{ -innerR*2 - outerR }}"/>

<!--
  wedge:
    - Move to the top edge, left-hand vertex of the slash
    - Line to the top edge, right-hand vertex
    - Smaller arc to left egde, upper vertex
    - Line to centre of the tile
    - Line back to the start
-->
<path
  id="wedge"
  d="M {{ padding + outerR }} {{ padding }}
     l {{ 2 * innerR }} 0
     a {{ outerR }} {{ outerR }} 0 0 0 {{ outerR }} {{ outerR }}
     l {{ 0 }} {{ 2 * innerR }}
     l {{ -innerR*2 - outerR }} 0"/>

<!--
  Bar: horizontal rectangle that spans the tile width and is the same height
  as a circle on the centre of an edge.
  -->
<rect
  id="bar"
  x="{{ padding }}" y="{{ padding + outerR }}"
  width="{{ tileSize }}"
  height="{{ 2 * innerR }}"/>

The foreground/background classes are defined in CSS, so I can choose the colour of each.

This template is more verbose than the rendered SVG, but I can see all the geometric expressions – I find this far more readable than a file full of numbers. This also allows easy experimentation – I can change an input, re-render the template, and instantly see the new result.

I can then compose the tiles by referencing these primitive shapes with a <use> element. For example, the “T” tile is made of a base and two wedge shapes:

<!-- The centre of rotation is the centre of the whole tile, including padding.
     centreRotation = outerR + innerR -->

<symbol id="carlsonT">
  <use href="#base"/>
  <use href="#wedge" class="foreground"/>
  <use href="#wedge" class="foreground" transform="rotate(90 {{ centreRotation }} {{ centreRotation }})"/>
</symbol>

After this, I write a similar <symbol> definition for all the other tiles, plus inverted versions that swap the background and foreground.

Now we have a bag full of tiles, let’s tell the computer how to place them.

Placing the tiles on the page

Suppose the computer has drawn a tile from the bag. To place it on the page, it needs to know:

• The x, y position, and
• The layer – should it place a full-size tile, or is it a smaller tile subdividing a larger tile

From these two properties, it can work out everything else – in particular, whether to invert the tile, and how large to scale it.

The procedure is straightforward: get the position of all the tiles in a layer, then decide if any of those tiles are going to be subdivided into smaller tiles. Use those to position the next layer, and repeat. Continue until the next layer is empty, or you hit the maximum number of layers you want.

Here’s an implementation of that procedure in JavaScript:

function getTilePositions({
  columns,
  rows,
  tileSize,
  maxLayers,
  subdivideChance,
}) {
  let tiles = [];
  
  // Draw layer 1 of tiles, which is a full-sized tile for
  // every row and column.
  for (i = 0; i < columns; i++) {
    for (j = 0; j < rows; j++) {
      tiles.push({ x: i * tileSize, y: j * tileSize, layer: 1 });
    }
  }
  
  // Now go through each layer up to maxLayers, and decide which
  // tiles from the previous layer to subdivide into four smaller tiles.
  for (layer = 2; layer <= maxLayers; layer++) {
    let previousLayer = tiles.filter(t => t.layer === layer - 1);
    
    // The size of tiles halves with each layer.
    // On layer 2, the tiles are 1/2 the size of the top layer.
    // On layer 3, the tiles are 1/4 the size of the top layer.
    // And so on.
    let layerTileSize = tileSize * (0.5 ** (layer - 1));
    
    previousLayer.forEach(tile => {
      if (Math.random() < subdivideChance) {
        tiles.push(
          { layer, x: tile.x,                 y: tile.y                 },
          { layer, x: tile.x + layerTileSize, y: tile.y                 },
          { layer, x: tile.x,                 y: tile.y + layerTileSize },
          { layer, x: tile.x + layerTileSize, y: tile.y + layerTileSize },
        )
      }
    })
  }
  
  return tiles;
}

Once we know the positions, we can lay them out in our SVG element.

We need to make sure we scale down smaller tiles to fit, and adjust the position – remember each Carlson tile only “owns” the red square in the middle, and the wings are meant to spill out of the tile area. Here’s the code:

function drawTruchetTiles(svg, tileTypes, tilePositions, padding) {
  tilePositions.forEach(c => {
    // We need to invert the tiles every time we subdivide, so we use
    // the inverted tiles on even-numbered layers.
    let tileName = c.layer % 2 === 0
      ? tileTypes[Math.floor(Math.random() * tileTypes.length)] + "-inverted"
      : tileTypes[Math.floor(Math.random() * tileTypes.length)];
      
    // The full-sized tiles are on layer 1, and every layer below
    // that halves the tile size.
    const scale = 0.5 ** (c.layer - 1);
    
    // We don't want to draw a tile exactly at (x, y) because that
    // would include the wings -- we add negative padding to offset.
    //
    // At layer 1, adjustment = padding
    // At layer 2, adjustment = padding * 1/2
    // At layer 3, adjustment = padding * 1/2 + padding * 1/4
    //
    const adjustment = -padding * Math.pow(0.5, c.layer - 1);

    svg.innerHTML += `
      <use
        href="${tileName}"
        x="${c.x / scale}"
        y="${c.y / scale}"
        transform="translate(${adjustment} ${adjustment}) scale(${scale})"/>`;
  });
}

The padding was fiddly and took me a while to work out, but now it works fine. The tricky bits are another reason I like defining my SVGs parametrically – it forces me to really understand what’s going on, rather than tweaking values until I get something that looks correct.

Demo

Here’s a drawing that uses this code to draw Carlson truchet tiles:

It was generated by your browser when you loaded the page, and there are so many possible combinations that it’s a unique image.

If you want a different picture, reload the page, or tell the computer to draw some new tiles.

These pictures put me in mind of an alien language – something I’d expect to see etched on the wall in a sci-fi movie. I can imagine eyes, tentacles, roads, and warnings left by a long-gone civilisation.

It’s fun, but not really the tone I want for this site – I’ve scrapped my plan to use Truchet tiles as header images. I’ll save them for something else, and in the meantime, I had a lot of fun.

[If the formatting of this post looks odd in your feed reader, visit the original article]

Browsing a list of S3 buckets is often an exercise in code archeology. Although people try to pick meaningful names, it’s easy for context to be forgotten and the purpose lost to time. Looking inside the bucket may not be helpful either, if all you see is binary objects in an unknown format named using UUIDs. A sentence or two of prose could really help a future reader.

We manage our infrastructure with Terraform and the Terraform AWS provider can upload objects to S3, so I only need to add a single resource:

resource "aws_s3_bucket" "example" {
  bucket = "alexwlchan-readme-example"
}

resource "aws_s3_object" "readme" {
  bucket  = aws_s3_bucket.example.id
  key     = "README.txt"
  content = <<EOF
This bucket stores log files for the Widget Wrangler Service.

These log files are anonymised and expire after 30 days.

Docs: http://internal-wiki.example.com/widget-logs
Contact: logging@example.com
EOF
  content_type = "text/plain"
}

Now when the bucket is created, it comes with its own explanation. When you open the bucket in the S3 console, the README appears as a regular object in the list of files.

This is an example, but a real README needn’t be much longer:

• What is the bucket for?
• Who do I talk to about what’s in this bucket?
• Where can I find out more?

This doesn’t replace longer documentation elsewhere, but it can be a useful pointer in the right direction. It’s a quick and easy way to help the future sysadmin who’s trying to understand an account full of half-forgotten S3 buckets, and my only regret is that I didn’t think to use aws_s3_object this way sooner.

[If the formatting of this post looks odd in your feed reader, visit the original article]

Some of the curves look a bit like Hebrew, but it’s definitely not that alphabet. I clicked on the first character (𐡱) and learnt that it’s Palmyrene Letter Pe, which is from the Palmyrene alphabet. I’d never heard of Palmyrene, so I knew I was about to learn something.

The Palmyrene Unicode block

These letters are part of the Palmyrene Unicode block, a set of 32 code points for the Palmyrene alphabet and digits. One of the cool things about Unicode is that the proposals for new characters are publicly available on the Unicode Consortium website, and they’re usually pretty readable.

Proposals have to provide some background on the characters they’re proposing. Here’s the introduction from the original proposal in 2010:

The Palmyrene alphabet was used from the first century BCE, in a small independent state established near the Red Sea, north of the Syrian desert between Damascus and the Euphrates. The alphabet was derived as a national script by modification of the customary forms that cursive Aramaic which themselves developed during the first Persian Empire.

Palmyrene is known from documents distributed over a period from the year 9 BCE until 273 CE, the date of the sack of Palmyra by Aurelian. […] No documents on perishable materials have survived; there are a few painted inscriptions, but many inscriptions on stone.

Here’s an example of a funerary stone inscribed with Palmyrene script, whose shapes match the Unicode characters I didn’t recognise:

The proposal was written by Michael Everson, a prolific contributor who’s submitted hundreds of proposals to add characters to Unicode. His Wikipedia article lists over seventy scripts. He was profiled by the New York Times in 2003 – seven years before proposing Palmyrene – which described his work and his “crucial role in developing Unicode”.

He takes a very long view of his work. Normally I’m sceptical of claims about the longevity of digital work, but Unicode is a rare area where I think it might just last:

“There’s satisfaction in knowing that the work of analyzing and encoding these languages, once done, will never need to be done again,” [Everson] said. “This will be used for the next thousand years.”

And I liked this part at the end:

He likes to tell about how he met the president of the Tibetan Calligraphy Society at a Unicode meeting in Copenhagen. Mr. Everson had helped the organization ensure that Tibetan was included in the standard. The president showed Mr. Everson how to write his name in Tibetan with a highlighter pen.

“He thanked me,” Mr. Everson said with reverence. “I couldn’t believe that, because his organization has been in existence for over a thousand years.”

I spent eight years working in cultural heritage and thinking about the longevity of digital collections, but I never gave much thought to the history or encoding of writing. This is cool and important work, and I should learn more about it.

What are the characters in Palmyrene?

Palmyrene has 22 letters in its alphabet, which expands to 32 Unicode codepoints when you include alternative letters, numbers, and a pair of symbols.

The only letter I recognise is aleph (𐡠), which looks similar to the Hebrew letter aleph ℵ. I know the latter because it’s used by mathematicians to describe the size of infinite sets. It turns out aleph or (alef) is the name of letters in a variety of languages, not all of which look the same – including Phoenician (𐤀), Syriac (ܐ), and Nabatean (𐢁/𐢀).

The other letters have names which are new to me, like heth (𐡧), samekh (𐡯), and gimel (𐡢).

One especially interesting letter is nun, which appears differently depending on whether it’s in the middle of the word (𐡮) or the end (𐡭). This reminds me of the ancient Greek letter sigma, which is either σ or ς. I can’t help but see a passing resemblance between final nun and final sigma, but surely it’s a coincidence – the rest of the alphabets are so different.

The Palmyrene numbers look similar to the Arabic numerals we use today, but not necessarily the same meaning. One, two, three and four are regular tally marks (𐡹, 𐡺, 𐡻, 𐡼). The more unusual characters are five (𐡽), ten (𐡾), and twenty (𐡿) – but again, it’s surely a coincidence that the latter resembles the modern digit 3.

Alongside the letters and numbers, there are two decorative symbols for left/right fleurons (𐡷/𐡸).

Writing about a right-to-left language

Palmyrene is written horizontally from right-to-left, which introduced some new challenges while writing this blog post.

The first issue was in my text editor, which is fairly old and doesn’t have good right-to-left support. I can include Palmyrene characters directly in my text, but it messes up the ordering and text selection. I can navigate the text with the arrow keys, but it behaves in weird ways. To get round this, I used HTML entities in all my source code (for example, 𐡠).

The second issue was in the rendered HTML page, where the Unicode characters affect the ordering on the page. In particular, I wanted to show the characters for 1, 2, 3, 4, in that order, so I wrote the four entities – but the browser uses a bidirectional algorithm and renders the sequence of characters as right-to-left. That’s the opposite of what I wanted:

The fix was to wrap each character in the bidirectional isolate <bdi> element. This tells the browser to isolate the direction of the text within that element, so the direction of each character doesn’t affect the overall sequence. This gave me what I wanted:

This is the first time the <bdi> element has appeared on this blog, and I think it’s the first time I’ve used it anywhere.

I took the original screenshot in September. It took me three months to dig into the detail, and I’m glad I did. This is a corner of history and writing that I’d never heard of, and even now I’ve only scratched the surface.

The Palmyrene alphabet is an example of what I call a “fractally interesting” topic. However deep you dig, however much you learn, there’s always more to uncover.

[If the formatting of this post looks odd in your feed reader, visit the original article]

But what about everything I’d saved before?

Across various disks, I’d accumulated over 150,000 posts from Twitter, Tumblr, and other platforms. These sites were important to me and I didn’t want to lose those memories, so I kept trying to back them up – but those snapshots had more enthusiasm than organisation. They were chaotic, devoid of context, and difficult to search – but the data was there.

After so many failed attempts, my scrapbook finally feels sustainable. It has a robust data model and a simple tech stack that I hope will last a long time. I wanted to bring in my older backups, but importing everything wholesale would just reproduce the problems of the past. I’d be polluting a clean space with a decade of disordered history. It was finally time to do some curation.

I went through each post, one-by-one, and asked: Is this worth keeping? Do I want this in the story of my life? Is this best left in the past?

That’s why I started looking back over fifteen years of a life lived online, which became an unexpectedly emotional meeting with my younger self.

The Internet as a teacher

One thing I’d forgotten is how much I learnt from being online, especially in fannish spaces. My timeline taught me about feminism and consent; about disability and the barriers of the built world; about racism in a way that went far deeper than anything I’d encountered before. I could learn about issues directly from the people who faced them, not filtered through a journalist’s lens. Today I take that social awareness for granted, but the Internet is where it started.

Social media was a crash course in humanity – broader, richer, and more diverse than anything I got from formal education.

Once I learned to shut up and just listen, Twitter let me follow conversations between people whose lives were nothing like mine. I got the answers to so many questions I’d never even known to ask, and I miss that. I stopped using Twitter after it was bought by Elon Musk, and I have yet to find another platform that replicates that passive, ambient learning.

Questioning and queer

More than anything else I saw online, queer culture has shaped my life. I’m queer, my partner is queer, and so are most of my friends. There are so many people I’d never have met if social media hadn’t introduced me to this world.

When I was realising I was queer, it all felt very difficult and angsty. Looking back, I can see myself following a classic path – talking to queer people, being a loud and enthusiastic ally, then starting to realise there might be a reason I cared so much. I went through it once when I realised I wasn’t straight, and again a few years later when I realised I wasn’t cis.

My younger self was oblivious, but it’s all so obvious in hindsight. I cringe at some of those older posts, but they helped me become who I am today, and I want to keep them.

I was annoying and rude, but I grew up

There are other posts I look back on with less fondness. I’m embarrassed by how annoying I was when I was younger. I spent too much time on self-indulgent moralising and pointless arguments, often with people I probably agreed with on almost everything else. I wanted to be right more than I wanted to listen, and that got in the way of useful conversations.

Those arguments were worthless then and they’re worthless now. Deleting them was a relief.

Among my less admirable behaviour was the performative outrage toward the “main character” of the day – the unlucky person whose viral tweet had summoned thousands of replies explaining why they were a terrible person. Looking back, it was a symptom of misplaced familiarity. I was reading a stranger’s posts as if I knew them, projecting motives from scraps of context, and joining dogpiles to fit in with the crowd.

Despite ruffling a lot of feathers, I was only the main character once, and in a small corner of the tech community. It was still an unpleasant weekend, and I got off lightly compared to some of my friends – but I’ve never forgotten how quickly online attention can turn to anger and hostility.

Learning about parasocial relationships helped me behave better. I realised how often my reactions were shaped by a false sense of intimacy, and how easy it was to be cruel when I forgot there was a person behind the avatar. I shifted my attention towards friends rather than strangers, and when I did talk to people I didn’t know, I tried to be constructive instead of showing off.

When I joined Twitter, I admired by people who were smart. Today, I look up to people who are kind. I’ve come to value generosity and empathy far more than cleverness and nitpicking.

The ghosts in the posts

Looking through old conversations, I see the ghosts of friendships and relationships I’ve since lost. Some of those could be recovered if either of us reached out; others are gone for good. A few people have even passed away. I don’t know where most of those friends ended up, but I hope life has been kind to them.

I’ve passed through so many spaces: the PyCon UK community; fandoms like the Marmfish and the Creampuffs; the trans elders who supported me during my transition; the small, loyal group of blog readers who always left thoughtful comments. Some I lost touch with while I was still on Twitter; others I left behind when I left Twitter altogether.

As my interests changed and I moved from one space to another, I often did a poor job of keeping up the friendships I already had. I’d pour my energy into chasing new connections in the spaces I’d just discovered, neglecting the people who had been there all along. That neglect is stark when I look at it over a decade-long span. It was sobering to realise how many more friends I might have today if I hadn’t taken so many past connections for granted.

I’ve tried to keep lingering traces of those friendships by saving my mentions as well as my own tweets. Here’s one I found that made me cry: “One of the things I miss the most from my pre-pandemic Twitter timeline is seeing @alexwlchan traveling on trains and taking train selfies”. I miss that culture too – selfies were such a source of joy and affirmation, especially in queer and trans spaces. I miss seeing pretty pictures of my friends, and sharing mine in return.

When Elon Musk bought Twitter, a lot of my remaining connections there were broken. Some friends went to other platforms; others left social media entirely. I was one of them! I still write here, but it’s a more professional, broadcast space – it’s not a back-and-forth conversation.

I miss the friendships I had, and the ones that might have been.

What I choose to remember

I started with 150,000 fragments, which I reduced to 4,000 conversations. A lot of it I was glad to forget, but there are gems I want to remember. I’m glad I’ve done this, and it reinforces my belief that social media is an important part of my life that I should preserve properly.

Curating these memories has made them feel smaller and more manageable. The mess of JSON files scattered across disks has been replaced by a meaningful, well-organised collection I can look back on with a smile.

What comes next?

My use of Tumblr fell away gradually, and I stopped tweeting when Elon Musk bought Twitter. I didn’t jump to another platform immediately, because I wanted to pause and reflect on what I wanted from social media. Currently, my social media usage is limited to linking to blog posts.

Looking back over my old posts has helped with those reflections. I’d like to think I’ve grown up a bit in the interim, and that I’d use it better if I made it a bigger part of my life again. A lot of good things started as conversations on social media, and I often wonder if I’m missing out. But I don’t miss the time sunk in pointless arguments, the performative anger, or the abuse from strangers.

I still don’t know what my future with social media will look like, but this project has me wondering. Until I decide, my scrapbook lets me see the best of what’s already been – the friendships, the joy, and the moments that mattered.

[If the formatting of this post looks odd in your feed reader, visit the original article]

The implementation is mine, but the ideas aren’t: cultural heritage institutions have been thinking about how to preserve social media for years. There’s decades of theory and practice behind digital preservation, but social media presents some unique challenges.

Institutional archiving has different constraints to individual collections – institutions serve a much wider audience, so their decisions need consistency and boundaries. My own scrapbook is tiny and personal, and comparing it alongside institutional efforts really highlights the differences and difficulties. It’s why I usually call it a “scrapbook”, not an “archive”: it’s informal and a bit chaotic, and that’s fine because it’s only for me.

In this post, I’ll explain what I see as the key issues facing institutional social media archiving: what can be saved, what resists preservation, and how context is so hard to keep.

What exists and what can be saved

The scale of social media is overwhelming

Social media exists at a scale that’s hard to comprehend: billions of posts, with millions more being added each day.

This makes it difficult for anyone to choose what to preserve, because any one person can only know a tiny fragment of the whole. Making a choice inevitably introduces selection bias, and I’ve spoken to many people who’d like to avoid that bias by “collecting everything” – but that’s far beyond the capacity of any institution.

Since they can’t collect everything, institutions create rules – collection policies that define what’s in-scope. These rules are meant to ensure consistency, fairness, and reduce individual bias, but they force archivists to draw boundaries in a medium that inherently resists them.

Social media isn’t a sequence of isolated posts; it’s a dense, interconnected graph. A single post only makes sense in context – the replies, the people, the topic du jour. How much of this context do you gather? How many hops out do you follow? Do you save the whole thread, every reply, every linked account? How do you prevent scope creep from sucking in everything?

My personal scrapbook is subjective and inconsistent, because the only audience is me. My “collection policy” is pure vibes – I save threads I think are interesting; I keep posts that I find moving; I prune replies that are embarrassing or unhelpful. If I’m inconsistent or I delete the wrong thing, nobody else is affected.

Institutions can’t be that casual. They need durable, defensible rules about where their collection starts and ends. On social media, where every post is context to a larger tangle of conversation, drawing that boundary is a major challenge.

Private and disappearing content

Social media archiving efforts often concentrate on publicly available, long-lasting content, which excludes other types of material – even though they make up an ever-growing proportion of social media. Two major categories stand out:

1. Private social media – direct messages, private accounts, closed groups, paywalled forums.
2. Ephemeral features – content that deliberately disappears or expires. Think Snapchat, Instagram Stories, or one-time messages.

Collecting this material is difficult. Technically, it’s behind authentication walls or interfaces that most web archiving tools can’t reach. And even if you save it, can you share it? Ethically, archivists must be careful not to violate social norms or user expectations.

It isn’t impossible, and I’ve seen a handful of projects capture private and ephemeral media – for example, researchers analysing Instagram Stories and their use in political campaigns. These efforts rely on a patchwork of methods: accessing content through user logins, browser plugins, even taking screenshots. They tend to be small, targeted, and short-lived.

My scrapbook has a small amount of private content, mostly conversations between me and locked accounts on Twitter. I’m comfortable with that because I was part of those conversations, and it’s a private archive. I’m not sharing it with anybody else, so I don’t think my friends would begrudge me keeping a copy. I haven’t saved any ephemeral content.

Private and ephemeral posts have a different dynamic from public timelines. People can be more personal, vulnerable, and candid when they know their posts can’t be seen by anyone, forever. Maybe those moments won’t appear in social media archives – but if so, we should acknowledge that limitation, and what stories it leaves out.

The experience of social media

Social media is more than just posts, words, and images – it’s the experience. The interface, interaction design, and the algorithms that shape our feeds are rarely captured in archives.

For example, consider TikTok and the rise of vertical-swipe video. Because the next video is just a swipe away, creators structure their content to hook you immediately, and keep your attention throughout – a shift from the slower pace of older videos. If you only save the video file and not the swiping experience, it’s harder to understand why the creator made those choices.

Even more elusive is the “algorithm”, the black box that decide what posts appear in our timeline. These algorithms shape culture itself – amplifying some voices, suppressing others, deciding which ideas can spread – but their inner workings are deliberately opaque and impossible to archive. Their behaviour is a closely-guarded commercial secret.

A purely technical approach to preserving the experience is doomed to fail – but that doesn’t mean all is lost. We can document how these experiences shaped the flow of content: screenshots, screen recordings, detailed descriptions. Oral histories can give future audiences a sense of what it was like to exist in these digital ecosystems.

One of my favourite parts of any archive is the everyday. Often, something isn’t written down because it seems “obvious” at the time – but decades later, that knowledge has vanished. Social media is evolving quickly, and now is the time to capture these experiences. Future generations, looking back once the landscape settles, will want to understand the path that led there.

Rules, resistance, and responsibility

What if platforms resist preservation?

In the early 2000s, many platforms were far more supportive of digital preservation. Public APIs were common, scraping was largely tolerated, and some companies even collaborated with heritage institutions.

Twitter is the poster child of this sort of corporate endorsement. Their public API allowed a flourishing ecosystem of third-party clients and research projects; researchers could easily assemble datasets; the Library of Congress even attempted to preserve every public tweet between 2006 and 2017. The project stalled and remains largely inaccessible today – but it would never even get started in 2025.

Today, most platforms resist being preserved, archived, or downloaded en masse. APIs are restricted or paywalled, rate limits are strict, and scraping is aggressively blocked. The rise of generative AI has accelerated this trend, as companies realise their data is valuable for model training. Why give it away for free when you can ask for money?

Reddit is the most recent example. They blocked the Internet Archive after some AI companies used it to access posts for free – posts for which Google pays Reddit millions to access.

Attempts to preserve content programmatically are increasingly limited, which makes it difficult to archive at scale. In my scrapbook, I replace APIs with entering data by hand, but that’s only practical if you’re saving a small amount of data.

Do people want to be preserved?

A lot of web archiving has historically ignored consent. If something is on the public web, many archives consider it eligible for capture – but preserving a post means it’s preserved forever. Embarrassing thoughts or personal pictures can’t be deleted once they’ve been archived.

Not everyone would agree to their posts being permanently preserved, even if they use services like the Wayback Machine. We see this in the popularity of private accounts, closed forums, and ephemeral posts – people want control over how and when their posts are seen. Generative AI and the use of social media for model training has made people even more sensitive about their data.

The general public often ignores copyright and privacy – how many people use images they found online with no regard for the creator? – but institutions hold themselves to a higher standard.

A strict ethical stance would require explicit consent from every creator. Institutions often use donor agreements, where you allow them to keep your material and sign away the right to remove it afterwards – but that solution is hard to scale to social media, where a single conversation may involve dozens of people.

It would also mean losing huge amounts of historically valuable material. It would exclude orphaned accounts, abandoned platforms, and users who have died or lost their password. And web archives preserve content from companies, politicians, and public figures, helping keep them accountable – but these figures would rarely consent to archiving they don’t control.

One interesting approach is Bluesky’s proposal User Intents for Data Reuse, letting users declare how they want their posts to be reused, such as for AI training or archiving. Technology alone is not the solution – you also need enforcement – but this feels like a step in the right direction.

I like the idea of a balanced approach – collecting material from public figures is fair game; anything from private citizens needs explicit consent. Of course, that’s easier said than done, and it’s tricky to codify that as a well-defined rule – but to me, “anything publicly available” feels increasingly insufficient as an ethical guideline.

In my personal scrapbook, I don’t have a formal consent process – something I feel comfortable with because my archive is small, private, and only my own reference. My guiding rule is “don’t be a creep”. I don’t save anything I think the original author would be uncomfortable knowing I kept.

Laws and legislation

Consent is a preference, but legislation is a hard boundary. Digital collections are affected by a patchwork of laws – copyright, privacy, data protection rules like the right to erasure, and even content-related restrictions. Institutions must ensure their collections comply with all relevant laws, even if those obligations conflict with the goals of long-term preservation.

Social media archiving is especially tricky. Automated, bulk collection can easily capture illegal or sensitive content, and mistakes may go unnoticed.

That’s why I prefer a targeted, human-reviewed approach. It slows you down, but reading all the material allows archivists to catch potential issues before content becomes a liability.

Understanding what you’ve saved

How do you search your collection?

An archive is useless if you can’t look at what you’ve saved. This is often a problem in social media archives: we can save posts at incredible speeds, but we can’t search them in any meaningful way.

Web archiving often saves page-by-page, one page per post, like the Wayback Machine. This scales beautifully for capture, but terribly for discovery. You can retrieve a post if you know its URL, but you can’t find everything about a single topic or written by a given author.

Traditional archives solve this with cataloguing: humans write descriptions, and researchers use those to find what they need. But that model can buckle if you try to save social media at scale: machines can save thousands of posts in the time it takes a human to describe just one.

In my personal scrapbook, I add keyword tags to every conversation. They’re fast, informal, and effective. If I want something specific, I can filter by tag and find it instantly. Since I’m the only person who uses these tags, I can define them in a way I like and change them when I decide. If I was in an institutional context, I’d use a controlled vocabulary like LCSH or MeSH.

These light-touch keywords feel like a realistic middle ground: human-scale data that’s quick to apply, but rich enough to cut through the fog.

Who’s the person behind the profile?

Identity on social media is a hard problem. Many accounts are anonymous or pseudonymous, and most people have accounts scattered across multiple platforms. This makes it tricky to track somebody’s presence on social media, because there’s rarely a mapping between a person’s real-world identity and the accounts they use online. Often, this anonymity is intentional.

This ambiguity creates a big headache for support teams at social media companies. When somebody asks for help regaining access to an account because they’ve lost the password or been hacked, how can the platform be sure they’re the real owner? That question is even harder to answer if you’re outside the company.

Institutions and researchers care about identity because it provides context and authority: who wrote this, and how much can we trust their words? Social media makes this hard, because many usernames don’t tell you anything about the person behind them. Although institutions have tools to connect people across records, you need to know who the person is first!

My personal scrapbook sidesteps this complexity. Nearly all of the conversations it contains are with friends I know well, so I can easily connect their identities across different services.

Implicit knowledge, cultural context, and memes

Social media relies on shared knowledge: current events, in-jokes, and memes. Without this context, the meaning of a post can fade – or an entirely new meaning can take its place.

This isn’t a new problem – all human communication requires context – but social media takes it to eleven. The pace and brevity are a fertile breeding ground for memes whose origins disappear almost immediately. Log off for a day, and you’ll return to posts that make no sense at all. You missed the moment that sparked the meme. Imagine how much harder it is to understand if you arrive years – or decades – later.

You can try to fill in the gap with catalogue descriptions, but that’s only possible if somebody understands the references well enough to describe them. With social media’s scale and speed, it’s impossible for anybody to know all the jokes, memes, and ideas that might affect a post.

In my personal scrapbook, I rely on my memory to provide that context. I don’t write longer descriptions, and I don’t know how much I’ll remember. Some posts that made sense in 2020 may be baffling in 2030, others will still be crystal clear. Only one way to find out!

We won’t save everything, but we can save something

Perhaps we can’t preserve social media perfectly, but that doesn’t mean we shouldn’t try. Every archive ever assembled is incomplete, but they still have immense value. Capturing public posts, threads, or conversations – even if we lose some of the context or ephemeral content – helps preserve a record of cultural history that could otherwise be lost.

Social media archiving may be a new endeavour for large institutions, but it’s not a new idea. There are small, ad-hoc projects happening everywhere, and there’s lots of prior art to learn from. Just today I came across Posty, a tool for creating archive a Mastodon account as a static site.

I’m always excited when I see people building tools to save tiny corners of the web – posts from a single account, fanworks from a tight-knit community, or shared advice from a community wiki. Whenever a platform disappears or looks shaky, there’s a renewed interest to minimise the loss.

Social media archiving will never be perfect, but it’s possible, and I’m excited to see how institutions rise to the challenge.

[If the formatting of this post looks odd in your feed reader, visit the original article]

That impact will last a long time. The posts themselves? Not so much.

Social media is fragile, and it can disappear quickly. Sites get sold, shut down or blocked. People close their accounts or flee the Internet. Posts get deleted, censored or lost by platforms that don’t care about permanence. We live in an era of abundant technology and storage, but the everyday record of our lives is disappearing before our eyes.

I want to remember social media, and not just as a vague memory. I want to remember exactly what I read, what I saw, what I wrote. If I was born 50 years ago, I’m the sort of person who’d keep a scrapbook full of letters and postcards – physical traces of the people who mattered to me. Today, those traces are digital.

I don’t trust the Internet to remember for me, so I’ve built my own scrapbook of social media. It’s a place where I can save the posts that shaped me, delighted me, or just stuck in my mind.

It’s a static site where I can save conversations from different services, enjoy them in my web browser, and search them using my own tags. It’s less than two years old, but it already feels more permanent than many social media sites. This post is the first in a three-part series about preserving social media, based on both my professional and personal experience.

The long road to a lasting archive

Before I ever heard the phrase “digital preservation”, I knew I wanted to keep my social media. I wrote scripts to capture my conversations and stash them away on storage I controlled.

Those scripts worked, technically, but the end result was a mess. I focusing on saving data, and organisation and presentation were an afterthought. I was left with disordered folders full of JSON and XML files – archives I couldn’t actually use, let along search or revisit with any joy.

I’ve tried to solve this problem more times than I can count. I have screenshots of at least a dozen different attempts, and there are probably just as many I’ve forgotten.

For the first time, though, I think I have a sustainable solution. I can store conversations, find them later, and the tech stack is simple enough to keep going for a long time. Saying something will last always has a whiff of hubris, especially if software is involved, but I have a good feeling.

Looking back, I realise my previous attempts failed because I focused too much on my tools. I kept thinking that if I just picked the right language, or found a better framework, or wrote cleaner code, I’d finally land on a permanent solution. The tools do matter – and a static site will easily outlive my hacky Python web apps – but other things are more important.

What I really needed was a good data model. Every earlier version started with a small schema that could hold simple conversations, which worked until I tried to save something more complex. Whenever that happened, I’d make a quick fix, thinking about the specific issue rather than the data model as a whole. Too many one-off changes and everything would become a tangled mess, which is usually when I’d start the next rewrite.

This time, I thought carefully about the shape of the data. What’s worth storing, and what’s the best way to store it? How do I clean, validate, and refine my data? How do I design a data schema that can evolve in a more coherent way? More than any language or framework choice, I think this is what will finally give this project some sticking power.

How it works

A static site, viewed in the browser

I store metadata in a machine-readable JSON/JavaScript file, and present it as a website that I can open in my browser. Static sites give me a lightweight, flexible way to save and view my data, in a format that’s widely supported and likely to remain usable for a long time.

This is a topic I’ve written about at length, including a detailed explanation of my code.

Conversations as the unit of storage

Within my scrapbook, the unit of storage is a conversation – a set of one or more posts that form a single thread. If I save one post in a conversation, I save them all. This is different to many other social media archives, which only save one post at a time.

The surrounding conversation is often essential to understanding a post. Without it, posts can be difficult to understand and interpret later. For example, a tweet where I said “that’s a great idea!” doesn’t make sense unless you know what I was replying to. Storing all the posts in a conversation together means I always have that context.

A different data model and renderer for each site

A big mistake I made in the past was trying to shoehorn every site into the same data model.

The consistency sounds appealing, but different sites are different. A tweet is a short fragment of plain text, sometimes with attached media. Tumblr posts are longer, with HTML and inline styles. On Flickr the photo is the star, with text-based metadata as a secondary concern.

It’s hard to create a single data model that can store a tweet and a Tumblr post and a Flickr picture and the dozen other sites I want to support. Trying to do so always led me to a reductive model that over-simplified the data.

For my scrapbook, I’m avoiding this problem by creating a different data model for each site I want to save. I can define the exact set of fields used by that site, and I can match the site’s terminology.

Here’s one example: a thread from Twitter, where I saved a tweet and one of the replies. The site, id, and meta are common to the data model across all sites, then there are site-specific fields in the body – in this example, the body is an array of tweets.

{
  "site": "twitter",
  "id": "1574527222374977559",
  "meta": {
    "tags": ["trans joy", "gender euphoria"],
    "date_saved": "2025-10-31T07:31:01Z",
    "url": "https://www.twitter.com/alexwlchan/status/1574527222374977559"
  },
  "body": [
    {
      "id": "1574527222374977559",
      "author": "alexwlchan",
      "text": "prepping for bed, I glanced in a mirror\n\nand i was struck by an overwhelming sense of feeling beautiful\n\njust from the angle of my face and the way my hair fell around over it\n\ni hope i never stop appreciating the sense of body confidence and comfort i got from Transition 🥰",
      "date_posted": "2022-09-26T22:31:57Z"
    },
    {
      "id": "1574527342470483970",
      "author": "oldenoughtosay",
      "text": "@alexwlchan you ARE beautiful!!",
      "date_posted": "2022-09-26T22:32:26Z",
      "entities": {
          "hashtags": [],
          "media": [],
          "urls": [],
          "user_mentions": ["alexwlchan"]
        },
        "in_reply_to": {
          "id": "1574527222374977559",
          "user": "alexwlchan"
        }
      }
    }
  ]
}

If this was a conversation from a different site, say Tumblr or Instagram, you’d see something different in the body.

I store all the data as JSON, and I keep the data model small enough that I can fill it in by hand.

I’ve been trying to preserve my social media for over a decade, so I have a good idea of what fields I look back on and what I don’t. For example, many social media websites have metrics – how many times a post was viewed, starred, or retweeted – but I don’t keep them. I remember posts because they were fun, thoughtful, or interesting, not because they hit a big number.

Writing my own data model means I know exactly when it changes. In previous tools, I only stored the raw API response I received from each site. That sounds nice – I’m saving as much information as I possibly can! – but APIs change and the model would subtly shift over time. The variation made searching tricky, and in practice I only looked at a small fraction of the saved data.

I try to reuse data structures where appropriate. Conversations from every site have the same meta scheme; conversations from microblogging services are all the same (Twitter, Mastodon, Bluesky, Threads); I have a common data structure for images and videos.

Each data model is accompanied by a rendering function, which reads the data and returns a snippet of HTML that appears in one of the “cards” in my web browser. I have a long switch statement that just picks the right rendering function, something like:

function renderConversation(props) {
    switch(props.site) {
        case 'flickr':
            return renderFlickrPicture(props);
        case 'twitter':
            return renderTwitterThread(props);
        case 'youtube':
            return renderYouTubeVideo(props);
        …
    }
}

This approach makes it easy for me to add support for new sites, without breaking anything I’ve already saved. It’s already scaled to twelve different sites (Twitter, Tumblr, Bluesky, Mastodon, Threads, Instagram, YouTube, Vimeo, TikTok, Flickr, Deviantart, Dribbble), and I’m going to add WhatsApp and email in future – which look and feel very different to public social media.

I also have a “generic media” data model, which is a catch-all for images and videos I’ve saved from elsewhere on the web. This lets me save something as a one-off from a blog or a forum without writing a whole new data model or rendering function.

Keyword tagging on every conversation

I tag everything with keywords as I save it. If I’m looking for a conversation later, I think of what tags I would have used, and I can filter for them in the web app. These tags mean I can find old conversations, and allows me to add my own interpretation to the posts I’m saving.

This is more reliable than full text search, because I can search a consistent set of terms. Social media posts don’t always mention their topic in a consistent, easy-to-find phrase – either because it just didn’t fit into the wording, or because they’re deliberately keeping it as subtext. For example, not all cat pictures include the word “cat”, but I tag them all with “cats” so I can find them later.

I use fuzzy string matching to find and fix mistyped tags.

Metadata in JSON/JavaScript, interpreted as a graph

Here’s a quick sketch of how my data and files are laid out on disk:

scrapbook/
 ├─ avatars/
 ├─ media/
 │   ├─ a/
 │   └─ b/
 │      └─ bananas.jpg
 ├─ posts.js
 └─ users.js

This metadata forms a little graph:

All of my post data is in posts.js, which contains objects like the Twitter example above.

Posts can refer to media files, which I store in the media/ directory and group by the first letter of their filename – this keeps the number of files in each subdirectory manageable.

Posts point to their author in users.js. My user model is small – the path of an avatar image in avatars/, and maybe a display name if the site supports it.

Currently, users are split by site, and I can’t correlate users across sites. For example, I have no way to record that @alexwlchan on Twitter and @alex@alexwlchan.net on Mastodon are the same person. That’s something I’d like to do in future.

A large suite of tests

I have a test suite written in Python and pytest that checks the consistency and correctness of my metadata. This includes things like:

• My metadata files match my data model
• Every media file described in the metadata is saved on disk, and every media file saved on disk is described in the metadata
• I have a profile image for the author of every post that I’ve saved
• Every timestamp uses a consistent format
• None of my videos are encoded in AV1 (which can’t play on my iPhone)

I’m doing a lot of manual editing of metadata, and these tests give me a safety net against mistakes. They’re pretty fast, so I run them every time I make a change.

Inspirations and influences

The static website in Twitter’s first-party archives

Pretty much every social media website has a way to export your data, but some exports are better than others. Some sites clearly offer it reluctantly – a zip archive full of JSON files, with minimal documentation or explanation. Enough to comply with data export laws, but nothing more.

Twitter’s archive was much better. When you downloaded your archive, the first thing you’d see was an HTML file called Your archive.html. Opening this would launch a static website where you could browse your data, including full-text search for your tweets:

This approach was a big inspiration for me, and put me on the path of using static websites for tiny archives. It’s a remarkably robust piece of engineering, and these archives will last long after Twitter or X have disappeared from the web.

The Twitter archive isn’t exactly what I want, because it only has my tweets. My favourite moments on Twitter were back-and-forth conversations, and my personal archive only contains my side of the conversation. In my custom scrapbook, I can capture both people’s contributions.

Data Lifeboat at the Flickr Foundation

Data Lifeboat is a project by the Flickr Foundation to create archival slivers of Flickr. I worked at the Foundation for nearly two years, and I built the first prototypes of Data Lifeboat. I joined because of my interest in archiving social media, and the ideas flowed in both directions: personal experiments informed my work, and vice versa.

Data Lifeboat and my scrapbook differ in some details, but the underlying principles are the same.

One of my favourite parts of that work was pushing static websites for tiny archives further than I ever have before. Each Data Lifeboat package includes a viewer app for browsing the contents, which is a static website built in vanilla JavaScript – very similar to the Twitter archive. It’s the most complex static site I’ve ever built, so much so that I had to write a test suite using Playwright.

That experience made me more ambitious about what I can do with static, self-contained sites.

My web bookmarks

Earlier this year I wrote about my bookmarks collection, which I also store in a static site. My bookmarks are mostly long-form prose and video – reference material with private notes. The scrapbook is typically short-form content, often with visual media, often with conversations I was a part of. Both give me searchable, durable copies of things I don’t want to lose.

I built my own bookmarks site because I didn’t trust a bookmarking service to last; I built my social media scrapbook because I don’t trust social media platforms to stick around. They’re two different manifestations of the same idea.

Tapestry, by the Iconfactory

Tapestry is an iPhone app that combines posts from multiple platforms into a single unified timeline – social media, RSS feeds, blogs. The app pulls in content using site-specific “connectors”, written with basic web technologies like JavaScript and JSON.

Although I don’t use Tapestry myself, I was struck by the design, especially the connectors. The idea that each site gets its own bit of logic is what inspired me to consider different data models for each site – and of course, I love the use of vanilla web tech.

Social media embeds on this site

When I embed social media posts on this site, I don’t use the native embeds offered by platforms, which pull in megabytes of of JavaScript and tracking. Instead, I use lightweight HTML snippets styled with my own CSS, an idea I first saw on Dr Drang’s site over thirteen years ago.

The visual appearance of these snippets isn’t a perfect match for the original site, but they’re close enough to be usable. The CSS and HTML templates were a good starting point for my scrapbook.

You can make your own scrapbook, too

I’ve spent a lot of time and effort on this project, and I had fun doing it, but you can build something similar with a fraction of the effort. There are lots of simpler ways to save an offline backup of an online page – a screenshot, a text file, a printout.

If there’s something online you care about and wouldn’t want to lose, save your own copy. The history of the Internet tells us that it will almost certainly disappear at some point.

The Internet forgets, but it doesn’t have to take your memories with it.

[If the formatting of this post looks odd in your feed reader, visit the original article]

<video style="aspect-ratio: 1920 / 1080">

If I also set a width or a height, the browser now knows exactly how much space this video will take up on the page – even if it hasn’t loaded the video file yet. When it initially renders the page, it can leave the right gap, so it doesn’t need to rearrange when the video eventually loads. (The technical term is “reducing cumulative layout shift”.)

That’s the idea, anyway.

I noticed that some of my videos weren’t fitting in their allocated boxes. When the video file loaded, it could be too small and get letterboxed, or be too big and force the page to rearrange to fit. Clearly there was a bug in my code for computing aspect ratios, but what?

Three aspect ratios, one video

I opened one of the problematic videos in QuickTime Player, and the resolution listed in the Movie Inspector was rather curious: Resolution: 1920×1080 (1350×1080).

The first resolution is what my code was reporting, but the second resolution is what I actually saw when I played the video. Why are there two?

The storage aspect ratio (SAR) of a video is the pixel resolution of a raw frame. If you extract a single frame as a still image, that’s the size of the image you’d get. This is the first resolution shown by QuickTime Player, and it’s what I was reading in my code.

I was missing a key value – the pixel aspect ratio (PAR). This describes the shape of each pixel, in particular the width-to-height ratio. It tells a video player how to stretch or squash the stored pixels when it displays them. This can sometimes cause square pixels in the stored image to appear as rectangles.

This reminds me of EXIF orientation for still images – a transformation that the viewer applies to the stored data. If you don’t apply this transformation properly, your media will look wrong when you view it. I wasn’t accounting for the pixel aspect ratio in my code.

According to Google, the primary use case for non-square pixels is standard-definition televisions which predate digital video. However, I’ve encountered several videos with an unusual PAR that were made long into the era of digital video, when that seems unlikely to be a consideration. It’s especially common in vertical videos like YouTube Shorts, where the stored resolution is a square 1080×1080, and the aspect ratio makes it a portrait.

I wonder if it’s being introduced by a processing step somewhere? I don’t understand why, but I don’t have to – I’m only displaying videos, not producing them.

The display aspect ratio (DAR) is the size of the video as viewed – what happens when you apply the pixel aspect ratio to the stored frames. This is the second resolution shown by QuickTime Player, and it’s the aspect ratio I should be using to preallocate space in my video player.

These three values are linked by a simple formula:

DAR = SAR × PAR

The size of the viewed video is the stored resolution times the shape of each pixel.

The stored frame may not be what you see

One video with a non-unit pixel aspect ratio is my download of Mars EDL 2020 Remastered. This video by Simeon Schmauß tries to match what the human eye would have seen during the landing of NASA’s Perseverance rover in 2021.

We can get the width, height, and sample aspect ratio (which is another name for pixel aspect ratio) using ffprobe:

$ ffprobe -v error \
      -select_streams v:0 \
      -show_entries stream=width,height,sample_aspect_ratio \
      "Mars 2020 EDL Remastered [HHhyznZ2u4E].mp4"
[STREAM]
width=1920
height=1080
sample_aspect_ratio=45:64
[/STREAM]

Here 1920 is the stored width, and 45:64 is the pixel aspect ratio. We can multiply them together to get the display width: 1920×45 / 64 = 1350. This matches what I saw in QuickTime Player.

Let’s extract a single frame using ffmpeg, to get the stored pixels. This command saves the 5000th frame as a PNG image:

$ ffmpeg -i "Mars 2020 EDL Remastered [HHhyznZ2u4E].mp4" \
    -filter:v "select=eq(n\,5000)" \
    -frames:v 1 \
    frame.png

The image is 1920×1080 pixels, and it looks wrong: the circular parachute is visibly stretched.

Suppose we take that same image, but now apply the pixel aspect ratio. This is what the image is meant to look like, and it’s not a small difference – now the parachute actually looks like a circle.

Seeing both versions side-by-side makes the problem obvious: the stored frame isn’t how the video is displayed. The video player in my browser will play it correctly using the pixel aspect ratio, but my layout code wasn’t doing that. I was telling the browser the wrong aspect ratio, and the browser had to update the page when it loaded the video file.

Getting the correct display dimensions in Python

This is my old function for getting the dimensions of a video file, which uses a Python wrapper around MediaInfo to extract the width and height fields. I now realise that this only gives me the storage aspect ratio, and may be misleading for some videos.

from pathlib import Path

from pymediainfo import MediaInfo


def get_storage_aspect_ratio(video_path: Path) -> tuple[int, int]:
    """
    Returns the storage aspect ratio of a video, as a width/height ratio.
    """
    media_info = MediaInfo.parse(video_path)
    
    try:
        video_track = next(
            tr
            for tr in media_info.tracks
            if tr.track_type == "Video"
        )
    except StopIteration:
        raise ValueError(f"No video track found in {video_path}")
    
    return video_track.width, video_track.height

I can’t find an easy way to extract the pixel aspect ratio using pymediainfo. It does expose a Track.aspect_ratio property, but that’s a string which has a rounded value – for example, 45:64 becomes 0.703. That’s close, but the rounding introduces a small inaccuracy. Since I can get the complete value from ffprobe, that’s what I’m doing in my revised function.

The new function is longer, but it’s more accurate:

from fractions import Fraction
import json
from pathlib import Path
import subprocess


def get_display_aspect_ratio(video_path: Path) -> tuple[int, int]:
    """
    Returns the display aspect ratio of a video, as a width/height fraction.
    """
    cmd = [
        "ffprobe",
        #
        # verbosity level = error
        "-v", "error",
        #
        # only get information about the first video stream
        "-select_streams", "v:0",
        #
        # only gather the entries I'm interested in
        "-show_entries", "stream=width,height,sample_aspect_ratio",
        #
        # print output in JSON, which is easier to parse
        "-print_format", "json",
        #
        # input file
        str(video_path)
    ]
    
    output = subprocess.check_output(cmd)
    ffprobe_resp = json.loads(output)
    
    # The output will be structured something like:
    #
    #   {
    #       "streams": [
    #           {
    #               "width": 1920,
    #               "height": 1080,
    #               "sample_aspect_ratio": "45:64"
    #           }
    #       ],
    #       …
    #   }
    #
    # If the video doesn't specify a pixel aspect ratio, then it won't
    # have a `sample_aspect_ratio` key.
    video_stream = ffprobe_resp["streams"][0]
    
    try:
        pixel_aspect_ratio = Fraction(
            video_stream["sample_aspect_ratio"].replace(":", "/")
        )
    except KeyError:
        pixel_aspect_ratio = 1
    
    width = round(video_stream["width"] * pixel_aspect_ratio)
    height = video_stream["height"]
    
    return width, height

This is calling the ffprobe command I showed above, plus -print_format json to print the data in JSON, which is easier for Python to parse.

I have to account for the case where a video doesn’t set a sample aspect ratio – in that case, the displayed video just uses square pixels.

Since the aspect ratio is expressed as a ratio of two integers, this felt like a good chance to try the fractions module. That avoids converting the ratio to a floating-point number, which potentially introduces inaccuracies. It doesn’t make a big difference, but in my video collection treating the aspect ratio as a float produces results that are 1 or 2 pixels different from QuickTime Player.

When I multiply the stored width and aspect ratio, I’m using the round() function to round the final width to the nearest integer. That’s more accurate than int(), which always rounds down.

Conclusion: use display aspect ratio

When you want to know how much space a video will take up on a web page, look at the display aspect ratio, not the stored pixel dimensions. Pixels can be squashed or stretched before display, and the stored width/height won’t tell you that.

Videos with non-square pixels are pretty rare, which is why I ignored this for so long. I’m glad I finally understand what’s going on.

After switching to ffprobe and using the display aspect ratio, my pre-allocated video boxes now match what the browser eventually renders – no more letterboxing, no more layout jumps.

[If the formatting of this post looks odd in your feed reader, visit the original article]