A day out to the Forth Bridge

While clearing out some boxes recently, I found a leaflet from an old holiday. It was a fun day out, and I’d always meant to share the pictures – so here’s the story a day trip from 2016.

I was spending a week in Edinburgh, relaxing and using up some holiday allowance at the end of my last job. My grandparents had suggested I might enjoy seeing the Forth Bridge, because I tend to like railways and railway-related things. I’d heard of the bridge, but I didn’t know that much about it – so while I was nearby, I decided to go take a look.

So on a cold December morning, I caught a train from Edinburgh station, up to a village on the north end of the Forth Bridge. I’d never heard of North Queensferry, but what little Googling I’d done suggested it was the best place to go if I wanted to see the bridge up close.

Here’s a map that shows the train line from Edinburgh to the village:

A map showing the railway line between Edinburgh and North Queensferry.
Map data from OpenStreetMap.

The train takes about 20 minutes, and it crosses the Forth Bridge just at the end of the journey. I wasn’t really aware from the bridge as we went across – not until I got out at the station, wandered into the village, and looked back towards the track.

A silhouette of the bridge and some trees against a grey sky.

The name of North Queensferry hints at its former life. It’s on the north side of the narrowest point of the Firth of Forth, which makes it a natural choice if you want to cross the water by boat.

It’s said that in 1068, Saint Margaret of Scotland (wife of King Malcolm III) created the village to ensure a safe crossing point for pilgrims heading to St. Andrew’s. Whether or not she actually created the village, she was a regular user of the ferry service to travel between Dunfurmline (the then-capital of Scotland) and Edinburgh Castle.

For centuries, there were regular crossings of boats and ferries. You can still see a handful of small boats in the harbour, but I’m sure it used to be a lot busier.

Photo from the water's edge, with two bridges in the background and a couple of boats in the water.

Update, 21 March 2019: It turns out the lighthouse isn’t the only reminder of the ferry service! Chris, an ex-Wellcome colleague and archivist extraordinaire, found some pictures from the Britten-Pears foundation (whose archive and library he runs), including a ticket from the ferry service:

Another lost transport world in @BrittenOfficial’s papers: the ferry over the Forth at Queensferry, which operated until the Forth Road Bridge opened in 1964. Here are 3 passenger tickets plus what I think is a counterfoil from a ticket for a car, from 1950. pic.twitter.com/mpSWZYeEiu

Although most of the boats are gone, one part of the ferry service survives – the lighthouse! This tiny hexagonal tower is the smallest working light tower in the world. It was built in 1817 by Robert Stevenson, a Scottish civil engineer who was famous for building lighthouses. (It was a name I recognised; I loved the story of the Bell Rock Lighthouse when I was younger.)

The light tower sits on the pier, where the ferries used to dock.

A yellowish-stone hexagonal tower, with a domed roof and windows around the top.

Unlike many lighthouses of the time, the keeper didn’t live in the lighthouse itself – but they were still responsible for keeping the flame lit, the oil topped out, and the lighthouse maintained. At night, it would have been an invaluable guide for boats crossing the Firth.

Today, the lighthouse is open to the public. (I think this is where I picked up my leaflet.) You can climb the 24 steps, see the lamp mechanism, and look out over the water. When lit, it gave a fixed white light, with a paraffin-burning lamp – and the large half-dome was the parabolic reflector that turned the candle light into a focused beam.

The back of a copper-coloured, parabolic lens looking out through a lighthouse window.

I wish I’d got a few more photos of the inside of the lighthouse, but it was a pretty small space, and I was struggling to find decent angles. Either way, the lighthouse was an unexpected treat – not something I was expecting at all!

But these days, North Queensferry isn’t known for its ferry service – it’s known for the famous bridge.

In the 1850s, the Edinburgh, Leith and Granton Railway ran a “train ferry” – a boat that carried railway carriages between Granton and Burntisland. There was a desire to build a continuous railway service, and the natural next step was a bridge. After a failed attempt to build a suspension bridge in the 1870s (axed after the Tay Bridge disaster), there was a second attempt in the 1880s. It was opened in March 1890, and it’s still standing today.

Here’s a photo of its original construction, taken from the North Queensferry hills:

A sepia-toned photograph of a partially constructed bridge, with three cantilevers visible above the water.
Image of the construction of the Forth Bridge, from the National Library of Scotland.

The Forth Bridge is a cantilever bridge. Each structure in the photo above is one of the cantilevers – a support structure fixed at one end – and in the finished bridge the load is spread between them. Spreading the load between multiple cantilevers allows you to build longer bridges, and the Forth Bridge uses this to great effect.

One of the advantages of cantilever bridges is that they don’t require any temporary support while they’re being built – once the initial structures are built, you can expand outwards and they’ll take the weight. Here’s another photo from the construction which shows off this idea:

Black-and-white photo from the construction of the Forth Bridge.
Another photo of the bridge under construction, George Washington Wilson.

You can see the shape of the bridge starting to expand out from the initial structure.

The Forth Bridge is famous for a couple of reasons. When it was built, it was the longest cantilever span in the world (not bested for another twenty-nine years, and still the second longest). It was also one of the first major structures in Britain to use steel – 55,000 tonnes in the girders, along with a mixture of granite and other materials in the masonry.

You get a great view of the finished bridge from inside the lighthouse:

Looking from inside the lighthouse window, with a red bridge visible outside and part of a copper-coloured lamp housing inside.

As I wandered around the village, I got lots of other pictures of the rail bridge. These are a few of my favourites:

The lighthouse in the foreground on the left, with the bridge set against a blue sky in the background.
Another photo with the lighthouse in the foreground, and the bridge running parallel to the horizon in the background.
The bridge dominating the background, with the jetty and a few boats in the foreground.
The bridge crossing the frame, set against a blue sky.

The last one was my favourite photo of the entire holiday, and I have a print of it on the wall of my flat. Blue skies galore, which made for a lovely day and some wonderful pictures – even if it’s not what you expect from a Scottish winter!

What’s great about wandering around the village is that you can see the bridge from all sorts of angles, not just from afar – you can get up and personal. You can see the approach viaduct towering over the houses as it approaches the village:

The bridge running across the image, with a few houses visible along the ground.

And you can get even closer, and walk right underneath the bridge itself. Here’s what part of the viaduct holding up the bridge looks like:

A yellowish-coloured stone viaduct, with the red girders of the bridge atop it.

They’re enormous – judging by the stairs, it’s quite a climb up!

A side-on view of one of the pillars and the red bridge, with some stairs going up the side of the pillar.

And you can look up through the girders, and see the thousands of beams that hold the bridge together:

A silhouette of the girders in the bridge, looking up from underneath.

It’s been standing for over a century, so I’m sure it’s quite safe – but it was still a bit disconcerting to hear a rattle as trains passed overhead!

I spent quite a while just wandering around under the bridge, looking up in awe at the structure. As you wander around the village, you never really get away from it – it always stands tall in the background. (Well, except when I popped into a café for some soup and a scone.)

After the rail bridge was built, the ferry crossings continued for many years – in part buoyed by the rise of personal cars, which couldn’t use the railway tracks. But it didn’t last – in 1964, a second bridge was built, the Forth Road Bridge – and it replaced the ferries. The day the bridge opened, the ferry service closed after eight centuries of continuous crossings.

When I visited in 2016, the Road Bridge was still open to cars. At the time, there was a second bridge under construction, but not yet open to the public – the Queensferry Crossing opened half a year later after my visit. The original Road Bridge is a public transport link (buses, cyclists, taxis and pedestrians), and the new bridge carries everything else.

Three bridges in a single day! The other bridges were on the other side of the bay, so I had to walk along the water’s edge to see them. Here’s a photo from midway along, with old and new both visible:

Two bridges spanning across a body of water.

These are both suspension bridges, whereas the rail bridge is a cantileverl.

Like the rail bridge, you can get up and close with the base of the road bridge. Here’s an attempt at an “artsy” shot of the bridge receding into the distance, with sun poking through the base:

The silhouette of a bridge on the right, with green grass and water visible.

And another “artsy” shot with more lens flare, and both the road bridges in the shot. I love the detail of the underside on the nearer bridge.

Looking at the underside of two bridges in silhouette, with lens flare in the middle of the photo.

Here’s the start of the bridge on the north side, starting to rise up over the houses:

A series of arched bridge supports rising up above some houses.

And one more close-up shot of one of the supports:

A single concrete support, with the ridged underside of the bridge visible.

Eventually it started getting dark, so I decided to head home. I considered walking back through North Queensferry to the station, but I decided to have a crack at crossing the road bridge instead, and catching the train from the other side. You can walk across it, although it’s nearly 2.5k long!

Looking onto a bridge, with a path directly ahead, fences and roadworks to the right, and water below on the left.

As you climb up to the bridge, I got some wonderful views back over the village, and in particular towards the rail bridge I’d originally come to see:

A village of houses in shadow in the foreground, with the bridge clearly visible in the background.

I didn’t take many photos from the bridge itself, although it’s a stunning view! It was extremely cold and windy, and I didn’t want to risk losing my camera while trying to take a photo. Here’s one of the few photos I did take, which I rather like. I took it near the midpoint, with the rail bridge set against a cloudy sky. (I’d forgotten about it until I came to write this post!)

Looking straight on to the side of the rail bridge, with blue sky and grey clouds behind it.

Safely across the bridge, I weaved my way through South Queensferry, found the station, and caught a train back to Edinburgh.

I didn’t plan this trip when I decided to visit Scotland, but over two years later, I still have fond memories of the day out. If you’re ever nearby and you like looking at impressive structures, it’s worth a trip. I’m glad I have pictures, but it’s hard to capture the sheer size and scale of a bridge this large – so if you have a chance, do visit in person.

Finding the latest screenshot in macOS Mojave

One of the things that changed in macOS Mojave was the format of screenshot filenames. On older versions of macOS, the filename would be something like:

Screen Shot 2016-10-10 at 18.34.18.png

On Mojave, the first two words got collapsed into one:

Screenshot 2019-03-08 at 18.38.41.png

I have a handful of scripts for doing something with screenshots – and in particular, a shortcut that grabs the newest screenshot. When I started updating to Mojave, I had to update the shell snippet that powers that shortcut. Because I couldn’t update to Mojave on every machine immediately, it had to work with both naming schemes.

This is what I’ve been using for the last few months (bound to last_screenshot in my shell config):

find ~/Desktop -name 'Screen Shot*' -print0 -o -name 'Screenshot*' -print0
  | xargs -0 stat -f '%m %N'
  | sort --numeric-sort --reverse
  | head -1
  | cut -f "2-" -d " "

Let’s break it down:

It’s certainly possible to do this with a higher-level language like Python or Ruby, but I like the elegance of chaining together tiny utilities like this. For non-critical code, I enjoy the brevity.

Atomic, cross-filesystem moves in Python

If you want to move a file around in Python, the standard library gives you at least two options: os.rename() or shutil.move(). Both of them work in certain circumstances, but they make different tradeoffs:

Sometimes you want both of those properties: you want to move across a filesystem boundary and have an atomic move.

For example: Loris, an image server. When a user requests an image, we start by downloading it from the source to a temporary folder. If the download succeeds, we move the saved image into another cache, and that known-good cache is used to serve the image to the user. We want that move to be atomic – so we won’t serve a partial image from the cache – and in some setups, the temporary download folder and the image cache are on different filesystems. We need a move function that can be both atomic and work across filesystems.

I’ve had to write code for this a couple of times now, so I’m writing it up here both as a reminder to myself, and an instruction for other people in case it’s useful.

Writing the code

If we’re copying within the same filesystem, os.rename() gives us everything we need. Let’s try that first, and only do something different if we get an error:

import os


def safe_move(src, dst):
    try:
        os.rename(src, dst)
    except OSError:
        # do something else...

This except clause is quite broad – it catches and retries any error thrown by os.rename(). There are lots of errors that have nothing to do with a cross-filesystem move – for example, if the source file just doesn’t exist! We should only catch and retry the specific error that comes from copying across a filesystem boundary.

If you try it, this is the error you get:

>>> import os
>>> os.rename("/mnt/semele/hello.txt", "/mnt/dionysus/hello.txt")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
OSError: [Errno 18] Cross-device link: '/mnt/semele/hello.txt' -> '/mnt/dionysus/hello.txt'

Error code 18 is what we want to retry – this is a standard Linux error number meaning “invalid cross-device link”. We can use the errno library to get 18 as a named variable that’s a little less of a magic number, like so:

import errno


def safe_move(src, dst):
    try:
        os.rename(src, dst)
    except OSError as err:
        if err.errno == errno.EXDEV:
            # do something else...
        else:
            raise

So now we need to decide what “something else” looks like.

To get the file onto the same filesystem, we can use shutil.move() to put it in the same directory as the intended destination, but with a different filename. As a first pass, we might try something like:

import shutil


def safe_move(src, dst):
    ...
    # do something else
    tmp_dst = dst + ".tmp"
    shutil.copyfile(src, tmp_dst)
    os.rename(tmp_dst, dst)
    os.unlink(tmp_dst)

This could be okay in certain circumstances, but if you have multiple worker processes you could end up with a corrupted destination. If you’re running multiple processes, and they both try to copy to the temporary destination, you could get garbage data in that file. One process might think it’s completed the copy, then rename the file as the other process is still writing to it.

To avoid processes treading on each other’s toes, add a unique ID to each copy – that way they can’t overlap. Closer to:

import uuid


def safe_move(src, dst):
    ...
    # do something else
    copy_id = uuid.uuid4()
    tmp_dst = "%s.%s.tmp" % (dst, copy_id)
    shutil.copyfile(src, tmp_dst)
    os.rename(tmp_dst, dst)
    os.unlink(tmp_dst)

This is an idea I originally got from a Stack Overflow answer about lock-free copy algorithms. This isn’t quite the same problem as that question – in particular, I don’t care if the file already exists – but the answer and the linked paper make interesting reading.

Putting it all together

If you just want the code, here’s the final version (with comments):

import errno
import os
import shutil


def safe_move(src, dst):
    """Rename a file from ``src`` to ``dst``.

    *   Moves must be atomic.  ``shutil.move()`` is not atomic.
        Note that multiple threads may try to write to the cache at once,
        so atomicity is required to ensure the serving on one thread doesn't
        pick up a partially saved image from another thread.

    *   Moves must work across filesystems.  Often temp directories and the
        cache directories live on different filesystems.  ``os.rename()`` can
        throw errors if run across filesystems.

    So we try ``os.rename()``, but if we detect a cross-filesystem copy, we
    switch to ``shutil.move()`` with some wrappers to make it atomic.
    """
    try:
        os.rename(src, dst)
    except OSError as err:

        if err.errno == errno.EXDEV:
            # Generate a unique ID, and copy `<src>` to the target directory
            # with a temporary name `<dst>.<ID>.tmp`.  Because we're copying
            # across a filesystem boundary, this initial copy may not be
            # atomic.  We intersperse a random UUID so if different processes
            # are copying into `<dst>`, they don't overlap in their tmp copies.
            copy_id = uuid.uuid4()
            tmp_dst = "%s.%s.tmp" % (dst, copy_id)
            shutil.copyfile(src, tmp_dst)

            # Then do an atomic rename onto the new name, and clean up the
            # source image.
            os.rename(tmp_dst, dst)
            os.unlink(src)
        else:
            raise

I’ve been running code like this in production for over a year (as part of our Loris installation at Wellcome), and used it in a few other places with no issues.

Checking Jekyll sites with HTMLProofer

Here’s a quick change I’ve just made to my Jekyll setup: I’ve added HTML linting with HTMLProofer.

HTMLProofer is a Ruby library that can validate HTML files – checking that links and internal references work, images have alt tags, HTML markup is valid, and so on. I was already doing a few of those checks with some hand-rolled scripts, but passing it off to a library is much nicer. I have less code to maintain, and I get more thorough error checking to boot.

Adding it to my setup was pleasantly easy. I added one line to my Gemfile:

gem "html-proofer", "~> 3.2"

and then I added a small plugin to my _plugins directory:

require "html-proofer"

Jekyll::Hooks.register :site, :post_write do |site|
  HTMLProofer.check_directory(site.config["destination"], opts = {
    :check_html => true,
    :check_img_http => true,
    :disable_external => true,
    :report_invalid_tags => true,
  }).run
end

This creates a Jekyll hook that runs after the entire site has been written to disk. It calls the HTMLProofer library, looking at all the HTML files that Jekyll has just created. If it finds any errors, it throws an exception and the build fails.

When I first ran with HTMLProofer enabled, it found nearly a thousand errors – mostly missing alt text on images. For now it’s not running on every build, but I’m planning to fix up the errors and then turn on these checks everywhere. That way, I’ll know immediately if I’ve done something wrong.

This sort of library is why I really like using Jekyll, rather than maintaining my own static site generator. There’s a rich ecosystem of plugins and tools that I can use straight away, rather than building from scratch. It was really easy to put this together – less than ten minutes from start to finish. That means less time programming, and more time for writing.

Working with really large objects in S3

One of our current work projects involves working with large ZIP files stored in S3. These are files in the BagIt format, which contain files we want to put in long-term digital storage. Part of this process involves unpacking the ZIP, and examining and verifying every file. So far, so easy – the AWS SDK allows us to read objects from S3, and there are plenty of libraries for dealing with ZIP files.

In Python, you can do something like:

import zipfile

import boto3

s3 = boto3.client("s3")
s3.download_file(Bucket="bukkit", Key="bagit.zip", Filename="bagit.zip")

with zipfile.ZipFile("bagit.zip") as zf:
    print(zf.namelist())

This is what most code examples for working with S3 look like – download the entire file first (whether to disk or in-memory), then work with the complete copy.

Where this breaks down is if you have an exceptionally large file, or you’re working in a constrained environment. Some of our BagIt files are tens of gigabytes, and the largest might be over half a terabyte (even if the individual files are small). And if you’ve gone serverless and you’re running in AWS Lambda, you only get 500 MB of disk space. What are we to do?

To process a ZIP file (like many formats), you don’t need to load the entire file at once – it has a well-defined internal structure, and you can read it a bit at a time. In a ZIP, there’s a table of contents that tells you what files it contains, and where they are in the overall ZIP. If you want to extract a single file, you can read the table of contents, then jump straight to that file – ignoring everything else. This is easy if you’re working with a file on disk, and S3 allows you to read a specific section of a object if you pass an HTTP Range header in your GetObject request.

So if we construct a wrapper for S3 objects that passes the correct Range headers, we can process a large object in S3 without downloading the whole thing.

I couldn’t find any public examples of somebody doing this, so I decided to try it myself. In this post, I’ll walk you through how I was able to stream a large ZIP file from S3. But fair warning: I wrote this as an experiment, not as production code. You’re welcome to use it, but you might want to test it first.

Getting a file-like object

In Python, there’s a notion of a “file-like object” – a wrapper around some I/O that behaves like a file, even if it isn’t actually a file on disk. It responds to calls like read() and write(), and you can use it in places where you’d ordinarily use a file. The docs for the io library explain the different methods that a file-like object can support, although not every file-like object supports every method – for example, you can’t write() to an HTTP response body.

Many libraries that work with local files can also work with file-like objects, including the zipfile module in the Python standard library. If we can get a file-like object from S3, we can pass that around and most libraries won’t know the difference!

The boto3 SDK actually already gives us one file-like object, when you call GetObject. Like so:

import boto3

s3 = boto3.client("s3")
s3_object = s3.get_object(Bucket="bukkit", Key="bagit.zip")

print(s3_object["Body"])
# <botocore.response.StreamingBody object at 0x10c46f438>

That StreamingBody is a file-like object responds to read(), which allows you to download the entire file into memory. So let’s try passing that into ZipFile:

import zipfile

import boto3

s3 = boto3.client("s3")
s3_object = s3.get_object(Bucket="bukkit", Key="bagit.zip")
streaming_body = s3_object["Body"]

with zipfile.ZipFile(streaming_body) as zf:
    print(zf.namelist())

Unfortunately, that throws an error:

Traceback (most recent call last):
  File "example.py", line 11, in <module>
    with zipfile.ZipFile(s3_object["Body"]) as zf:
  File "/usr/local/Cellar/python/3.6.4_4/Frameworks/Python.framework/Versions/3.6/lib/python3.6/zipfile.py", line 1108, in __init__
    self._RealGetContents()
  File "/usr/local/Cellar/python/3.6.4_4/Frameworks/Python.framework/Versions/3.6/lib/python3.6/zipfile.py", line 1171, in _RealGetContents
    endrec = _EndRecData(fp)
  File "/usr/local/Cellar/python/3.6.4_4/Frameworks/Python.framework/Versions/3.6/lib/python3.6/zipfile.py", line 241, in _EndRecData
    fpin.seek(0, 2)
AttributeError: 'StreamingBody' object has no attribute 'seek'

So although StreamingBody is file-like, it doesn’t support the methods we need. We’ll have to create our own file-like object, and define those methods ourselves.

Creating our own file-like object

The io docs suggest a good base for a read-only file-like object that returns bytes (the S3 SDK deals entirely in bytestrings) is RawIOBase, so let’s start with a skeleton class:

import io


class S3File(io.RawIOBase):
    def __init__(self, s3_object):
        self.s3_object = s3_object

Note: the constructor expects an instance of boto3.S3.Object, which you might create directly or via a boto3 resource. This means our class doesn’t have to create an S3 client or deal with authentication – it can stay simple, and just focus on I/O operations.

Implementing the seek() method

When we tried to load a ZIP file the first time, we discovered that somewhere the zipfile module is using the seek() method. Let’s implement that as our first operation. The io docs explain how seek() works:

seek(offset[, whence])

Change the stream position to the given byte offset. offset is interpreted relative to the position indicated by whence. The default value for whence is SEEK_SET. Values for whence are:

Return the new absolute position.

This hints at the key part of doing selective reads: we need to know how far through we are. What part of the object are we currently looking at? When we open a file on disk, the OS handles that for us – but in this case, we’ll need to track it ourselves.

This is what a seek() method might look like:

import io


class S3File(io.RawIOBase):
    def __init__(self, s3_object):
        self.s3_object = s3_object
        self.position = 0

    def seek(self, offset, whence=io.SEEK_SET):
        if whence == io.SEEK_SET:
            self.position = offset
        elif whence == io.SEEK_CUR:
            self.position += offset
        elif whence == io.SEEK_END:
            self.position = self.s3_object.content_length + offset
        else:
            raise ValueError("invalid whence (%r, should be %d, %d, %d)" % (
                whence, io.SEEK_SET, io.SEEK_CUR, io.SEEK_END
            ))

        return self.position

    def seekable(self):
        return True

We’ve added the position attribute to track where we are in the stream, and that’s what we update when we call seek().

The content_length attribute on the S3 object tells us its length in bytes, which corresponds to the end of the stream.

For the ValueError, I copied the error you get if you pass an unexpected whence to a regular open() call:

>>> open("example.py").seek(5, 5)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: invalid whence (5, should be 0, 1 or 2)

Now let’s try using this updated version:

s3 = boto3.resource("s3")
s3_object = s3.Object(bucket_name="bukkit", key="bag.zip")

s3_file = S3File(s3_object)

with zipfile.ZipFile(s3_file) as zf:
    print(zf.namelist())

This gets further, but now it throws a different error:

Traceback (most recent call last):
  File "example.py", line 38, in <module>
    with zipfile.ZipFile(s3_file) as zf:
  File "/usr/local/Cellar/python/3.6.4_4/Frameworks/Python.framework/Versions/3.6/lib/python3.6/zipfile.py", line 1108, in __init__
    self._RealGetContents()
  File "/usr/local/Cellar/python/3.6.4_4/Frameworks/Python.framework/Versions/3.6/lib/python3.6/zipfile.py", line 1171, in _RealGetContents
    endrec = _EndRecData(fp)
  File "/usr/local/Cellar/python/3.6.4_4/Frameworks/Python.framework/Versions/3.6/lib/python3.6/zipfile.py", line 251, in _EndRecData
    data = fpin.read()
NotImplementedError

So we know what to implement next!

Implementing the read() method

The io docs explain how read() works:

read(size=-1)

Read up to size bytes from the object and return them. As a convenience, if size is unspecified or -1, all bytes until EOF are returned. Otherwise, only one system call is ever made. Fewer than size bytes may be returned if the operating system call returns fewer than size bytes.

If 0 bytes are returned, and size was not 0, this indicates end of file. If the object is in non-blocking mode and no bytes are available, None is returned.

To implement this method, we have to remember that we read from the position set by seek() – not necessarily the start of object. And when we’ve read some bytes, we need to advance the position.

To read a specific section of an S3 object, we pass an HTTP Range header into the get() call, which defines what part of the object we want to read.

So let’s add a read() method:

class S3File(io.RawIOBase):
    ...

    @property
    def size(self):
        return self.s3_object.content_length

    def read(self, size=-1):
        if size == -1:
            # Read to the end of the file
            range_header = "bytes=%d-" % self.position
            self.seek(offset=0, whence=io.SEEK_END)
        else:
            new_position = self.position + size

            # If we're going to read beyond the end of the object, return
            # the entire object.
            if new_position >= self.size:
                return self.read()

            range_header = "bytes=%d-%d" % (self.position, new_position - 1)
            self.seek(offset=size, whence=io.SEEK_CUR)

        return self.s3_object.get(Range=range_header)["Body"].read()

    def readable(self):
        return True

This is a little more complicated than seek().

I’ve added a size property that exposes the length of the stream, and wraps the content_length attribute. This is for convenience.

If the user doesn’t specify a size for read(), we create an open-ended Range header and seek to the end of the file. Note that I’m calling seek() rather than updating the position manually – it saves me writing a second copy of the logic for tracking the position.

If the caller passes a size to read(), we need to work out if this size goes beyond the end of the object – in which case we should truncate it! If it is too big, we fall back to reading the entire object, by making a second call to read() – we don’t need to duplicate that logic. If it’s not, we create a Range header (and note that byte ranges include the upper value, hence the -1), and seek to the appropriate position.

Then we call the get() method on the object, pass the Range header, and read all the bytes that come back.

If you use this version of the code, we can load the list of files in the ZIP correctly:

s3 = boto3.resource("s3")
s3_object = s3.Object(bucket_name="bukkit", key="bag.zip")

s3_file = S3File(s3_object)

with zipfile.ZipFile(s3_file) as zf:
    print(zf.namelist())

And that’s all you need to do selective reads from S3.

Is it worth it?

There’s a small cost to making GetObject calls in S3 – both in money and performance.

This wrapper class uses more GetObject calls than downloading the object once. In my brief experiments, it took 3 calls to load the table of contents, and another 3 calls to load an individual file. If you can, it’s cheaper and faster to download the entire object to disk, and do all the processing locally – but only if you have the resources to do so! This wrapper is useful when you can’t do that.

In practice, I’d probably use a hybrid approach: download the entire object if it’s small enough, or use this wrapper if not. I’d trade some extra performance and lower costs for a bit more code complexity.

At work, we write everything in Scala, so I don’t think we’ll ever use this code directly. (At best, we’ll use the ideas it contains.) But this post proves the general idea: you can process a large object in S3 without having to download the whole thing first.

Putting it all together: the final code

Adding a couple of extra convenience methods (a repr() for pretty-printing, and tell() is a useful convenience), this is the final code, along with the example:

import io


class S3File(io.RawIOBase):
    def __init__(self, s3_object):
        self.s3_object = s3_object
        self.position = 0

    def __repr__(self):
        return "<%s s3_object=%r>" % (type(self).__name__, self.s3_object)

    @property
    def size(self):
        return self.s3_object.content_length

    def tell(self):
        return self.position

    def seek(self, offset, whence=io.SEEK_SET):
        if whence == io.SEEK_SET:
            self.position = offset
        elif whence == io.SEEK_CUR:
            self.position += offset
        elif whence == io.SEEK_END:
            self.position = self.size + offset
        else:
            raise ValueError("invalid whence (%r, should be %d, %d, %d)" % (
                whence, io.SEEK_SET, io.SEEK_CUR, io.SEEK_END
            ))

        return self.position

    def seekable(self):
        return True

    def read(self, size=-1):
        if size == -1:
            # Read to the end of the file
            range_header = "bytes=%d-" % self.position
            self.seek(offset=0, whence=io.SEEK_END)
        else:
            new_position = self.position + size

            # If we're going to read beyond the end of the object, return
            # the entire object.
            if new_position >= self.size:
                return self.read()

            range_header = "bytes=%d-%d" % (self.position, new_position - 1)
            self.seek(offset=size, whence=io.SEEK_CUR)

        return self.s3_object.get(Range=range_header)["Body"].read()

    def readable(self):
        return True


if __name__ == "__main__":
    import zipfile

    import boto3

    s3 = boto3.resource("s3")
    s3_object = s3.Object(bucket_name="bukkit", key="bagit.zip")

    s3_file = S3File(s3_object)

    with zipfile.ZipFile(s3_file) as zf:
        print(zf.namelist())

As I said at the top, I wrote this as an experiment, not as production code. Feel free to use it (MIT licence), but you probably want to do some more testing first!

More advice on running inclusive and welcoming events

Last year, I wrote a blog post with my ideas for running inclusive conferences. I was thrilled by the number of people who read, shared, or gave me feedback and suggestions. I kept all the suggestions in a text file, but I never quite got around to writing an updated post.

Just before the weekend, I sent a link to the post in a reply to this tweet by Ada Powers:

If you have the spoons, tell me about the conference accessibility fails you’ve witnessed or experienced. Can cover disability, MH, and related concerns. (Name names if you want, but I’m primarily soliciting feedback to help others from making the same mistakes.)

Her original tweet got a huge response (so many good replies), and sent a spike of traffic to my old blog post. I realised it was time to update it properly, so I dusted it off and wrote a new version. You can read my updated advice at https://alexwlchan.net/ideas-for-inclusive-events/.

The new version includes:

And because it’s no longer a point-in-time snapshot (like a blog post), I’m hoping to update this version a little more regularly.

Monki Gras 2019: The Curb Cut Effect

Earlier today I did a talk at Monki Gras 2019. The theme of the conference is “Accessible Craft: Creating great experiences for everyone”, so I did a talk about inclusive design – and in particular, something called the Curb Cut Effect.

I originally pitched a talk based on Assume Worst Intent, because if you’re thinking about inclusion you might think about ways to avoid exclusion (specifically, exclusion of harassment and abuse victims). That talk was definitely too narrow for this conference, but James isolated a key idea – making a service better for vulnerable uses makes it better for everyone – and I wrote an entirely new talk around it.

The talk went really well – everybody was very nice afterwards, and I’m enjoying the rest of talks too. (Notes on those will be a separate post.) It was a lot of fun to write and present.

The talk was recorded, and you can watch it on YouTube:

You can read the slides and my notes on this page, or download the slides as a PDF.

Slides and notes

Title slide.

Hi, I’m Alex. I’m going to talk about the Curb Cut Effect, what it is, and how we might use it.

The Wellcome Collection building, lit up in purple.

The Wellcome Collection building lit up in purple to mark International Day of Persons with Disabilities in December 2018. Image credit: Wellcome Collection.

I’m a software developer at Wellcome Collection, a free museum and library on Euston Road.

At Wellcome, we’ve been thinking a lot about architecture recently. Our current exhibition, Living with Buildings, is all about the effect of architecture on human health, so I wanted to start today by telling you all the story of architecture – specifically, the story of a common architectural feature that we all walk past every day without a second glance.

A dropped kerb against a black background, with the caption “dropped kerb aka curb cut”.

A dropped kerb around the back of UCL, near the Wellcome offices. Image credit: me!

I’m talking about those areas where the kerb dips to form a ramp – giving a level, step-free path from the road to the pavement. In the UK, these are usually accompanied by a textured yellow surface (pictured).

They go by several names – in the UK we call them dropped kerbs, in the US they’re called curb cuts (one of the few American spellings that adds the letter U), and they have other names around the world. The American spelling is mostly common, and that’s what I’ll use for the rest of the talk, but you can mentally substitute “curb cut” for your preferred term.

A map, with circled areas “Kalamazoo” and “Battle Creek”.

A map showing part of Michigan, highlighting Kalamazoo and Battle Creek. Image credit: original map from the US Geological Survey.

One of the earliest examples of curb cuts was in Kalamazoo, MI. (Great name!)

There was a man called Jack Fisher, born in Kalamazoo in 1918. Like many young men of his age, he enlisted in the Army during the Second World War.

He was involved in a jeep accident in 1943. While he was recovering in hospital, he read the records of other patients with similar injuries to keep himself occupied. (Because 1940s privacy laws.)

He returned to Kalamazoo with steel braces from his hip to his neck, and a heavy limp. After the war, he graduated from Harvard law school (very impressive), but none of the established firms would take him – he was a disabled veteran, who might trip and injure himself, and need compensation. So they chose not to hire him. (Because 1940s labour rights.)

So he worked as an attorney in his own practice. He became well-known among other disabled veterans in and near Kalamazoo – of which there were many, because there was a nearby hospital at Battle Creek which specialised in amputee treatment and rehabilitation.

A black and white photo of a wheelchair standing at the edge of a raised curb.

A wheelchair standing at an raised, inaccessible curb. Image from an article by the Smithsonian.

Working closely with them, he became aware of the problems and challenges they faced.

One of those problems: Kalamazoo had tall curbs (up to 6 inches). This was a problem – people would trip, injure themselves, damage prosthetic limbs, and for wheelchair users they’re a total nightmare. Tall kerbs are inaccessible, and prevent people getting around, socialising, working and so on.

A black and white photo of a street, with somebody walking up a ramp with hand rails cut into the kerb.

A ramp with hand rails on the streets of Kalamazoo. Image from an article in Encore Magazine.

So in 1945, Jack Fisher took it upon himself to fix this, and petitioned the city commission for curb cuts and hand-rails. Ditching the step would make it easier for people to get around. The city authorised their construction and ran a small pilot program, they were very successful, and they grew in number.

Kalamazoo is one of the earliest examples of dropped kerbs, but the same story plays out in lots of other places – curb cuts were installed in lots of places to make the streets more accessible for disabled people and wheelchair users.

Text slide, white text on purple. “Curb cuts make the roads more accessible for wheelchair users and disabled people, but they aren't the only people who benefit!”

So curb cuts make the roads more accessible for wheelchair users and disabled people. Yay!

But they’re not the only people to benefit, so do:

And probably others.

Wcould call this a “force multipler” or a “happy accident”, but really this is the original example of the “Curb Cut Effect”.

Text slide, white text on purple. “Making something better for disabled people can make it better for everyone.”

The Curb Cut Effect comes in many forms, but the way I think of it is:

Making something better for disabled people can make it better for everyone.

Text slide, white text on purple. “What it means for us: Designs that include disabled people are better designs for everybody.”

What this means for us, as people who build things:

Designs that think about disabled people are better designs for everyone

And we reflect this is the words we use: it’s why we don’t talk about handicapped design or barrier-free design. We talk universal design. We talk about good design.

Text slide, white text on purple. “Making something better for disabled people can make it better for everyone.”

(Repeat the Curb Cut Effect.)

Let’s look at a few examples.

A black-and-white artwork of a woman sitting at a table. There’s a large machine with a keyboard on the table.

One of the earliest examplse predates Kalamazoo by more than a century.

This is the Countess Carolina Fantoni da Fivizzano. She lived in the early nineteenth century, and was the friend and lover of the Italian inventor Pellegrino Turri.

They’d write each other letters, but she lost her sight as an adult, and was unable to write herself. That meant the only way to send letters was to dictate to somebody else – but she wanted to send letters in private. Together they built a machine that let her write letters by pressing a key for each letter – allowing her to “type” letters.

Writing with type… a type-writer, of sorts.

This was one of the earliest iterations of the typewriter. It made writing accessible to the blind, and the derivatives became the modern-day keyboard. A machine created to help one blind woman write love letters was the basis for a fundamental input device for modern computing.

A photo of a laptop with an email client open.

Somebody using an email client on a laptop. Image credit: rawpixel.com on Pexels.

Speaking of love letters… let’s talk about email!

(You don’t use email to write love letters? Sounds fake.)

Email has become a ubiquitous part of modern comms, but where did it come from? Why was it invented?

A photo of a man in a suit (Vint Cerf), with a quote overlaid. “Because I’m hearing-impaired, emails are a tremendously valuable tool because of the precision that you get. I can read what’s typed as opposed to straining to hear what’s being said.”

A picture of Vinton Cerf, taken from his Royal Society photo and overlaid with a quote from a CNET article.

This is Vinton Cerf. He’s often called the “Father of the Internet”, did a lot of work on the early Internet (then-ARPANET) protocols, is a strong advocate for accessibility, and led the work on the first commercial email program.

Why?

Because he’s hard-of-hearing, and his wife Sigrid is deaf. His work on email started, in part, as a way for them to stay connected when they weren’t in the same room. At the time, the alternative was the telephone, which was a bit of a non-starter.

Here’s a quote he gave to CNET that sums it up nicely:

Because I’m hearing-impaired, emails are a tremendously valuable tool because of the precision that you get.

I can read what’s typed as opposed to straining to hear what’s being said.

Email started as a key technology for people who are deaf or have hearing loss – or who are just separated by time and space.

A screenshot from Sesame Street, with four characters on screen and a caption “Now everyone can read it”.

Sticking with hearing loss, let’s talk about captions.

It was originally intended to help deaf people understand movies with dialogue or sound effects. Closed captions were first broadcast in the 1970s. They were expensive, needed specialist equipment, and most content wasn’t captioned – today things are generally better. And indeed, they help people who are deaf or hard-of-hearing, but lots of other people besides:

And even if you can hear the sound fine, you can still benefit from captions:

A person with their hands on a special “stenographic” keyboard, looking up to hear someone talking.

A photo of one of the captioners at PyCon UK 2017, by Mark Hawkins.

And it helps with conferences too!

I’m being captioned right now – literally as I speak! [Monki Gras has live captioning.] This is one of the captioners at PyCon UK, and our experience is that lots of people find it useful during talks, not just the deaf or hard-of-hearing – maybe a word you couldn’t hear, somebody’s speaking with an accent, or you stopped to check twitter halfway through the session.

A printed page titled “Optical character recognition”, being scanned with a handheld OCR scanner with a red light.

A photo of a handheld OCR scanner, from Wikipedia.

Let’s look at another bit of early technology: optical character recognition, or OCR.

A sepia drawing of a machine with a scanning frame and a pair of headphones on a cord.

A scanned image of an optophone, taken from Wikipedia.

Early research into OCR was done to help the blind.

This is a machine called an optophone, a device for helping blind people read. You put printed text on the scanner, it identifies the characters and translates them into audio pulses (sort of like Morse code), then plays them through headphones. This was pioneering work for computer vision and text-to-speech synthesis.

These have become widely-used technologies: for making textual versions of scanned documents, Google Books, even those smartphone apps that let you translate signs in a foreign language.

A room with a row of shelves, with the spines of some large books visible on the nearest shelves.

A room full of grey shelves, with books visible on the nearest shelves. Image credit: Wellcome Collection.

And in fact, this is what we do at Wellcome Collection!

For those unfamiliar with Wellcome: we have an archive about human health and medicine. This is one of our “data centres”…

A close-up photo of some shelves, with the spines of large and old books closest to the camera.

Four shelves, each with a couple of large books on each shelf. Image credit: Wellcome Collection.

…using advanced container technology, like “shelves” and “books”.

Like many institutions, we’re scanning our archives to make them more easily available, and then we use OCR to make them searchable.

A screenshot of an ebook viewer, with a page titled “Die Radioaktivität”.

A page from the notebooks of Marie Curie, with a search highlighting instances of the word “radioaktiv”. Image credit: Wellcome Collection.

Here’s one example of our books: a notebook from Marie Curie, freely available to browse online. (Which is preferable to the original, which is slightly radioactive!) And using OCR, we can see that the word “radioaktiv” appears 730 times.

This so cool, but it wouldn’t exist without the pioneering work done into OCR to help blind people.

A purple door with a silver handle.

A silver door handle set against a purple door, with raindrops on the door’s surface. Image credit: MabelAmber on Pixabay, and recoloured by me.

One final example, less high technology and more small convenience: door handles.

Compared to door knobs, handles provide a larger area to grip or rest your hand against, so they’re easier to operate for people with a range of fine motor disabilities that prevent them from grasping small objects.

But they also make it easier if your hands are full, or you’re carrying things – you can lean on the door with an elbow without dropping something. It just makes life a bit easier.

Text slide, white text on purple. “Making something better for disabled people can make it better for everyone.”

So those are just a few examples of the Curb Cut Effect.

Text slide, white text on purple. “It isn’t just about disability!”

And that’s often where discussion of the Curb Cut Effect stops, which is a shame, because it isn’t just about disability!

Although disabled people are the most visibly excluded, there are plenty of excluded groups to whom this effect applies: women in tech, people of colour, victims of harassment and abuse…

Let’s look at one more example.

A photo with a blue sign “inclusive” and a trans and wheelchair icon, and a toilet visible in the room marked by the sign.

In the last few years, there’s been a big uptick in single stall, gender-neutral bathrooms. These are bathrooms that contain their own toilet and sink, maybe a shelf, all in a single private, enclosed space. And their installation has mostly been driven by a desire to accommodate trans and non-binary people who feel uncomfortable in traditionally gendered bathrooms.

And this is great for them… but it helps lots of other people too.

Text slide, white text on purple. “Making something better for people who are excluded or marginalised makes it better for everyone.”

So we can take the Curb Cut Effect, and make it stronger:

Making something better for people who are excluded or marginalised makes it better for everyone.

It’s not just about disability.

Text slide, white text on purple. “These are good stories.”

So why am I telling you all this? Two reasons.

First, these are good stories! They’re little love letters to inclusion, and a nice way to get people talking about inclusion or accessibility.

It’s been really fun to research this talk, and get to go to friends and say, “Hey, did you know this really cool story about the invention of the bendy straw?” (Talk to me in the break if you want to know this one.)

Text slide, white text on purple. “Things don’t happen because they’re ‘fair’ or ‘right’.”

There is a moe serious point.

If you come to a conference titled “Accessible Craft”, you probably already care about inclusion. I don’t need to convince you – you want to do it because it’s the right thing to do.

Unfortunately, things dopn’t happen because they’re “right” or “fair” (what a world that would be!).

Text slide, white text on purple. “We often have to justify the value of inclusion.”

We often have to justify the value of inclusion. (If you’re at this conference, you might be the person who does this advocacy, who has to make the business case.)

How often have you heard questions like “Do we have to do this? How many people will use it? Can we really afford it?”

And the Curb Cut Effect is a great tool to remember: it shows the value of inclusion. Spread the cost across a wide group, and changes to support inclusion suddenly seem much more attractive.

Text slide, white text on purple. “We can help one group without hurting another.”

It also serves to dispel a powerful myth.

There’s a common suspicion that helping one group must somehow, invisibly, hurt another group. As Spock said, “The needs of the many outweigh the needs of the few”. But in reality, we don’t have to choose!

The Curb Cut Effect shows us this is false: in fact, it shows us the opposite is true. Making the world a better place for a small number of people can make it better for a much wider number too. Yay!

Text slide, white text on purple. “Making something better for people who are excluded or marginalised makes it better for everyone.”

So let’s bring this all together.

I’ve shown you the Curb Cut Effect (repeat), and a handful of my favourite examples. Email. OCR. Door handles. And of course, curb cuts!

But there are many more – go away and try to think of some, maybe even ones in your own work. Keep it in mind, see how you can apply it to the things you build, and remember it the next time you’re asked to justify the value of inclusion.

Closing slide, with a reminder of the curb cut effect and a link to the slides.

(Exit to rapturous applause.)

Debugging a stuck Terraform plan

While working on some Terraform today, I had a problem that it would hang in the plan stage. Adding the following setting to my provider block exposed the issue:

provider "aws" {
  # other settings
  max_retries = 1
}

Rather than retrying a flaky AWS API, it crashed immediately and told me which API had an issue.

Notes from You Got This 2019

About a week ago, I was at You Got This 2019, a conference that bills itself as an event for “juniors on skills needed for a happy, healthy work life”. I’d seen the schedule on Twitter, thought it looked interesting, and decided to attend on a whim. All the topics had a strong focus on wellbeing, not technical content.

This post has my lightly edited notes from the conference and the sessions.

General vibes

Everybody I met was friendly and nice.

All the talks were good, and perhaps more notably, consistent. Usually when I go to a conference, there’s at least one or two talks that leave me feeling a bit “meh”, and that wasn’t the case here. Every session was interesting, well-presented, and left me with at least a few new ideas.

The emphasis in the introduction was that these are “topics we should be talking about more” – which is precisely why I decided to go.

I care a lot about running inclusive events, and have a long list of ideas for doing so. For a first-time conference, I was pleased by how many of them they hit:

And here’s some stuff that felt a little odd:

No conference gets inclusivity perfect, and the gold standard for me is still AlterConf London – but this was pretty good, overall. I’d happily attend again.

If you read the post and think sounds interesting, there’s a sequel planned for January 2020, with a mailing list on the website.

The talks

These are based on my handwritten notes, which are the bits I found most interesting, not a comprehensive summary of the talks. They’ll give you a flavour, but they’re not a substitute for watching the talks.


Impostor syndrome, perfectionism and anxiety, by Jo Francetti

She started with a story of “Mo”, a young developer who was feeling overwhelmed at work, and the sort of things she was feeling. Wondering if she knows enough, if she’s being judged by her co-workers, overworking and ruining her mental health. It was a good way to set up the topic – a sort of “Do you recognise this?” moment.

Editor note: I was wondering how she chose the name “Mo”, and can’t believe it took me to now to spot it.

There are some common issues, which she went through in turn:


Knowledge sharing and self-worth, by Sascha Wolf

Knowledge sharing can take many forms: blog posts, answering questions, speaking at conferences.

A big part of this talk was Sascha talking about his struggles with depression and exhaustion, and what felt a lot like burnout. (I don’t remember if he actually used the word “burnout”.)

He talked about how he built a new frame of mind, and that “we’re more than just a walking tech stack”*. In other words: we shouldn’t define ourselves solely by our technical work or our technical skills. As part of that, he talked about his experiences with therapy, which made me happy. (I think therapy is a really useful tool, I’ve been to therapy multiple times, and it’s good to hear it discussed on stage.)

A key attribute of a successful team is psychological safety – people feeling like they can take risks, and b supported by their team.

“Tech enables us, but it doesn’t define us.”

How to find knowledge you can share:

I really liked the framing of this talk: the introduction was “Hi, I’m Sascha”, and a list of technical skills and background. Then at the end of the talk, another slide “Hi, I’m still Sascha”, which had a similar list that covered much more of his life.

There is more to you than your profession.


Where did all my money go? by Paula Muldoon

Notable for starting and ending with a violin recital. (Although not the first time I’ve heard music on stage at a tech conference!)

Some general advice:

If you want to get your spending under control, there are two key steps:

  1. Understand your beliefs about money.

    For example: “I’ll never have enough money”, “I’ll always have enough money” or “money is evil”. You might believe all, some or none of these – whatever, knowing your beliefs is useful.

    Some useful resources:

    Talk about money! It’s often a taboo topic.

  2. Take some practical steps.

    You want to understand your current money habits – but exercise self-compassion. Don’t beat yourself up over everything you think you’re doing wrong.

    Suggestions:

    • Have a spreadsheet. Calculate all the numbers: exactly how much you’re earning each month, and your regular bills (rent, utilities, tax, etc.). What’s left is your budget.
    • Automate everything. If you have to remember to pay bills, there’s a risk you’ll forget and take a hit to your credit rating.
    • Max out your workplace pension. (Note: I scribbled down “needs research” next to this, because I’m not sure how I’d do this, what it means, and suspect the best thing to do with pension stuff is probably more complicated than simple.)
    • Use bank accounts that help you track your spending, like Monzo or Starling.
    • Get free credit reports from Experian and Equifax.
    • If you have a credit card, keep your credit limit at an affordable level. (I’ve heard conflicting advice on this one. My credit limit is several times my monthly salary, and I rarely max it out – but by paying it off regularly it’s slowly inched up, and might be useful in an emergency. Hasn’t bitten me so far.)
    • Advanced moneying: have side gigs, negotiate a pay rise.

Paula also wrote a blog post with a summary and some resources.


Junior.next(), by Tara Ojo

Slides: https://speakerdeck.com/tyeraqii/junior-dot-next-yougotthisconf

I love the phrase “career crushes”.

Everyone has to start as a junior, so how do you progress?

If you’re a junior looking to progress:

There’s a sweet spot of “stretch” between comfort (easy, minimal learning) and panic (bad, a source of stress) – that’s where you want to be.

You want a mixture of base knowledge skills and depth in specific areas.

If you’re a supporter trying to help a junior:


Self-care beyond the hashtags, by Taylor Elyse-Morrison

Looking at the #selfcare hashtag. What is it?

Taylor is a “ritual builder” – helping people build rituals.

In order to care for yourself, you need to listen to your body. You want stillness, observation, reflection. Be consciously aware of your body.

When you reflect:

  1. Pick an interval
  2. Ask “When did I experience tension?”
  3. Ask “When did I feel supported?”

For building rituals, you want an emergency toolkit for when you’re in a bad moment. This might include:

Self-care means listening to your body and responding in the most loving way possible.

Our self-care needs will change with time, so keep listening and responding.


Morality and ethics, by Sam Warner

What’s the definition of ethical technology?

Notions of good/bad are fuzzy; one possible alternative is effective altruism.

Tech is a relatively young and immature industry, but “we shouldn’t be ashamed of being immature if we mature well”.

A Stack Overflow survey (admittedly not representative, with “some demographic challenges”) – a majority of developers think they should think about the ethics of the software they build, but only a minority think the ultimate responsibility lies with them. We should be empowering developers to feel like they have that responsibility (and power).

Three principles of ethical software:

  1. People come first
  2. Create accessible and useful services for everyone
  3. More diverse teams give better visibility ethical factors

We all make mistakes; what matters is how we get better. (The speaker used an example of Facebook – Mark Zuckerberg didn’t realise he’d create the monster he did – but some of the early iterations of Facebook were pretty sketchy, so I’m reluctant to give Zuckerberg a pass.)

A useful resource: the Ethical OS toolkit. A checklist of useful things to think about when building ethical software. (I wondered: does it cover online harassment and abuse?)

Things to think about:

  1. Unethical software isn’t more profitable
  2. Don’t hide behind contracts
  3. You can make a difference (e.g. Project Maven, in which Google’s engineers successfully protested a contract with the US DoD)


Understanding and building independence, by Violet Peña

Slides and notes: https://violet.is/files/you-got-this.pdf

We are problem solvers.

Independence is the ability to understand and construct solutions. It’s not about working alone.

Here’s a framework for solving problems:

  1. Have a plan.

    Check you understand the problem: the data, the conditions, the unknowns.

    Break your plan down into steps. How long will each step take? How can you track progress?

    Be honest with yourself and your team. Are you progressing?

    Be aware of how you’re performing. It’s okay to say “I need more time”, “I need more help”, “I need to do something differently”. (And she had us all say them out loud to get used to saying the words!)

  2. Use your tools.

    Use the tools you have, but don’t be afraid to change modalities; to try new tools or technologies if needed.

    Find a related problem (similar data, conditions, unknowns), and try to solve your problem in a similar way.

  3. Ask for help.

    It’s a great thing to do: it builds your knowledge, shows your respect for your colleagues, shows you don’t have a massive ego.

    You want to “build a Voltron” – a group opf people who can help with particular topics.

    You should ask for help when you’re stuck, can’t get past a particular problem, or need a rethink.

    The more certain you are somebody has seen this before, the earlier you can ask for help.

    How do you ask for help?

    • Give context
    • Ask a specific question
    • Tell them what you’ve already tried
    • Thank them for their time/help


The power of junior developers, by Sam Morgan

Two topics:

  1. Be suspicious of the word “junior”
  2. Advice for people who want to support junior developers

If you say “junior”, what are you suggesting? You’re saying that:

You only have permission to learn if you’re:

This is a false dichotomy.

Here’s how to build a structure for learning:

Provide context for problems:

Set goals:

Provide resources and help:

There are several types of question somebody might ask/opportunities for learning, and they need to be treated differently:

Have a structure for feedback:

  1. Require intentionality. Ask: What do you want feedback on?

    This allows you to get feedback early, rather than waiting for perfect or completed specimens, and early feedback is better.

  2. Keep it tight. Don’t overwhelm somebody with information, or provide comments on irrelevant areas.

  3. Demand iteration.

    When they’ve addressed your feedback, they should come back for more. The model of “get feedback once, then finish” is an odd one, and not so helpful (exams are weird).

A question for the junior devs in the room: how are you going to transform the businesses you work in?

How much sunlight affects my mood levels

Over the last few months, it’s become really obvious how much sunlight affects my mood levels.

If I commute in darkness, I’m miserable.
If I commute in daylight, I’m a lot happier.

This might sound a bit like seasonal affective disorder (SAD), and I do wonder if it’s something like that. SAD is a type of depression that follows the seasons, and is usually worse in winter.

I have friends with fairly severe SAD, who feel tired all the time during the winter months, and it’s never been as bad as that for me, so until now I’ve assumed I didn’t have any form of SAD.

Maybe I do, maybe I don’t – but I can look to SAD treatment for tips and ideas to make my life easier. In particular, getting as much natural light as possible, and a sun lamp or six. (I have a friend with a small solar system in their front room, and the effect on their mood is ~noticeable~.)

If any of this sounds familiar to you, you might want to check out medical advice for SAD, and try some of the coping mechanisms yourself.

In the meantime, spring is fast approaching – just check out this blue sky from today’s commute:

(This post originally appeared on Twitter.)