Monki Gras 2019: The Curb Cut Effect

Earlier today I did a talk at Monki Gras 2019. The theme of the conference is “Accessible Craft: Creating great experiences for everyone”, so I did a talk about inclusive design – and in particular, something called the Curb Cut Effect.

I originally pitched a talk based on Assume Worst Intent, because if you’re thinking about inclusion you might think about ways to avoid exclusion (specifically, exclusion of harassment and abuse victims). That talk was definitely too narrow for this conference, but James isolated a key idea – making a service better for vulnerable uses makes it better for everyone – and I wrote an entirely new talk around it.

The talk went really well – everybody was very nice afterwards, and I’m enjoying the rest of talks too. (Notes on those will be a separate post.) It was a lot of fun to write and present.

The talk was recorded, and you can watch it on YouTube:

You can read the slides and my notes on this page, or download the slides as a PDF.

Slides and notes

Title slide.

Hi, I’m Alex. I’m going to talk about the Curb Cut Effect, what it is, and how we might use it.

The Wellcome Collection building, lit up in purple.

The Wellcome Collection building lit up in purple to mark International Day of Persons with Disabilities in December 2018. Image credit: Wellcome Collection.

I’m a software developer at Wellcome Collection, a free museum and library on Euston Road.

At Wellcome, we’ve been thinking a lot about architecture recently. Our current exhibition, Living with Buildings, is all about the effect of architecture on human health, so I wanted to start today by telling you all the story of architecture – specifically, the story of a common architectural feature that we all walk past every day without a second glance.

A dropped kerb against a black background, with the caption “dropped kerb aka curb cut”.

A dropped kerb around the back of UCL, near the Wellcome offices. Image credit: me!

I’m talking about those areas where the kerb dips to form a ramp – giving a level, step-free path from the road to the pavement. In the UK, these are usually accompanied by a textured yellow surface (pictured).

They go by several names – in the UK we call them dropped kerbs, in the US they’re called curb cuts (one of the few American spellings that adds the letter U), and they have other names around the world. The American spelling is mostly common, and that’s what I’ll use for the rest of the talk, but you can mentally substitute “curb cut” for your preferred term.

A map, with circled areas “Kalamazoo” and “Battle Creek”.

A map showing part of Michigan, highlighting Kalamazoo and Battle Creek. Image credit: original map from the US Geological Survey.

One of the earliest examples of curb cuts was in Kalamazoo, MI. (Great name!)

There was a man called Jack Fisher, born in Kalamazoo in 1918. Like many young men of his age, he enlisted in the Army during the Second World War.

He was involved in a jeep accident in 1943. While he was recovering in hospital, he read the records of other patients with similar injuries to keep himself occupied. (Because 1940s privacy laws.)

He returned to Kalamazoo with steel braces from his hip to his neck, and a heavy limp. After the war, he graduated from Harvard law school (very impressive), but none of the established firms would take him – he was a disabled veteran, who might trip and injure himself, and need compensation. So they chose not to hire him. (Because 1940s labour rights.)

So he worked as an attorney in his own practice. He became well-known among other disabled veterans in and near Kalamazoo – of which there were many, because there was a nearby hospital at Battle Creek which specialised in amputee treatment and rehabilitation.

A black and white photo of a wheelchair standing at the edge of a raised curb.

A wheelchair standing at an raised, inaccessible curb. Image from an article by the Smithsonian.

Working closely with them, he became aware of the problems and challenges they faced.

One of those problems: Kalamazoo had tall curbs (up to 6 inches). This was a problem – people would trip, injure themselves, damage prosthetic limbs, and for wheelchair users they’re a total nightmare. Tall kerbs are inaccessible, and prevent people getting around, socialising, working and so on.

A black and white photo of a street, with somebody walking up a ramp with hand rails cut into the kerb.

A ramp with hand rails on the streets of Kalamazoo. Image from an article in Encore Magazine.

So in 1945, Jack Fisher took it upon himself to fix this, and petitioned the city commission for curb cuts and hand-rails. Ditching the step would make it easier for people to get around. The city authorised their construction and ran a small pilot program, they were very successful, and they grew in number.

Kalamazoo is one of the earliest examples of dropped kerbs, but the same story plays out in lots of other places – curb cuts were installed in lots of places to make the streets more accessible for disabled people and wheelchair users.

Text slide, white text on purple. “Curb cuts make the roads more accessible for wheelchair users and disabled people, but they aren't the only people who benefit!”

So curb cuts make the roads more accessible for wheelchair users and disabled people. Yay!

But they’re not the only people to benefit, so do:

And probably others.

Wcould call this a “force multipler” or a “happy accident”, but really this is the original example of the “Curb Cut Effect”.

Text slide, white text on purple. “Making something better for disabled people can make it better for everyone.”

The Curb Cut Effect comes in many forms, but the way I think of it is:

Making something better for disabled people can make it better for everyone.

Text slide, white text on purple. “What it means for us: Designs that include disabled people are better designs for everybody.”

What this means for us, as people who build things:

Designs that think about disabled people are better designs for everyone

And we reflect this is the words we use: it’s why we don’t talk about handicapped design or barrier-free design. We talk universal design. We talk about good design.

Text slide, white text on purple. “Making something better for disabled people can make it better for everyone.”

(Repeat the Curb Cut Effect.)

Let’s look at a few examples.

A black-and-white artwork of a woman sitting at a table. There’s a large machine with a keyboard on the table.

One of the earliest examplse predates Kalamazoo by more than a century.

This is the Countess Carolina Fantoni da Fivizzano. She lived in the early nineteenth century, and was the friend and lover of the Italian inventor Pellegrino Turri.

They’d write each other letters, but she lost her sight as an adult, and was unable to write herself. That meant the only way to send letters was to dictate to somebody else – but she wanted to send letters in private. Together they built a machine that let her write letters by pressing a key for each letter – allowing her to “type” letters.

Writing with type… a type-writer, of sorts.

This was one of the earliest iterations of the typewriter. It made writing accessible to the blind, and the derivatives became the modern-day keyboard. A machine created to help one blind woman write love letters was the basis for a fundamental input device for modern computing.

A photo of a laptop with an email client open.

Somebody using an email client on a laptop. Image credit: rawpixel.com on Pexels.

Speaking of love letters… let’s talk about email!

(You don’t use email to write love letters? Sounds fake.)

Email has become a ubiquitous part of modern comms, but where did it come from? Why was it invented?

A photo of a man in a suit (Vint Cerf), with a quote overlaid. “Because I’m hearing-impaired, emails are a tremendously valuable tool because of the precision that you get. I can read what’s typed as opposed to straining to hear what’s being said.”

A picture of Vinton Cerf, taken from his Royal Society photo and overlaid with a quote from a CNET article.

This is Vinton Cerf. He’s often called the “Father of the Internet”, did a lot of work on the early Internet (then-ARPANET) protocols, is a strong advocate for accessibility, and led the work on the first commercial email program.

Why?

Because he’s hard-of-hearing, and his wife Sigrid is deaf. His work on email started, in part, as a way for them to stay connected when they weren’t in the same room. At the time, the alternative was the telephone, which was a bit of a non-starter.

Here’s a quote he gave to CNET that sums it up nicely:

Because I’m hearing-impaired, emails are a tremendously valuable tool because of the precision that you get.

I can read what’s typed as opposed to straining to hear what’s being said.

Email started as a key technology for people who are deaf or have hearing loss – or who are just separated by time and space.

A screenshot from Sesame Street, with four characters on screen and a caption “Now everyone can read it”.

Sticking with hearing loss, let’s talk about captions.

It was originally intended to help deaf people understand movies with dialogue or sound effects. Closed captions were first broadcast in the 1970s. They were expensive, needed specialist equipment, and most content wasn’t captioned – today things are generally better. And indeed, they help people who are deaf or hard-of-hearing, but lots of other people besides:

And even if you can hear the sound fine, you can still benefit from captions:

A person with their hands on a special “stenographic” keyboard, looking up to hear someone talking.

A photo of one of the captioners at PyCon UK 2017, by Mark Hawkins.

And it helps with conferences too!

I’m being captioned right now – literally as I speak! [Monki Gras has live captioning.] This is one of the captioners at PyCon UK, and our experience is that lots of people find it useful during talks, not just the deaf or hard-of-hearing – maybe a word you couldn’t hear, somebody’s speaking with an accent, or you stopped to check twitter halfway through the session.

A printed page titled “Optical character recognition”, being scanned with a handheld OCR scanner with a red light.

A photo of a handheld OCR scanner, from Wikipedia.

Let’s look at another bit of early technology: optical character recognition, or OCR.

A sepia drawing of a machine with a scanning frame and a pair of headphones on a cord.

A scanned image of an optophone, taken from Wikipedia.

Early research into OCR was done to help the blind.

This is a machine called an optophone, a device for helping blind people read. You put printed text on the scanner, it identifies the characters and translates them into audio pulses (sort of like Morse code), then plays them through headphones. This was pioneering work for computer vision and text-to-speech synthesis.

These have become widely-used technologies: for making textual versions of scanned documents, Google Books, even those smartphone apps that let you translate signs in a foreign language.

A room with a row of shelves, with the spines of some large books visible on the nearest shelves.

A room full of grey shelves, with books visible on the nearest shelves. Image credit: Wellcome Collection.

And in fact, this is what we do at Wellcome Collection!

For those unfamiliar with Wellcome: we have an archive about human health and medicine. This is one of our “data centres”…

A close-up photo of some shelves, with the spines of large and old books closest to the camera.

Four shelves, each with a couple of large books on each shelf. Image credit: Wellcome Collection.

…using advanced container technology, like “shelves” and “books”.

Like many institutions, we’re scanning our archives to make them more easily available, and then we use OCR to make them searchable.

A screenshot of an ebook viewer, with a page titled “Die Radioaktivität”.

A page from the notebooks of Marie Curie, with a search highlighting instances of the word “radioaktiv”. Image credit: Wellcome Collection.

Here’s one example of our books: a notebook from Marie Curie, freely available to browse online. (Which is preferable to the original, which is slightly radioactive!) And using OCR, we can see that the word “radioaktiv” appears 730 times.

This so cool, but it wouldn’t exist without the pioneering work done into OCR to help blind people.

A purple door with a silver handle.

A silver door handle set against a purple door, with raindrops on the door’s surface. Image credit: MabelAmber on Pixabay, and recoloured by me.

One final example, less high technology and more small convenience: door handles.

Compared to door knobs, handles provide a larger area to grip or rest your hand against, so they’re easier to operate for people with a range of fine motor disabilities that prevent them from grasping small objects.

But they also make it easier if your hands are full, or you’re carrying things – you can lean on the door with an elbow without dropping something. It just makes life a bit easier.

Text slide, white text on purple. “Making something better for disabled people can make it better for everyone.”

So those are just a few examples of the Curb Cut Effect.

Text slide, white text on purple. “It isn’t just about disability!”

And that’s often where discussion of the Curb Cut Effect stops, which is a shame, because it isn’t just about disability!

Although disabled people are the most visibly excluded, there are plenty of excluded groups to whom this effect applies: women in tech, people of colour, victims of harassment and abuse…

Let’s look at one more example.

A photo with a blue sign “inclusive” and a trans and wheelchair icon, and a toilet visible in the room marked by the sign.

In the last few years, there’s been a big uptick in single stall, gender-neutral bathrooms. These are bathrooms that contain their own toilet and sink, maybe a shelf, all in a single private, enclosed space. And their installation has mostly been driven by a desire to accommodate trans and non-binary people who feel uncomfortable in traditionally gendered bathrooms.

And this is great for them… but it helps lots of other people too.

Text slide, white text on purple. “Making something better for people who are excluded or marginalised makes it better for everyone.”

So we can take the Curb Cut Effect, and make it stronger:

Making something better for people who are excluded or marginalised makes it better for everyone.

It’s not just about disability.

Text slide, white text on purple. “These are good stories.”

So why am I telling you all this? Two reasons.

First, these are good stories! They’re little love letters to inclusion, and a nice way to get people talking about inclusion or accessibility.

It’s been really fun to research this talk, and get to go to friends and say, “Hey, did you know this really cool story about the invention of the bendy straw?” (Talk to me in the break if you want to know this one.)

Text slide, white text on purple. “Things don’t happen because they’re ‘fair’ or ‘right’.”

There is a moe serious point.

If you come to a conference titled “Accessible Craft”, you probably already care about inclusion. I don’t need to convince you – you want to do it because it’s the right thing to do.

Unfortunately, things dopn’t happen because they’re “right” or “fair” (what a world that would be!).

Text slide, white text on purple. “We often have to justify the value of inclusion.”

We often have to justify the value of inclusion. (If you’re at this conference, you might be the person who does this advocacy, who has to make the business case.)

How often have you heard questions like “Do we have to do this? How many people will use it? Can we really afford it?”

And the Curb Cut Effect is a great tool to remember: it shows the value of inclusion. Spread the cost across a wide group, and changes to support inclusion suddenly seem much more attractive.

Text slide, white text on purple. “We can help one group without hurting another.”

It also serves to dispel a powerful myth.

There’s a common suspicion that helping one group must somehow, invisibly, hurt another group. As Spock said, “The needs of the many outweigh the needs of the few”. But in reality, we don’t have to choose!

The Curb Cut Effect shows us this is false: in fact, it shows us the opposite is true. Making the world a better place for a small number of people can make it better for a much wider number too. Yay!

Text slide, white text on purple. “Making something better for people who are excluded or marginalised makes it better for everyone.”

So let’s bring this all together.

I’ve shown you the Curb Cut Effect (repeat), and a handful of my favourite examples. Email. OCR. Door handles. And of course, curb cuts!

But there are many more – go away and try to think of some, maybe even ones in your own work. Keep it in mind, see how you can apply it to the things you build, and remember it the next time you’re asked to justify the value of inclusion.

Closing slide, with a reminder of the curb cut effect and a link to the slides.

(Exit to rapturous applause.)

Debugging a stuck Terraform plan

While working on some Terraform today, I had a problem that it would hang in the plan stage. Adding the following setting to my provider block exposed the issue:

provider "aws" {
  # other settings
  max_retries = 1
}

Rather than retrying a flaky AWS API, it crashed immediately and told me which API had an issue.

Notes from You Got This 2019

About a week ago, I was at You Got This 2019, a conference that bills itself as an event for “juniors on skills needed for a happy, healthy work life”. I’d seen the schedule on Twitter, thought it looked interesting, and decided to attend on a whim. All the topics had a strong focus on wellbeing, not technical content.

This post has my lightly edited notes from the conference and the sessions.

General vibes

Everybody I met was friendly and nice.

All the talks were good, and perhaps more notably, consistent. Usually when I go to a conference, there’s at least one or two talks that leave me feeling a bit “meh”, and that wasn’t the case here. Every session was interesting, well-presented, and left me with at least a few new ideas.

The emphasis in the introduction was that these are “topics we should be talking about more” – which is precisely why I decided to go.

I care a lot about running inclusive events, and have a long list of ideas for doing so. For a first-time conference, I was pleased by how many of them they hit:

And here’s some stuff that felt a little odd:

No conference gets inclusivity perfect, and the gold standard for me is still AlterConf London – but this was pretty good, overall. I’d happily attend again.

If you read the post and think sounds interesting, there’s a sequel planned for January 2020, with a mailing list on the website.

The talks

These are based on my handwritten notes, which are the bits I found most interesting, not a comprehensive summary of the talks. They’ll give you a flavour, but they’re not a substitute for watching the talks.


Impostor syndrome, perfectionism and anxiety, by Jo Francetti

She started with a story of “Mo”, a young developer who was feeling overwhelmed at work, and the sort of things she was feeling. Wondering if she knows enough, if she’s being judged by her co-workers, overworking and ruining her mental health. It was a good way to set up the topic – a sort of “Do you recognise this?” moment.

Editor note: I was wondering how she chose the name “Mo”, and can’t believe it took me to now to spot it.

There are some common issues, which she went through in turn:


Knowledge sharing and self-worth, by Sascha Wolf

Knowledge sharing can take many forms: blog posts, answering questions, speaking at conferences.

A big part of this talk was Sascha talking about his struggles with depression and exhaustion, and what felt a lot like burnout. (I don’t remember if he actually used the word “burnout”.)

He talked about how he built a new frame of mind, and that “we’re more than just a walking tech stack”*. In other words: we shouldn’t define ourselves solely by our technical work or our technical skills. As part of that, he talked about his experiences with therapy, which made me happy. (I think therapy is a really useful tool, I’ve been to therapy multiple times, and it’s good to hear it discussed on stage.)

A key attribute of a successful team is psychological safety – people feeling like they can take risks, and b supported by their team.

“Tech enables us, but it doesn’t define us.”

How to find knowledge you can share:

I really liked the framing of this talk: the introduction was “Hi, I’m Sascha”, and a list of technical skills and background. Then at the end of the talk, another slide “Hi, I’m still Sascha”, which had a similar list that covered much more of his life.

There is more to you than your profession.


Where did all my money go? by Paula Muldoon

Notable for starting and ending with a violin recital. (Although not the first time I’ve heard music on stage at a tech conference!)

Some general advice:

If you want to get your spending under control, there are two key steps:

  1. Understand your beliefs about money.

    For example: “I’ll never have enough money”, “I’ll always have enough money” or “money is evil”. You might believe all, some or none of these – whatever, knowing your beliefs is useful.

    Some useful resources:

    Talk about money! It’s often a taboo topic.

  2. Take some practical steps.

    You want to understand your current money habits – but exercise self-compassion. Don’t beat yourself up over everything you think you’re doing wrong.

    Suggestions:

    • Have a spreadsheet. Calculate all the numbers: exactly how much you’re earning each month, and your regular bills (rent, utilities, tax, etc.). What’s left is your budget.
    • Automate everything. If you have to remember to pay bills, there’s a risk you’ll forget and take a hit to your credit rating.
    • Max out your workplace pension. (Note: I scribbled down “needs research” next to this, because I’m not sure how I’d do this, what it means, and suspect the best thing to do with pension stuff is probably more complicated than simple.)
    • Use bank accounts that help you track your spending, like Monzo or Starling.
    • Get free credit reports from Experian and Equifax.
    • If you have a credit card, keep your credit limit at an affordable level. (I’ve heard conflicting advice on this one. My credit limit is several times my monthly salary, and I rarely max it out – but by paying it off regularly it’s slowly inched up, and might be useful in an emergency. Hasn’t bitten me so far.)
    • Advanced moneying: have side gigs, negotiate a pay rise.

Paula also wrote a blog post with a summary and some resources.


Junior.next(), by Tara Ojo

Slides: https://speakerdeck.com/tyeraqii/junior-dot-next-yougotthisconf

I love the phrase “career crushes”.

Everyone has to start as a junior, so how do you progress?

If you’re a junior looking to progress:

There’s a sweet spot of “stretch” between comfort (easy, minimal learning) and panic (bad, a source of stress) – that’s where you want to be.

You want a mixture of base knowledge skills and depth in specific areas.

If you’re a supporter trying to help a junior:


Self-care beyond the hashtags, by Taylor Elyse-Morrison

Looking at the #selfcare hashtag. What is it?

Taylor is a “ritual builder” – helping people build rituals.

In order to care for yourself, you need to listen to your body. You want stillness, observation, reflection. Be consciously aware of your body.

When you reflect:

  1. Pick an interval
  2. Ask “When did I experience tension?”
  3. Ask “When did I feel supported?”

For building rituals, you want an emergency toolkit for when you’re in a bad moment. This might include:

Self-care means listening to your body and responding in the most loving way possible.

Our self-care needs will change with time, so keep listening and responding.


Morality and ethics, by Sam Warner

What’s the definition of ethical technology?

Notions of good/bad are fuzzy; one possible alternative is effective altruism.

Tech is a relatively young and immature industry, but “we shouldn’t be ashamed of being immature if we mature well”.

A Stack Overflow survey (admittedly not representative, with “some demographic challenges”) – a majority of developers think they should think about the ethics of the software they build, but only a minority think the ultimate responsibility lies with them. We should be empowering developers to feel like they have that responsibility (and power).

Three principles of ethical software:

  1. People come first
  2. Create accessible and useful services for everyone
  3. More diverse teams give better visibility ethical factors

We all make mistakes; what matters is how we get better. (The speaker used an example of Facebook – Mark Zuckerberg didn’t realise he’d create the monster he did – but some of the early iterations of Facebook were pretty sketchy, so I’m reluctant to give Zuckerberg a pass.)

A useful resource: the Ethical OS toolkit. A checklist of useful things to think about when building ethical software. (I wondered: does it cover online harassment and abuse?)

Things to think about:

  1. Unethical software isn’t more profitable
  2. Don’t hide behind contracts
  3. You can make a difference (e.g. Project Maven, in which Google’s engineers successfully protested a contract with the US DoD)


Understanding and building independence, by Violet Peña

Slides and notes: https://violet.is/files/you-got-this.pdf

We are problem solvers.

Independence is the ability to understand and construct solutions. It’s not about working alone.

Here’s a framework for solving problems:

  1. Have a plan.

    Check you understand the problem: the data, the conditions, the unknowns.

    Break your plan down into steps. How long will each step take? How can you track progress?

    Be honest with yourself and your team. Are you progressing?

    Be aware of how you’re performing. It’s okay to say “I need more time”, “I need more help”, “I need to do something differently”. (And she had us all say them out loud to get used to saying the words!)

  2. Use your tools.

    Use the tools you have, but don’t be afraid to change modalities; to try new tools or technologies if needed.

    Find a related problem (similar data, conditions, unknowns), and try to solve your problem in a similar way.

  3. Ask for help.

    It’s a great thing to do: it builds your knowledge, shows your respect for your colleagues, shows you don’t have a massive ego.

    You want to “build a Voltron” – a group opf people who can help with particular topics.

    You should ask for help when you’re stuck, can’t get past a particular problem, or need a rethink.

    The more certain you are somebody has seen this before, the earlier you can ask for help.

    How do you ask for help?

    • Give context
    • Ask a specific question
    • Tell them what you’ve already tried
    • Thank them for their time/help


The power of junior developers, by Sam Morgan

Two topics:

  1. Be suspicious of the word “junior”
  2. Advice for people who want to support junior developers

If you say “junior”, what are you suggesting? You’re saying that:

You only have permission to learn if you’re:

This is a false dichotomy.

Here’s how to build a structure for learning:

Provide context for problems:

Set goals:

Provide resources and help:

There are several types of question somebody might ask/opportunities for learning, and they need to be treated differently:

Have a structure for feedback:

  1. Require intentionality. Ask: What do you want feedback on?

    This allows you to get feedback early, rather than waiting for perfect or completed specimens, and early feedback is better.

  2. Keep it tight. Don’t overwhelm somebody with information, or provide comments on irrelevant areas.

  3. Demand iteration.

    When they’ve addressed your feedback, they should come back for more. The model of “get feedback once, then finish” is an odd one, and not so helpful (exams are weird).

A question for the junior devs in the room: how are you going to transform the businesses you work in?

How much sunlight affects my mood levels

Over the last few months, it’s become really obvious how much sunlight affects my mood levels.

If I commute in darkness, I’m miserable.
If I commute in daylight, I’m a lot happier.

This might sound a bit like seasonal affective disorder (SAD), and I do wonder if it’s something like that. SAD is a type of depression that follows the seasons, and is usually worse in winter.

I have friends with fairly severe SAD, who feel tired all the time during the winter months, and it’s never been as bad as that for me, so until now I’ve assumed I didn’t have any form of SAD.

Maybe I do, maybe I don’t – but I can look to SAD treatment for tips and ideas to make my life easier. In particular, getting as much natural light as possible, and a sun lamp or six. (I have a friend with a small solar system in their front room, and the effect on their mood is ~noticeable~.)

If any of this sounds familiar to you, you might want to check out medical advice for SAD, and try some of the coping mechanisms yourself.

In the meantime, spring is fast approaching – just check out this blue sky from today’s commute:

(This post originally appeared on Twitter.)

Thoughts on privacy and Oyster cards

This ship has long since sailed, but yesterday I found another reason to be a bit sad that TfL has phased out cash as a payment mechanism.

I logged into my Oyster account for the first time today, and I was struck by how much location data they have. You can reconstruct a large chunk of my diary from that data – for any Tube station or bus stop outside the core, there’s only really one person I would be visiting, and now you know when and for how long I was with them.

Intellectually, I knew this was happening – if I’m using the same card each time, I’d almost be more surprised if they weren’t recording the fine-grained data. But it’s sobering to see it all laid out, in complete and unerring detail.

I’ve never had to worry about stalkers or people coming after me, but I’d be pretty unnerved if I was.

Cash lets you travel anonymously, but you can’t use it on buses, and paper tickets on the Tube are more expensive. So if you’re somebody who doesn’t want your location tracked – or doesn’t have a card you can use – public transport in London is a bit harder for you to use.

A couple of examples sprang to mind:

There are probably others, and while your Tube data along might not give away your secrets, it could prompt awkward questions.

(This post originally appeared on Twitter.)

A delightful store for speciality spreads: Paul Rothe & Sons

Late last year, I wanted to find a jar of loganberry jelly as a Christmas present for my granddad. This turned out to be surprisingly difficult to find – it’s a rare flavour that isn’t in any of the supermarkets, and even Fortnum & Mason and Harrods both came up short. I did find it on Amazon, but something about buying jam on Amazon just feels wrong.

Googling turned up an article about a place called Paul Rothe & Sons, a small family-run deli in Marylebone. I walked over on my lunch break, and found a cosy little shop – tables for eating sandwiches and drinks, and walls stacked with a huge variety of jams and spreads. I had a little trouble finding what I wanted, but one of the owners (Stephen Rothe) helped me find exactly what I wanted.

Sadly I couldn’t stay for lunch (and it was so busy, I’m not sure I’d have found a seat!), but there was a very relaxed air among the tables. I did grab a sausage sandwich to go, and it was warm and tasty. Perfect for a cold December day!

If you’re ever in London and need an unusual jam or a delicious sandwich, give it a look.

Two shelves, each stacked with two rows of jam in glass jars.
A very small sample of the jams on offer. The gaps in the shelves are from the jars I’d just picked up – a pair of loganberries, and one of gooseberry.

Notes on reading a UTF-8 encoded CSV in Python

Here’s a problem I solved today: I have a CSV file to parse which contained UTF-8 strings, and I want to parse it using Python. I want to do it in a way that works in both Python 2.7 and Python 3.

This proved to be non-trivial, so this blog post is a quick brain dump of what I did, in the hope it’s useful to somebody else and/or my future self.

Problem statement

Consider the following minimal example of a CSV file:

1,alïce
2,bøb
3,cárol

We want to parse this into a list of lists:

[
    ["1", "alïce"],
    ["2", "bøb"],
    ["3", "cárol"],
]

Experiments

The following code can read the file in Python 2.7; here we treat the file as a bag of bytes and only decode after the CSV parsing is done:

import csv

with open("example.csv", "rb") as csvfile:
    csvreader = csv.reader(csvfile, delimiter=",")

    for row in csvreader:
        row = [entry.decode("utf8") for entry in row]
        print(": ".join(row))

But if you run that code in Python 3, you get the following error:

Traceback (most recent call last):
  File "reader2.py", line 6, in <module>
    for row in csvreader:
_csv.Error: iterator should return strings, not bytes (did you open the file in text mode?)

The following code can read the file in Python 3:

import csv

with open("example.csv", encoding="utf8") as csvfile:
    csvreader = csv.reader(csvfile, delimiter=",")

    for row in csvreader:
        print(": ".join(row))

But the encoding argument to open() is only in Python 3 or later, so you can’t use this in Python 2.

In theory this is backported as codecs.open(), but I get a different error if I use codecs.open() in this file with Python 2.7:

Traceback (most recent call last):
  File "reader3.py", line 7, in <module>
    for row in csvreader:
UnicodeEncodeError: 'ascii' codec can't encode character u'\xef' in position 4: ordinal not in range(128)

This feels like it should be possible using only the standard library, but it was becoming sufficiently complicated that I didn’t want to bother.

I considered defining these as two separate functions, and running:

import sys

if sys.version_info[0] == 2:
    read_csv_python2()
else:
    read_csv_python3()

but that felt a little icky, and would have been annoying for code coverage. Having two separate functions also introduces a source of bugs – I might remember to update one function, but not the other.

I found csv23 on PyPI, whose description sounded similar to what I wanted. The following snippet does what I want:

import csv23

with csv23.open_reader("example.csv") as csvreader:
    for row in csvreader:
        print(": ".join(row))

This reads the CSV file as UTF-8 in both Python 2 and 3. Having a third-party library is mildly annoying, but it’s easier than trying to write, test and maintain this functionality myself.

tl;dr

Python 2 only:

import csv

with open("example.csv", "rb") as csvfile:
    csvreader = csv.reader(csvfile, delimiter=",")

    for row in csvreader:
        row = [entry.decode("utf8") for entry in row]
        print(": ".join(row))

Python 3 only:

import csv

with open("example.csv", encoding="utf8") as csvfile:
    csvreader = csv.reader(csvfile, delimiter=",")

    for row in csvreader:
        print(": ".join(row))

Both Python 2 and 3:

import csv23

with csv23.open_reader("example.csv") as csvreader:
    for row in csvreader:
        print(": ".join(row))

Iterating in fixed-size chunks in Python

Here’s a fairly common problem I have: I have an iterable, and I want to go through it in “chunks”. Rather than looking at every item of the sequence one-by-one, I want to process multiple elements at once.

For example, when I’m using the bulk APIs in Elasticsearch, I can index many document with a single API call, which is more efficient than making a new API call for every document.

Here’s the sort of output I want:

for c in chunked_iterable(range(14), size=4):
    print(c)

# (0, 1, 2, 3)
# (4, 5, 6, 7)
# (8, 9, 10, 11)
# (12, 13)

I have two requirements which are often missed in Stack Overflow answers or other snippets I’ve found:

So to save me having to find it again, this is what I usually use:

import itertools


def chunked_iterable(iterable, size):
    it = iter(iterable)
    while True:
        chunk = tuple(itertools.islice(it, size))
        if not chunk:
            break
        yield chunk

Most of the heavy lifting is done by itertools.islice(); I call that repeatedly until it returns an empty sequence. The itertools module has lots of useful functions for this sort of thing.

The it = iter(iterable) line may be non-obvious – this ensures that the value it is using the same iterator throughout. If you pass certain fixed iterables to islice(), it creates a new iterator each time – and then you only ever get the first handful of elements.

For example, trying to call chunked_iterable([1, 2, 3, 4, 5], size=2) without this line would emit [1, 2] forever.

I think it’s the difference between a container (for which iter(…) returns a new object each time) and an iterator (for which iter(…) returns itself). I forget the exact details, but I remember first reading about this in Brett Slatkin’s book Effective Python.

Getting credentials for an assumed IAM Role

In AWS, everybody has a user account, and you can give each user very granular permissions. For example, you might allow some users complete access to your S3 buckets, databases and EC2 instances, while other users just have read-only permissions. Maybe you have another user who can only see billing information. These permissions are all managed by AWS IAM.

Sometimes you want to give somebody temporary permissions that aren’t part of their usual IAM profile – maybe for an unusual operation, or to let them access resources in a different AWS account. The mechanism for managing this is an IAM role. An IAM role is an identity with certain permissions and privileges that can be assumed by a user. When you assume a role, you get the associated permissions.

For example, at work, the DNS entries for wellcomecollection.org are managed in a different AWS account to the one I usually work in – but I can assume a role that lets me edit the DNS config.

If you’re using the AWS console, you can assume a role in the GUI – there’s a dropdown menu with a button for it:

A screenshot from the IAM console, showing a dropdown menu with a green arrow pointing at “Switch Role”.

If you’re using the SDK or the CLI, it can be a little trickier – so I wrote a script to help me.

The “proper” approach

According to the AWS docs, you can define an IAM role as a profile in ~/.aws/config.

This example shows a role profile called dns_editor_profile.

[profile dns_editor_profile]
role_arn = arn:aws:iam::123456789012:role/dns_editor
source_profile = user1

When I use this profile, the CLI automatically creates temporary credentials for the dns_editor role, and uses those during my session. When the credentials expire, it renews them. Seamless!

This config is also supported in the Python SDK, and I’d guess it works with SDKs in other languages as well – but when I tried it with Terraform, it was struggling to find credentials. I don’t know if this is a gap in the Go SDK, or in Terraform’s use of it – either way, I needed an alternative. So rather than configuring credentials implicitly, I wrote a script to create them explicitly.

Creating temporary AWS credentials for a role

There are a couple of ways to pass AWS credentials to the SDK: as environment variables, with SDK-specific arguments, or with the shared credentials profile file in ~/.aws/credentials. I store the credentials in the shared profile file because all the SDKs can use it, so my script has two steps:

  1. Create a set of temporary credentials
  2. Store them in ~/.aws/credentials

By keeping those as separate steps, it’s easier to change the storage later if, for example, I want to use environment variables.

Create a set of temporary credentials

AWS credentials are managed by AWS Security Token Service (STS). You get a set of temporary credentials by calling the assume_role() API.

Let’s suppose we already have the account ID (the 13-digit number in the role ARN above) and the role name. We can get some temporary credentials like so:

import boto3


def get_credentials(*, account_id, role_name):
    sts_client = boto3.client("sts")

    role_arn = f"arn:aws:iam::{account_id}:role/{role_name}"
    role_session_name = "..."

    resp = sts_client.assume_role(
        RoleArn=role_arn,
        RoleSessionName=role_session_name
    )

    return resp["Credentials"]

Here RoleArn is the ARN (AWS identifier) of the IAM role we want to assume, and RoleSessionName is an identifier for the session. If multiple people assume a role at the same time, we want to distinguish the different sessions.

You can put any alphanumeric string there (no spaces, but a few punctuation characters). I use my IAM username and the details of the role I’m assuming, so it’s easy to understand in audit logs:

    iam_client = boto3.client("iam")
    username = iam_client.get_user()["User"]["UserName"]
    role_session_name = f"{username}@{role_name}.{account_id}"

We could also set the DurationSeconds parameter, which configures how long the credentials are valid for. It defaults to an hour, which is fine for my purposes – but you might want to change it if you have longer sessions, and don’t want to keep re-issuing credentials.

Note that I’m using two Python 3 features here: f-strings for interpolation, which I find much cleaner, and the * in the argument list creates keyword-only arguments, to enforce clarity when this function is called.

Store the credentials in ~/.aws/credentials

The format of the credentials file is something like this:

[profile_name]
aws_access_key_id=ABCDEFGHIJKLM1234567890
aws_secret_access_key=ABCDEFGHIJKLM1234567890

[another_profile]
aws_access_key_id=ABCDEFGHIJKLM1234567890
aws_secret_access_key=ABCDEFGHIJKLM1234567890
aws_session_token=ABCDEFGHIJKLM1234567890

Each section is a new AWS profile, and contains an access key, a secret key, and optionally a session token. That session token is tied to the RoleSessionName we gave when assuming the role.

We could try to edit this file by hand – or easier, we could use the configparser module in the Python standard library, which is meant for working with this type of file.

First we have to load the existing credentials, then look for a profile with this name. If it’s present, we replace it; if not, we create it. Then we store the new credentials, and rewrite the file. Like so:

import configparser
import os


def update_credentials_file(*, profile_name, credentials):
    aws_dir = os.path.join(os.environ["HOME"], ".aws")

    credentials_path = os.path.join(aws_dir, "credentials")
    config = configparser.ConfigParser()
    config.read(credentials_path)

    if profile_name not in config.sections():
        config.add_section(profile_name)

    assert profile_name in config.sections()

    config[profile_name]["aws_access_key_id"] = credentials["AccessKeyId"]
    config[profile_name]["aws_secret_access_key"] = credentials["SecretAccessKey"]
    config[profile_name]["aws_session_token"] = credentials["SessionToken"]

    config.write(open(credentials_path, "w"), space_around_delimiters=False)

Most of this is fairly standard use of the configparser library. The one item of note: I remove the spaces around delimiters, because when I tried leaving them in, boto3 got upset – I think it read the extra space as part of the credentials.

Read command-line parameters

Finally, we need to get some command-line parameters to tell us what the account ID and role name are, and optionally a profile name to store in ~/.aws/credentials. Recently I’ve been trying click for command-line parameters, and I quite like it. Here’s the code:

import click


@click.command()
@click.option("--account_id", required=True)
@click.option("--role_name", required=True)
@click.option("--profile_name")
def save_assumed_role_credentials(account_id, role_name, profile_name):
    if profile_name is None:
        profile_name = account_id

    credentials = get_credentials(
        account_id=account_id,
        role_name=role_name
    )

    update_credentials_file(profile_name=profile_name, credentials=credentials)


if __name__ == "__main__":
    save_assumed_role_credentials()

This defines a command-line interface with @click.command(), then sets up two required command-line parameters – account ID and role name. The profile name is a third, optional parameter, and defaults to the account ID if you don’t supply one. These parameters are passed into the save_assumed_role_credentials() method, which calls the two helpers methods.

Now I can call the script like so:

$python issue_temporary_credentials.py --account_id=123456789012 --role_name=dns_editor --profile_name=dns_editor_profile

and it creates a set of credentials and writes them to ~/.aws/credentials.

To use this profile, I set the AWS_PROFILE variable:

$AWS_PROFILE=dns_editor_profile aws s3 ls

and this command now runs with the credentials for that profile.

tl;dr

If you just want the code, here’s the final copy of the script:

# issue_temporary_credentials.py

import configparser
import os
import sys

import boto3
import click


def get_credentials(*, account_id, role_name):
    iam_client = boto3.client("iam")
    sts_client = boto3.client("sts")

    username = iam_client.get_user()["User"]["UserName"]

    role_arn = f"arn:aws:iam::{account_id}:role/{role_name}"
    role_session_name = f"{username}@{role_name}.{account_id}"

    resp = sts_client.assume_role(
        RoleArn=role_arn,
        RoleSessionName=role_session_name
    )

    return resp["Credentials"]


def update_credentials_file(*, profile_name, credentials):
    aws_dir = os.path.join(os.environ["HOME"], ".aws")

    credentials_path = os.path.join(aws_dir, "credentials")
    config = configparser.ConfigParser()
    config.read(credentials_path)

    if profile_name not in config.sections():
        config.add_section(profile_name)

    assert profile_name in config.sections()

    config[profile_name]["aws_access_key_id"] = credentials["AccessKeyId"]
    config[profile_name]["aws_secret_access_key"] = credentials["SecretAccessKey"]
    config[profile_name]["aws_session_token"] = credentials["SessionToken"]

    config.write(open(credentials_path, "w"), space_around_delimiters=False)


@click.command()
@click.option("--account_id", required=True)
@click.option("--role_name", required=True)
@click.option("--profile_name")
def save_assumed_role_credentials(account_id, role_name, profile_name):
    if profile_name is None:
        profile_name = account_id

    credentials = get_credentials(
        account_id=account_id,
        role_name=role_name
    )

    update_credentials_file(profile_name=profile_name, credentials=credentials)


if __name__ == "__main__":
    save_assumed_role_credentials()

A script for backing up Tumblr posts and likes

A few days ago, Tumblr announced some new content moderation policies that include the mass deletion of any posts deemed to contain “adult content”. (If you missed the news, the Verge has a good summary.)

If my dashboard and Twitter feed are anything to go by, this is bad news for Tumblr. Lots of people are getting flagged for innocuous posts, the timeline seems to be falling apart, and users are leaving in droves. The new policies don’t solve the site’s problems (porn bots, spam, child grooming, among others), but it hurts their marginalised users.

For all its faults, Tumblr was home to communities of sex workers, queer kids, fan artists, people with disabilities – and it gave many of them a positive, encouraging, empowering online space. I’m sad that those are going away.

Some people are leaving the site and deleting their posts as they go (rather than waiting for Tumblr do it for them). Totally understandable, but it leaves a hole in the Internet. A lot of that content just isn’t available anywhere else.

In theory you can export your posts with the official export tool, but I’ve heard mixed reports of its usefulness – I suspect it’s clogged up as lots of people try to leave. In the meantime, I’ve posted a couple of my scripts that I use for backing up my posts from Tumblr. It includes posts and likes, saves the full API responses, and optionally includes the media files (photos, videos, and so on).

They’re a bit scrappy – not properly tested or documented – but content is already being deleted by Tumblr and others, so getting it out quickly seemed more useful. If you use Tumblr, you might want to give them a look.