Earlier today I did a talk at Monki Gras 2019. The theme of the conference is “Accessible Craft: Creating great experiences for everyone”, so I did a talk about inclusive design – and in particular, something called the Curb Cut Effect.
I originally pitched a talk based on Assume Worst Intent, because if you’re thinking about inclusion you might think about ways to avoid exclusion (specifically, exclusion of harassment and abuse victims). That talk was definitely too narrow for this conference, but James isolated a key idea – making a service better for vulnerable uses makes it better for everyone – and I wrote an entirely new talk around it.
The talk went really well – everybody was very nice afterwards, and I’m enjoying the rest of talks too. (Notes on those will be a separate post.) It was a lot of fun to write and present.
The talk was recorded, and you can watch it on YouTube:
You can read the slides and my notes on this page, or download the slides as a PDF.
Mismatch, by Kat Holmes, was a useful book. It’s a short but detailed book that I’d recommend for anybody wanting to learn more about inclusive design. The penultimate chapter was especially helpful for finding a good framing.
Some general articles about the Curb Cut Effect that I found useful:
Electronic Curb Cuts, by Steve Jacobs, is a long list of other examples of the curb cut effect. I drew a couple of examples from here (the typewriter and OCR), and there are plenty of others I didn’t have time to mention.
Slides and notes
Hi, I’m Alex. I’m going to talk about the Curb Cut Effect, what it is, and how we might use it.
At Wellcome, we’ve been thinking a lot about architecture recently. Our current exhibition, Living with Buildings, is all about the effect of architecture on human health, so I wanted to start today by telling you all the story of architecture – specifically, the story of a common architectural feature that we all walk past every day without a second glance.
I’m talking about those areas where the kerb dips to form a ramp – giving a level, step-free path from the road to the pavement. In the UK, these are usually accompanied by a textured yellow surface (pictured).
They go by several names – in the UK we call them dropped kerbs, in the US they’re called curb cuts (one of the few American spellings that adds the letter U), and they have other names around the world. The American spelling is mostly common, and that’s what I’ll use for the rest of the talk, but you can mentally substitute “curb cut” for your preferred term.
One of the earliest examples of curb cuts was in Kalamazoo, MI. (Great name!)
There was a man called Jack Fisher, born in Kalamazoo in 1918. Like many young men of his age, he enlisted in the Army during the Second World War.
He was involved in a jeep accident in 1943. While he was recovering in hospital, he read the records of other patients with similar injuries to keep himself occupied. (Because 1940s privacy laws.)
He returned to Kalamazoo with steel braces from his hip to his neck, and a heavy limp. After the war, he graduated from Harvard law school (very impressive), but none of the established firms would take him – he was a disabled veteran, who might trip and injure himself, and need compensation. So they chose not to hire him. (Because 1940s labour rights.)
So he worked as an attorney in his own practice. He became well-known among other disabled veterans in and near Kalamazoo – of which there were many, because there was a nearby hospital at Battle Creek which specialised in amputee treatment and rehabilitation.
Working closely with them, he became aware of the problems and challenges they faced.
One of those problems: Kalamazoo had tall curbs (up to 6 inches). This was a problem – people would trip, injure themselves, damage prosthetic limbs, and for wheelchair users they’re a total nightmare. Tall kerbs are inaccessible, and prevent people getting around, socialising, working and so on.
So in 1945, Jack Fisher took it upon himself to fix this, and petitioned the city commission for curb cuts and hand-rails. Ditching the step would make it easier for people to get around. The city authorised their construction and ran a small pilot program, they were very successful, and they grew in number.
Kalamazoo is one of the earliest examples of dropped kerbs, but the same story plays out in lots of other places – curb cuts were installed in lots of places to make the streets more accessible for disabled people and wheelchair users.
So curb cuts make the roads more accessible for wheelchair users and disabled people. Yay!
But they’re not the only people to benefit, so do:
Parents with prams
Workers pulling heavy carts
Travellers with wheeled suitcases
And probably others.
Wcould call this a “force multipler” or a “happy accident”, but really this is the original example of the “Curb Cut Effect”.
The Curb Cut Effect comes in many forms, but the way I think of it is:
Making something better for disabled people can make it better for everyone.
What this means for us, as people who build things:
Designs that think about disabled people are better designs for everyone
And we reflect this is the words we use: it’s why we don’t talk about handicapped design or barrier-free design. We talk universal design. We talk about good design.
(Repeat the Curb Cut Effect.)
Let’s look at a few examples.
One of the earliest examplse predates Kalamazoo by more than a century.
This is the Countess Carolina Fantoni da Fivizzano. She lived in the early nineteenth century, and was the friend and lover of the Italian inventor Pellegrino Turri.
They’d write each other letters, but she lost her sight as an adult, and was unable to write herself. That meant the only way to send letters was to dictate to somebody else – but she wanted to send letters in private. Together they built a machine that let her write letters by pressing a key for each letter – allowing her to “type” letters.
Writing with type… a type-writer, of sorts.
This was one of the earliest iterations of the typewriter. It made writing accessible to the blind, and the derivatives became the modern-day keyboard. A machine created to help one blind woman write love letters was the basis for a fundamental input device for modern computing.
Speaking of love letters… let’s talk about email!
(You don’t use email to write love letters? Sounds fake.)
Email has become a ubiquitous part of modern comms, but where did it come from? Why was it invented?
This is Vinton Cerf. He’s often called the “Father of the Internet”, did a lot of work on the early Internet (then-ARPANET) protocols, is a strong advocate for accessibility, and led the work on the first commercial email program.
Because he’s hard-of-hearing, and his wife Sigrid is deaf. His work on email started, in part, as a way for them to stay connected when they weren’t in the same room. At the time, the alternative was the telephone, which was a bit of a non-starter.
Because I’m hearing-impaired, emails are a tremendously valuable tool because of the precision that you get.
I can read what’s typed as opposed to straining to hear what’s being said.
Email started as a key technology for people who are deaf or have hearing loss – or who are just separated by time and space.
Sticking with hearing loss, let’s talk about captions.
It was originally intended to help deaf people understand movies with dialogue or sound effects. Closed captions were first broadcast in the 1970s. They were expensive, needed specialist equipment, and most content wasn’t captioned – today things are generally better. And indeed, they help people who are deaf or hard-of-hearing, but lots of other people besides:
MAYBE YOU’RE IN A NOISY ENVIRONMENT, LIKE A BAR OR GYM, AND YOU CAN’T HEAR THE TELEVISON OVER THE BACKGROUND NOISE
(or maybe you’re a quiet environment, where somebody is sleeping and you can’t turn on the sound)
Or perhaps you have the bad luck to be with those monsters who talk during films
And even if you can hear the sound fine, you can still benefit from captions:
Children learning to read
Somebody learning a foreign language
And it helps with conferences too!
I’m being captioned right now – literally as I speak! [Monki Gras has live captioning.] This is one of the captioners at PyCon UK, and our experience is that lots of people find it useful during talks, not just the deaf or hard-of-hearing – maybe a word you couldn’t hear, somebody’s speaking with an accent, or you stopped to check twitter halfway through the session.
Early research into OCR was done to help the blind.
This is a machine called an optophone, a device for helping blind people read. You put printed text on the scanner, it identifies the characters and translates them into audio pulses (sort of like Morse code), then plays them through headphones. This was pioneering work for computer vision and text-to-speech synthesis.
These have become widely-used technologies: for making textual versions of scanned documents, Google Books, even those smartphone apps that let you translate signs in a foreign language.
And in fact, this is what we do at Wellcome Collection!
For those unfamiliar with Wellcome: we have an archive about human health and medicine. This is one of our “data centres”…
…using advanced container technology, like “shelves” and “books”.
Like many institutions, we’re scanning our archives to make them more easily available, and then we use OCR to make them searchable.
Here’s one example of our books: a notebook from Marie Curie, freely available to browse online. (Which is preferable to the original, which is slightly radioactive!) And using OCR, we can see that the word “radioaktiv” appears 730 times.
This so cool, but it wouldn’t exist without the pioneering work done into OCR to help blind people.
One final example, less high technology and more small convenience: door handles.
Compared to door knobs, handles provide a larger area to grip or rest your hand against, so they’re easier to operate for people with a range of fine motor disabilities that prevent them from grasping small objects.
But they also make it easier if your hands are full, or you’re carrying things – you can lean on the door with an elbow without dropping something. It just makes life a bit easier.
So those are just a few examples of the Curb Cut Effect.
And that’s often where discussion of the Curb Cut Effect stops, which is a shame, because it isn’t just about disability!
Although disabled people are the most visibly excluded, there are plenty of excluded groups to whom this effect applies: women in tech, people of colour, victims of harassment and abuse…
Let’s look at one more example.
In the last few years, there’s been a big uptick in single stall, gender-neutral bathrooms. These are bathrooms that contain their own toilet and sink, maybe a shelf, all in a single private, enclosed space. And their installation has mostly been driven by a desire to accommodate trans and non-binary people who feel uncomfortable in traditionally gendered bathrooms.
And this is great for them… but it helps lots of other people too.
Parents with children (especially a child of a different gender, especially men)
Anybody accompanied by a carer
Men who need baby changing facilities
Somebody having a period emergency and wants a bit of privacy
So we can take the Curb Cut Effect, and make it stronger:
Making something better for people who are excluded or marginalised makes it better for everyone.
It’s not just about disability.
So why am I telling you all this? Two reasons.
First, these are good stories! They’re little love letters to inclusion, and a nice way to get people talking about inclusion or accessibility.
It’s been really fun to research this talk, and get to go to friends and say, “Hey, did you know this really cool story about the invention of the bendy straw?” (Talk to me in the break if you want to know this one.)
There is a moe serious point.
If you come to a conference titled “Accessible Craft”, you probably already care about inclusion. I don’t need to convince you – you want to do it because it’s the right thing to do.
Unfortunately, things dopn’t happen because they’re “right” or “fair” (what a world that would be!).
We often have to justify the value of inclusion. (If you’re at this conference, you might be the person who does this advocacy, who has to make the business case.)
How often have you heard questions like “Do we have to do this? How many people will use it? Can we really afford it?”
And the Curb Cut Effect is a great tool to remember: it shows the value of inclusion. Spread the cost across a wide group, and changes to support inclusion suddenly seem much more attractive.
It also serves to dispel a powerful myth.
There’s a common suspicion that helping one group must somehow, invisibly, hurt another group. As Spock said, “The needs of the many outweigh the needs of the few”. But in reality, we don’t have to choose!
The Curb Cut Effect shows us this is false: in fact, it shows us the opposite is true. Making the world a better place for a small number of people can make it better for a much wider number too. Yay!
So let’s bring this all together.
I’ve shown you the Curb Cut Effect (repeat), and a handful of my favourite examples. Email. OCR. Door handles. And of course, curb cuts!
But there are many more – go away and try to think of some, maybe even ones in your own work. Keep it in mind, see how you can apply it to the things you build, and remember it the next time you’re asked to justify the value of inclusion.
About a week ago, I was at You Got This 2019, a conference that bills itself as an event for “juniors on skills needed for a happy, healthy work life”. I’d seen the schedule on Twitter, thought it looked interesting, and decided to attend on a whim. All the topics had a strong focus on wellbeing, not technical content.
This post has my lightly edited notes from the conference and the sessions.
Everybody I met was friendly and nice.
All the talks were good, and perhaps more notably, consistent. Usually when I go to a conference, there’s at least one or two talks that leave me feeling a bit “meh”, and that wasn’t the case here. Every session was interesting, well-presented, and left me with at least a few new ideas.
The emphasis in the introduction was that these are “topics we should be talking about more” – which is precisely why I decided to go.
When I arrived, I was asked if I was happy to appear in photographs – and if not, I could wear a bright yellow lanyard. In the introductory remarks, we were all reminded not to take photographs of people with yellow lanyards.
All the food was vegan and delicious (except the milk for the coffee, I think), including ice cream in the afternoon break. They said several times “if you have other allergies, talk to us”, so I’m assuming other food was available if you couldn’t eat the default options.
The Code of Conduct was prominently featured in the signs, tickets, and the introductory remarks.
The ground floor bathroom was gender neutral, and had free sanitary, dental and haircare products for people to take. Apparently this is a feature of the Monzo offices, where the conference was held – very nice indeed! Sanitary products are the most important, but dental and haircare are useful too, and a new one to me.
The entire day was MC’d by Scary Boots, who helped warm up the audience, support the speakers, and keep us all laughing. Especially notable when somebody was having A/V issues.
There was an inclusion programme, including a variety of scholarships for people who couldn’t afford tickets, travel or childcare. And when I bought my ticket, there was an option to pay over the ticket price, contributing to the scholarship pool.
And here’s some stuff that felt a little odd:
I only saw a handful of name badges, and mostly on speakers, who are more likely to be recognised. I’m not sure why they weren’t given to everybody.
The badge design itself was quite good – rather than pre-printed with wallet names or companies, they were blank with a large space for “name” and “preferred pronouns”.
I’m told the venue was wheelchair accessible, but I didn’t see anybody using a wheelchair. The food and breakout spaces were downstairs, and I assume there was a lift, but I didn’t see it. (Better signposting!)
I also felt the spaces were a bit small, and felt crowded during the lunch break.
No conference gets inclusivity perfect, and the gold standard for me is still AlterConf London – but this was pretty good, overall. I’d happily attend again.
If you read the post and think sounds interesting, there’s a sequel planned for January 2020, with a mailing list on the website.
These are based on my handwritten notes, which are the bits I found most interesting, not a comprehensive summary of the talks. They’ll give you a flavour, but they’re not a substitute for watching the talks.
Impostor syndrome, perfectionism and anxiety, by Jo Francetti
She started with a story of “Mo”, a young developer who was feeling overwhelmed at work, and the sort of things she was feeling. Wondering if she knows enough, if she’s being judged by her co-workers, overworking and ruining her mental health. It was a good way to set up the topic – a sort of “Do you recognise this?” moment.
Editor note: I was wondering how she chose the name “Mo”, and can’t believe it took me to now to spot it.
There are some common issues, which she went through in turn:
A new concept to me was pluralistic ignorance, in which nobody believes something, but everybody believes that everybody else belives it. For example, nobody believes that a task is easy, but everybody believes that everybody else finds it easy, and thus impostor syndrome sets in. (“Everybody else finds this easy but I can’t do it.”)
My peers are all more intelligent and accomplished than me.
My peers are so gullible that I’ve tricked them into believing I’m one of them.
Open and frank conversations around our challenges can help combat impostor syndrome.
Perfectionism. Expresses itself as catastrophic thinking (an alternative wording to “black and white thinking”), paralysis and procrastination. The result is anxiety, overwork and stress.
We need to remember that perfect is the enemy of good – we’re not expected to come up with a perfect answer every time. If you give yourself time to solve a problem (with an explicit deadline), you can work through it.
Anxiety. Don’t neglect your mental health! Be kind to yourself, and don’t just look externally for validation (it’s fleeting).
Knowledge sharing can take many forms: blog posts, answering questions, speaking at conferences.
A big part of this talk was Sascha talking about his struggles with depression and exhaustion, and what felt a lot like burnout. (I don’t remember if he actually used the word “burnout”.)
He talked about how he built a new frame of mind, and that “we’re more than just a walking tech stack”*. In other words: we shouldn’t define ourselves solely by our technical work or our technical skills. As part of that, he talked about his experiences with therapy, which made me happy. (I think therapy is a really useful tool, I’ve been to therapy multiple times, and it’s good to hear it discussed on stage.)
A key attribute of a successful team is psychological safety – people feeling like they can take risks, and b supported by their team.
“Tech enables us, but it doesn’t define us.”
How to find knowledge you can share:
Keep a TIL (Today I Learned) record
Focus on learnings, not failures
Ask colleagues what they’d ask you about (i.e. what do they think are your areas of expertise)
Document the journey, not the goal
I really liked the framing of this talk: the introduction was “Hi, I’m Sascha”, and a list of technical skills and background. Then at the end of the talk, another slide “Hi, I’m still Sascha”, which had a similar list that covered much more of his life.
You want to understand your current money habits – but exercise self-compassion. Don’t beat yourself up over everything you think you’re doing wrong.
Have a spreadsheet. Calculate all the numbers: exactly how much you’re earning each month, and your regular bills (rent, utilities, tax, etc.). What’s left is your budget.
Automate everything. If you have to remember to pay bills, there’s a risk you’ll forget and take a hit to your credit rating.
Max out your workplace pension. (Note: I scribbled down “needs research” next to this, because I’m not sure how I’d do this, what it means, and suspect the best thing to do with pension stuff is probably more complicated than simple.)
Use bank accounts that help you track your spending, like Monzo or Starling.
Get free credit reports from Experian and Equifax.
If you have a credit card, keep your credit limit at an affordable level. (I’ve heard conflicting advice on this one. My credit limit is several times my monthly salary, and I rarely max it out – but by paying it off regularly it’s slowly inched up, and might be useful in an emergency. Hasn’t bitten me so far.)
Advanced moneying: have side gigs, negotiate a pay rise.
Paula also wrote a blog post with a summary and some resources.
Tech is a relatively young and immature industry, but “we shouldn’t be ashamed of being immature if we mature well”.
A Stack Overflow survey (admittedly not representative, with “some demographic challenges”) – a majority of developers think they should think about the ethics of the software they build, but only a minority think the ultimate responsibility lies with them. We should be empowering developers to feel like they have that responsibility (and power).
Three principles of ethical software:
People come first
Create accessible and useful services for everyone
More diverse teams give better visibility ethical factors
We all make mistakes; what matters is how we get better. (The speaker used an example of Facebook – Mark Zuckerberg didn’t realise he’d create the monster he did – but some of the early iterations of Facebook were pretty sketchy, so I’m reluctant to give Zuckerberg a pass.)
A useful resource: the Ethical OS toolkit. A checklist of useful things to think about when building ethical software. (I wondered: does it cover online harassment and abuse?)
Things to think about:
Unethical software isn’t more profitable
Don’t hide behind contracts
You can make a difference (e.g. Project Maven, in which Google’s engineers successfully protested a contract with the US DoD)
Understanding and building independence, by Violet Peña
Independence is the ability to understand and construct solutions. It’s not about working alone.
Here’s a framework for solving problems:
Have a plan.
Check you understand the problem: the data, the conditions, the unknowns.
Break your plan down into steps. How long will each step take? How can you track progress?
Be honest with yourself and your team. Are you progressing?
Be aware of how you’re performing. It’s okay to say “I need more time”, “I need more help”, “I need to do something differently”. (And she had us all say them out loud to get used to saying the words!)
Use your tools.
Use the tools you have, but don’t be afraid to change modalities; to try new tools or technologies if needed.
Find a related problem (similar data, conditions, unknowns), and try to solve your problem in a similar way.
Ask for help.
It’s a great thing to do: it builds your knowledge, shows your respect for your colleagues, shows you don’t have a massive ego.
You want to “build a Voltron” – a group opf people who can help with particular topics.
You should ask for help when you’re stuck, can’t get past a particular problem, or need a rethink.
The more certain you are somebody has seen this before, the earlier you can ask for help.
Advice for people who want to support junior developers
If you say “junior”, what are you suggesting? You’re saying that:
Learning isn’t for everyone
You’re old school (there are some industries where being conservative about change is good, e.g. healthcare or aviation safety; this is not true in most companies)
You undervalue skills
You only have permission to learn if you’re:
Not expected to deliver
This is a false dichotomy.
Here’s how to build a structure for learning:
Provide context for problems:
Run whiteboarding and diagramming sessions.
Link to the domain – every ticket should link to an end-user case.
Treat everything as a learning exercise
Provide your time and expertise to help people
Mentioned: A theory of goal setting and task performance, by Edwin Locke
What can I say about myself?
A good goal is specific, aspirational, expected and believed. (i.e., somebody else should believe you can achieve it, and then say so!)
Provide resources and help:
Provide a summary of reading, what you’ve learnt, prepared exercises. If a junior asks a question and you point them at a 400-page book, they won’t read it and they’re no better off – give distilled knowledge.
Be socratic, i.e., ask questions. Don’t just answer somebody’s question straight up: it robs them the chance to ever learn the answer. Instead, increase their power to ask questions, and lead themselves to the answer.
Build a link library of useful advice, tips, technical info.
There are several types of question somebody might ask/opportunities for learning, and they need to be treated differently:
Concepts. These can lead to interesting conversations and discussion (e.g. “what is a function?”).
Skills, processes, behaviours. These are learnt by pairing, observing, feedback (e.g. “what is a good amount of code for a single function?”)
Have a structure for feedback:
Require intentionality. Ask: What do you want feedback on?
This allows you to get feedback early, rather than waiting for perfect or completed specimens, and early feedback is better.
Keep it tight. Don’t overwhelm somebody with information, or provide comments on irrelevant areas.
When they’ve addressed your feedback, they should come back for more. The model of “get feedback once, then finish” is an odd one, and not so helpful (exams are weird).
A question for the junior devs in the room: how are you going to transform the businesses you work in?
Over the last few months, it’s become really obvious how much sunlight affects my mood levels.
If I commute in darkness, I’m miserable. If I commute in daylight, I’m a lot happier.
This might sound a bit like seasonal affective disorder (SAD), and I do wonder if it’s something like that. SAD is a type of depression that follows the seasons, and is usually worse in winter.
I have friends with fairly severe SAD, who feel tired all the time during the winter months, and it’s never been as bad as that for me, so until now I’ve assumed I didn’t have any form of SAD.
Maybe I do, maybe I don’t – but I can look to SAD treatment for tips and ideas to make my life easier. In particular, getting as much natural light as possible, and a sun lamp or six. (I have a friend with a small solar system in their front room, and the effect on their mood is ~noticeable~.)
If any of this sounds familiar to you, you might want to check out medical advice for SAD, and try some of the coping mechanisms yourself.
In the meantime, spring is fast approaching – just check out this blue sky from today’s commute:
This ship has long since sailed, but yesterday I found another reason to be a bit sad that TfL has phased out cash as a payment mechanism.
I logged into my Oyster account for the first time today, and I was struck by how much location data they have. You can reconstruct a large chunk of my diary from that data – for any Tube station or bus stop outside the core, there’s only really one person I would be visiting, and now you know when and for how long I was with them.
Intellectually, I knew this was happening – if I’m using the same card each time, I’d almost be more surprised if they weren’t recording the fine-grained data. But it’s sobering to see it all laid out, in complete and unerring detail.
I’ve never had to worry about stalkers or people coming after me, but I’d be pretty unnerved if I was.
Cash lets you travel anonymously, but you can’t use it on buses, and paper tickets on the Tube are more expensive. So if you’re somebody who doesn’t want your location tracked – or doesn’t have a card you can use – public transport in London is a bit harder for you to use.
A couple of examples sprang to mind:
Sex workers predominantly use cash, and probably don’t want a paper trail of where they’ve been
If you’re living with somebody abusive (with whom you might share a card), and you’re visiting refuges or safe spaces but don’t want them to know
If you’re a queer teen who’s not out to your parents, and you’re going to support groups
There are probably others, and while your Tube data along might not give away your secrets, it could prompt awkward questions.
Late last year, I wanted to find a jar of loganberry jelly as a Christmas present for my granddad. This turned out to be surprisingly difficult to find – it’s a rare flavour that isn’t in any of the supermarkets, and even Fortnum & Mason and Harrods both came up short. I did find it on Amazon, but something about buying jam on Amazon just feels wrong.
Googling turned up an article about a place called Paul Rothe & Sons, a small family-run deli in Marylebone. I walked over on my lunch break, and found a cosy little shop – tables for eating sandwiches and drinks, and walls stacked with a huge variety of jams and spreads. I had a little trouble finding what I wanted, but one of the owners (Stephen Rothe) helped me find exactly what I wanted.
Sadly I couldn’t stay for lunch (and it was so busy, I’m not sure I’d have found a seat!), but there was a very relaxed air among the tables. I did grab a sausage sandwich to go, and it was warm and tasty. Perfect for a cold December day!
If you’re ever in London and need an unusual jam or a delicious sandwich, give it a look.
But the encoding argument to open() is only in Python 3 or later, so you can’t use this in Python 2.
In theory this is backported as codecs.open(), but I get a different error if I use codecs.open() in this file with Python 2.7:
Traceback (most recent call last):
File "reader3.py", line 7, in <module>
for row in csvreader:
UnicodeEncodeError: 'ascii' codec can't encode character u'\xef' in position 4: ordinal not in range(128)
This feels like it should be possible using only the standard library, but it was becoming sufficiently complicated that I didn’t want to bother.
I considered defining these as two separate functions, and running:
but that felt a little icky, and would have been annoying for code coverage. Having two separate functions also introduces a source of bugs – I might remember to update one function, but not the other.
I found csv23 on PyPI, whose description sounded similar to what I wanted. The following snippet does what I want:
Here’s a fairly common problem I have: I have an iterable, and I want to go through it in “chunks”. Rather than looking at every item of the sequence one-by-one, I want to process multiple elements at once.
For example, when I’m using the bulk APIs in Elasticsearch, I can index many document with a single API call, which is more efficient than making a new API call for every document.
Most of the heavy lifting is done by itertools.islice(); I call that repeatedly until it returns an empty sequence. The itertools module has lots of useful functions for this sort of thing.
The it = iter(iterable) line may be non-obvious – this ensures that the value it is using the same iterator throughout. If you pass certain fixed iterables to islice(), it creates a new iterator each time – and then you only ever get the first handful of elements.
For example, trying to call chunked_iterable([1, 2, 3, 4, 5], size=2) without this line would emit [1, 2] forever.
I think it’s the difference between a container (for which iter(…) returns a new object each time) and an iterator (for which iter(…) returns itself). I forget the exact details, but I remember first reading about this in Brett Slatkin’s book Effective Python.
In AWS, everybody has a user account, and you can give each user very granular permissions. For example, you might allow some users complete access to your S3 buckets, databases and EC2 instances, while other users just have read-only permissions. Maybe you have another user who can only see billing information. These permissions are all managed by AWS IAM.
Sometimes you want to give somebody temporary permissions that aren’t part of their usual IAM profile – maybe for an unusual operation, or to let them access resources in a different AWS account. The mechanism for managing this is an IAM role. An IAM role is an identity with certain permissions and privileges that can be assumed by a user. When you assume a role, you get the associated permissions.
For example, at work, the DNS entries for wellcomecollection.org are managed in a different AWS account to the one I usually work in – but I can assume a role that lets me edit the DNS config.
If you’re using the AWS console, you can assume a role in the GUI – there’s a dropdown menu with a button for it:
If you’re using the SDK or the CLI, it can be a little trickier – so I wrote a script to help me.
The “proper” approach
According to the AWS docs, you can define an IAM role as a profile in ~/.aws/config.
This example shows a role profile called dns_editor_profile.
When I use this profile, the CLI automatically creates temporary credentials for the dns_editor role, and uses those during my session. When the credentials expire, it renews them. Seamless!
This config is also supported in the Python SDK, and I’d guess it works with SDKs in other languages as well – but when I tried it with Terraform, it was struggling to find credentials. I don’t know if this is a gap in the Go SDK, or in Terraform’s use of it – either way, I needed an alternative. So rather than configuring credentials implicitly, I wrote a script to create them explicitly.
Creating temporary AWS credentials for a role
There are a couple of ways to pass AWS credentials to the SDK: as environment variables, with SDK-specific arguments, or with the shared credentials profile file in ~/.aws/credentials. I store the credentials in the shared profile file because all the SDKs can use it, so my script has two steps:
Create a set of temporary credentials
Store them in ~/.aws/credentials
By keeping those as separate steps, it’s easier to change the storage later if, for example, I want to use environment variables.
Create a set of temporary credentials
AWS credentials are managed by AWS Security Token Service (STS). You get a set of temporary credentials by calling the assume_role() API.
Let’s suppose we already have the account ID (the 13-digit number in the role ARN above) and the role name. We can get some temporary credentials like so:
Here RoleArn is the ARN (AWS identifier) of the IAM role we want to assume, and RoleSessionName is an identifier for the session. If multiple people assume a role at the same time, we want to distinguish the different sessions.
You can put any alphanumeric string there (no spaces, but a few punctuation characters). I use my IAM username and the details of the role I’m assuming, so it’s easy to understand in audit logs:
We could also set the DurationSeconds parameter, which configures how long the credentials are valid for. It defaults to an hour, which is fine for my purposes – but you might want to change it if you have longer sessions, and don’t want to keep re-issuing credentials.
Each section is a new AWS profile, and contains an access key, a secret key, and optionally a session token. That session token is tied to the RoleSessionName we gave when assuming the role.
We could try to edit this file by hand – or easier, we could use the configparser module in the Python standard library, which is meant for working with this type of file.
First we have to load the existing credentials, then look for a profile with this name. If it’s present, we replace it; if not, we create it. Then we store the new credentials, and rewrite the file. Like so:
Most of this is fairly standard use of the configparser library. The one item of note: I remove the spaces around delimiters, because when I tried leaving them in, boto3 got upset – I think it read the extra space as part of the credentials.
Read command-line parameters
Finally, we need to get some command-line parameters to tell us what the account ID and role name are, and optionally a profile name to store in ~/.aws/credentials. Recently I’ve been trying click for command-line parameters, and I quite like it. Here’s the code:
This defines a command-line interface with @click.command(), then sets up two required command-line parameters – account ID and role name. The profile name is a third, optional parameter, and defaults to the account ID if you don’t supply one. These parameters are passed into the save_assumed_role_credentials() method, which calls the two helpers methods.
A few days ago, Tumblr announced some new content moderation policies that include the mass deletion of any posts deemed to contain “adult content”. (If you missed the news, the Verge has a good summary.)
If my dashboard and Twitter feed are anything to go by, this is bad news for Tumblr. Lots of people are getting flagged for innocuous posts, the timeline seems to be falling apart, and users are leaving in droves. The new policies don’t solve the site’s problems (porn bots, spam, child grooming, among others), but it hurts their marginalised users.
For all its faults, Tumblr was home to communities of sex workers, queer kids, fan artists, people with disabilities – and it gave many of them a positive, encouraging, empowering online space. I’m sad that those are going away.
Some people are leaving the site and deleting their posts as they go (rather than waiting for Tumblr do it for them). Totally understandable, but it leaves a hole in the Internet. A lot of that content just isn’t available anywhere else.
In theory you can export your posts with the official export tool, but I’ve heard mixed reports of its usefulness – I suspect it’s clogged up as lots of people try to leave. In the meantime, I’ve posted a couple of my scripts that I use for backing up my posts from Tumblr. It includes posts and likes, saves the full API responses, and optionally includes the media files (photos, videos, and so on).
They’re a bit scrappy – not properly tested or documented – but content is already being deleted by Tumblr and others, so getting it out quickly seemed more useful. If you use Tumblr, you might want to give them a look.