Getting credentials for an assumed IAM Role

In AWS, everybody has a user account, and you can give each user very granular permissions. For example, you might allow some users complete access to your S3 buckets, databases and EC2 instances, while other users just have read-only permissions. Maybe you have another user who can only see billing information. These permissions are all managed by AWS IAM.

Sometimes you want to give somebody temporary permissions that aren’t part of their usual IAM profile – maybe for an unusual operation, or to let them access resources in a different AWS account. The mechanism for managing this is an IAM role. An IAM role is an identity with certain permissions and privileges that can be assumed by a user. When you assume a role, you get the associated permissions.

For example, at work, the DNS entries for wellcomecollection.org are managed in a different AWS account to the one I usually work in – but I can assume a role that lets me edit the DNS config.

If you’re using the AWS console, you can assume a role in the GUI – there’s a dropdown menu with a button for it:

If you’re using the SDK or the CLI, it can be a little trickier – so I wrote a script to help me.

The “proper” approach

According to the AWS docs, you can define an IAM role as a profile in ~/.aws/config.

This example shows a role profile called dns_editor_profile.

[profile dns_editor_profile]
role_arn = arn:aws:iam::123456789012:role/dns_editor
source_profile = user1

When I use this profile, the CLI automatically creates temporary credentials for the dns_editor role, and uses those during my session. When the credentials expire, it renews them. Seamless!

This config is also supported in the Python SDK, and I’d guess it works with SDKs in other languages as well – but when I tried it with Terraform, it was struggling to find credentials. I don’t know if this is a gap in the Go SDK, or in Terraform’s use of it – either way, I needed an alternative. So rather than configuring credentials implicitly, I wrote a script to create them explicitly.

Creating temporary AWS credentials for a role

There are a couple of ways to pass AWS credentials to the SDK: as environment variables, with SDK-specific arguments, or with the shared credentials profile file in ~/.aws/credentials. I store the credentials in the shared profile file because all the SDKs can use it, so my script has two steps:

  1. Create a set of temporary credentials
  2. Store them in ~/.aws/credentials

By keeping those as separate steps, it’s easier to change the storage later if, for example, I want to use environment variables.

Create a set of temporary credentials

AWS credentials are managed by AWS Security Token Service (STS). You get a set of temporary credentials by calling the assume_role() API.

Let’s suppose we already have the account ID (the 13-digit number in the role ARN above) and the role name. We can get some temporary credentials like so:

import boto3


def get_credentials(*, account_id, role_name):
    sts_client = boto3.client("sts")

    role_arn = f"arn:aws:iam::{account_id}:role/{role_name}"
    role_session_name = "..."

    resp = sts_client.assume_role(
        RoleArn=role_arn,
        RoleSessionName=role_session_name
    )

    return resp["Credentials"]

Here RoleArn is the ARN (AWS identifier) of the IAM role we want to assume, and RoleSessionName is an identifier for the session. If multiple people assume a role at the same time, we want to distinguish the different sessions.

You can put any alphanumeric string there (no spaces, but a few punctuation characters). I use my IAM username and the details of the role I’m assuming, so it’s easy to understand in audit logs:

    iam_client = boto3.client("iam")
    username = iam_client.get_user()["User"]["UserName"]
    role_session_name = f"{username}@{role_name}.{account_id}"

We could also set the DurationSeconds parameter, which configures how long the credentials are valid for. It defaults to an hour, which is fine for my purposes – but you might want to change it if you have longer sessions, and don’t want to keep re-issuing credentials.

Note that I’m using two Python 3 features here: f-strings for interpolation, which I find much cleaner, and the * in the argument list creates keyword-only arguments, to enforce clarity when this function is called.

Store the credentials in ~/.aws/credentials

The format of the credentials file is something like this:

[profile_name]
aws_access_key_id=ABCDEFGHIJKLM1234567890
aws_secret_access_key=ABCDEFGHIJKLM1234567890

[another_profile]
aws_access_key_id=ABCDEFGHIJKLM1234567890
aws_secret_access_key=ABCDEFGHIJKLM1234567890
aws_session_token=ABCDEFGHIJKLM1234567890

Each section is a new AWS profile, and contains an access key, a secret key, and optionally a session token. That session token is tied to the RoleSessionName we gave when assuming the role.

We could try to edit this file by hand – or easier, we could use the configparser module in the Python standard library, which is meant for working with this type of file.

First we have to load the existing credentials, then look for a profile with this name. If it’s present, we replace it; if not, we create it. Then we store the new credentials, and rewrite the file. Like so:

import configparser
import os


def update_credentials_file(*, profile_name, credentials):
    aws_dir = os.path.join(os.environ["HOME"], ".aws")

    credentials_path = os.path.join(aws_dir, "credentials")
    config = configparser.ConfigParser()
    config.read(credentials_path)

    if profile_name not in config.sections():
        config.add_section(profile_name)

    assert profile_name in config.sections()

    config[profile_name]["aws_access_key_id"] = credentials["AccessKeyId"]
    config[profile_name]["aws_secret_access_key"] = credentials["SecretAccessKey"]
    config[profile_name]["aws_session_token"] = credentials["SessionToken"]

    config.write(open(credentials_path, "w"), space_around_delimiters=False)

Most of this is fairly standard use of the configparser library. The one item of note: I remove the spaces around delimiters, because when I tried leaving them in, boto3 got upset – I think it read the extra space as part of the credentials.

Read command-line parameters

Finally, we need to get some command-line parameters to tell us what the account ID and role name are, and optionally a profile name to store in ~/.aws/credentials. Recently I’ve been trying click for command-line parameters, and I quite like it. Here’s the code:

import click


@click.command()
@click.option("--account_id", required=True)
@click.option("--role_name", required=True)
@click.option("--profile_name")
def save_assumed_role_credentials(account_id, role_name, profile_name):
    if profile_name is None:
        profile_name = account_id

    credentials = get_credentials(
        account_id=account_id,
        role_name=role_name
    )

    update_credentials_file(profile_name=profile_name, credentials=credentials)


if __name__ == "__main__":
    save_assumed_role_credentials()

This defines a command-line interface with @click.command(), then sets up two required command-line parameters – account ID and role name. The profile name is a third, optional parameter, and defaults to the account ID if you don’t supply one. These parameters are passed into the save_assumed_role_credentials() method, which calls the two helpers methods.

Now I can call the script like so:

$python issue_temporary_credentials.py --account_id=123456789012 --role_name=dns_editor --profile_name=dns_editor_profile

and it creates a set of credentials and writes them to ~/.aws/credentials.

To use this profile, I set the AWS_PROFILE variable:

$AWS_PROFILE=dns_editor_profile aws s3 ls

and this command now runs with the credentials for that profile.

tl;dr

If you just want the code, here’s the final copy of the script:

# issue_temporary_credentials.py

import configparser
import os
import sys

import boto3
import click


def get_credentials(*, account_id, role_name):
    iam_client = boto3.client("iam")
    sts_client = boto3.client("sts")

    username = iam_client.get_user()["User"]["UserName"]

    role_arn = f"arn:aws:iam::{account_id}:role/{role_name}"
    role_session_name = f"{username}@{role_name}.{account_id}"

    resp = sts_client.assume_role(
        RoleArn=role_arn,
        RoleSessionName=role_session_name
    )

    return resp["Credentials"]


def update_credentials_file(*, profile_name, credentials):
    aws_dir = os.path.join(os.environ["HOME"], ".aws")

    credentials_path = os.path.join(aws_dir, "credentials")
    config = configparser.ConfigParser()
    config.read(credentials_path)

    if profile_name not in config.sections():
        config.add_section(profile_name)

    assert profile_name in config.sections()

    config[profile_name]["aws_access_key_id"] = credentials["AccessKeyId"]
    config[profile_name]["aws_secret_access_key"] = credentials["SecretAccessKey"]
    config[profile_name]["aws_session_token"] = credentials["SessionToken"]

    config.write(open(credentials_path, "w"), space_around_delimiters=False)


@click.command()
@click.option("--account_id", required=True)
@click.option("--role_name", required=True)
@click.option("--profile_name")
def save_assumed_role_credentials(account_id, role_name, profile_name):
    if profile_name is None:
        profile_name = account_id

    credentials = get_credentials(
        account_id=account_id,
        role_name=role_name
    )

    update_credentials_file(profile_name=profile_name, credentials=credentials)


if __name__ == "__main__":
    save_assumed_role_credentials()

A script for backing up Tumblr posts and likes

A few days ago, Tumblr announced some new content moderation policies that include the mass deletion of any posts deemed to contain “adult content”. (If you missed the news, the Verge has a good summary.)

If my dashboard and Twitter feed are anything to go by, this is bad news for Tumblr. Lots of people are getting flagged for innocuous posts, the timeline seems to be falling apart, and users are leaving in droves. The new policies don’t solve the site’s problems (porn bots, spam, child grooming, among others), but it hurts their marginalised users.

For all its faults, Tumblr was home to communities of sex workers, queer kids, fan artists, people with disabilities – and it gave many of them a positive, encouraging, empowering online space. I’m sad that those are going away.

Some people are leaving the site and deleting their posts as they go (rather than waiting for Tumblr do it for them). Totally understandable, but it leaves a hole in the Internet. A lot of that content just isn’t available anywhere else.

In theory you can export your posts with the official export tool, but I’ve heard mixed reports of its usefulness – I suspect it’s clogged up as lots of people try to leave. In the meantime, I’ve posted a couple of my scripts that I use for backing up my posts from Tumblr. It includes posts and likes, saves the full API responses, and optionally includes the media files (photos, videos, and so on).

They’re a bit scrappy – not properly tested or documented – but content is already being deleted by Tumblr and others, so getting it out quickly seemed more useful. If you use Tumblr, you might want to give them a look.

Keeping track of my book recommendations

I have a text file where I write down every book recommendation I receive. It has three lists:

  1. Personal recommendations. Anything recommended specifically to me – usually from somebody who knows my reading tastes, so there’s a good chance this will be a book I enjoy.

  2. General recommendations from friends. Any recommendations from somebody I trust, but not specifically to me – for example, somebody tweeting “I really enjoyed this book and you should all read it”. I enjoy a lot of the same books as my friends, but this is a softer recommendation. I feel less obliged to follow up on these.

  3. Everything else. Recommendations from retweets, strangers at parties, people I don’t know very well, stuff I saw while browsing in Waterstones, and so on. These are mostly valuable in the aggregate – a single recommendation for a book isn’t very useful, but knowing that six different people recommended it might be.

I’ve tried slicing in other ways – fiction and non-fiction, by author, genre, date, and so on – but sorting by quality of recommendation is the one that I keep going back to.

A lot of what I read comes from these lists. (I have similar lists for films and TV shows.) I’m not bound by the list, and a bit of spontaneity is helpful to avoid an echo chamber effect – but using it as a starting point means I know I’m likely to enjoy something before I pick it up.

As a bonus, having this list means that if I read something and enjoy it, I get to go back and thank the person who gave me the recommendation. Sometimes it’s years later, but better late than never!

David has been thinking a lot about reading on Twitter recently, and I wrote about my system in a reply. This blog post is the expanded version of that thought.

My visit to the Aberdulais Falls

In September, I was in Cardiff to help organise PyCon UK. I had a huge amount of fun at the conference, but running a five-day event gets quite tiring. This year I took a few days of extra holiday, so I could unwind before returning home. Unfortunately heavy storms kept me inside for several days, but I did venture out at the end of the week to the Aberdulais Falls.

Aberdulais is a village in south Wales, about 45 minutes drive from Cardiff, and a place with a long industrial history. It had easy supplies of coal and wood, and it sits atop a powerful river – the River Dulais. Today the former tin plate works are owned and managed by the National Trust, and I decided to go have a look.

Etymology note: the word Aberdulais is Welsh for mouth of the river Dulais. Aber is a Celtic prefix that appears in lots of place names. Well-known examples are places like Aberdeen and Aberystwyth.

In 1584, German engineer Ulrich Frosse had developed a new way to smelt copper, but he wanted to keep his process safe from “pryinge eyes”. The Welsh countryside is nice and quiet, so he set up a smelting works in Aberdulais – the first of its kind in Wales. The copper ore was mined in Cornwall, the coal and charcoal supplied from nearby Neath, and a waterwheel on the river powered the site. (If you’re interested, I found an 1880 lecture that gives more detail about the history of smelting in Wales.)

A sixteenth-century woodcut of copper smelting. Taken from Wikimedia Commons; public domain.

The National Trust site says the copper was used in coins minted for Queen Elizabeth I. I tried to find some pictures of the coins in question, but I couldn’t find not enough detail to pick them out. Based on dates, I think it would have been something like this gold pound, but that’s only a guess.

Over time, Aberdulais became a site of different industries – textile milling, cloth production, even a flour mill – then in 1831, it became the site of a tinplate works. Tin plating is the process of coating thin sheets of iron or steel with tin, so they don’t rust. Tin plate is used for things like cooking utensils and canned food, and versions of it are still in use today. Wales had an incredibly successful tin plating industry – so much so that the US slapped massive tariffs on it, and shut the whole thing down.

The National Trust site is based in the remains of one of the old tin plating works.

So what’s it like to visit?

Read more →

Finding SNS topics without any subscriptions

I make regular use of Amazon SNS when working with message queues in AWS.

SNS is a notification service. A user can send a notification to a topic. Each topic can have multiple subscribers, which receive a copy of every message sent to the topic – something like an HTTP endpoint, an email address, or an Amazon SQS queue. Sending a single notification can go to multiple places.

A common use case is something like push notifications on your phone. For example, when a game sends you a notification to tell you about new content – that could be powered by SNS.

We use SNS as an intermediary for SQS at work. Rather than sending a message directly to a queue, we send messages to an SNS topic, and the queue subscribes to the topic. Usually there’s a 1-to-1 relationship between topics and queues, but SNS is useful if we ever want to do some debugging or monitoring. We can create a second subscription to the topic, get a copy of the messages, and inspect them without breaking the original queue.

We’ve had a few bugs recently where the subscription between the SNS topic and SQS queue gets broken. When nothing subscribes to a topic, any notifications it receives are silently discarded – because there’s nowhere for them to be sent.

I wanted a way to detect if this had happened – do we have any topics without any subscribers?

You can see this information in the console, but it’s a little cumbersome. Anything more than a handful of topics becomes unwieldy, so I wrote a script. Normally I’d reach for Python, but I’m trying to learn some new languages, so I decided to write it in Go. I’ve only dabbled in Go, and this was a chance to write a useful program and test my Go skills.

In this post, I’ll explain how I wrote the script. Even if the Go isn’t very idiomatic, I hope it’s a useful insight into how I write this sort of thing, and what I’m learning as a Go novice.

Read more →

How do you hide a coin for 400 years?

As part of an upcoming blog post, I’ve been trawling the Internet for information about Elizabethan coins. Mostly for curiosity, as I know very little about old coins. Something I’ve learnt: historical coins are valuable. Five-figure prices aren’t unheard of, and I found one coin selling for nearly £100k:

A Triple Unite from the Oxford mint, a gold coin produced for Charles I in 1642. It was worth sixty shillings.

Which got me thinking: suppose you were an unscrupulous time traveller, and you wanted to make some extra cash. Going back in time, getting some coins, and then “discovering” them in the present day could be quite lucrative. But how do you do it in practice?

You can’t just bring the coins straight to the present. They’d be in much better condition than a coin that actually waited for 400 years, and carbon dating would be thrown off. Your coins would be derided as fakes, or at least prompt some tricky questions. You need to hide them somewhere in the past, and retrieve them in the present.

But where do you leave a seventeenth century coin so it’s still safe in 2018?

It’s certainly possible, if expensive – the Tower of London have been looking after the same jewels for centuries. But I don’t think it’s trivial, at least not without attracting some attention. It’s very tricky if you don’t have outside help. And the further back you go, the harder it becomes – imagine trying to save not coins, but dinosaur bones.

I wrote this as a late night musing, but while sleeping I realised it’s not just a theoretical problem. Nuclear power creates nuclear waste, and that waste has to go somewhere. Most plans involve putting it in a bunker, and sealing the bunker for at least 10,000 years – but how do you stop future humans exploring? How do you ensure nobody opens your radioactive death bunker?

Suggestions on the back of a postcard.

How to set the clock on a Horstmann Electronic 7 water heater

The clocks went back last night, which means changing the clock on my appliances. One of my few remaining appliances that has a clock but no Internet connection is the timer on my boiler. It’s a new boiler (I moved in June), so I’ve never had to set the clock on it before.

It turns out it selected the correct winter/summer setting itself, but it’s drifted by twenty minutes, so I decided to set it anyway.

This is the timer on my boiler:

My boiler is in a utility cupboard next to my front door. When I turn on the hot water boost, I put the post-it note on the cupboard door, so I don’t leave it on when I go out.

The name on the bottom right says “Horstmann Electronic 7”, so I did the obvious thing and googled “set clock horstmann electronic 7 boiler”.

It took me several minutes to find the answer – in the user guide for the timer – so for the sake of future!me and other Googlers, here are the instructions for setting the clock. Unless you have this exact appliance, you can stop reading.

Read more →

Custom 404 responses in Finatra

This post is a quick writeup of a problem I had to solve at work today. I’m writing it so I can find the information later, but if you don’t use Finatra, it’s unlikely to be of much interest.

Context

The Catalogue API we’ve built at Wellcome is a Finatra app. It usually returns JSON responses, including for errors – we have a custom error model. A few simplified examples:

$curl "https://api.wellcomecollection.org/catalogue/v2/works?query=fish"
{
  "type": "ResultList",
  "results": [
    {
      "id":  "bqzs9649",
      "title":  "A discourse of fish and fish-ponds, by the Hon. Roger North."
    },
    ...
  ]
}

$curl "https://api.wellcomecollection.org/catalogue/v2/works/bqzs9649"
{
  "id":  "bqzs9649",
  "title":  "A discourse of fish and fish-ponds, by the Hon. Roger North."
}

$curl "https://api.wellcomecollection.org/catalogue/v2/works/doesnotexist"
{
  "errorType": "http",
  "httpStatus": 404,
  "label": "Not Found",
  "description": "Work not found for identifier doesnotexist",
  "type": "Error"
}

The Finatra app is listening for the endpoint /catalogue/v2/works/:id. We’ve written a function that handles all requests to that endpoint, and inside that function, we can return our custom error model if the ID doesn’t exist.

What if you look at a different endpoint?

Over the weekend, I was poking around, and discovered that requests to unhandled endpoints would return an empty 404. For example:

$curl "https://api.wellcomecollection.org/catalogue/"
<404 response with an empty body>

We’d rather this endpoint returned instances of our custom error model. It took me a while to figure out how to do this – the Finatra docs weren’t so helpful – so I’m going to write down how I got it working.

You can see the pull request and tests on our public repo, but I’ll use an example app in this post to make it easier to follow.

Read more →

Open consultation on the Gender Recognition Act

In July, the UK Government launched a consultation on reforming the Gender Recognition Act 2004.

This is the law that lets people change their legal gender, but the current process isn’t as good as it could be. It’s complicated, slow, and expensive – and completely ignores non-binary people and anybody under 18. If you’re married, you can’t change your legal gender without permission from your spouse. This consultation is an opportunity to improve the process!

There’s a good overview of the current issues on GenderBen, and the consultation itself also includes lots of detail.

You can fill in the consultation on the Government website:

Open consultation: Reform of the Gender Recognition Act 2004 – GOV.UK

If you live in the UK and you haven’t already, please consider taking the time to respond. Tell the Government that you support trans rights, and that legal protection shouldn’t require such excessive gatekeeping hoops. There’s probably going to be a strong transphobic response (such as from the protestors at London Pride), so countering that with positive responses from trans allies is very welcome!

It took me about 30 minutes to finish. If you can’t finish it in one go, you can save your progress and come back to it later – but don’t delay! The consultation closes at 11pm on 19 October, in just over a week.

Content warnings

Content warning in the first section for sexual assault and rape – description of a graphic image, and discussion of recent US politics. If those topics are upsetting for you, consider skipping.

There’s a cartoon of Lady Justice going around Twitter. I won’t link to it here, but it shows her being pinned down, and the implication is that she’s about to be (or being) raped. The cartoon is about this week’s confirmation hearings for Brett Kavanaugh. It’s in part a reference to specific events described in Christine Blasey Ford’s testimony, in part the Republican decision to ignore her testimony and advance his nomination anyway.

In a week where many survivors/victims of sexual assault and rape have been forced to relive traumatic experiences, this cartoon makes me deeply uncomfortable. It touches an already raw nerve, especially when shared without content warnings.

What’s a content warning?

A content warning is a note about potentially sensitive topics – something a reader might find upsetting or traumatic. Just as you’d warn about a risk to somebody’s physical health, so you can warn about something that might affect their mental health. With a warning, a reader can make sure they’re prepared (and safe) to read the content, or choose to skip over it.

There’s an example at the top of this post. When characters are limited, like on Twitter, it’s sometimes shortened to “CW”. Here’s another example:

CW death

*talks about death, links to a news article that mentions death*

Here’s a list of common topics I try to warn for:

I might include an extra warning, or be more specific, if I know more detail about somebody in my audience. For example, several of my friends have traumas around parental death, so I’ll use a more specific content warning when appropriate. In general, more detail is usually better (as long as the warning itself doesn’t become upsetting!).

If you’re writing or talking about topics that some people could find distressing, I’d encourage you to learn about content warnings, and use them!

(This post is an expanded version of a Twitter thread.)