When I’m writing scripts, I often have some tabular data that I need to present. This data might show the number of website hits per day, or which pages had the most errors. Here’s an example of the sort of tabular data I mean:
data = [
('Clare', 64),
('Donegal', 48),
('Mayo', 57),
('Meath', 67),
('Offaly', 58),
('Tipperary', 59),
('Wicklow', 74),
]
I want to print it in a way that’s easy for me to read, and makes the trends stand out. It’s hard to get a sense of the overall picture without reading the individual numbers – I’d much rather have a bar chart.
If I was being fancy, I could use matplotlib and draw a graphical chart – but if I’m running a shell script in the terminal, it’s not always convenient to display images. I’d rather print something I can view in my terminal – an ASCII bar chart.
There are probably libraries that can do this for you, but I found it simpler to write my own snippet to draw bar charts.
Read more →
◇
Last week was Google I/O, and there was a lot of discussion around one of their keynote demos: Google Duplex. This is a service that acts like a human, and makes phone calls on your behalf. You can watch the demos on YouTube – one has Duplex booking a haircut, another trying to book a table at a restaurant.
From a technical perspective, I think it’s very impressive. I still have memories of primitive speech-to-text voices, so the vocal quality of that demo and the understanding of the other person and the near-instant responses feels very futuristic.
But I’ve heard people dismissing it as a toy for rich lazy people, and that feels a bit ableist to me.
Lots of people have trouble with phone calls, for a variety of reasons. Maybe hearing is difficult. Perhaps they can’t speak, or they have speech difficulties or an accent that make it hard to be understood. Or they have anxiety talking to strangers on the phone, or waiting on hold uses energy they don’t have.
Giving those people a way to use phone trees/voice-only interfaces? That could be a great step forward for accessibility.
Calling it “lazy” is like shaming somebody for not using the stairs, or for buying pre-cut fruit and veg. You might not need it, but maybe they do.
I’m not somebody who needs this, but I feel icky seeing people so quick to pass judgement.
◇
Yesterday, Twitter wrote a blog post about a recent security bug:
When you set a password for your Twitter account, we use technology that masks it so no one at the company can see it. We recently identified a bug that stored passwords unmasked in an internal log. We have fixed the bug, and our investigation shows no indication of breach or misuse by anyone.
Quite by chance, I spent yesterday fixing a similar bug. I was a bit careless when using the subprocess module, and leaked some AWS credentials into a public CI log.
Read more →
◇
- Posted on 27 April 2018
- Tagged with #aws
I often find myself needing to edit or inspect the contents of a text file stored in S3.
For example, at work we have a Terraform variables file kept in a private S3 bucket. This contains configuration that we don’t want to put in a public repository – passwords, API credentials, usernames, and so on. If I want to add a new secret to this file, I need to download the existing file, make an edit, then re-upload the file under the same key. It isn’t hard, but it’s moderately tedious to do these steps manually.
Any time you have a repetitive and tedious task, it’s worth trying to find a way to automate it. To that end, I have a function in my shell config that simplifies the process of editing an text file in S3. The function is written for fish, but the concept could be adapted for any shell. It opens the file in my preferred text editor, which is TextMate (invoked with mate).
function s3mate
set s3key $argv[1]
set localname (basename $argv[1])
pushd (mktemp -d)
# Download the object from S3. Although we're in a temporary
# directory, give it a nice name for the sake of the editor.
aws s3 cp "$s3key" "$localname"
cp "$localname" "$localname.copy"
# Open the file in an editor. The '-w' flag to 'mate' means
# "wait until the file has closed before continuing".
mate -w "$localname"
# Is the file different to the original version? If so, save
# a new copy to S3.
cmp "$localname" "$localname.copy" >/dev/null
if [ "$status" != "0" ]
aws s3 cp "$localname" "$s3key"
end
popd
end
I call it from a shell by passing it an s3:// URI. For example:
$ s3mate s3://private-bucket/terraform.tfvars
This download the object terraform.tfvars from private-bucket into a temporary directory, and opens it in TextMate. I can edit the file as much as I like, then I save and close it. Once the file is closed, it checks to see if I’ve changed anything with cmp(1). If I’ve made changes, it uploads a new copy of the file to the original key.
If lots of people were editing this file at once, this approach wouldn’t be safe – I could download and start editing, and somebody else could change the file at the same time. When I uploaded my new version, I’d delete their changes. There’s no safe way to protect against this in S3 – it has no support for transactional updates. Even if you checked the object in S3 hadn’t changed before uploading, it could still change between the check and the upload.
In practice, it’s rare for me to work on a file that has multiple editors, so this isn’t an issue for me – but it is worth noting.
Once I had this function, it was only a small tweak to get a version that inspects files, but doesn’t edit them. Viz:
function s3cat
set s3key $argv[1]
set localname (basename $argv[1])
pushd (mktemp -d)
aws s3 cp "$s3key" "$localname"
cat "$localname"
popd
end
The name is to match cat(1).
I’ve had both of these in my shell config for a while now, and they’ve been quite useful. Neither is a massive time saver, but they save me a few seconds each time I use them.
S3 isn’t actually a local filesystem, and it’s risky to treat it as such. But for quick file edits, these functions help paper over the difference.
◇
- Posted on 18 April 2018
- Tagged with , #slides
This is a talk I gave today for students on Bournemouth University’s Cyber Security Management course. It’s loosely inspired by a talk about privilege and inclusion I gave at PyCon UK last year, focusing on a specific area – online harassment.
The idea is to discuss harassment, and how the design of online services can increase (or decrease) the risk to users. A common mantra is “imagine how an abusive ex will use your service” – this talk is the expanded version of that.
Here’s a brief outline:
What does online harassment look like? With specific examples: harassment, bullying, doxing, threats, and so on. Not everyone faces harassment to the same degree (or at all!), so I wanted to illustrate the sort of risks a user might face.
Threat models: why some groups are more at risk, and the sort of people we should worry about. The abusive ex is an important risk to consider, but who else?
What are some possible good practices? How can service operators reduce the risk to their users? Reviewing some common suggestions – things like blocking, shadow bans, restricting anonymity – what works and what doesn’t.
The aim isn’t to be a comprehensive resource, but to get students thinking about these risks. Harassment is a constantly moving target, and it’s better to anticipate them before they happen.
You can read the slides and notes on this page, or download the slides as a PDF. The notes are my lightly edited thoughts about what I was going to say with each slide – but they may not be exactly what I said on the day!
(Caveat: I didn’t quite finish writing up all the notes before the lecture. The PDF slides are the most up-to-date, and I’ll try to go back and update the inline notes soon.)
Content warning: this talk includes discussion of online harassment, misogyny, racism, suicide, domestic abuse, police violence, sexual violence and assault, rape threats and death threats.
Read more →
◇
A few months back, I tried making a change to the way I handle Instapaper: I added a 24-hour limit to my queue. If I haven’t read something within a day of saving it, I delete it without reading it.
I made the same rule for RSS, Twitter, GitHub issues, and so on. Dealt with in a day, or not at all.
Dealing with something could mean several things. It might be replying to an email, reading an Instapaper item, or it could mean moving it to my todo list. Once something goes on my todo list, it gets prioritised against everything else I have to do. Reading something important becomes the same as doing something important.
As part of this change, I started redirecting as many notifications as possible to email. I can work through all my messages in one go, not poke around in a dozen different apps and inboxes.
I’ve eliminated a (admittedly self-imposed) source of stress – I don’t have an ever-growing backlog to deal with or worry about. Most of this stuff just isn’t that important, and keeping it in check helps me focus on the messages I actually care about.
Overall, I’m very pleased with this change, and I’ll be keeping it.
◇
- Posted on 13 March 2018
- Tagged with #git
- ∞
On Tuesday, I ran my workshop A Plumber’s Guide to Git for the Cambridge Python User Group. I’ve also run it at PyCon UK and in my workplace. On all three occasions, it’s been very popular and I’ve heard people find it useful – on Tuesday, we actually ran out of space in the room! In an attempt to make it more accessible, I’ve written up the entire workshop and posted it on my site.
The aim of the workshop is to understand of the underlying concepts of Git. We learn exactly what happens in a typical Git workflow (add, branch, commit, and so on). We look inside the .git directory, and explain what Git stores internally to make those workflows happen.
If that sounds interesting to you, start reading the introduction, and continue from there. I’ve uploaded everything I use in the workshop – my notes, whiteboard sketches, and exercises. Typically it takes about two hours to complete. Enjoy!
◇
About a year ago, David built a powerful continuous release system for the hypothesis-python repo. If you push to master with a release note, our CI bumps the version, updates the changelog, tags a new version on GitHub and uploads a new release to PyPI. With no manual intervention.
This sort of change permanently ruins you. I’m now so used to continuous deployment, I get annoyed when I work on projects that don’t have it (and have thus copied it to several other repos).
If you’re interested, I wrote about the process on the Hypothesis blog – how it works, why we do it, and why we find it so useful.
◇
- Posted on 1 March 2018
- Tagged with #work
For the last few days, there’s been a lot of bad weather across the UK. The Beast from the East has caused snow, subzero temperatures, and problems on the transport network. Many trains have been delayed or cancelled, and hundreds of drivers have been trapped in their cars. The Met Office have even issued rare red snow warnings, meaning possible loss of life.
In such bad weather, the best thing to do is to avoid travelling. If you usually work in an office, it’s better to work from home or skip work entirely – or if you decide to go in, leave plenty of extra travel time. Sometimes you can’t avoid going to work, so here’s a bit of advice:
Keep an overnight bag in your workplace.
It doesn’t have to be large, but enough that you could sleep in or near your workplace if you really had to. For me, the limit is “could I walk home quickly (less than 15 minutes) in foul weather” – if not, I assume I might get stuck at the office one day, and prepare accordingly.
Is that fun? No. But in weeks like this, it may be safer than trying to make the journey home.
The sort of things you might want:
- Toiletries – I keep a toothbrush and razor; you’ll also want menstrual products if you need those
- Any regular/prescription medication you need
- Clean clothing
- A phone charger
- Glasses and/or a spare contact lens case, if you wear those
- Food and water, depending on how easily available they are near your workplace
Some of these are things I carry anyway – for example, I usually have paracetamol in my bag – but I like to have a ring-fenced supply just in case. Sometimes, you can venture out for supplies – but depending on the weather and where you work, that may not always be possible or safe. In both cases, it’s much easier to have it pre-packed.
I’ve had a bag like this for years; so far I’ve only used it once. During Storm Doris last year, all the train lines from London to Cambridge were knocked out, and I slept on the floor of my office. (It helps that we have showers!) And although I’ve never used it since, it’s been reassuring to know that it’s there if I need it.
This is the sort of thing you hope you never have to use, but when bad conditions strike, it’s really nice to have it handy.
◇
For the last year, I’ve been commuting from Cambridge to London. The office is near King’s Cross station, so on a fast train with no delays, it’s about 90 minutes each way – or about 15 hours of commuting a week. Turns out, that’s quite a lot!
About a month ago, I got home on Thursday evening, and I was exhausted. I’d had severe train delays that week, and I was crying tied. I worked from home on the Friday, and I finally realised that the amount of commuting I was doing was unsustainable for my long-term health. Something had to change.
With agreement from everyone at work, and encouragement from David and Camilla on Twitter, I’ve started working from home for one day a week. I have a nice home office, and my team is set up to support remote development. I’m very lucky – from a practical standpoint, it’s easy for me to work from home.
What are the benefits?
There are several reasons I want to try this.
I’ll spend less time commuting. Just the free time I get back from an extra day at home each week is significant. Over the rest of this year, I’ll get back five days of time. That’s time to do other things – cook a nice dinner, spend time with friends, work on a personal project – that I’m often too tired to do after three hours spent on trains.
It breaks up my week. I’m working from home on either Tuesdays or Thursdays, so I never have to spend more than three consecutive days on a commute. On the off days, I can get some extra sleep, and still start work at the same time.
It’s not as disruptive as moving to a new flat. I’m often asked “Why don’t you move to and/or closer to London?”, which would be another way to reduce my commute. But I’m happy in my current flat, and moving would be a big upheaval compared to a bit of remote working. Maybe I’ll be forced to move to London eventually, but I want to try this first.
I want to try regular remote working. All of my jobs so far have been on-site jobs in offices, but a lot of companies are work entirely remotely, with no office. The two lifestyles are very different, and I’ve never tried the latter. This is a way to dip my toe in the water without total immersion, so I can decide if I want to go fully remote in a future job.
Not being in an open office. Like a lot of places, Wellcome has an open office, which I know some people find harder to work in. I don’t usually feel like an open office is a problem for me, so the quiet working space is less of a benefit – but it’s another reason to consider working from home.
What are the risks?
Before I started, I knew this would make some things more difficult. These are the challenges I was more worried about upfront.
The impact on my mental health. When I’ve worked from home before, it’s often been at short notice – for example, when the trains to London aren’t running. These days consistently end with me feeling miserable. I think it’s a combination of the unpleasant surprise, and the loneliness of unexpectedly spending the day on my own.
I’m hoping that if I plan it in advance, I can avoid some of that. Being at home will no longer be an unpleasant surprise, and I can make plans to see people for lunch or in the evening.
Communication with the rest of the team. The rest of my team (currently) spend five days a week in the office, and we have a lot of in person conversations. It’s easy to keep up with what we’re all doing. When I’m in the office, I’ll lose that visibility and the casual conversations. We’ll need to make more use of tools like email, Slack, and GitHub to stay in touch.
I don’t end up taking the day at home. Right now, I don’t have a fixed day to work from home; I’m choosing it on a week-by-week basis. Because it’s not a regular fixture in my calendar, there’s a chance I may end up skipping it, which defeats the point – I need to be disciplined about booking in that day, and following through with it.
So far
I’ve been doing this for a couple of weeks, and already I feel much better. I’m less tired at the end of the week, and I have less dread about commuting on the days when I do. If you can work from home for a day or so every week, I’d recommend giving it a try.
As with all big changes, this probably came three to six months after I really needed it, but I’m glad I’m doing it now.