s3tree: viewing a tree of objects in S3 in my terminal

Tagged with aws, aws:amazon s3, drawing things
Posted 17 May 2023
∞

I’ve been doing a bunch of work to clear up some old S3 buckets, and I wanted a quick way to see what’s in a given prefix or bucket. I found the console a bit slow, because you can only see one folder at a time, so I wrote a command-line tool that prints a tree in my terminal:

A screenshot of my terminal after running the command `s3tree s3://wellcomecollection-storage/digitised/b21466154`. It shows a hierarchical tree with several nested folders, and under each folder is a list of object names and then '...N other objects'. The object names are shown in blue; the folder names are underlined with a dashed line because they're hyperlinks.

This isn’t a complete listing – it just prints the first few objects in each folder, so I can get a sense of what’s in there. The objects I was looking at have a pretty regular naming structure, so that’s plenty.

It’s particularly useful when I have several layers of nested folders – this is much easier than clicking back and forth in the console.

There are a few neat tricks in here:

The folder names are clickable links to the S3 console, so I can dig into the contents in more detail. I found a Gist by Egmont Koblinger which describes how to do hyperlinks in terminal emulators, which made it easy to get this working – and I’ve already used this technique in a couple of other places.
I work with buckets in several AWS accounts, and normally I have to pick my credentials with an AWS_PROFILE environment variable or something similar. But this script looks at the bucket name, works out which account it’s in, then selects the right credentials.

(This is based on a manually populated mapping of bucket to account; I don’t think you can get the AWS account ID given an arbitrary bucket name.)
Although S3 object keys are sorted alphabetically, I’m using the naturalsort package to reorder the keys into natural ordering. This is particularly useful with some of the numeric-like keys I deal with; the zero padding isn’t always consistent. Using natural sorting makes the object summaries easier to skim.

If this might be useful to you, I’ve put the script on GitHub. I’m especially pleased with the tree-printing code, which is reused from another script and I expect to reuse again – a way to display hierarchical data in a terminal could have all sorts of uses.