amazon s3

Large things living in cold places

Using cold storage tiers to reduce the cost of storing Wellcome’s digital collections in the cloud.
Illustrating lifecycle transitions in Amazon S3

A picture speaks a thousand words, which is why I always have pen and paper to hand.

s3tree: viewing a tree of objects in S3 in my terminal

A script to give me a quick overview of some objects in a hierarchical view.
A Python function to iterate through an S3 Bucket Inventory

Getting something that looks more like the output of the ListObjectsV2 API.
Downloading objects from/uploading files to S3 with progress bars in Python

Making it easier to see how long a file transfer will take, in the terminal.
S3 prefixes are not directories

Although an S3 prefix looks a lot like a directory path, they aren’t the same. Whether or not you include a trailing slash can change the behaviour.
S3 keys are not file paths

Although an S3 key looks a lot like a file path, they aren’t always the same, and the distinction can trip you up.
A deletion canary: testing your S3 bucket permissions

If you’ve tried to disable deletions in your S3 buckets, how do you know they’re working?
Streaming large objects from S3 with ranged GET requests

Reliably reading a large object by stitching together multiple GetObject requests into a single Java InputStream.
Listing even more keys in an S3 bucket with Python

Python functions for getting a list of keys and objects in an S3 bucket.
Working with really large objects in S3

Code for processing large objects in S3 without downloading the whole thing first, using file-like objects in Python.
Two shortcuts for using S3 in the shell

Two shell functions for editing and inspecting S3 objects as if they were local files.
Listing keys in an S3 bucket with Python, redux

Python functions for getting a list of keys and objects in an S3 bucket.
Listing keys in an S3 bucket with Python

A short Python function for getting a list of keys in an S3 bucket.