Tags » aws » amazon s3

• 

  Large things living in cold places

  Using cold storage tiers to reduce the cost of storing Wellcome’s digital collections in the cloud.

• 

  Illustrating lifecycle transitions in Amazon S3

  A picture speaks a thousand words, which is why I always have pen and paper to hand.

• s3tree: viewing a tree of objects in S3 in my terminal

  A script to give me a quick overview of some objects in a hierarchical view.

• A Python function to iterate through an S3 Bucket Inventory

  Getting something that looks more like the output of the ListObjectsV2 API.

• Downloading objects from/uploading files to S3 with progress bars in Python

  Making it easier to see how long a file transfer will take, in the terminal.

• S3 prefixes are not directories

  Although an S3 prefix looks a lot like a directory path, they aren’t the same. Whether or not you include a trailing slash can change the behaviour.

• S3 keys are not file paths

  Although an S3 key looks a lot like a file path, they aren’t always the same, and the distinction can trip you up.

• A deletion canary: testing your S3 bucket permissions

  If you’ve tried to disable deletions in your S3 buckets, how do you know they’re working?

• Streaming large objects from S3 with ranged GET requests

  Reliably reading a large object by stitching together multiple GetObject requests into a single Java InputStream.

• Listing even more keys in an S3 bucket with Python

  Python functions for getting a list of keys and objects in an S3 bucket.

• Working with really large objects in S3

  Code for processing large objects in S3 without downloading the whole thing first, using file-like objects in Python.

• Creating a data store from S3 and DynamoDB

  A new storage layer for large records in the Catalogue pipeline.

• Two shortcuts for using S3 in the shell

  Two shell functions for editing and inspecting S3 objects as if they were local files.

• Listing keys in an S3 bucket with Python, redux

  Python functions for getting a list of keys and objects in an S3 bucket.

• Listing keys in an S3 bucket with Python

  A short Python function for getting a list of keys in an S3 bucket.