AWS
AWS is a collection of cloud services offered by Amazon, including virtual computers, databases, and storage. I first used AWS to build digital services when I worked at Wellcome Collection, and I also use it in my current job at Tailscale.
30 articles
Adding a README to S3 buckets with Terraform
If you create an S3 bucket in Terraform, you can also create a README to help a future sysadmin understand what the bucket is for.
What comes after AWS?
Whatever displaces public cloud as the default model for large-scale computing has to be more than “AWS, but 3% better”.
Spotting spam in our CloudFront logs
Looking for search queries that came from robots, not real people.
Finding a mystery IAM access key
Using the GetAccessKeyInfo and GetAccessKeyLastUsed APIs can help us trace an IAM key back to its source.
Publishing lots and lots of messages to SNS
Careful use of the
PublishBatchAPI makes it quick and easy for me to send thousands of messages into SNS.Filtering AWS CLI output by tags using jq
Using
from_entriesis a nicer way to deal with the list of Name/Value pairs returned by the AWS CLI.A Python function to iterate through an S3 Bucket Inventory
Getting something that looks more like the output of the ListObjectsV2 API.
Experimenting with jq as a tool for filtering JSON
I wanted to learn jq’s more powerful features, so I tried to filter some JSON from the AWS Secrets Manager CLI.
Beware delays in SQS metric delivery
A mysterious problem with SQS-based autoscaling and an over-eager CloudWatch Alarm.
When is my EventBridge cron expression going to run next?
The AWS console will tell you when your EventBridge rule is going to run… if you know where to look.
Finding misconfigured or dangling CloudWatch Alarms
A Python script that finds CloudWatch Alarms which are based on a now non-existent source.
Listing deleted secrets in AWS Secrets Manager with boto3 and the AWS CLI
Diving into the internals of the AWS SDK to find deleted secrets.
Downloading objects from/uploading files to S3 with progress bars in Python
Making it easier to see how long a file transfer will take, in the terminal.
Two Python functions for getting CloudTrail events
S3 prefixes are not directories
Although an S3 prefix looks a lot like a directory path, they aren’t the same. Whether or not you include a trailing slash can change the behaviour.
S3 keys are not file paths
Although an S3 key looks a lot like a file path, they aren’t always the same, and the distinction can trip you up.
Moving messages between SQS queues
You can send messages to a DLQ if they fail processing. What if you fix the bug, and you want to resend the failed messages?
A deletion canary: testing your S3 bucket permissions
If you’ve tried to disable deletions in your S3 buckets, how do you know they’re working?
Finding the CPU and memory bottlenecks in an ECS cluster
Streaming large objects from S3 with ranged GET requests
Reliably reading a large object by stitching together multiple GetObject requests into a single Java InputStream.
Listing even more keys in an S3 bucket with Python
Python functions for getting a list of keys and objects in an S3 bucket.
Working with really large objects in S3
Code for processing large objects in S3 without downloading the whole thing first, using file-like objects in Python.
Finding SNS topics without any subscriptions
I’m trying out Go, and I wrote a tool to help me find SNS topics that don’t have any subscriptions.
A basic error logger for Python Lambdas
A snippet to make it a bit easier to debug errors in AWS Lambda functions written in Python.
Implementing parallel scan in DynamoDB with Scanamo
Prototype code for running a parallel scan against a DynamoDB table, and using Scanamo to serialise rows as Scala case classes.
Creating a data store from S3 and DynamoDB
A new storage layer for large records in the Catalogue pipeline.
Two shortcuts for using S3 in the shell
Two shell functions for editing and inspecting S3 objects as if they were local files.
Getting every message in an SQS queue
Code for saving every message from an SQS queue, and then saving the messages to a file, or resending them to another queue.
Downloading logs from Amazon CloudWatch
A detailed breakdown of how I wrote a Python script to download logs from CloudWatch.
Listing keys in an S3 bucket with Python
A short Python function for getting a list of keys in an S3 bucket.
4 notes
The LastModified date of an S3 multipart upload is when the upload started, not when it finished
Using zipstream to stream new zip files to object storage with boto3
You can construct a
zipstream.ZipFile, add files, then wrap it in a file-like object to upload it withS3.upload_fileobj.Getting a boto3 Session for an IAM role using Python
Why I use Sessions in boto3, and the Python function I use to create them.
Conditional updates on nested fields in DynamoDB