Tags » aws

45 posts

Large things living in cold places
Using cold storage tiers to reduce the cost of storing Wellcome’s digital collections in the cloud.
Illustrating lifecycle transitions in Amazon S3
A picture speaks a thousand words, which is why I always have pen and paper to hand.
Using DynamoDB as a calculator
Taking advantage of Amazon’s least-loved compute platform.

What comes after AWS?
Whatever displaces public cloud as the default model for large-scale computing has to be more than “AWS, but 3% better”.
TIL: Getting a boto3 Session for an IAM role using Python
Why I use Sessions in boto3, and the Python function I use to create them.
Finding a mystery IAM access key
Using the GetAccessKeyInfo and GetAccessKeyLastUsed APIs can help us trace an IAM key back to its source.
Tag your infrastructure-as-code resources with a link to their definitions
Applying a default tag that points to the IaC definition makes it easy to go from the console to the code.
Parsing CloudFront logs with Python
A couple of functions I use to get access to CloudFront logs as easy-to-use iterators.
s3tree: viewing a tree of objects in S3 in my terminal
A script to give me a quick overview of some objects in a hierarchical view.
Getting alerts about flaky ECS tasks in Slack
When ECS is “unable to consistently start tasks successfully”, we get a Slack alert that tells us to investigate.
Publishing lots and lots of messages to SNS
Careful use of the PublishBatch API makes it quick and easy for me to send thousands of messages into SNS.
A Python function to iterate through an S3 Bucket Inventory
Getting something that looks more like the output of the ListObjectsV2 API.
Experimenting with jq as a tool for filtering JSON
I wanted to learn jq’s more powerful features, so I tried to filter some JSON from the AWS Secrets Manager CLI.
Beware delays in SQS metric delivery
A mysterious problem with SQS-based autoscaling and an over-eager CloudWatch Alarm.
Why is Amazon Route 53 named that way?
Digging into the history of Route 53, DNS, and port number assignments.
Getting a monthly cloud costs report in Slack
Sending the AWS bill to Slack, so everyone can be more informed and intentional about spending.
When is my EventBridge cron expression going to run next?
The AWS console will tell you when your EventBridge rule is going to run… if you know where to look.
Finding misconfigured or dangling CloudWatch Alarms
A Python script that finds CloudWatch Alarms which are based on a now non-existent source.
Listing deleted secrets in AWS Secrets Manager with boto3 and the AWS CLI
Diving into the internals of the AWS SDK to find deleted secrets.
Downloading objects from/uploading files to S3 with progress bars in Python
Making it easier to see how long a file transfer will take, in the terminal.
Screaming in the Cloud: Using the Cloud to Preserve the Future
I joined Corey Quinn to discuss my DynamoDB calculator and using the cloud to preserve digital collections.
Creating short-lived, temporary roles for experimenting with AWS IAM policy documents
Create short-lived, temporary roles for experimenting with AWS IAM policy documents
A script to copy images from Docker Hub to Amazon ECR
Two Python functions for getting CloudTrail events
S3 prefixes are not directories
Although an S3 prefix looks a lot like a directory path, they aren’t the same. Whether or not you include a trailing slash can change the behaviour.
S3 keys are not file paths
Although an S3 key looks a lot like a file path, they aren’t always the same, and the distinction can trip you up.
Getting every item from a DynamoDB table with Python
A Python function that generates every item in a DynamoDB table.
Moving messages between SQS queues
You can send messages to a DLQ if they fail processing. What if you fix the bug, and you want to resend the failed messages?
A deletion canary: testing your S3 bucket permissions
If you’ve tried to disable deletions in your S3 buckets, how do you know they’re working?
Finding the CPU and memory bottlenecks in an ECS cluster
November 2019 scripts: downloading podcasts, retrying flaky errors, Azure and AWS
An AWS costs graph that works for me
How I get a Cost Explorer graph for the last 30 days of spending, broken down by service.
Streaming large objects from S3 with ranged GET requests
Reliably reading a large object by stitching together multiple GetObject requests into a single Java InputStream.
Listing even more keys in an S3 bucket with Python
Python functions for getting a list of keys and objects in an S3 bucket.
TIL: DynamoDB: Conditional updates on nested fields
Working with really large objects in S3
Code for processing large objects in S3 without downloading the whole thing first, using file-like objects in Python.
Getting credentials for an assumed IAM Role
A script that creates temporary credentials for an assumed IAM role, and stores them in ~/.aws/credentials.
Finding SNS topics without any subscriptions
I’m trying out Go, and I wrote a tool to help me find SNS topics that don’t have any subscriptions.
A basic error logger for Python Lambdas
A snippet to make it a bit easier to debug errors in AWS Lambda functions written in Python.
Implementing parallel scan in DynamoDB with Scanamo
Prototype code for running a parallel scan against a DynamoDB table, and using Scanamo to serialise rows as Scala case classes.
Two shortcuts for using S3 in the shell
Two shell functions for editing and inspecting S3 objects as if they were local files.
Getting every message in an SQS queue
Code for saving every message from an SQS queue, and then saving the messages to a file, or resending them to another queue.
Listing keys in an S3 bucket with Python, redux
Python functions for getting a list of keys and objects in an S3 bucket.
Downloading logs from Amazon CloudWatch
A detailed breakdown of how I wrote a Python script to download logs from CloudWatch.
Listing keys in an S3 bucket with Python
A short Python function for getting a list of keys in an S3 bucket.