Tags » aws

Tag your infrastructure-as-code resources with a link to their definitions

Applying a default tag that points to the IaC definition makes it easy to go from the console to the code.

Finding a mystery IAM access key

Using the GetAccessKeyInfo and GetAccessKeyLastUsed APIs can help us trace an IAM key back to its source.

Publishing lots and lots of messages to SNS

Careful use of the PublishBatch API makes it quick and easy for me to send thousands of messages into SNS.

Getting alerts about flaky ECS tasks in Slack

When ECS is “unable to consistently start tasks successfully”, we get a Slack alert that tells us to investigate.

s3tree: viewing a tree of objects in S3 in my terminal

A script to give me a quick overview of some objects in a hierarchical view.

Parsing CloudFront logs with Python

A couple of functions I use to get access to CloudFront logs as easy-to-use iterators.

A Python function to iterate through an S3 Bucket Inventory

Getting something that looks more like the output of the ListObjectsV2 API.

Working with really large objects in S3

Code for processing large objects in S3 without downloading the whole thing first, using file-like objects in Python.

An AWS costs graph that works for me

How I get a Cost Explorer graph for the last 30 days of spending, broken down by service.

Streaming large objects from S3 with ranged GET requests

Reliably reading a large object by stitching together multiple GetObject requests into a single Java InputStream.

Listing even more keys in an S3 bucket with Python

Python functions for getting a list of keys and objects in an S3 bucket.

November 2019 scripts: downloading podcasts, retrying flaky errors, Azure and AWS

What comes after AWS?

Whatever displaces public cloud as the default model for large-scale computing has to be more than “AWS, but 3% better”.

Creating short-lived, temporary roles for experimenting with AWS IAM policy documents

Create short-lived, temporary roles for experimenting with AWS IAM policy documents

S3 keys are not file paths

Although an S3 key looks a lot like a file path, they aren’t always the same, and the distinction can trip you up.

S3 prefixes are not directories

Although an S3 prefix looks a lot like a directory path, they aren’t the same. Whether or not you include a trailing slash can change the behaviour.

Getting every item from a DynamoDB table with Python

A Python function that generates every item in a DynamoDB table.

A deletion canary: testing your S3 bucket permissions

If you’ve tried to disable deletions in your S3 buckets, how do you know they’re working?

Finding the CPU and memory bottlenecks in an ECS cluster

Two Python functions for getting CloudTrail events

Moving messages between SQS queues

You can send messages to a DLQ if they fail processing. What if you fix the bug, and you want to resend the failed messages?

A script to copy images from Docker Hub to Amazon ECR

Getting credentials for an assumed IAM Role

A script that creates temporary credentials for an assumed IAM role, and stores them in ~/.aws/credentials.

Implementing parallel scan in DynamoDB with Scanamo

Prototype code for running a parallel scan against a DynamoDB table, and using Scanamo to serialise rows as Scala case classes.

Two shortcuts for using S3 in the shell

Two shell functions for editing and inspecting S3 objects as if they were local files.

Listing keys in an S3 bucket with Python, redux

Python functions for getting a list of keys and objects in an S3 bucket.

Getting every message in an SQS queue

Code for saving every message from an SQS queue, and then saving the messages to a file, or resending them to another queue.

Finding SNS topics without any subscriptions

I’m trying out Go, and I wrote a tool to help me find SNS topics that don’t have any subscriptions.

A basic error logger for Python Lambdas

A snippet to make it a bit easier to debug errors in AWS Lambda functions written in Python.

Listing keys in an S3 bucket with Python

A short Python function for getting a list of keys in an S3 bucket.

Downloading logs from Amazon CloudWatch

A detailed breakdown of how I wrote a Python script to download logs from CloudWatch.

Finding misconfigured or dangling CloudWatch Alarms

A Python script that finds CloudWatch Alarms which are based on a now non-existent source.

Downloading objects from/uploading files to S3 with progress bars in Python

Making it easier to see how long a file transfer will take, in the terminal.

Getting a monthly cloud costs report in Slack

Sending the AWS bill to Slack, so everyone can be more informed and intentional about spending.

When is my EventBridge cron expression going to run next?

The AWS console will tell you when your EventBridge rule is going to run… if you know where to look.

Listing deleted secrets in AWS Secrets Manager with boto3 and the AWS CLI

Diving into the internals of the AWS SDK to find deleted secrets.

Screaming in the Cloud: Using the Cloud to Preserve the Future

I joined Corey Quinn to discuss my DynamoDB calculator and using the cloud to preserve digital collections.

Beware delays in SQS metric delivery

A mysterious problem with SQS-based autoscaling and an over-eager CloudWatch Alarm.

Why is Amazon Route 53 named that way?

Digging into the history of Route 53, DNS, and port number assignments.

Experimenting with jq as a tool for filtering JSON

I wanted to learn jq’s more powerful features, so I tried to filter some JSON from the AWS Secrets Manager CLI.

Adding a README to S3 buckets with Terraform

If you create an S3 bucket in Terraform, you can also create a README to help a future sysadmin understand what the bucket is for.

TIL: DynamoDB: Conditional updates on nested fields

TIL: Getting a boto3 Session for an IAM role using Python

Why I use Sessions in boto3, and the Python function I use to create them.