Tags » aws

Adding a README to S3 buckets with Terraform

If you create an S3 bucket in Terraform, you can also create a README to help a future sysadmin understand what the bucket is for.

What comes after AWS?

Whatever displaces public cloud as the default model for large-scale computing has to be more than “AWS, but 3% better”.

TIL: Getting a boto3 Session for an IAM role using Python

Why I use Sessions in boto3, and the Python function I use to create them.

Finding a mystery IAM access key

Using the GetAccessKeyInfo and GetAccessKeyLastUsed APIs can help us trace an IAM key back to its source.

Tag your infrastructure-as-code resources with a link to their definitions

Applying a default tag that points to the IaC definition makes it easy to go from the console to the code.

Parsing CloudFront logs with Python

A couple of functions I use to get access to CloudFront logs as easy-to-use iterators.

s3tree: viewing a tree of objects in S3 in my terminal

A script to give me a quick overview of some objects in a hierarchical view.

Getting alerts about flaky ECS tasks in Slack

When ECS is “unable to consistently start tasks successfully”, we get a Slack alert that tells us to investigate.

Publishing lots and lots of messages to SNS

Careful use of the PublishBatch API makes it quick and easy for me to send thousands of messages into SNS.

A Python function to iterate through an S3 Bucket Inventory

Getting something that looks more like the output of the ListObjectsV2 API.

Experimenting with jq as a tool for filtering JSON

I wanted to learn jq’s more powerful features, so I tried to filter some JSON from the AWS Secrets Manager CLI.

Beware delays in SQS metric delivery

A mysterious problem with SQS-based autoscaling and an over-eager CloudWatch Alarm.

Why is Amazon Route 53 named that way?

Digging into the history of Route 53, DNS, and port number assignments.

Getting a monthly cloud costs report in Slack

Sending the AWS bill to Slack, so everyone can be more informed and intentional about spending.

When is my EventBridge cron expression going to run next?

The AWS console will tell you when your EventBridge rule is going to run… if you know where to look.

Finding misconfigured or dangling CloudWatch Alarms

A Python script that finds CloudWatch Alarms which are based on a now non-existent source.

Listing deleted secrets in AWS Secrets Manager with boto3 and the AWS CLI

Diving into the internals of the AWS SDK to find deleted secrets.

Downloading objects from/uploading files to S3 with progress bars in Python

Making it easier to see how long a file transfer will take, in the terminal.

Screaming in the Cloud: Using the Cloud to Preserve the Future

I joined Corey Quinn to discuss my DynamoDB calculator and using the cloud to preserve digital collections.

Creating short-lived, temporary roles for experimenting with AWS IAM policy documents

Create short-lived, temporary roles for experimenting with AWS IAM policy documents

A script to copy images from Docker Hub to Amazon ECR

Two Python functions for getting CloudTrail events

S3 prefixes are not directories

Although an S3 prefix looks a lot like a directory path, they aren’t the same. Whether or not you include a trailing slash can change the behaviour.

S3 keys are not file paths

Although an S3 key looks a lot like a file path, they aren’t always the same, and the distinction can trip you up.

Getting every item from a DynamoDB table with Python

A Python function that generates every item in a DynamoDB table.

Moving messages between SQS queues

You can send messages to a DLQ if they fail processing. What if you fix the bug, and you want to resend the failed messages?

A deletion canary: testing your S3 bucket permissions

If you’ve tried to disable deletions in your S3 buckets, how do you know they’re working?

Finding the CPU and memory bottlenecks in an ECS cluster

November 2019 scripts: downloading podcasts, retrying flaky errors, Azure and AWS

An AWS costs graph that works for me

How I get a Cost Explorer graph for the last 30 days of spending, broken down by service.

Streaming large objects from S3 with ranged GET requests

Reliably reading a large object by stitching together multiple GetObject requests into a single Java InputStream.

Listing even more keys in an S3 bucket with Python

Python functions for getting a list of keys and objects in an S3 bucket.

TIL: DynamoDB: Conditional updates on nested fields

Working with really large objects in S3

Code for processing large objects in S3 without downloading the whole thing first, using file-like objects in Python.

Getting credentials for an assumed IAM Role

A script that creates temporary credentials for an assumed IAM role, and stores them in ~/.aws/credentials.

Finding SNS topics without any subscriptions

I’m trying out Go, and I wrote a tool to help me find SNS topics that don’t have any subscriptions.

A basic error logger for Python Lambdas

A snippet to make it a bit easier to debug errors in AWS Lambda functions written in Python.

Implementing parallel scan in DynamoDB with Scanamo

Prototype code for running a parallel scan against a DynamoDB table, and using Scanamo to serialise rows as Scala case classes.

Two shortcuts for using S3 in the shell

Two shell functions for editing and inspecting S3 objects as if they were local files.

Getting every message in an SQS queue

Code for saving every message from an SQS queue, and then saving the messages to a file, or resending them to another queue.

Listing keys in an S3 bucket with Python, redux

Python functions for getting a list of keys and objects in an S3 bucket.

Downloading logs from Amazon CloudWatch

A detailed breakdown of how I wrote a Python script to download logs from CloudWatch.

Listing keys in an S3 bucket with Python

A short Python function for getting a list of keys in an S3 bucket.