Tags » aws

• 

  Large things living in cold places

  Using cold storage tiers to reduce the cost of storing Wellcome’s digital collections in the cloud.

• 

  Illustrating lifecycle transitions in Amazon S3

  A picture speaks a thousand words, which is why I always have pen and paper to hand.

• 

  Using DynamoDB as a calculator

  Taking advantage of Amazon’s least-loved compute platform.

• What comes after AWS?

  Whatever displaces public cloud as the default model for large-scale computing has to be more than “AWS, but 3% better”.

• TIL: Getting a boto3 Session for an IAM role using Python

  Why I use Sessions in boto3, and the Python function I use to create them.

• Finding a mystery IAM access key

  Using the GetAccessKeyInfo and GetAccessKeyLastUsed APIs can help us trace an IAM key back to its source.

• Tag your infrastructure-as-code resources with a link to their definitions

  Applying a default tag that points to the IaC definition makes it easy to go from the console to the code.

• Parsing CloudFront logs with Python

  A couple of functions I use to get access to CloudFront logs as easy-to-use iterators.

• s3tree: viewing a tree of objects in S3 in my terminal

  A script to give me a quick overview of some objects in a hierarchical view.

• Getting alerts about flaky ECS tasks in Slack

  When ECS is “unable to consistently start tasks successfully”, we get a Slack alert that tells us to investigate.

• Publishing lots and lots of messages to SNS

  Careful use of the PublishBatch API makes it quick and easy for me to send thousands of messages into SNS.

• A Python function to iterate through an S3 Bucket Inventory

  Getting something that looks more like the output of the ListObjectsV2 API.

• Experimenting with jq as a tool for filtering JSON

  I wanted to learn jq’s more powerful features, so I tried to filter some JSON from the AWS Secrets Manager CLI.

• Beware delays in SQS metric delivery

  A mysterious problem with SQS-based autoscaling and an over-eager CloudWatch Alarm.

• Why is Amazon Route 53 named that way?

  Digging into the history of Route 53, DNS, and port number assignments.

• Getting a monthly cloud costs report in Slack

  Sending the AWS bill to Slack, so everyone can be more informed and intentional about spending.

• When is my EventBridge cron expression going to run next?

  The AWS console will tell you when your EventBridge rule is going to run… if you know where to look.

• Finding misconfigured or dangling CloudWatch Alarms

  A Python script that finds CloudWatch Alarms which are based on a now non-existent source.

• Listing deleted secrets in AWS Secrets Manager with boto3 and the AWS CLI

  Diving into the internals of the AWS SDK to find deleted secrets.

• Downloading objects from/uploading files to S3 with progress bars in Python

  Making it easier to see how long a file transfer will take, in the terminal.

• Screaming in the Cloud: Using the Cloud to Preserve the Future

  I joined Corey Quinn to discuss my DynamoDB calculator and using the cloud to preserve digital collections.

• Creating short-lived, temporary roles for experimenting with AWS IAM policy documents

  Create short-lived, temporary roles for experimenting with AWS IAM policy documents

• A script to copy images from Docker Hub to Amazon ECR

• Two Python functions for getting CloudTrail events

• S3 prefixes are not directories

  Although an S3 prefix looks a lot like a directory path, they aren’t the same. Whether or not you include a trailing slash can change the behaviour.

• S3 keys are not file paths

  Although an S3 key looks a lot like a file path, they aren’t always the same, and the distinction can trip you up.

• Getting every item from a DynamoDB table with Python

  A Python function that generates every item in a DynamoDB table.

• Moving messages between SQS queues

  You can send messages to a DLQ if they fail processing. What if you fix the bug, and you want to resend the failed messages?

• A deletion canary: testing your S3 bucket permissions

  If you’ve tried to disable deletions in your S3 buckets, how do you know they’re working?

• Finding the CPU and memory bottlenecks in an ECS cluster

• November 2019 scripts: downloading podcasts, retrying flaky errors, Azure and AWS

• An AWS costs graph that works for me

  How I get a Cost Explorer graph for the last 30 days of spending, broken down by service.

• Streaming large objects from S3 with ranged GET requests

  Reliably reading a large object by stitching together multiple GetObject requests into a single Java InputStream.

• Listing even more keys in an S3 bucket with Python

  Python functions for getting a list of keys and objects in an S3 bucket.

• TIL: DynamoDB: Conditional updates on nested fields

• Working with really large objects in S3

  Code for processing large objects in S3 without downloading the whole thing first, using file-like objects in Python.

• Getting credentials for an assumed IAM Role

  A script that creates temporary credentials for an assumed IAM role, and stores them in ~/.aws/credentials.

• Finding SNS topics without any subscriptions

  I’m trying out Go, and I wrote a tool to help me find SNS topics that don’t have any subscriptions.

• A basic error logger for Python Lambdas

  A snippet to make it a bit easier to debug errors in AWS Lambda functions written in Python.

• Implementing parallel scan in DynamoDB with Scanamo

  Prototype code for running a parallel scan against a DynamoDB table, and using Scanamo to serialise rows as Scala case classes.

• Two shortcuts for using S3 in the shell

  Two shell functions for editing and inspecting S3 objects as if they were local files.

• Getting every message in an SQS queue

  Code for saving every message from an SQS queue, and then saving the messages to a file, or resending them to another queue.

• Listing keys in an S3 bucket with Python, redux

  Python functions for getting a list of keys and objects in an S3 bucket.

• Downloading logs from Amazon CloudWatch

  A detailed breakdown of how I wrote a Python script to download logs from CloudWatch.

• Listing keys in an S3 bucket with Python

  A short Python function for getting a list of keys in an S3 bucket.