Tags » python

• 

  How KempisBot works

  How I reincarnated a fifteenth-century monk and taught him to use Twitter.

• 

  Using DynamoDB as a calculator

  Taking advantage of Amazon’s least-loved compute platform.

• 

  Adventures in Python with concurrent.futures

  Some examples of how I’ve been using concurrent.futures to speed up my batch scripting in Python.

• 

  Getting a tint colour from an image with Python and k-means

  I want to display text next to an image that looks visually similar to the image, so I need to extract a tint colour.

• TIL: What errors can you get from hyperlink.parse?

  Mostly you get hyperlink.URLParseError, but you can occasionally get a ValueError as well.

• TIL: With Flask-Login, you want current_user == None, not current_user is None

  current_user is a proxy object that happens to be wrapping None, but isn’t actually None.

• TIL: How to get the expiry date of an HTTPS certificate

• TIL: Write to the middle of a file with Python

  Open the file with mode r+ to be able to seek around the file and write to it.

• TIL: How to highlight Python console sessions in Jekyll

  Adding a couple of options to the console lexer (console?lang=python&prompt=>>>) gets you syntax highlighting for a Python console session.

• The surprising utility of a Flickr URL parser

  I made a library that knows how to read lots of different forms of Flickr.com URL, and I used hyperlink to do it.

• TIL: How to simulate an [Errno 54] Connection reset by peer when using pytest

  You can run a TCP server in the background using a fixture, and using the SO_LINGER socket option can reset the connection.

• TIL: How to see the HTTP requests being made by pywikibot

  To see exactly what HTTP requests were being made, I modified the library so that betamax would record requests.

• TIL: Checking if a URL has changed when you fetch it over HTTP

  When you make an HTTP request, you can use the If-Modified-Since header to get a 304 Not Modified if nothing has changed since your last request.

• TIL: Why is Pillow rotating my image when I save it?

  Images can have orientation specified in their EXIF metadata, which isn’t preserved when you open and save an image with Pillow.

• TIL: Getting a boto3 Session for an IAM role using Python

  Why I use Sessions in boto3, and the Python function I use to create them.

• Moving my YouTube Likes from one account to another

  Some experimenting with the YouTube API to merge two accounts into one.

• TIL: You need to call resp.close() to close the file opened by send_file()

• TIL: Custom error pages in Flask

  You can use app.error_handler to add custom responses for HTTP status codes, so the errors match the look and feel of the rest of the site.

• TIL: How to do offline geo-lookups of IP addresses

  MaxMind offer databases you can do to look up IP addresses without sending the address off to a remote service.

• TIL: How to gracefully restart a gunicorn app

• TIL: How to find the longest common suffix in a list of strings in Python

• TIL: How to simulate shell pipes with the subprocess module

• TIL: Manage MP3 metadata from iTunes with eyed3

• TIL: How to create flag emojis for countries in Python

• TIL: Use shlex.split() to parse log files quickly

• TIL: Use the regex library to get Unicode property escapes in Python

• TIL: Run a randomly selected subset of tests with pytest

  By reading the code for the pytest-random-order plugin, I was able to write a new plugin that runs a random subset of tests.

• TIL: Using sqlite-utils to convert a CSV into a SQLite database

  You can use sqlite-utils on the command line to create a SQLite database from a CSV file.

• TIL: Python’s sqlite3 context manager doesn’t close connections

  The sqlite3.connect(…) context manager will hold connections open, so you need to remember to close it manually or write your own context manager.

• Setting up Fish to make virtualenv easier

  I wrote some shell config to smooth out the experience of using virtual environments in Python.

• TIL: Telling mechanize how to find local issuer certificates

  Calling browser.set_ca_data(cafile=certifi.where()) will tell where mechanize can find some local SSL certificates.

• Spotting spam in our CloudFront logs

  Looking for search queries that came from robots, not real people.

• Adding locations to my photos from my Apple Watch workouts

  My Apple Watch knows where I am, which is handy when I have a camera that doesn’t.

• Parsing CloudFront logs with Python

  A couple of functions I use to get access to CloudFront logs as easy-to-use iterators.

• My Python snippet for walking a file tree

  A function to find all the files in a directory is one of my most-used snippets.

• Creating a Python dictionary with multiple, equivalent keys

  Using collections.UserDict, we can create a dictionary where dict[key1] and dict[key2] always point to the same value.

• Splitting a class into balanced groups

  How do you make sure everyone gets to work with everyone else?

• A Python function to iterate through an S3 Bucket Inventory

  Getting something that looks more like the output of the ListObjectsV2 API.

• I always want StrictUndefined in Jinja

  When I’m writing templates with Jinja, strict behaviour is what I want, even if it’s not the default.

• Creating animated GIFs from fruit and veg

  Some Python code for turning MRI scans of fruit and veg into animated GIFs.

• Why is os.sep insufficient for path operations?

  Digging into a throwaway comment in the Python documentation.

• Picking perfect planks with Python

  How do you pick the right combination of planks to lay a wooden floor? Python and itertools to the rescue!

• How to ignore lots of folders in Spotlight

  A script that allows me to ignore folders like “target” and “node_modules”, so they don’t appear in search results.

• Finding misconfigured or dangling CloudWatch Alarms

  A Python script that finds CloudWatch Alarms which are based on a now non-existent source.

• Listing deleted secrets in AWS Secrets Manager with boto3 and the AWS CLI

  Diving into the internals of the AWS SDK to find deleted secrets.

• Visualising how often I write in my journal

  A Python script that shows me how often I’ve been journalling, so I can track my progress.

• Downloading objects from/uploading files to S3 with progress bars in Python

  Making it easier to see how long a file transfer will take, in the terminal.

• Drawing coloured squares/text in my terminal with Python

• A Python function to ignore a path with .git/info/exclude

  If your Python script creates a file that you don’t want to track in Git, here’s how you can ignore it.

• Two Python functions for getting CloudTrail events

• Using fuzzy string matching to find duplicate tags

• Finding the months between two dates in Python

• Getting every item from a DynamoDB table with Python

  A Python function that generates every item in a DynamoDB table.

• Downloading the AO3 fics that I’ve saved in Pinboard

  A script that downloads the nicely formatted AO3 downloads for everything I’ve saved in Pinboard.

• Taking tuple unpacking to terrible places

  I want to assign a bunch of variables to True, but I don’t know how many there are. Reflection to the rescue!

• A snippet for downloading files with Python

• Storing language vocabulary as a graph

  Experimenting with a way to store words and phrases that highlights the connections between them.

• Creating striped flag wallpapers with Pillow

• Adjusting the dominant colour of an image

  Adjusting the hue to get different colour variants of the same image.

• Generating pride-themed Norse valknuts with Python 🌈

  A web app to generate mashups of Norse valknuts and Pride flags.

• This YAML file will self-destruct in five seconds!

  YAML allows you to execute arbitrary code in a parser, even if you really really shouldn’t.

• November 2019 scripts: downloading podcasts, retrying flaky errors, Azure and AWS

• Saving a copy of a tweet by typing ;twurl

• The rough edges of filecmp

  The filecmp module has a confusing API, and it just caught me out.

• Finding divisors of a number with Python

  Using unique prime factorisations and itertools to find all the divisors of a number.

• Listing even more keys in an S3 bucket with Python

  Python functions for getting a list of keys and objects in an S3 bucket.

• Atomic, cross-filesystem moves in Python

  Explaining some code for moving files around in a way that’s atomic and works across filesystem boundaries.

• Working with really large objects in S3

  Code for processing large objects in S3 without downloading the whole thing first, using file-like objects in Python.

• Notes on reading a UTF-8 encoded CSV in Python

  Some notes on trying to do this in a way that supports both Python 2 and 3, and the frustration of doing so.

• Iterating in fixed-size chunks in Python

  A snippet for iterating over an arbitrary iterable in chunks, and returning a smaller chunk if the boundaries don’t line up.

• Getting credentials for an assumed IAM Role

  A script that creates temporary credentials for an assumed IAM role, and stores them in ~/.aws/credentials.

• TIL: Create compact JSON with Python

  Calling json.dumps(…, separators=(',', ':')) will reduce the amount of whitespace in the final string.

• A basic error logger for Python Lambdas

  A snippet to make it a bit easier to debug errors in AWS Lambda functions written in Python.

• Drawing ASCII bar charts

  A Python snippets for drawing bar charts in command-line applications.

• Beware of logged errors from subprocess

  If you use Python’s subprocess module, be careful you don’t leak sensitive information in your error logs.

• Getting every message in an SQS queue

  Code for saving every message from an SQS queue, and then saving the messages to a file, or resending them to another queue.

• Listing keys in an S3 bucket with Python, redux

  Python functions for getting a list of keys and objects in an S3 bucket.

• Downloading logs from Amazon CloudWatch

  A detailed breakdown of how I wrote a Python script to download logs from CloudWatch.

• Using hooks for custom behaviour in requests

  I often have code I want to run against every HTTP response (logging, error checking) — event hooks give me a nice way to do that without repetition.

• Using pip-tools to manage my Python dependencies

  How I use pip-tools to ensure my Python dependencies are pinned, precise, and as minimal as possible.

• Ode to docopt

  Why I love docopt as a tool for writing clean, simple command-line interfaces.

• A Python module for lazy reading of file objects

  I wrote a small Python module for lazy file reading, ideal for efficient batch processing.

• Listing keys in an S3 bucket with Python

  A short Python function for getting a list of keys in an S3 bucket.

• A Python interface to AO3

  AO3 doesn’t have an official API for scraping data - but with a bit of Python, it might not be necessary.

• Experiments with AO3 and Python

  AO3 doesn’t have an official API for scraping data - but with a bit of Python, it might not be necessary.

• TIL: Why does Hypothesis try the same example three times before failing?

• Another example of why strings are terrible

  Pop quiz: if I lowercase a string, does it still have the same length as the original string?

• Use keyring to store your credentials

  If you need to store passwords in a Python application, use the keyring module to keep them safe.

• Creating low contrast wallpapers with Pillow

  Take a regular tiling of the plane, apply a random colouring, and voila: a unique wallpaper, courtesy of the Python Imaging Library.

• Tiling the plane with Pillow

  Using the Python Imaging Library to draw regular tilings of squares, triangles and hexagons.

• Why I use py.test

  Why py.test is my unit test framework of choice in Python.

• Python snippets: Cleaning up empty/nearly empty directories

  A pair of Python scripts I’ve been using to clean up my mess of directories.

• Python snippets: Chasing redirects and URL shorteners

  A quick Python function to follow a redirect to its eventual conclusion.

• Reading web pages on my Kindle

  A Python script I wrote that let me sends web pages from my Mac and my iPhone to my Kindle.

• Introduction to property-based testing

  Testing with randomly generated examples can be a good way to uncover bugs in your code.

• Finding 404s and broken pages in my Apache logs

  A Python script for finding 404 errors in my Apache web logs - and by extension, broken pages.

• A Python smtplib wrapper for Fastmail

  A quick python-smtplib wrapper for sending emails through Fastmail.

• Get images from the iTunes/App/Mac App Stores with Alfred

  Using Alfred and a Python script to retrieve artwork from the iTunes, App and Mac App Stores.

• Exclusively create a file in Python 3

  If you want to create a file, but only if it doesn’t already exist, Python 3 has a helpful new file mode x.

• Python and the BBC micro:bit

  Playing with a tiny computer that runs Python.

• Review: Effective Python

  A review of Effective Python, by Brett Slatkin.

• Persistent IPython notebooks in Windows

  Configuring an IPython notebook server that is always running and easily accessible in Windows.

• Safer file copying in Python

  A Python script for non-destructive file copying/moving.

• Pygmentizr

  A web app for applying syntax highlighting to code using the Pygments library.

• Acronyms

• Unpacking sets and ranges from a single string

• Playing with 404 pages

• Finding untagged posts on Tumblr

• Automatic Pinboard backups

  A script for automatically backing up bookmarks from Pinboard