Python

• 

  Using vcrpy to test HTTP interactions in Python

  How I record HTTP requests to get fast, reliable, and consistent tests, and the patterns I use in a production codebase.

• 

  Adventures in Python with concurrent.futures

  How I use concurrent.futures to speed up my batch scripting in Python.

44 articles

• Creating Caddyfiles with Cog

  Cog is a tool for doing in-place text generation for static files. It’s useful for generating repetitive config, like my web server redirects.

• Cleaning up messy dates in JSON

  I wrote a Python script to help me get timestamps in a consistent format in my JSON.

• Detecting AV1-encoded videos with Python

  I wrote a Python test to find videos that are encoded with AV1, so I can convert them to a codec my iPhone can play.

• Create space-saving clones on macOS with Python

  You can shell out to cp -c using subprocess, or you can make a clonefile() syscall using the ctypes library.

• Handling JSON objects with duplicate names in Python

  It’s possible, although uncommon, for a JSON object to contain the same name multiple times. Here are some ways to handle that in Python.

• Beyond None: actionable error messages for keyring.get_password()

  I have a wrapper for get_password() so that if it can’t find a password, you get an error that explains how to set the password, and what password you should choose.

• Setting up Fish to make virtualenv easier

  I wrote some shell config to smooth out the experience of using virtual environments in Python.

• Spotting spam in our CloudFront logs

  Looking for search queries that came from robots, not real people.

• My Python snippet for walking a file tree

  A function to find all the files in a directory is one of my most-used snippets.

• Creating a Python dictionary with multiple, equivalent keys

  Using collections.UserDict, we can create a dictionary where dict[key1] and dict[key2] always point to the same value.

• Splitting a class into balanced groups

  How do you make sure everyone gets to work with everyone else?

• A Python function to iterate through an S3 Bucket Inventory

  Getting something that looks more like the output of the ListObjectsV2 API.

• I always want StrictUndefined in Jinja

  When I’m writing templates with Jinja, strict behaviour is what I want, even if it’s not the default.

• Why is os.sep insufficient for path operations?

  Digging into a throwaway comment in the Python documentation.

• Picking perfect planks with Python

  How do you pick the right combination of planks to lay a wooden floor? Python and itertools to the rescue!

• Listing deleted secrets in AWS Secrets Manager with boto3 and the AWS CLI

  Diving into the internals of the AWS SDK to find deleted secrets.

• Visualising how often I write in my journal

  A Python script that shows me how often I’ve been journalling, so I can track my progress.

• Downloading objects from/uploading files to S3 with progress bars in Python

  Making it easier to see how long a file transfer will take, in the terminal.

• Two Python functions for getting CloudTrail events

• Finding the months between two dates in Python

• A snippet for downloading files with Python

• Adjusting the dominant colour of an image

  Adjusting the hue to get different colour variants of the same image.

• Saving a copy of a tweet by typing ;twurl

• The rough edges of filecmp

  The filecmp module has a confusing API, and it just caught me out.

• Listing even more keys in an S3 bucket with Python

  Python functions for getting a list of keys and objects in an S3 bucket.

• Atomic, cross-filesystem moves in Python

  Explaining some code for moving files around in a way that’s atomic and works across filesystem boundaries.

• Working with really large objects in S3

  Code for processing large objects in S3 without downloading the whole thing first, using file-like objects in Python.

• Notes on reading a UTF-8 encoded CSV in Python

  Some notes on trying to do this in a way that supports both Python 2 and 3, and the frustration of doing so.

• Iterating in fixed-size chunks in Python

  A snippet for iterating over an arbitrary iterable in chunks, and returning a smaller chunk if the boundaries don’t line up.

• A basic error logger for Python Lambdas

  A snippet to make it a bit easier to debug errors in AWS Lambda functions written in Python.

• Beware of logged errors from subprocess

  If you use Python’s subprocess module, be careful you don’t leak sensitive information in your error logs.

• Getting every message in an SQS queue

  Code for saving every message from an SQS queue, and then saving the messages to a file, or resending them to another queue.

• Using hooks for custom behaviour in requests

  I often have code I want to run against every HTTP response (logging, error checking) — event hooks give me a nice way to do that without repetition.

• A Python module for lazy reading of file objects

  I wrote a small Python module for lazy file reading, ideal for efficient batch processing.

• Listing keys in an S3 bucket with Python

  A short Python function for getting a list of keys in an S3 bucket.

• Tiling the plane with Pillow

  Using the Python Imaging Library to draw regular tilings of squares, triangles and hexagons.

• Python snippets: Cleaning up empty/nearly empty directories

  A pair of Python scripts I’ve been using to clean up my mess of directories.

• Exclusively create a file in Python 3

  If you want to create a file, but only if it doesn’t already exist, Python 3 has a helpful new file mode x.

• Export a list of URLs from Safari Reading List

  A Python script for getting a list of URLs from Safari Reading List.

• Safer file copying in Python

  A Python script for non-destructive file copying/moving.

• Pygmentizr

  A web app for applying syntax highlighting to code using the Pygments library.

• Acronyms

• Unpacking sets and ranges from a single string

• Darwin, pancakes and birthdays

  Looking at whether Darwin ever missed out on birthday cake for pancakes

41 notes

• My preferred options for SmartyPants in Python

  smartypants.smartypants(…, Attr.q | Attr.D | Attr.e | Attr.u)

• How to find the Pygments token type for a short name

  Look at the source code of pygments.token.

• Python f-strings cheat sheet

  Some Python f-string examples that I find helpful.

• Remove the microsecond precision from a datetime in Python

  Call datetime.replace(microsecond=0).

• Look at the __annotations__ to learn about the definition of a TypedDict

• Using zipstream to stream new zip files to object storage with boto3

  You can construct a zipstream.ZipFile, add files, then wrap it in a file-like object to upload it with S3.upload_fileobj.

• Adding a string to a tarfile in Python

  Wrap the string in an io.BytesIO first, then create a TarInfo object and pass them both to addfile() to add the string to your tarfile.

• Collecting pytest markers from failing tests

  You can annotate tests with markers, and using the pytest_collection_modifyitems and pytest_terminal_summary hooks you can get a list of markers for tests that failed.

• Python’s f-strings support = for self-documenting expressions

  The f-string f"{x=}" expands to f"x={x}".

• Use typing.getargs() to get a list of typing.Literal[…] values

• The error “No aggregated item, sequence was empty” comes from Jinja2

  You get this error message if you try to use Jinja2’s filters to get the min/max of an empty sequence.

• Get a string representation of a Python traceback with traceback.format_exc()

• Use text=true with subprocess functions to get stdout/stderr as str, not bytes

• You can set/update the total of a progress bar tqdm after it starts

  Update the .total attribute, then call .refresh().

• How to stream lines from stdout with subprocess

  Use subprocess.Popen() with bufsize=1 and text=True, then you can interate over the lines with proc.stdout.

• Python 3.14 includes a pi-related Easter egg

  You can start the interpreter with 𝜋thon!

• Why isn’t delete_where() deleting rows in sqlite-utils?

  The delete_where() function doesn’t auto-commit, so you need to wrap it with db.conn or something else to trigger the commit.

• Python 3.13 throws a ResourceWarning for SQLite databases that aren’t explicitly closed

• How to find platform-specific directories

  Two Python libraries for this task are appdirs and platformdirs, which tell you the location of the platform-specific cache directory and other similar directories.

• Use shutil.copyfileobj and xb to avoid overwriting files when copying in Python

• What errors can you get from hyperlink.parse?

  Mostly you get hyperlink.URLParseError, but you can occasionally get a ValueError as well.

• With Flask-Login, you want current_user == None, not current_user is None

  current_user is a proxy object that happens to be wrapping None, but isn’t actually None.

• How to get the expiry date of an HTTPS certificate with Python

  Connect to the domain using the socket module, then use the getpeercert() method on the connection to get information about the HTTPS certificate.

• Write to the middle of a file with Python

  Open the file with mode r+ to be able to seek around the file and write to it.

• Custom error pages in Flask

  You can use app.error_handler to add custom responses for HTTP status codes, so the errors match the look and feel of the rest of the site.

• How to simulate an [Errno 54] Connection reset by peer when using pytest

  You can run a TCP server in the background using a fixture, and using the SO_LINGER socket option can reset the connection.

• Getting a boto3 Session for an IAM role using Python

  Why I use Sessions in boto3, and the Python function I use to create them.

• You need to call resp.close() to close the file opened by send_file()

• How to gracefully restart a gunicorn app

• How to find the longest common suffix in a list of strings in Python

• How to simulate shell pipes with the subprocess module

• Use shlex.split() to parse log files quickly

• Use the regex library to get Unicode property escapes in Python

• Run a randomly selected subset of tests with pytest

  By reading the code for the pytest-random-order plugin, I was able to write a new plugin that runs a random subset of tests.

• Using sqlite-utils to convert a CSV into a SQLite database

  You can use sqlite-utils on the command line to create a SQLite database from a CSV file.

• Python’s sqlite3 context manager doesn’t close connections

  The sqlite3.connect(…) context manager will hold connections open, so you need to remember to close it manually or write your own context manager.

• Telling mechanize how to find local issuer certificates

  Calling browser.set_ca_data(cafile=certifi.where()) will tell where mechanize can find some local SSL certificates.

• A Python function to ignore a path with .git/info/exclude

  If your Python script creates a file that you don’t want to track in Git, here’s how you can ignore it.

• Create compact JSON with Python

  Calling json.dumps(…, separators=(',', ':')) will reduce the amount of whitespace in the final string.

• Python throws a TypeError if you return a non-string from a custom __repr__ or __str__ method

  It fails with the error “__repr__/__str__ returned non-string”.

• You can use shutil.which to check if an executable is in your PATH

  This is useful for checking if something’s installed.