Tags » python

Python is a programming language that I use for scripting, automation and writing web apps. It’s the language I’m most familiar with, and my go-to language for new projects.

• 

  How KempisBot works

  How I reincarnated a fifteenth-century monk and taught him to use Twitter.

• 

  Using DynamoDB as a calculator

  Taking advantage of Amazon’s least-loved compute platform.

• 

  Adventures in Python with concurrent.futures

  Some examples of how I’ve been using concurrent.futures to speed up my batch scripting in Python.

• 

  Getting a tint colour from an image with Python and k-means

  I want to display text next to an image that looks visually similar to the image, so I need to extract a tint colour.

• Create space-saving clones on macOS with Python

  You can shell out to cp -c using subprocess, or you can make a clonefile() syscall using the ctypes library.

• TIL: Using zipstream to stream new zip files to object storage with boto3

  You can construct a zipstream.ZipFile, add files, then wrap it in a file-like object to upload it with S3.upload_fileobj.

• TIL: Using Linode object storage and boto3

  If you’re calling put_object, you need the config option request_checksum_calculation = "when_required".

• TIL: Adding a string to a tarfile in Python

  Wrap the string in an io.BytesIO first, then create a TarInfo object and pass them both to addfile() to add the string to your tarfile.

• TIL: Collecting pytest markers from failing tests

  You can annotate tests with markers, and using the pytest_collection_modifyitems and pytest_terminal_summary hooks you can get a list of markers for tests that failed.

• TIL: Show a list of checkboxes in a WTForms form

  Subclass SelectMultipleField and override the widget and option_widget fields.

• TIL: Redacting sensitive information from gunicorn access logs

  Create a subclass of gunicorn.glogging.Logger, and redact information in the atoms() method.

• TIL: Python’s f-strings support = for self-documenting expressions

  The f-string f"{x=}" expands to f"x={x}".

• TIL: Use typing.getargs() to get a list of typing.Literal[…] values

• TIL: The error “No aggregated item, sequence was empty” comes from Jinja2

  You get this error message if you try to use Jinja2’s filters to get the min/max of an empty sequence.

• TIL: Get a string representation of a Python traceback with traceback.format_exc()

• TIL: Use text=true with subprocess functions to get stdout/stderr as str, not bytes

• TIL: Print a comma-separated number in Python with {num:,}

  You can use {num:,} to insert a comma every three digits, {num:_} to insert an underscore every three digits, and {num:n} to insert a locale-aware digit separator.

• TIL: You can set/update the total of a progress bar tqdm after it starts

  Update the .total attribute, then call .refresh().

• TIL: How to stream lines from stdout with subprocess

  Use subprocess.Popen() with bufsize=1 and text=True, then you can interate over the lines with proc.stdout.

• Handling JSON objects with duplicate names in Python

  It’s possible, although uncommon, for a JSON object to contain the same name multiple times. Here are some ways to handle that in Python.

• TIL: Python 3.14 includes a pi-related Easter egg

  You can start the interpreter with 𝜋thon!

• TIL: When fixing mojibake, use ftfy.fix_and_explain() to understand how it’s fixing a piece of text

• TIL: Why isn’t delete_where() deleting rows in sqlite-utils?

  The delete_where() function doesn’t auto-commit, so you need to wrap it with db.conn or something else to trigger the commit.

• Beyond None: actionable error messages for keyring.get_password()

  I have a wrapper for get_password() so that if it can’t find a password, you get an error that explains how to set the password, and what password you should choose.

• TIL: Python 3.13 throws a ResourceWarning for SQLite databases that aren’t explicitly closed

• Creating static map images with OpenStreetMap, Web Mercator, and Pillow

  I made some simple map visualisations by downloading tiles from OpenStreetMap, then annotating the tiles with Pillow.

• TIL: How to find platform-specific directories

  Two Python libraries for this task are appdirs and platformdirs, which tell you the location of the platform-specific cache directory and other similar directories.

• TIL: Use shutil.copyfileobj and xb to avoid overwriting files when copying in Python

• TIL: What errors can you get from hyperlink.parse?

  Mostly you get hyperlink.URLParseError, but you can occasionally get a ValueError as well.

• TIL: With Flask-Login, you want current_user == None, not current_user is None

  current_user is a proxy object that happens to be wrapping None, but isn’t actually None.

• TIL: How to get the expiry date of an HTTPS certificate with Python

  Connect to the domain using the socket module, then use the getpeercert() method on the connection to get information about the HTTPS certificate.

• TIL: Write to the middle of a file with Python

  Open the file with mode r+ to be able to seek around the file and write to it.

• TIL: How to highlight Python console sessions in Jekyll

  Adding a couple of options to the console lexer (console?lang=python&prompt=>>>) gets you syntax highlighting for a Python console session.

• The surprising utility of a Flickr URL parser

  I made a library that knows how to read lots of different forms of Flickr.com URL, and I used hyperlink to do it.

• TIL: How to simulate an [Errno 54] Connection reset by peer when using pytest

  You can run a TCP server in the background using a fixture, and using the SO_LINGER socket option can reset the connection.

• TIL: How to see the HTTP requests being made by pywikibot

  To see exactly what HTTP requests were being made, I modified the library so that betamax would record requests.

• TIL: Checking if a URL has changed when you fetch it over HTTP

  When you make an HTTP request, you can use the If-Modified-Since header to get a 304 Not Modified if nothing has changed since your last request.

• TIL: Why is Pillow rotating my image when I save it?

  Images can have orientation specified in their EXIF metadata, which isn’t preserved when you open and save an image with Pillow.

• TIL: Getting a boto3 Session for an IAM role using Python

  Why I use Sessions in boto3, and the Python function I use to create them.

• Moving my YouTube Likes from one account to another

  Some experimenting with the YouTube API to merge two accounts into one.

• TIL: You need to call resp.close() to close the file opened by send_file()

• TIL: Custom error pages in Flask

  You can use app.error_handler to add custom responses for HTTP status codes, so the errors match the look and feel of the rest of the site.

• TIL: How to do offline geo-lookups of IP addresses

  MaxMind offer databases you can do to look up IP addresses without sending the address off to a remote service.

• TIL: How to gracefully restart a gunicorn app

• TIL: How to find the longest common suffix in a list of strings in Python

• TIL: How to simulate shell pipes with the subprocess module

• TIL: Manage MP3 metadata from iTunes with eyed3

• TIL: How to create flag emojis for countries in Python

• TIL: Use shlex.split() to parse log files quickly

• TIL: Use the regex library to get Unicode property escapes in Python

• TIL: Run a randomly selected subset of tests with pytest

  By reading the code for the pytest-random-order plugin, I was able to write a new plugin that runs a random subset of tests.

• TIL: Using sqlite-utils to convert a CSV into a SQLite database

  You can use sqlite-utils on the command line to create a SQLite database from a CSV file.

• TIL: Python’s sqlite3 context manager doesn’t close connections

  The sqlite3.connect(…) context manager will hold connections open, so you need to remember to close it manually or write your own context manager.

• Setting up Fish to make virtualenv easier

  I wrote some shell config to smooth out the experience of using virtual environments in Python.

• TIL: Telling mechanize how to find local issuer certificates

  Calling browser.set_ca_data(cafile=certifi.where()) will tell where mechanize can find some local SSL certificates.

• Spotting spam in our CloudFront logs

  Looking for search queries that came from robots, not real people.

• Adding locations to my photos from my Apple Watch workouts

  My Apple Watch knows where I am, which is handy when I have a camera that doesn’t.

• Parsing CloudFront logs with Python

  A couple of functions I use to get access to CloudFront logs as easy-to-use iterators.

• My Python snippet for walking a file tree

  A function to find all the files in a directory is one of my most-used snippets.

• Creating a Python dictionary with multiple, equivalent keys

  Using collections.UserDict, we can create a dictionary where dict[key1] and dict[key2] always point to the same value.

• Splitting a class into balanced groups

  How do you make sure everyone gets to work with everyone else?

• A Python function to iterate through an S3 Bucket Inventory

  Getting something that looks more like the output of the ListObjectsV2 API.

• I always want StrictUndefined in Jinja

  When I’m writing templates with Jinja, strict behaviour is what I want, even if it’s not the default.

• Creating animated GIFs from fruit and veg

  Some Python code for turning MRI scans of fruit and veg into animated GIFs.

• Why is os.sep insufficient for path operations?

  Digging into a throwaway comment in the Python documentation.

• Picking perfect planks with Python

  How do you pick the right combination of planks to lay a wooden floor? Python and itertools to the rescue!

• How to ignore lots of folders in Spotlight

  A script that allows me to ignore folders like “target” and “node_modules”, so they don’t appear in search results.

• Finding misconfigured or dangling CloudWatch Alarms

  A Python script that finds CloudWatch Alarms which are based on a now non-existent source.

• Listing deleted secrets in AWS Secrets Manager with boto3 and the AWS CLI

  Diving into the internals of the AWS SDK to find deleted secrets.

• Visualising how often I write in my journal

  A Python script that shows me how often I’ve been journalling, so I can track my progress.

• Downloading objects from/uploading files to S3 with progress bars in Python

  Making it easier to see how long a file transfer will take, in the terminal.

• Drawing coloured squares/text in my terminal with Python

• A Python function to ignore a path with .git/info/exclude

  If your Python script creates a file that you don’t want to track in Git, here’s how you can ignore it.

• Two Python functions for getting CloudTrail events

• Using fuzzy string matching to find duplicate tags

• Finding the months between two dates in Python

• Getting every item from a DynamoDB table with Python

  A Python function that generates every item in a DynamoDB table.

• Downloading the AO3 fics that I’ve saved in Pinboard

  A script that downloads the nicely formatted AO3 downloads for everything I’ve saved in Pinboard.

• Taking tuple unpacking to terrible places

  I want to assign a bunch of variables to True, but I don’t know how many there are. Reflection to the rescue!

• A snippet for downloading files with Python

• Storing language vocabulary as a graph

  Experimenting with a way to store words and phrases that highlights the connections between them.

• Creating striped flag wallpapers with Pillow

• Adjusting the dominant colour of an image

  Adjusting the hue to get different colour variants of the same image.

• Generating pride-themed Norse valknuts with Python 🌈

  A web app to generate mashups of Norse valknuts and Pride flags.

• This YAML file will self-destruct in five seconds!

  YAML allows you to execute arbitrary code in a parser, even if you really really shouldn’t.

• November 2019 scripts: downloading podcasts, retrying flaky errors, Azure and AWS

• Saving a copy of a tweet by typing ;twurl

• The rough edges of filecmp

  The filecmp module has a confusing API, and it just caught me out.

• Finding divisors of a number with Python

  Using unique prime factorisations and itertools to find all the divisors of a number.

• Listing even more keys in an S3 bucket with Python

  Python functions for getting a list of keys and objects in an S3 bucket.

• TIL: Using TransportAPI and geopy to get the distance between stations

• Atomic, cross-filesystem moves in Python

  Explaining some code for moving files around in a way that’s atomic and works across filesystem boundaries.

• Working with really large objects in S3

  Code for processing large objects in S3 without downloading the whole thing first, using file-like objects in Python.

• Notes on reading a UTF-8 encoded CSV in Python

  Some notes on trying to do this in a way that supports both Python 2 and 3, and the frustration of doing so.

• Iterating in fixed-size chunks in Python

  A snippet for iterating over an arbitrary iterable in chunks, and returning a smaller chunk if the boundaries don’t line up.

• Getting credentials for an assumed IAM Role

  A script that creates temporary credentials for an assumed IAM role, and stores them in ~/.aws/credentials.

• TIL: Create compact JSON with Python

  Calling json.dumps(…, separators=(',', ':')) will reduce the amount of whitespace in the final string.

• A basic error logger for Python Lambdas

  A snippet to make it a bit easier to debug errors in AWS Lambda functions written in Python.

• Drawing ASCII bar charts

  A Python snippets for drawing bar charts in command-line applications.

• Beware of logged errors from subprocess

  If you use Python’s subprocess module, be careful you don’t leak sensitive information in your error logs.

• Getting every message in an SQS queue

  Code for saving every message from an SQS queue, and then saving the messages to a file, or resending them to another queue.

• Listing keys in an S3 bucket with Python, redux

  Python functions for getting a list of keys and objects in an S3 bucket.

• Downloading logs from Amazon CloudWatch

  A detailed breakdown of how I wrote a Python script to download logs from CloudWatch.

• Using hooks for custom behaviour in requests

  I often have code I want to run against every HTTP response (logging, error checking) — event hooks give me a nice way to do that without repetition.

• Using pip-tools to manage my Python dependencies

  How I use pip-tools to ensure my Python dependencies are pinned, precise, and as minimal as possible.

• Ode to docopt

  Why I love docopt as a tool for writing clean, simple command-line interfaces.

• A Python module for lazy reading of file objects

  I wrote a small Python module for lazy file reading, ideal for efficient batch processing.

• Listing keys in an S3 bucket with Python

  A short Python function for getting a list of keys in an S3 bucket.

• A Python interface to AO3

  AO3 doesn’t have an official API for scraping data - but with a bit of Python, it might not be necessary.

• Experiments with AO3 and Python

  AO3 doesn’t have an official API for scraping data - but with a bit of Python, it might not be necessary.

• TIL: Why does Hypothesis try the same example three times before failing?

• Another example of why strings are terrible

  Pop quiz: if I lowercase a string, does it still have the same length as the original string?

• Use keyring to store your credentials

  If you need to store passwords in a Python application, use the keyring module to keep them safe.

• Creating low contrast wallpapers with Pillow

  Take a regular tiling of the plane, apply a random colouring, and voila: a unique wallpaper, courtesy of the Python Imaging Library.

• Tiling the plane with Pillow

  Using the Python Imaging Library to draw regular tilings of squares, triangles and hexagons.

• Why I use py.test

  Why py.test is my unit test framework of choice in Python.

• Python snippets: Cleaning up empty/nearly empty directories

  A pair of Python scripts I’ve been using to clean up my mess of directories.

• Python snippets: Chasing redirects and URL shorteners

  A quick Python function to follow a redirect to its eventual conclusion.

• Reading web pages on my Kindle

  A Python script I wrote that let me sends web pages from my Mac and my iPhone to my Kindle.

• Introduction to property-based testing

  Testing with randomly generated examples can be a good way to uncover bugs in your code.

• Finding 404s and broken pages in my Apache logs

  A Python script for finding 404 errors in my Apache web logs - and by extension, broken pages.

• A Python smtplib wrapper for Fastmail

  A quick python-smtplib wrapper for sending emails through Fastmail.

• Get images from the iTunes/App/Mac App Stores with Alfred

  Using Alfred and a Python script to retrieve artwork from the iTunes, App and Mac App Stores.

• Exclusively create a file in Python 3

  If you want to create a file, but only if it doesn’t already exist, Python 3 has a helpful new file mode x.

• Python and the BBC micro:bit

  Playing with a tiny computer that runs Python.

• Review: Effective Python

  A review of Effective Python, by Brett Slatkin.

• Persistent IPython notebooks in Windows

  Configuring an IPython notebook server that is always running and easily accessible in Windows.

• Safer file copying in Python

  A Python script for non-destructive file copying/moving.

• Pygmentizr

  A web app for applying syntax highlighting to code using the Pygments library.

• Acronyms

• Unpacking sets and ranges from a single string

• Playing with 404 pages

• Finding untagged posts on Tumblr

• Automatic Pinboard backups

  A script for automatically backing up bookmarks from Pinboard