Today I Learned (TIL)

TIL stands for today I learned. This is a collection of small, practical things I’ve learnt while writing software, which I thought were worth remembering and sharing with other people.

If you want to follow along, these posts have their own RSS feed.

• How can I work out what program is keeping a disk open?

  Use sudo lsof and grep for the name of the disk you’re trying to eject.

• You can reset the start of a regex in Ruby

  The \K escape is the “Match-Reset Anchor”, which resets the start of the reported match, and skip any previously consumed characters.

• How to get the expiry date of an HTTPS certificate with Ruby

  Connect to the domain using net/http, then you can inspect the peer_cert/not_after property on the response.

• Debugging some confusing behaviour with find and xargs

  Use the --verbose flag to see what xargs is doing; don’t rely on find to return files in a consistent order.

• How to install an asset from a GitHub release

  Use gh release download, which includes a pattern matcher if you want to pick a specific asset.

• Creating a reverse proxy to a multi-site server with Caddy

  You need to add Host headers and HTTPS configuration to your reverse_proxy block.

• How to find all of Apple’s system icons

  You need to look for files named *.icns inside any subdirectory of CoreTypes.bundle.

• Drawing a diagonal banner over the corner of an image

• Find the shortest prefix to identify a string in Ruby

  The built-in Abbrev module can calculate a set of unambiguous abbreviations for a set of strings, and then you can look for the shortest result for each string.

• How to get the target of an HTTP redirect with curl

• What is the author_name in the list of tags on a Flickr photo?

  When you call the flickr.photos.getInfo API, each tag is attributed to an author. The author_name is their username, not their realname.

• How to get a user’s email address with the Flickr API

  The flickr.profile.getProfile API returns somebody’s email address, but only if you’re allowed to see it.

• Making an “under construction” element in pure CSS

  Using a repeating-linear-gradient as the border gets you something that looks a bit like hazard tape.

• What errors can you get from hyperlink.parse?

  Mostly you get hyperlink.URLParseError, but you can occasionally get a ValueError as well.

• HTML strings may not be equivalent if you minify them

  There’s a lot of whitespace in HTML which looks irrelevant at first glance, but may be significant and cause the document to render differently.

• How do the ispublic, isfriend and isfamily flags work in the Flickr API?

• Use std::io::IsTerminal to detect if you’re running in the terminal in Rust

  This allows me to suppress ANSI escape codes if the output is going somewhere other than the terminal.

• Use $_ to get the last argument to the previous bash command

  This allows you to write commands like mkdir myfolder && cd $_ or git init myrepo && cd $_.

• With Flask-Login, you want current_user == None, not current_user is None

  current_user is a proxy object that happens to be wrapping None, but isn’t actually None.

• How to get the expiry date of an HTTPS certificate with Python

  Connect to the domain using the socket module, then use the getpeercert() method on the connection to get information about the HTTPS certificate.

• Write to the middle of a file with Python

  Open the file with mode r+ to be able to seek around the file and write to it.

• Create solid-colour image placeholders to show before an image loads

• Get an image from a video with ffmpeg

• Get the embedded artwork from an MP3 file

  Use the command eyeD3 [MP3_FILE] --write-images [FOLDER].

• Convert an animated GIF to an MP4 with ffmpeg

• Using the Wikimedia Commons API to tell if a file has been deleted

• Build a URL with query string parameters with curl

  A combination of --get and --data/--data-urlencode allows you to write curl commands which are readable and expressive.

• How to get the selected item in Finder using AppleScript

• Removing letterboxing from a video screenshot with ImageMagick

  Using -trim will remove the black portions and leave you the unletterboxed image.

• How to check the quality of a network connection

  Using an NWPathMonitor and inspecting the value of NWPath.status, NWPath.isExpensive and NWPath.isConstrained can tell you what sort of connection you’re running on.

• How to highlight Python console sessions in Jekyll

  Adding a couple of options to the console lexer (console?lang=python&prompt=>>>) gets you syntax highlighting for a Python console session.

• Not all coal is the same

• How to simulate an [Errno 54] Connection reset by peer when using pytest

  You can run a TCP server in the background using a fixture, and using the SO_LINGER socket option can reset the connection.

• What does “insomnolent” mean?

• Open a Safari webarchive from Twitter/X without being redirected

  Disabling JavaScript when you open the webarchive file will prevent you from redirecting you to twitter.com.

• Writing a file in Swift, but only if it doesn’t already exist

  Adding .withoutOverwriting to your write() call will prevent you from overwriting a file that already exists.

• What’s inside a Safari webarchive?

  The inside of a .webarchive file is a binary property list with the complete responses and some request metadata.

• Get a Palette colour as a command-line argument with Clap

  Wrapping a Palette:Srgb in a struct and implementing FromStr for the struct allows you to take hexadecimal colours as command-line inputs.

• How to get the filename/size without downloading a file in curl

  You can do some fun stuff with the --write-out flag and variables.

• There’s a musical that tells you the number of minutes in a year

  The song Seasons of Love from Rent starts with the line “Five hundred twenty-five thousand, six hundred minutes”.

• How to get a tally of tally counts in SQLite

  Using a nested query allows me to perform a two-level aggregation of the values in a column – how many values appear once, how many twice, and so on.

• How to get a list of captures from the Wayback Machine

  Use the CDX Server API to get a list of captures for a particular URL.

• How to take a screenshot of a page in the Wayback Machine

  Using Playwright to take screenshots and adding some custom styles gets a screenshot of a page without the Wayback Machine overlay.

• How to count how many Discord messages were sent on a given day

  Using the During filter gives me a count of how many messages were being sent.

• How to see the HTTP requests being made by pywikibot

  To see exactly what HTTP requests were being made, I modified the library so that betamax would record requests.

• How to embed an inline SVG in a CSS rule

  Modern browsers allow you to embed the SVG almost as-is, with just a couple of characters that need escaping – no base64 required!

• How to shuffle an array in a Jekyll template

  If you want an array in random order, you can use the sample filter to get a random sample of the same size as the original array.

• Checking if a URL has changed when you fetch it over HTTP

  When you make an HTTP request, you can use the If-Modified-Since header to get a 304 Not Modified if nothing has changed since your last request.

• The open command can ask questions

  If you pass an argument that can’t be easily identified as a file or a URL, open will ask you what to do next. This may be a surprise if you were trying to use it in a script.

• Why is Pillow rotating my image when I save it?

  Images can have orientation specified in their EXIF metadata, which isn’t preserved when you open and save an image with Pillow.

• How to change the name of an internal link in an Obsidian table

  Escaping the pipe like [[filename\|display text]] allows you to customise the of a link in a table.

• Getting a boto3 Session for an IAM role using Python

  Why I use Sessions in boto3, and the Python function I use to create them.

• My config for running youtube-dl

  The flags and arguments I find useful when I’m using youtube-dl.

• How much will Mastodon instances try to re-send messages?

• Get my Netlify bandwidth usage from the API

• How to restrict a page to specific IP addresses

• Use the -n/-i flags to avoid overwriting files with cp and mv

• How to parse URLs in JXA

• Using errexit and arithmetic expressions in bash

• How to check when an HTTPS certificate expires

• The COUNT(X) function only counts non-null values

• What happens when you replace a photo on Flickr?

• Exclude files from Time Machine with tmutil addexclusion

• You need to call resp.close() to close the file opened by send_file()

• Add the -v flag to see what rm is deleting

• Custom error pages in Flask

  You can use app.error_handler to add custom responses for HTTP status codes, so the errors match the look and feel of the rest of the site.

• Use the {% raw %} tag to describe Liquid in Liquid

  If you’re trying to write about using Liquid tags in a Liquid-based site, wrapping your tags in the {% raw %} tag will prevent them being rendered.

• What characters are allowed in titles on Wikimedia Commons?

• Run a script on macOS on a schedule using a LaunchAgent

• Beware of using test -n with command expansion

• How to do offline geo-lookups of IP addresses

  MaxMind offer databases you can do to look up IP addresses without sending the address off to a remote service.

• How to tally combinations of values across multiple columns

• How to move files when you need sudo on the remote server

• How to gracefully restart a gunicorn app

• How to find the longest common suffix in a list of strings in Python

• How to simulate shell pipes with the subprocess module

• Use the {% capture %} tag to assign complex strings to variables

  If you want to get a string that’s semi-complicated to construct, you can put a “mini-template” in the {% capture %} tag to build it over multiple lines.

• How to create a footer that’s always at the bottom of the page

• WordPress URLs that get hammered by spammers

• Manage MP3 metadata from iTunes with eyed3

• Sort a list of DOM elements by sorting and calling appendChild()

• How to create flag emojis for countries in Python

• Use the IMAGE function to insert an image into a spreadsheet

• How to style a <details> element differently depending on whether it’s open or closed

• Pausing the animation of <svg> elements can affect child <svg> elements differently in different browsers

• Create a directory before you cp or mv a file to it

• Running the Netlify CLI in GitHub Actions

• Use shlex.split() to parse log files quickly

• Use the regex library to get Unicode property escapes in Python

• Get and manipulate the contents of a page in Safari with "do JavaScript"

• SVGs are only rendered on GitHub if you use an <img> that points to another file

• Animate an attribute of an element with <animate>

• Run a randomly selected subset of tests with pytest

  By reading the code for the pytest-random-order plugin, I was able to write a new plugin that runs a random subset of tests.

• Getting a tally of SQLite values as a CSV

• Using sqlite-utils to convert a CSV into a SQLite database

  You can use sqlite-utils on the command line to create a SQLite database from a CSV file.

• Python’s sqlite3 context manager doesn’t close connections

  The sqlite3.connect(…) context manager will hold connections open, so you need to remember to close it manually or write your own context manager.

• How to delete albums

• Live Text is aware of how hyphenation works (kinda)

• Go between M-IDs and filenames on Wikimedia Commons

• Telling mechanize how to find local issuer certificates

  Calling browser.set_ca_data(cafile=certifi.where()) will tell where mechanize can find some local SSL certificates.

• Why I prefer XML to JSON in the Wikimedia Commons APIs

  The XML-to-JSON conversion leads to some inconsistent behaviour, especially in corner cases of the API.

• How to do resumable downloads with curl

• Find files that use a particular SDC field

• Why the term “snak” keeps appearing in the Wikidata API

• The acronym “woe” in the Flickr API stands for “Where On Earth”

• Use Unicode property escapes to detect emoji in JavaScript

• Finding the original page for a post on Mastodon

  Following the logged-out 302 Redirect takes you to the original post.

• Use concurrency gates to prevent concurrent deployments

• How to profile Swift code

• How to use hex colours with the palette crate

  You can use Srgb::from_str() to parse a hexadecimal string as a colour in the palette crate.

• Calley-ope (calliope) Syndrome is pronouncing a word wrong because you’ve only ever read it on the page

• Use git check-ignore to debug your .gitignore

  Running git check-ignore --verbose <PATH> will tell you which rule applies to a given path, and where that rule is defined.

• Installing mimetype on Alpine Linux

• How do Dreamwidth posts IDs work?

  They were deliberately non-sequential as an anti-spam technique. It’s no longer required, but it’s in the codebase now and hasn’t been changed since it was written.

• DynamoDB: Conditional updates on nested fields

• How to iterate over the lines of an InputStream in Scala

• Manipulating URL query parameters in JavaScript

• The Content-Disposition header can be used to name a downloaded file

• How to suppress installing rdoc/ri docs when running gem install

• Replace black/white parts of an image with ImageMagick

• How to set the clock on a Horstmann Electronic 7 water heater

• Pushing with --force-with-lease is safer than with --force

  It checks the remote state of the origin hasn’t changed since you last fetched, so you don’t risk overwriting anybody else’s commits.

• Create compact JSON with Python

  Calling json.dumps(…, separators=(',', ':')) will reduce the amount of whitespace in the final string.

• List all Git object IDs and their type

• Why does Hypothesis try the same example three times before failing?

• How to set the clock on a Tricity Bendix SIE454 electric cooker

• How to set the clock on a Worcester 28CDi boiler

• Get an RSS feed of external audio posts on Tumblr

  If you add /podcast to a Tumblr site, you get a podcast-like RSS feed for all the external audio posts on that site.