Today I Learned (TIL)
TIL stands for today I learned. This is a collection of small, practical things I’ve learnt while writing software, which I thought were worth remembering and sharing with other people.
If you want to follow along, these posts have their own RSS feed.
-
Use
shutil.copyfileobj
andxb
to avoid overwriting files when copying in Python -
How to tally the attributes of the top N rows in a SQLite table
Use a
WITH
clause to do a nested query for the top N rows in the table, then do a tally over that result. -
How to get the IP address of a device in my Tailnet
Use
tailscale status --json
and filter the output usingjq
. -
How to find the biggest files backed up by Backblaze
Look at the file
/Library/Backblaze.bzpkg/bzdata/bzfilelists/bigfilelist.dat
. -
There are limits on the styles you can apply with
:visited
Because the
:visited
selector will tell you whether somebody has been to a URL, browsers limit what styles you can apply to such links – to prevent somebody nefarious stealing your browsing history. -
Editing a filename in Finder will convert it to NFD
Even if the filename looks the same, it may be invisibly converted to a different sequence of bytes.
-
How can I work out what program is keeping a disk open?
Use
sudo lsof
and grep for the name of the disk you’re trying to eject. -
You can reset the start of a regex in Ruby
The
\K
escape is the “Match-Reset Anchor”, which resets the start of the reported match, and skip any previously consumed characters. -
How to get the expiry date of an HTTPS certificate with Ruby
Connect to the domain using
net/http
, then you can inspect thepeer_cert
/not_after
property on the response. -
Debugging some confusing behaviour with
find
andxargs
Use the
--verbose
flag to see whatxargs
is doing; don’t rely onfind
to return files in a consistent order. -
How to install an asset from a GitHub release
Use
gh release download
, which includes a pattern matcher if you want to pick a specific asset. -
Creating a reverse proxy to a multi-site server with Caddy
You need to add Host headers and HTTPS configuration to your
reverse_proxy
block. -
How to find all of Apple’s system icons
You need to look for files named
*.icns
inside any subdirectory ofCoreTypes.bundle
. -
Drawing a diagonal banner over the corner of an image
-
Find the shortest prefix to identify a string in Ruby
The built-in
Abbrev
module can calculate a set of unambiguous abbreviations for a set of strings, and then you can look for the shortest result for each string. -
How to get the target of an HTTP redirect with curl
-
What is the
author_name
in the list of tags on a Flickr photo?When you call the
flickr.photos.getInfo
API, each tag is attributed to an author. Theauthor_name
is their username, not their realname. -
How to get a user’s email address with the Flickr API
The
flickr.profile.getProfile
API returns somebody’s email address, but only if you’re allowed to see it. -
Making an “under construction” element in pure CSS
Using a
repeating-linear-gradient
as theborder
gets you something that looks a bit like hazard tape. -
What errors can you get from
hyperlink.parse
?Mostly you get
hyperlink.URLParseError
, but you can occasionally get aValueError
as well. -
HTML strings may not be equivalent if you minify them
There’s a lot of whitespace in HTML which looks irrelevant at first glance, but may be significant and cause the document to render differently.
-
How do the
ispublic
,isfriend
andisfamily
flags work in the Flickr API? -
Use
std::io::IsTerminal
to detect if you’re running in the terminal in RustThis allows me to suppress ANSI escape codes if the output is going somewhere other than the terminal.
-
Use
$_
to get the last argument to the previous bash commandThis allows you to write commands like
mkdir myfolder && cd $_
orgit init myrepo && cd $_
. -
With Flask-Login, you want
current_user == None
, notcurrent_user is None
current_user
is a proxy object that happens to be wrappingNone
, but isn’t actuallyNone
. -
How to get the expiry date of an HTTPS certificate with Python
Connect to the domain using the
socket
module, then use thegetpeercert()
method on the connection to get information about the HTTPS certificate. -
Write to the middle of a file with Python
Open the file with mode
r+
to be able to seek around the file and write to it. -
Create solid-colour image placeholders to show before an image loads
-
Get an image from a video with ffmpeg
-
Get the embedded artwork from an MP3 file
Use the command
eyeD3 [MP3_FILE] --write-images [FOLDER]
. -
Convert an animated GIF to an MP4 with ffmpeg
-
Using the Wikimedia Commons API to tell if a file has been deleted
-
Build a URL with query string parameters with curl
A combination of
--get
and--data
/--data-urlencode
allows you to write curl commands which are readable and expressive. -
How to get the selected item in Finder using AppleScript
-
Removing letterboxing from a video screenshot with ImageMagick
Using
-trim
will remove the black portions and leave you the unletterboxed image. -
How to check the quality of a network connection
Using an
NWPathMonitor
and inspecting the value ofNWPath.status
,NWPath.isExpensive
andNWPath.isConstrained
can tell you what sort of connection you’re running on. -
How to highlight Python console sessions in Jekyll
Adding a couple of options to the
console
lexer (console?lang=python&prompt=>>>
) gets you syntax highlighting for a Python console session. -
Not all coal is the same
-
How to simulate an
[Errno 54] Connection reset by peer
when using pytestYou can run a TCP server in the background using a fixture, and using the
SO_LINGER
socket option can reset the connection. -
What does “insomnolent” mean?
-
Open a Safari webarchive from Twitter/X without being redirected
Disabling JavaScript when you open the webarchive file will prevent you from redirecting you to twitter.com.
-
Writing a file in Swift, but only if it doesn’t already exist
Adding
.withoutOverwriting
to yourwrite()
call will prevent you from overwriting a file that already exists. -
What’s inside a Safari webarchive?
The inside of a
.webarchive
file is a binary property list with the complete responses and some request metadata. -
Get a Palette colour as a command-line argument with Clap
Wrapping a
Palette:Srgb
in a struct and implementingFromStr
for the struct allows you to take hexadecimal colours as command-line inputs. -
How to get the filename/size without downloading a file in curl
You can do some fun stuff with the
--write-out
flag and variables. -
There’s a musical that tells you the number of minutes in a year
The song Seasons of Love from Rent starts with the line “Five hundred twenty-five thousand, six hundred minutes”.
-
How to get a tally of tally counts in SQLite
Using a nested query allows me to perform a two-level aggregation of the values in a column – how many values appear once, how many twice, and so on.
-
How to get a list of captures from the Wayback Machine
Use the CDX Server API to get a list of captures for a particular URL.
-
How to take a screenshot of a page in the Wayback Machine
Using Playwright to take screenshots and adding some custom styles gets a screenshot of a page without the Wayback Machine overlay.
-
How to count how many Discord messages were sent on a given day
Using the
During
filter gives me a count of how many messages were being sent. -
How to see the HTTP requests being made by pywikibot
To see exactly what HTTP requests were being made, I modified the library so that betamax would record requests.
-
How to embed an inline SVG in a CSS rule
Modern browsers allow you to embed the SVG almost as-is, with just a couple of characters that need escaping – no base64 required!
-
How to shuffle an array in a Jekyll template
If you want an array in random order, you can use the
sample
filter to get a random sample of the same size as the original array. -
Checking if a URL has changed when you fetch it over HTTP
When you make an HTTP request, you can use the
If-Modified-Since
header to get a 304 Not Modified if nothing has changed since your last request. -
The
open
command can ask questionsIf you pass an argument that can’t be easily identified as a file or a URL,
open
will ask you what to do next. This may be a surprise if you were trying to use it in a script. -
Why is Pillow rotating my image when I save it?
Images can have orientation specified in their EXIF metadata, which isn’t preserved when you open and save an image with Pillow.
-
How to change the name of an internal link in an Obsidian table
Escaping the pipe like
[[filename\|display text]]
allows you to customise the of a link in a table. -
Getting a boto3 Session for an IAM role using Python
Why I use Sessions in boto3, and the Python function I use to create them.
-
My config for running youtube-dl
The flags and arguments I find useful when I’m using youtube-dl.
-
How much will Mastodon instances try to re-send messages?
-
Get my Netlify bandwidth usage from the API
-
How to restrict a page to specific IP addresses
-
Use the
-n
/-i
flags to avoid overwriting files withcp
andmv
-
How to parse URLs in JXA
-
Using
errexit
and arithmetic expressions in bash -
How to check when an HTTPS certificate expires
-
The COUNT(X) function only counts non-null values
-
What happens when you replace a photo on Flickr?
-
Exclude files from Time Machine with
tmutil addexclusion
-
You need to call
resp.close()
to close the file opened bysend_file()
-
Add the
-v
flag to see whatrm
is deleting -
Custom error pages in Flask
You can use
app.error_handler
to add custom responses for HTTP status codes, so the errors match the look and feel of the rest of the site. -
Use the
{% raw %}
tag to describe Liquid in LiquidIf you’re trying to write about using Liquid tags in a Liquid-based site, wrapping your tags in the
{% raw %}
tag will prevent them being rendered. -
What characters are allowed in titles on Wikimedia Commons?
-
Run a script on macOS on a schedule using a LaunchAgent
-
Beware of using
test -n
with command expansion -
How to do offline geo-lookups of IP addresses
MaxMind offer databases you can do to look up IP addresses without sending the address off to a remote service.
-
How to tally combinations of values across multiple columns
-
How to move files when you need sudo on the remote server
-
How to gracefully restart a gunicorn app
-
How to find the longest common suffix in a list of strings in Python
-
How to simulate shell pipes with the subprocess module
-
Use the
{% capture %}
tag to assign complex strings to variablesIf you want to get a string that’s semi-complicated to construct, you can put a “mini-template” in the
{% capture %}
tag to build it over multiple lines. -
How to create a footer that’s always at the bottom of the page
-
WordPress URLs that get hammered by spammers
-
Manage MP3 metadata from iTunes with eyed3
-
Sort a list of DOM elements by sorting and calling
appendChild()
-
How to create flag emojis for countries in Python
-
Use the IMAGE function to insert an image into a spreadsheet
-
How to style a
<details>
element differently depending on whether it’s open or closed -
Pausing the animation of <svg> elements can affect child <svg> elements differently in different browsers
-
Create a directory before you
cp
ormv
a file to it -
Running the Netlify CLI in GitHub Actions
-
Use shlex.split() to parse log files quickly
-
Use the regex library to get Unicode property escapes in Python
-
Get and manipulate the contents of a page in Safari with
"do JavaScript"
-
SVGs are only rendered on GitHub if you use an
<img>
that points to another file -
Animate an attribute of an element with <animate>
-
Run a randomly selected subset of tests with pytest
By reading the code for the
pytest-random-order
plugin, I was able to write a new plugin that runs a random subset of tests. -
Getting a tally of SQLite values as a CSV
-
Using sqlite-utils to convert a CSV into a SQLite database
You can use sqlite-utils on the command line to create a SQLite database from a CSV file.
-
Python’s sqlite3 context manager doesn’t close connections
The
sqlite3.connect(…)
context manager will hold connections open, so you need to remember to close it manually or write your own context manager. -
How to delete albums
-
Live Text is aware of how hyphenation works (kinda)
-
Go between M-IDs and filenames on Wikimedia Commons
-
Telling mechanize how to find local issuer certificates
Calling
browser.set_ca_data(cafile=certifi.where())
will tell where mechanize can find some local SSL certificates. -
Why I prefer XML to JSON in the Wikimedia Commons APIs
The XML-to-JSON conversion leads to some inconsistent behaviour, especially in corner cases of the API.
-
How to do resumable downloads with curl
-
Find files that use a particular SDC field
-
Why the term “snak” keeps appearing in the Wikidata API
-
The acronym “woe” in the Flickr API stands for “Where On Earth”
-
Use Unicode property escapes to detect emoji in JavaScript
-
Finding the original page for a post on Mastodon
Following the logged-out 302 Redirect takes you to the original post.
-
Use concurrency gates to prevent concurrent deployments
-
How to profile Swift code
-
Getting the base directory of an sbt project
Some notes on printing sbt settings, so you can use them as the input to another script.
-
How to use hex colours with the palette crate
You can use
Srgb::from_str()
to parse a hexadecimal string as a colour in the palette crate. -
Calley-ope (calliope) Syndrome is pronouncing a word wrong because you’ve only ever read it on the page
-
Use
git check-ignore
to debug your.gitignore
Running
git check-ignore --verbose <PATH>
will tell you which rule applies to a given path, and where that rule is defined. -
Installing mimetype on Alpine Linux
-
How do Dreamwidth posts IDs work?
They were deliberately non-sequential as an anti-spam technique. It’s no longer required, but it’s in the codebase now and hasn’t been changed since it was written.
-
DynamoDB: Conditional updates on nested fields
-
How to iterate over the lines of an InputStream in Scala
-
Manipulating URL query parameters in JavaScript
-
The Content-Disposition header can be used to name a downloaded file
-
How to suppress installing rdoc/ri docs when running
gem install
-
Replace black/white parts of an image with ImageMagick
-
How to set the clock on a Horstmann Electronic 7 water heater
-
Pushing with
--force-with-lease
is safer than with--force
It checks the remote state of the origin hasn’t changed since you last fetched, so you don’t risk overwriting anybody else’s commits.
-
Create compact JSON with Python
Calling
json.dumps(…, separators=(',', ':'))
will reduce the amount of whitespace in the final string. -
List all Git object IDs and their type
-
Why does Hypothesis try the same example three times before failing?
-
How to set the clock on a Tricity Bendix SIE454 electric cooker
-
How to set the clock on a Worcester 28CDi boiler
-
Get an RSS feed of external audio posts on Tumblr
If you add
/podcast
to a Tumblr site, you get a podcast-like RSS feed for all the external audio posts on that site.