Finding the target of an HTTP redirect with the Python standard library
Create an HTTPRedirectHandler that doesn’t follow redirects, then inspect the headers of the “error” you receive from a redirect.
I use the Python standard library to do HTTP requests rather than relying on third-party packages. One gap in my toolbox was the ability to inspect the redirect location of a request. I have a lot of redirects on this site, and I want tests to check that they’re working correctly.
I’ve now written a function that fetches a URL, checks it got an HTTP 3xx redirect, and returns the URL it redirects to. Here are the key steps:
- Define an
HTTPRedirectHandlerthat overrides theredirect_requestmethod to always returnNone, so redirects are never followed. - Pass that handler to
urllib.request.build_opener. - Open the target URL. This function throws an
HTTPErrorif it gets a non-2xx response, which we can catch and inspect. - An
HTTPErrorcan also be treated as an HTTP response, so we check it has the desired 3xx status code and look for the HTTP Location header. (We have to close this, or we’ll have a dangling open file.)
Here’s the complete function:
from contextlib import closing
from email.message import Message
import ssl
from typing import IO
from urllib.error import HTTPError
import urllib.request
from urllib.request import HTTPRedirectHandler, HTTPSHandler, Request
import certifi
def get_redirect_target(url: str) -> str:
"""
Fetch a URL, and return the URL it redirects to.
This throws an error if the URL does not respond with a response
that includes a Location header.
"""
class NoRedirectHandler(HTTPRedirectHandler):
"""
A redirect handler that never follows redirects.
"""
def redirect_request(
self,
req: Request,
fp: IO[bytes],
code: int,
msg: str,
headers: Message,
newurl: str,
) -> urllib.request.Request | None:
"""
Return None so we don't follow the redirect.
"""
return None
https_handler = HTTPSHandler(
context=ssl.create_default_context(cafile=certifi.where())
)
opener = urllib.request.build_opener(NoRedirectHandler, https_handler)
try:
with opener.open(url):
raise RuntimeError("url did not respond with a redirect")
except HTTPError as err:
with closing(err):
if 300 <= err.code < 400:
try:
return err.headers["location"]
except KeyError:
raise RuntimeError("url did not respond with a Location header")
else:
raise
def test_redirects_from_http_to_https() -> None:
"""
Check my website redirects from http to https.
"""
assert get_redirect_target("http://alexwlchan.net/") == "https://alexwlchan.net/"