Skip to main content

Finding the target of an HTTP redirect with the Python standard library

  • Posted
  • Filed in Python

Create an HTTPRedirectHandler that doesn’t follow redirects, then inspect the headers of the “error” you receive from a redirect.

I use the Python standard library to do HTTP requests rather than relying on third-party packages. One gap in my toolbox was the ability to inspect the redirect location of a request. I have a lot of redirects on this site, and I want tests to check that they’re working correctly.

I’ve now written a function that fetches a URL, checks it got an HTTP 3xx redirect, and returns the URL it redirects to. Here are the key steps:

  1. Define an HTTPRedirectHandler that overrides the redirect_request method to always return None, so redirects are never followed.
  2. Pass that handler to urllib.request.build_opener.
  3. Open the target URL. This function throws an HTTPError if it gets a non-2xx response, which we can catch and inspect.
  4. An HTTPError can also be treated as an HTTP response, so we check it has the desired 3xx status code and look for the HTTP Location header. (We have to close this, or we’ll have a dangling open file.)

Here’s the complete function:

from contextlib import closing
from email.message import Message
import ssl
from typing import IO
from urllib.error import HTTPError
import urllib.request
from urllib.request import HTTPRedirectHandler, HTTPSHandler, Request

import certifi


def get_redirect_target(url: str) -> str:
    """
    Fetch a URL, and return the URL it redirects to.

    This throws an error if the URL does not respond with a response
    that includes a Location header.
    """

    class NoRedirectHandler(HTTPRedirectHandler):
        """
        A redirect handler that never follows redirects.
        """

        def redirect_request(
            self,
            req: Request,
            fp: IO[bytes],
            code: int,
            msg: str,
            headers: Message,
            newurl: str,
        ) -> urllib.request.Request | None:
            """
            Return None so we don't follow the redirect.
            """
            return None

    https_handler = HTTPSHandler(
        context=ssl.create_default_context(cafile=certifi.where())
    )

    opener = urllib.request.build_opener(NoRedirectHandler, https_handler)
    try:
        with opener.open(url):
            raise RuntimeError("url did not respond with a redirect")
    except HTTPError as err:
        with closing(err):
            if 300 <= err.code < 400:
                try:
                    return err.headers["location"]
                except KeyError:
                    raise RuntimeError("url did not respond with a Location header")
            else:
                raise


def test_redirects_from_http_to_https() -> None:
    """
    Check my website redirects from http to https.
    """
    assert get_redirect_target("http://alexwlchan.net/") == "https://alexwlchan.net/"