Skip to main content

Verifying my web archives with Playwright

To protect against link rot and disappearing content, I save local copies of web pages I like into my web archive.

Ideally, this archive is completely self-contained. I want a local copy of each page’s HTML and all the assets it depends on (CSS, images, scripts, and so on). I don’t want my archived pages fetching resources from the original website, which could disappear at any time.

Initially, I was verifying self-containedness by hand. I’d open each saved page in a web browser, and check the browser dev tools. I’d check the Console tab for errors and failed requests, and watch the Network tab for requests going out to the internet. That approach was tedious and error-prone, and I wanted something better.

I now automate the process using Playwright, a testing framework that allows you to control web browsers. I open each page using Playwright, and check if it’s making external requests.

Detecting external requests in Playwright

When you open a page in Playwright, you can subscribe to request events which tell you when the page issues a request.

In this Python snippet, we start Playwright, start a WebKit browser, and open a new page. We tell the page to append external requests to a list, and then load example.com. We see that example.com makes two requests on page load: one for https://example.com, and one for a JavaScript file.

from contextlib import closing
from playwright.sync_api import sync_playwright

with (
    sync_playwright() as p,
    closing(p.webkit.launch()) as browser,
    closing(browser.new_page()) as page,
):
    requests = []
    page.on("request", lambda msg: requests.append(msg))

    page.goto("https://example.com")

print("requests:", requests)
# requests: [<Request url='https://example.com/' method='GET'>,
#            <Request url='https://example.com/s.js' method='GET'>]

In a more complex scenario, we might use Playwright to interact with the page after loading – for example, clicking a button or filling in a form, and then we could see the requests triggered by each action.

Once we have a list of requests made by the page, we can inspect the url property to check if any of them were fetching from an external website. I wrote a function is_external_request to carry out this check:

from playwright.sync_api import Request
import urllib.parse


def is_external_request(req: Request, original_url: str) -> bool:
    """
    Check if a request has a different origin than the original URL.

    A request is considered internal if:
      - it matches the original URL exactly
      - it has the same scheme, host, and port as the original URL
      - it uses a `blob:` scheme (for example, in-memory media representations)

    """
    if req.url == original_url:
        return False

    # Compare the scheme/host of the request to that of the original URL.
    org_url = urllib.parse.urlsplit(original_url)
    req_url = urllib.parse.urlsplit(req.url)

    if org_url.scheme == req_url.scheme and org_url.netloc == req_url.netloc:
        return False

    # Ignore blob: URLs, which are in-memory representation of some data,
    # and not an external request.
    #
    # Playwright detects blob: URLs when loading media files; for example
    # using the <audio> or <video> tag.
    if req_url.scheme == "blob":
        return False

    # Fail open: if we can't decide if a request was internal or external,
    # assume it was external.
    return True

assert_page_is_self_contained(ctx, url)

I’ve wrapped these two steps in a function assert_page_is_self_contained(ctx, url), which is part of my chives utility library.

The function takes a Playwright Browser (or BrowserContext) and a URL, then:

  1. Opens a new page in the browser
  2. Loads the URL in the page
  3. Checks what happened when the page completes loading

It throws an exception if there are any:

The function tries to trigger important network requests that might be skipped on the initial page load. For example, it evaluates the body’s offsetHeight to force a synchronous reflow, which causes the browser to request any lazily loaded images or fonts that might otherwise be missed.

Here’s an example which verifies a file example.html from the working directory:

from chives.browser_fixtures import assert_page_is_self_contained, file_uri
from contextlib import closing
from playwright.sync_api import sync_playwright

with (
    sync_playwright() as p,
    closing(p.webkit.launch()) as browser,
):
    assert_page_is_self_contained(browser, url=file_uri("example.html"))

I originally passed a Page as the first argument, but changing the function to take a Browser or BrowserContext lets it manage its own pages. It gets a fresh page for every verification, and closes it when it’s done. (The first time I ran this, I created a new page for every verification, and never closed them – my computer ground to a crawl as my script opened hundreds of browser tabs.)

If you create a new context, you can emulate different browser settings, like location, locale, and window size. Here’s an example where we use contexts to test the same page in light/dark mode:

with (
    sync_playwright() as p,
    closing(p.webkit.launch()) as browser,
):
    with closing(browser.new_context(color_scheme="light")) as light_ctx:
        assert_page_is_self_contained(light_ctx, url=file_uri("example.html"))

    with closing(browser.new_context(color_scheme="dark")) as dark_ctx:
        assert_page_is_self_contained(dark_ctx, url=file_uri("example.html"))

In my web archive, I check every page in light/dark mode, and at 1×/2× pixel densities. It takes at least a second to run each verification, so I cache the results. As I save new pages, they’re added to a verification queue, and I’m warned of any missing assets.

Unsurprisingly, the first run flagged many missing fonts and images, which I added to my local archive. Every saved page now passes cleanly.

Browser is best-effort, but may still miss things

Even with automated testing, it’s difficult to guarantee a completely self-contained archive. Different instances of a browser might load different assets, and a single page load might miss different variations:

You can mitigate some of this by testing with multiple contexts or adding post-load interactions, but every step makes the process take longer. Fortunately, most of the pages I save are simple, blog-style pages where I’ve stripped out the JavaScript. For my specific use case, initial load verification is more than good enough.

Other uses for this code

There are other projects where it could be useful to intercept and audit the requests made by a web page:

I don’t do much front-end web development, but Playwright has become a key tool in my toolbox. I read a lot about people using AI agents and MCPs to interact with websites, but I prefer Playwright. It’s more fiddly and requires explicit instructions, but it’s constrained, deterministic, and I never have to worry about it doing something unexpected.