Skip to main content

Run a randomly selected subset of tests with pytest

By reading the code for the pytest-random-order plugin, I was able to write a new plugin that runs a random subset of tests.

This is a question asked by Ned Batchelder on Mastodon.

There’s a plugin pytest-random-order that runs your pytest suite in random order, but it runs a complete set of tests.

I looked for a way to limit the number of tests that pytest runs, and I found a Stack Overflow answer by hoefling that implements a --limit=N flag for pytest. I was able to adapt it to run a random subset of tests.

Add this to

import random

def pytest_addoption(parser):
        help="Only run random selected subset of N tests.",

def pytest_collection_modifyitems(session, config, items):
    random_sample_size = config.getoption("--random-selection")

    if random_sample_size >= 0:
        items[:] = random.sample(items, k=random_sample_size)

I created a file with 100 empty tests, and tested this new flag was working correctly:

$ pytest --verbose --random-selection 5
===================== test session starts ======================
collected 100 items PASSED                            [ 20%] PASSED                            [ 40%] PASSED                            [ 60%] PASSED                             [ 80%] PASSED                            [100%]

====================== 5 passed in 0.01s =======================

By running it multiple times, I could see it selecting a different subset of 5 tests.