Article

Snapshot Testing CLI Help and Error Output

Store the rendered --help output and each error message as a checked-in text file, normalise the parts that legitimately vary — terminal width, temporary paths, version numbers — and assert equality. Help text is the one artefact where a stored expectation is right, because the exact wording is the contract users read. It builds on the Testing Geospatial CLI Tools guide, part of the broader CLI Architecture & Design Patterns reference.

Prerequisites

  • Python 3.10 or later
  • pip install pytest typer (or click)
  • A CLI with at least one subcommand group, as described in CLI Subcommand Organization

What Varies, and What Must Not

A snapshot is only useful if it fails when the interface changes and passes otherwise. Four things in a rendered help page vary for reasons that have nothing to do with your code, and each needs normalising before comparison.

Normalising a captured help page Terminal width changes wrapping and is pinned with the COLUMNS variable. Temporary paths differ per run and are replaced with a placeholder. The version string changes every release and is replaced. ANSI colour codes depend on the capture environment and are stripped. what varies why normalisation line wrapping terminal width differs set COLUMNS=100 in the fixture temporary paths tmp_path is per run substitute a TMP placeholder version strings changes every release substitute a VERSION placeholder colour codes depends on the capture disable colour at the source

Everything else — option names, argument order, the wording of a validation failure, the exit code shown in the epilogue — is exactly what the snapshot exists to pin.

Complete Working Implementation

# tests/snapshots.py
import os
import re
from pathlib import Path

SNAPSHOT_DIR = Path(__file__).parent / "snapshots"
_ANSI = re.compile(r"\x1b\[[0-9;]*m")


def normalise(text: str, tmp_path: Path | None = None, version: str | None = None) -> str:
    out = _ANSI.sub("", text)
    if tmp_path is not None:
        out = out.replace(str(tmp_path), "<TMP>")
    if version is not None:
        out = out.replace(version, "<VERSION>")
    out = re.sub(r"[ \t]+$", "", out, flags=re.M)     # trailing whitespace
    return out.rstrip("\n") + "\n"


def assert_snapshot(name: str, actual: str) -> None:
    """Compare against tests/snapshots/<name>.txt.

    Set UPDATE_SNAPSHOTS=1 to rewrite them after an intentional change.
    """
    path = SNAPSHOT_DIR / f"{name}.txt"
    if os.environ.get("UPDATE_SNAPSHOTS"):
        path.parent.mkdir(parents=True, exist_ok=True)
        path.write_text(actual, encoding="utf-8")
        return
    assert path.exists(), (
        f"no snapshot at {path}; run UPDATE_SNAPSHOTS=1 pytest to create it"
    )
    expected = path.read_text(encoding="utf-8")
    assert actual == expected, (
        f"{name} changed.\n"
        f"If the change is intended: UPDATE_SNAPSHOTS=1 pytest -k {name}"
    )

And the tests themselves stay short:

# tests/test_help_snapshots.py
import pytest
from typer.testing import CliRunner

from gistools import __version__
from gistools.cli import app
from tests.snapshots import assert_snapshot, normalise

runner = CliRunner()


@pytest.fixture(autouse=True)
def fixed_width(monkeypatch):
    monkeypatch.setenv("COLUMNS", "100")
    monkeypatch.setenv("NO_COLOR", "1")


@pytest.mark.parametrize("argv,name", [
    ([], "help_root"),
    (["raster", "--help"], "help_raster"),
    (["raster", "warp", "--help"], "help_raster_warp"),
    (["vector", "--help"], "help_vector"),
])
def test_help_text(argv, name):
    result = runner.invoke(app, argv + (["--help"] if not argv else []))
    assert result.exit_code == 0
    assert_snapshot(name, normalise(result.output, version=__version__))


def test_invalid_crs_message(raster_path, tmp_path):
    result = runner.invoke(app, ["raster", "warp", str(raster_path),
                                 str(tmp_path / "out.tif"), "--dst-crs", "EPSG:4362"])
    assert result.exit_code == 2
    assert_snapshot("error_invalid_crs",
                    normalise(result.output, tmp_path=tmp_path))

Snapshots suit a narrow band of output, and knowing where that band starts and stops prevents the technique spreading into places it makes tests worse.

What should and should not be snapshotted Help text and validation error messages are stable, human-facing and worth pinning exactly. A rendered progress bar contains timing and is never reproducible. Structured JSON output should be parsed and asserted on by field, so key order and whitespace do not matter. help text stable, human-facing, and the thing users read first snapshot it validation error messages changes here signal behaviour changes, not wording changes snapshot it structured JSON output key order and spacing are not part of the contract parse, then assert on fields a rendered progress bar contains elapsed time — never reproducible assert on a property instead

The third row is the one people get wrong most often. Snapshotting a JSON payload works until someone adds a key, at which point every snapshot in the file needs regenerating and the diff is too large to review — which is precisely when a real regression slips through unnoticed.

Keeping Snapshots Reviewable

A snapshot suite is only as good as the diffs it produces, and three habits keep those diffs readable a year later.

Split by command rather than by test file. One snapshot per help page means renaming an option in the raster group produces a one-file diff, and a reviewer can see the whole before-and-after without scrolling. A single combined snapshot of every help page produces a diff whose size is unrelated to the size of the change.

Keep the files free of anything a human would not write. Trailing whitespace, in particular, is invisible in review and produces diffs that appear empty; the normalise helper above strips it for exactly that reason. The same argument applies to a trailing newline: normalising to exactly one means an editor that adds or removes it does not create a spurious change.

Finally, treat a snapshot change as an interface change in the commit message. Update snapshots is not a useful log entry; rename --target-crs to --dst-crs is, and the snapshot diff below it is the evidence. Six months later, when someone is trying to work out when a flag changed name, that commit is the answer.

There is one more benefit that emerges once the snapshots exist: they document the CLI without anyone maintaining documentation. A new contributor reading tests/snapshots/help_raster.txt gets the current, accurate list of what the raster group does, generated by the code rather than written alongside it. That is not a reason to snapshot on its own, but it is a pleasant consequence, and it is why keeping the files as plain text rather than a serialised blob repays itself.

Step Annotations

  1. NO_COLOR rather than a runner flag. Setting the environment variable exercises the same detection path production uses, described in Detecting Non-TTY Output and Disabling Rich Color. A runner-level override would bypass it and leave that logic untested.
  2. COLUMNS=100, not 80. Eighty characters wraps most geospatial option names badly, producing snapshots that are hard to read in review. A hundred is wide enough to keep one option per line and narrow enough to fit a diff view.
  3. Parametrising over commands, not writing one test each. Adding a new subcommand becomes a one-line change plus one UPDATE_SNAPSHOTS run, which is low enough friction that people actually do it.
  4. The failure message contains the fix command. A maintainer who has just renamed an option should not have to search the repository to find out how to accept the change.
  5. Error messages are snapshotted too, and are the more valuable half. Help text changes when someone edits a docstring; error wording changes when someone alters behaviour, which is exactly when a reviewer should be looking.

One Named Gotcha: An Accepted Snapshot Hides a Real Regression

UPDATE_SNAPSHOTS=1 is a loaded gun. Run it while several tests are failing and you commit whatever the code currently does, including the bug you were about to find. The safeguard is procedural rather than technical, and it is worth writing into the contributing guide: update snapshots in their own commit, with nothing else in it, so a reviewer sees the diff of the interface on its own.

It also helps to make the snapshot directory conspicuous in review. Keeping the files as plain .txt under a single tests/snapshots/ directory means a diff shows the before and after text inline, rather than a binary blob or an unreadable single-line JSON string.

The workflow around a snapshot change has one shape that keeps reviews honest, and it is worth making explicit because the tempting shortcut is right there in the failure message.

What to do when a snapshot fails The safe path reads the diff first, confirms the change was intended, regenerates the snapshot and commits it on its own. The unsafe path regenerates immediately and mixes the result into a larger commit, where a real regression is invisible. snapshot fails read the diff was it intended? own commit regenerate at once mix into a big commit regression ships The two paths take about the same time. Only one of them leaves evidence a reviewer can use. A CI check that fails when snapshot files change alongside source files enforces the top path.

That CI check is three lines of shell against the diff of changed paths, and it is the only mechanical defence against the lower path. Everything else is a habit.

Verification

Confirm the snapshots are load-bearing by making a harmless-looking change and checking that exactly one fails:

# rename an option's help text by one word, then:
pytest tests/test_help_snapshots.py -q
# expect: 1 failed, the rest passed, and the diff shows the changed line

Then confirm the normalisation is doing its job by running the same suite twice from different directories and at a different terminal width:

(cd /tmp && COLUMNS=40 pytest tests/test_help_snapshots.py -q)

Both runs should pass. If the second fails, something environment-dependent is leaking into the captured output and needs adding to normalise.

Snapshots as a Deprecation Trail

Because a snapshot records exactly what users saw at a point in time, the file’s history becomes a usable record of how the interface has changed. That turns out to be worth more than it sounds when a support question arrives about behaviour someone remembers from an older release.

The practical use is checking a deprecation actually reached users before the removal did. A flag that was deprecated in one release and removed in the next should show a deprecation notice in the intermediate snapshot; if it does not, the notice was never rendered and the removal will be a surprise. Grepping the snapshot directory at release time — for the word deprecated, and for the names of anything scheduled for removal — is a cheap check that the warnings people planned actually shipped.

The same record helps with the opposite question. When a user reports that a command used to accept some option, the snapshot history answers it in seconds and without guesswork. Compare that with reconstructing the parser from an old tag, which requires checking out the tag, installing its dependencies, and running it.

None of this justifies snapshotting output you would not otherwise pin. It is a by-product of storing help text as plain files in version control rather than asserting on substrings, and it is one more reason to prefer the whole page over assert "--dst-crs" in result.output. A substring assertion tells you the flag exists; it tells you nothing about how it was described, which group it appeared under, or whether the surrounding text still made sense after a refactor moved it.