Benchmarking Spatial Batch Pipelines

Most geospatial performance work is guesswork wearing a stopwatch. A benchmark that runs on a representative fixture, separates cold from warm, reports the tail rather than the mean, and records the environment it ran in turns tuning into something you can defend. It is part of the Spatial Batch Processing & Async Workflows guide.

Prerequisites

• Python 3.10 or later, with pip install pytest-benchmark py-spy rasterio pyogrio.
• A pipeline that already runs correctly. Benchmarking a pipeline with a bug measures the bug.
• Somewhere consistent to run. A laptop with a thermal limit produces numbers that vary by a third between runs.
• Structured records from the run itself, as set up in Structured Logging for Geospatial CLIs — the log usually contains most of what a benchmark wants.

Problem framing

Three habits make geospatial benchmarks misleading, and all three are easy to fall into.

The first is benchmarking on a fixture that does not resemble production. A hundred tiles of the same size, all cached, all in the same CRS, measure a code path that never runs against real data — where sizes vary by two orders of magnitude, half the inputs are cold, and a tenth need a datum shift.

The second is reporting the mean. A batch’s wall clock is set by its slowest units, and a mean over ten thousand tiles hides the forty that take a minute each. The number that predicts how long the job takes is the ninety-fifth percentile, not the average.

The third is not recording the environment. A result measured with a warm page cache, on a machine with different GDAL, is not comparable to one taken last month, and without the environment recorded there is no way to know.

What to measure, and what it tells you

Reporting all four is what stops an optimisation trading one for another silently. A change that doubles throughput and triples peak memory is not an improvement if the node has no headroom.

Step-by-step implementation

1. Build a fixture that represents the real distribution

# bench/fixture.py
import numpy as np
import rasterio
from rasterio.transform import from_origin

# Sizes drawn from a real archive: mostly small, a long tail of large ones.
SIZES = [(256, 256)] * 40 + [(1024, 1024)] * 12 + [(4096, 4096)] * 3 + [(8192, 8192)]
CRS_MIX = ["EPSG:4326"] * 40 + ["EPSG:27700"] * 12 + ["EPSG:3857"] * 4


def build_corpus(root, seed: int = 0) -> list[str]:
    rng = np.random.default_rng(seed)
    paths = []
    for i, ((h, w), crs) in enumerate(zip(SIZES, CRS_MIX)):
        path = root / f"scene_{i:03d}.tif"
        profile = dict(driver="GTiff", height=h, width=w, count=1,
                       dtype="float32", crs=crs, tiled=True,
                       blockxsize=256, blockysize=256,
                       transform=from_origin(-1.0, 51.6, 0.001, 0.001))
        with rasterio.open(path, "w", **profile) as dst:
            dst.write(rng.random((h, w), dtype="float32"), 1)
        paths.append(str(path))
    return paths

The distribution is the point. A corpus of identical tiles measures the median case and tells you nothing about the tail that actually sets the wall clock.

2. Separate cold from warm explicitly

import subprocess


def drop_caches() -> None:
    """Best-effort page-cache drop. Requires privileges; skip cleanly without."""
    try:
        subprocess.run(["sync"], check=True)
        with open("/proc/sys/vm/drop_caches", "w") as handle:
            handle.write("3")
    except (OSError, PermissionError):
        print("warning: could not drop caches — results are warm-cache only")

Both numbers matter and they answer different questions. Cold measures what a scheduled job sees on a fresh node; warm measures what an interactive user sees on a second run. Quoting one as if it were the other is the most common way a benchmark misleads.

3. Record the environment alongside the result

Without the fourth row, a benchmark that improved is indistinguishable from a corpus that shrank.

Configuration integration

Benchmarks belong in the repository next to the tests, not in a notebook on someone’s machine. The practical arrangement is a bench/ directory that shares fixtures with the test suite described in Testing Geospatial CLI Tools, a make target that runs them, and a JSON result file committed per release rather than per commit.

Committing results per commit produces noise; committing them per release produces a history that answers when did this get slower. That question comes up far more often than is this faster than yesterday, and only the second arrangement can answer it.

Error handling and gotchas

A benchmark that shares a cache with the previous iteration measures the cache. Rebuild the corpus, or at least drop caches, between variants.

Thermal throttling makes the last variant look worst. Randomise the order of variants and take the minimum of several runs rather than the mean of one.

A single-threaded benchmark on a busy machine measures the other tenants. Run on an idle host, or report the interquartile range so the noise is visible.

Timing a run that failed halfway looks like an improvement. Assert the output count before recording any number.

A benchmark result is only comparable against another taken the same way, so it is worth recording which of the four conditions each run used.

Recording all four in the result file makes a comparison self-checking: a script can refuse to compare two runs whose conditions differ rather than producing a misleading percentage.

Verification

# Two runs of the same variant should agree within a few percent.
python -m bench.run --variant baseline --repeat 3 --json bench/baseline.json
jq '.summary | {p50, p95, throughput}' bench/baseline.json

If two runs of the same variant differ by more than about five percent, the harness is measuring noise and no comparison between variants is meaningful yet. Fix that before trusting anything else.

Performance notes

Profiling and benchmarking answer different questions and neither substitutes for the other. A benchmark tells you the pipeline got slower; a profile tells you where. Running py-spy against a live worker for thirty seconds is usually enough to find the hot frame, and it needs no code change — which matters because instrumenting the code changes what you are measuring.

For geospatial work specifically, the profile is often dominated by frames inside GDAL, which tells you the answer is a different access pattern rather than better Python. Block-aligned reads, fewer opens, and overviews for reduced-resolution work are the three changes that move that number.

Turning measurements into a decision

A benchmark exists to settle a question, and the questions worth settling in a spatial pipeline are narrower than they first appear. Four cover most of what teams actually argue about.

Is this change faster? Needs two runs under identical conditions and a difference larger than the harness’s own variance. If the harness varies by five percent, a four percent improvement is not a result — which is why establishing the variance comes before any comparison.

Will this fit on the node? Needs peak memory across the whole pool at the busiest moment, not the average and not the per-worker figure. That number is what the container limit is compared against, and it is the one that decides whether a batch survives a larger input next month.

Where should I spend the next day? Needs a breakdown by stage rather than a total. A pipeline whose time is sixty percent in decode has a different answer from one whose time is sixty percent in waiting, and a single wall-clock figure cannot distinguish them.

Is it getting worse? Needs a history, which is the argument for committing results rather than printing them. Without a stored series the question is unanswerable, and the version of it that matters — when did this start — is unanswerable even with two data points.

The corpus is the experiment

More benchmarking effort is wasted on the harness than on the corpus, and the corpus is what decides whether the numbers transfer to production.

Three properties matter. The size distribution should match the real archive, including the tail: a corpus of uniform tiles measures the median case and misses the units that set the wall clock. The CRS mix should match, because a datum shift with a grid file is materially more expensive than one without and a single-CRS corpus never exercises it. And the cache state should be controlled, because a corpus small enough to sit entirely in the page cache measures a machine’s memory rather than its storage.

Building such a corpus synthetically, as the fixture above does, is usually better than copying real data. It is small enough to keep in the repository as code, it is reproducible from a seed, and it can be adjusted deliberately — doubling the tail, or adding a CRS — to answer a specific question. Copying production data gives realism at the cost of reproducibility, and the realism is usually in properties you could have specified.

Keeping the result honest over time

A measurement is a claim about a moment, and the claim decays. Three habits keep a benchmark useful rather than decorative.

Record the conditions with the number, not beside it. A result file that carries the library versions, the machine shape and a corpus hash can be compared automatically; a number in a commit message cannot, and within a quarter nobody remembers whether it was cold or warm.

Re-measure the baseline when anything underneath changes. A GDAL upgrade, a base-image bump or a move to a different instance family invalidates every stored comparison, and continuing to compare against the old baseline produces a percentage that means nothing. Re-running the baseline takes minutes and restores the meaning.

Prefer a smaller corpus run more often to a larger one run once. A benchmark that takes forty minutes will be run before releases and forgotten in between; one that takes four will be run whenever someone is curious, and the curiosity is what catches regressions early.

What to do with a regression

Finding one is the easy part. The sequence that resolves them quickly is narrow, then profile, then fix — in that order, because each step makes the next cheaper.

Narrowing means bisecting on the measurement rather than on a test. A benchmark harness that runs from the command line against a fixed corpus can be driven by git bisect run, which turns it got slower somewhere in the last twenty commits into a specific commit in a few minutes of unattended compute.

Profiling then answers where, against the identified commit rather than against the whole history. The profile of a known-bad revision compared with the profile of its parent usually makes the cause obvious, because only one thing changed.

Fixing is then ordinary work, with one caveat worth stating: verify the fix against the same corpus and the same conditions, and record the new number. A regression fixed without re-measuring has a way of coming back, because nobody knows whether the fix recovered all of it or half.

FAQ

Benchmarks as a release artefact

The most useful thing a benchmark produces is not a number but a record. Committing the result file with each release turns the series into something a team can reason about, and three questions become answerable that otherwise are not.

Did this release change performance? A diff between two result files answers it in seconds, and the answer is specific: throughput up four percent, p95 unchanged, peak memory up eleven percent. That last figure is the one a wall-clock comparison would have missed entirely.

Which release introduced this? A series of committed results bisects by inspection. Without one, the investigation starts by rebuilding old versions, which for a geospatial tool means also reconstructing the dependency set they were built against.

Is the machine we benchmark on still representative? A result file carrying the machine shape makes drift visible — a runner replaced with a different instance family shows up as a step change across every metric at once, which is recognisably different from a code regression.

The overhead is one file per release and a job that produces it. The alternative, which most projects have, is a performance history that exists only in people’s memories and is consulted by asking whoever has been there longest.

The measurements that pay for themselves first

For a team adding benchmarking to an existing spatial pipeline, three measurements return the most for the least effort, and they can be taken before any harness exists.

Peak memory across the pool, sampled during a normal run, answers whether the pipeline has headroom. It needs no fixture and no baseline — the number is either comfortably under the limit or it is not, and a pipeline running at ninety percent is one dataset away from an incident.

Request count for one representative unit, taken with GDAL’s tracing on, answers whether cloud reads are behaving. A windowed read making dozens of requests is misconfigured, and the finding does not depend on comparing against anything.

Per-unit duration percentiles, computed from the structured records the pipeline already emits, answer where the wall clock goes. If p95 is ten times p50, the batch is dominated by a small number of units, and that fact alone redirects the next day of work away from the median case.

None of the three needs a corpus, a baseline or a dedicated machine. They are worth taking first precisely because they turn up the largest problems before any effort goes into measuring smaller ones accurately.

A short checklist before quoting a number

Before a benchmark result leaves the machine it was taken on, five things are worth confirming. The corpus is representative of the real size distribution. The cache state is recorded and matches whatever it is being compared against. The run completed — the output count matches the input count, so a partial failure is not being reported as a speed-up. The variance between repeats is small enough that the difference being claimed is larger than the noise. And the library versions are in the result file, so the comparison remains valid when someone repeats it in six months.

Each takes seconds. Skipping them is how a number that meant something in one context gets quoted in another where it does not.

Related

• Measuring Throughput of a Raster Batch Pipeline — the harness that produces the numbers above.
• Profiling Native GDAL Memory with tracemalloc and RSS — the memory half of the same question.
• Spatial Batch Processing & Async Workflows — the pipelines these benchmarks measure.