Article

Profiling Geospatial Python with py-spy

Attach py-spy to a running worker by process id, record for thirty seconds, and read the flame graph. Because it samples from outside the process it needs no code change and no restart, which matters when the slow behaviour only appears against production data. It builds on the Benchmarking Spatial Batch Pipelines guide, part of the broader Spatial Batch Processing & Async Workflows reference.

Prerequisites

  • Python 3.10 or later
  • pip install py-spy
  • Permission to attach to the target process — on Linux that usually means the same user, or --pid with elevated privileges
  • A worker that is currently slow; a profile of an idle process shows nothing

Which Profiler Answers Which Question

Three tools overlap and are good at different things, and picking wrongly wastes an afternoon.

Three profilers, three questions py-spy samples a running process from outside and shows where wall clock goes, including native frames. cProfile counts every Python call deterministically but adds overhead and cannot see into GDAL. GDAL's own CPL timing reports driver-level costs that neither Python profiler can attribute. py-spy samples a live process no code change, no restart sees native frames as one opaque block start here cProfile deterministic call counts significant overhead blind inside GDAL — it is all one C call for pure-Python hot spots CPL_DEBUG / CPL_TIMING driver-level detail shows block reads and decompression cost no Python context when GDAL dominates

For a geospatial batch the sequence is almost always py-spy first — because it costs nothing and runs against the real workload — and only then one of the other two, chosen by what py-spy pointed at.

Complete Working Implementation

# 1. Find the worker that is actually busy.
py-spy top --pid "$(pgrep -f 'gistools warp' | head -1)"

# 2. Record a flame graph over 30 seconds, following into subprocesses.
py-spy record --pid "$(pgrep -f 'gistools warp' | head -1)" \
              --duration 30 --subprocesses --output profile.svg

# 3. Dump every thread's stack right now — useful when nothing is moving.
py-spy dump --pid "$(pgrep -f 'gistools warp' | head -1)"

Wiring it into the tool itself, so a slow batch can be profiled without anyone finding the pid:

# gistools/profiling.py
from __future__ import annotations

import os
import shutil
import signal
import subprocess
from pathlib import Path


def install_profile_signal(out_dir: Path, duration: int = 30) -> None:
    """On SIGUSR1, record a flame graph of this process to out_dir."""
    if shutil.which("py-spy") is None:
        return

    def handler(signum, frame):
        out_dir.mkdir(parents=True, exist_ok=True)
        target = out_dir / f"profile-{os.getpid()}.svg"
        subprocess.Popen([
            "py-spy", "record", "--pid", str(os.getpid()),
            "--duration", str(duration), "--subprocesses",
            "--output", str(target),
        ])

    signal.signal(signal.SIGUSR1, handler)

With that installed, kill -USR1 <pid> against any worker produces a flame graph without stopping the batch — which is the only way to profile a problem that takes four hours to appear.

Step Annotations

  1. --subprocesses is not the default. A pool’s work happens in children, and a profile of the parent shows it waiting on a queue.
  2. Thirty seconds, not three. A sampling profiler needs enough samples for the shape to be stable; a very short record produces a graph dominated by whatever happened to be running.
  3. py-spy dump for a stalled process. When throughput has gone to zero, the question is not where time is going but what everything is blocked on, and a stack dump answers it immediately.
  4. The signal handler spawns rather than blocks. Recording inline would stall the worker for the duration, which changes the thing being measured.
  5. The shutil.which guard. Production images often omit the profiler; the handler should be a no-op there rather than a crash on an operator’s signal.

One Named Gotcha: GDAL Frames Look Like One Flat Block

py-spy shows native frames without Python context, so a batch spending its time inside reproject appears as a single wide bar labelled with a C symbol and nothing beneath it. That is correct and unhelpful — it tells you the time is in GDAL, not what GDAL is doing.

Turning that into an answer needs GDAL’s own instrumentation:

CPL_DEBUG=ON GDAL_CACHEMAX=256 python -m gistools warp in.tif out.tif \
  --dst-crs EPSG:3857 2>&1 | grep -E 'GDAL:|GTiff:' | sort | uniq -c | sort -rn | head

The counts usually make the cause obvious: many block reads for the same block means the cache is too small or the windows are misaligned, as covered in Streaming Raster Windows to Cap Memory in Mosaics. Many opens of the same file means the loop is reopening rather than reusing a handle.

A flame graph over geospatial code has a small number of recognisable shapes, and each points at a different fix.

Reading four common shapes A wide native block means the time is inside GDAL and the fix is an access-pattern change. A wide open call means the loop reopens datasets. A deep pure-Python stack means a per-feature loop that should be vectorised. A flat profile with little on the stack means the process is waiting. one wide native block time is inside GDAL — change the access pattern block alignment, overviews many frames in open() the loop reopens the same dataset hoist the open a deep Python stack a per-feature loop that should be vectorised operate on arrays almost nothing on the stack the process is waiting, not computing look at I/O

The last shape is the one people misread as a failed profile. A process that is waiting has little on the stack because there is little to sample, and that is itself the finding.

Verification

Confirm the profile is representative before acting on it:

# Record twice and check the top frames agree.
py-spy record --pid "$PID" --duration 30 --output p1.svg
py-spy record --pid "$PID" --duration 30 --output p2.svg
py-spy top --pid "$PID" --duration 10

If the two recordings disagree about the dominant frame, the workload is heterogeneous and one profile is not enough — record over a longer window, or profile a single unit type at a time.

The other check worth doing is whether the profile matches the benchmark. A profile saying the time is in decompression, alongside a benchmark whose p95 is set by a handful of large tiles, is a consistent story. A profile that disagrees with the benchmark usually means one of them sampled the wrong thing.

Profiling a distributed batch needs one extra decision: which process to attach to.

Choosing which process to profile The parent shows dispatch and collection, which is rarely the bottleneck. A single busy worker shows the real work. Every worker at once, with the subprocesses flag, shows aggregate behaviour but mixes units of different kinds. the parent dispatch and collection usually shows a queue wait and nothing else rarely useful one busy worker the actual work clearest picture of a single unit's cost start here all workers together aggregate behaviour mixes unit types — good for balance, not detail for imbalance

Profiling the parent and concluding that the pipeline is dominated by waiting is a common and misleading result: the parent is supposed to be waiting.

Profiling Memory Rather Than Time

py-spy answers where time goes and says nothing about where memory goes, which for a geospatial batch is frequently the more urgent question. Two tools cover it, and they see different halves.

tracemalloc sees Python allocations and attributes them to the line that made them. It is the right tool for a list that grows without bound or a cache nobody clears, and it is blind to anything GDAL allocates natively — which is most of the memory in a raster pipeline.

Resident set size, sampled from outside, sees everything and attributes nothing. It is the right tool for answering is the process growing and useless for answering why.

Used together they narrow the answer by elimination, which is the technique Profiling Native GDAL Memory with tracemalloc and RSS sets out. Growth visible in both is Python’s; growth visible only in RSS is native, and in a rasterio pipeline that almost always means datasets left open.

Profiling in Production Without Fear

The reason to reach for a sampling profiler is that it is safe to run against a real workload, and it is worth being precise about why.

It runs as a separate process and reads the target’s memory, so it does not modify the target, does not require a restart, and cannot introduce a bug into the code path being measured. The overhead is the cost of the samples, which at the default rate is a low single-digit percentage.

Two caveats apply. Attaching requires permission — the same user, or elevated privileges — which in a container usually means the profiler must run inside it or the container must be started with the appropriate capability. And a process under heavy memory pressure will be slowed slightly more by the profiler’s reads, which matters only if you are profiling the memory pressure itself.

Neither is a reason to avoid it. The alternative — reproducing a production workload locally to profile it — is slower, less accurate, and frequently impossible for a pipeline whose behaviour depends on data volume.

One habit makes profiles far more useful over time: save them with the commit hash and the corpus in the filename. A directory of flame graphs named by revision turns a vague memory that something used to be faster into a side-by-side comparison, and it costs nothing beyond a naming convention. The same applies to the py-spy dump output from a stalled run, which is often the only record that the stall ever happened.

Profiling a short-lived process

The signal-handler arrangement suits a long-running worker. A command that finishes in ten seconds gives no time to attach, and the answer is to profile the whole run rather than a window of it.

py-spy record -- python -m gistools warp in.tif out.tif starts the process under the profiler and records for its lifetime, which covers the short case completely. The output includes start-up, so the import cost appears as a wide block at the base — useful when that is the question, and worth subtracting mentally when it is not.

One further practical note: keep the recording duration short enough that the profile describes a single kind of work. A thirty-second window over a batch whose units all do the same thing is clean; the same window over a pipeline that alternates between reading and writing mixes two profiles into one flame graph and makes both harder to read.