Article

Capturing GDAL Native Errors in Python Logs

Install a handler with gdal.PushErrorHandler(fn) and enable gdal.UseExceptions() at start-up. The first routes GDAL’s C-level messages — driver warnings, PROJ notices, HTTP diagnostics — into your Python logger as structured records; the second turns failures into exceptions your try/except can actually see. Without both, roughly half of what goes wrong in a geospatial batch never appears in the log at all. It builds on the Structured Logging for Geospatial CLIs guide, part of the broader CLI Architecture & Design Patterns reference.

Prerequisites

  • Python 3.10 or later
  • pip install rasterio — which vendors the osgeo bindings, or install GDAL directly if your build provides them separately
  • A configured logger, as set up in Emitting JSON Logs from a Typer CLI

Where GDAL’s Messages Go by Default

GDAL has its own error reporting, older and separate from Python’s. Understanding the two paths is what makes the fix obvious.

Two error paths, only one of which you see A GDAL call emits native messages through the CPL error handler, which writes raw text to stderr and never reaches Python logging. Python-level failures raise exceptions only when UseExceptions is on; otherwise the call returns None and the failure surfaces later as an AttributeError. a GDAL call Open, Warp, Translate CPL error handler default: raw text to stderr the return value None on failure, by default PushErrorHandler UseExceptions your logger one record per message

Only the two green arrows are things you add. Everything else happens whether you want it to or not, which is why a batch that appears to have logged nothing often has a great deal of diagnostic text sitting in a captured stderr stream nobody thought to look at.

Complete Working Implementation

# gistools/gdal_bridge.py
from __future__ import annotations

import logging
import re
from contextlib import contextmanager

from osgeo import gdal

_LEVELS = {
    gdal.CE_None: logging.DEBUG,
    gdal.CE_Debug: logging.DEBUG,
    gdal.CE_Warning: logging.WARNING,
    gdal.CE_Failure: logging.ERROR,
    gdal.CE_Fatal: logging.CRITICAL,
}

# Signed URLs and credentials turn up in GDAL's HTTP diagnostics.
_REDACT = re.compile(r"([?&])(X-Amz-[A-Za-z-]+|Signature|token|sig)=[^&\s]+", re.I)


def _clean(message: str) -> str:
    return _REDACT.sub(r"\1\2=<redacted>", message.strip())


def install_gdal_bridge(log: logging.LoggerAdapter | logging.Logger) -> None:
    """Route CPL messages into `log` and make GDAL failures raise."""

    def handler(err_class: int, err_num: int, message: str) -> None:
        level = _LEVELS.get(err_class, logging.INFO)
        log.log(level, "gdal_message", extra={
            "gdal_class": int(err_class),
            "gdal_code": int(err_num),
            "gdal_text": _clean(message),
        })

    gdal.PushErrorHandler(handler)
    gdal.UseExceptions()


@contextmanager
def gdal_quiet():
    """Temporarily silence CPL messages — for probes that are expected to fail."""
    gdal.PushErrorHandler("CPLQuietErrorHandler")
    try:
        yield
    finally:
        gdal.PopErrorHandler()

Calling it in the root callback, alongside the rest of the logging setup:

# gistools/cli.py  (inside the root callback, after _configure)
from gistools.gdal_bridge import install_gdal_bridge

log = get_logger("gistools").bind(command=ctx.invoked_subcommand or "root")
install_gdal_bridge(logging.getLogger("gistools.gdal"))

And the probe case, where a failure is an expected outcome rather than a problem:

from gistools.gdal_bridge import gdal_quiet


def is_readable(path: str) -> bool:
    """True if GDAL can open `path`. Failures here are normal, not worth logging."""
    with gdal_quiet():
        try:
            ds = gdal.Open(path)
        except RuntimeError:
            return False
    return ds is not None

Step Annotations

  1. gdal.CE_None maps to DEBUG, not to nothing. GDAL emits informational messages through the same channel, and dropping them loses useful context about which driver was chosen.
  2. The message is redacted before it becomes a field. GDAL’s HTTP diagnostics include the full request URL, which for a signed object-store read contains a credential. Redacting at the bridge means every downstream consumer is safe, rather than each one remembering.
  3. err_num is kept as gdal_code. It is the stable identifier — the text changes between versions, the number does not — so any alert or filter should key on the code.
  4. UseExceptions() is called after the handler is installed. Order does not strictly matter, but installing the handler first means any message emitted during the switch is captured too.
  5. gdal_quiet uses GDAL’s own quiet handler rather than removing yours. Pushing and popping keeps the stack correct even if the block raises, and CPLQuietErrorHandler is the documented way to discard rather than a homemade no-op.

Not every GDAL message deserves the same treatment. Sorting them by what a reader should do turns a noisy stream into a useful one.

What level each kind of GDAL message deserves Metadata and overview notices are debug detail. A projection fallback or a datum shift approximation is a warning because it changes results. A driver or I/O failure is an error. A probe that was expected to fail should be suppressed entirely. metadata and overview notices "overviews are out of date", driver selection detail debug projection fallbacks a missing grid file, an approximated datum shift — results change warning driver and I/O failures unsupported format, unreadable object, write refused error expected probe failures wrap in gdal_quiet — not a problem, so not a record

The second row is the one people under-rate. A missing PROJ grid file downgrades a transform to an approximation silently as far as Python is concerned, and the only notice you get is a CPL warning — which is precisely the message the bridge exists to capture.

One Named Gotcha: The Handler Is Per Thread, Not Per Process

GDAL’s error-handler stack is thread-local. A handler installed on the main thread does not apply to messages emitted from a worker thread, so a pipeline that offloads warps to a ThreadPoolExecutor — a reasonable choice, since the warp releases the interpreter lock — silently loses every message from inside the pool.

The fix is to install the bridge in each thread, which is what a thread pool’s initializer is for:

from concurrent.futures import ThreadPoolExecutor

pool = ThreadPoolExecutor(
    max_workers=8,
    initializer=install_gdal_bridge,
    initargs=(logging.getLogger("gistools.gdal"),),
)

The same applies to processes, for a different reason: a spawned process starts with no handler at all. In both cases the initializer is the right place, and forgetting it produces the same symptom — a log that looks clean while stderr fills with text.

Verification

Trigger each class of message deliberately and confirm it lands as a record:

# A warning: open a file whose overviews are out of date.
gis raster info stale-overviews.tif --log-json run.jsonl
jq -c 'select(.event=="gdal_message") | {level, gdal_code, gdal_text}' run.jsonl

Expected output shows the message as fields rather than as loose text:

{"level":"warning","gdal_code":0,"gdal_text":"overviews are out of date"}

Then confirm the redaction works, which is the check worth automating:

# Read from a signed URL and grep the log for anything that looks like a credential.
gis raster info "https://bucket.example/tile.tif?X-Amz-Signature=abc123" --log-json run.jsonl
grep -c 'X-Amz-Signature=abc123' run.jsonl        # expect 0
grep -c 'Signature=<redacted>' run.jsonl          # expect at least 1

Finally, confirm UseExceptions is actually in force, since its absence is silent:

python - <<'PY'
from osgeo import gdal
from gistools.gdal_bridge import install_gdal_bridge
import logging
install_gdal_bridge(logging.getLogger("x"))
try:
    gdal.Open("/definitely/not/here.tif")
except RuntimeError as exc:
    print("raises correctly:", exc)
else:
    print("NOT raising — UseExceptions did not take effect")
PY

Restoring the Default When You Are a Library

If your package is imported by other people’s code, installing a global error handler is a decision you are making on their behalf. The polite arrangement is to scope it.

Push and pop rather than install The host application's handler sits at the bottom of the stack. A library pushes its own handler for the duration of a call and pops it on exit, so the host's handler is restored whether the call succeeded or raised. host handler installed yours pushed on top popped — host restored with gdal_logging(log): result = do_the_work(...) the finally clause pops even when the block raises, so the stack never drifts

An application entry point can still install once and never pop — that is the right behaviour there, because the application owns the process. The distinction is ownership, not correctness: whoever owns the process gets to make the global decision, and a library does not.

Two further details make the scoped version safe under concurrency. The handler stack is thread-local, so a context manager entered on one thread does not affect another — which is convenient here, since it means a library call on a worker thread cannot disturb the main thread’s handler. And because the stack is a stack, nesting works: an inner gdal_quiet inside an outer logging handler suppresses messages for its own block and restores logging afterwards, with no bookkeeping of your own.

Which Messages Are Worth Alerting On

Once GDAL’s messages are records rather than text, they become alertable — and the temptation is to alert on all of them, which produces a channel nobody reads. Three codes are worth a rule and the rest are worth a dashboard.

A datum-shift approximation warning means results are less accurate than the CRS definition claims, usually because a grid file is missing from the image. It is silent otherwise, it changes numbers, and it is fixed by installing the grid — which makes it exactly the shape of thing an alert should catch.

An unsupported-driver failure means the image is missing a format the pipeline needs. It fails every unit of that type, so the alert fires once and saves the rest of the run.

A repeated I/O failure against the same object means something between you and the storage is degraded. Individually these are noise; a rate above baseline is signal, which is why the rule should be on the rate rather than on the event.

Everything else — overview notices, driver selection detail, metadata warnings — belongs on a dashboard where a spike is visible and a single occurrence is not an interruption.

Message Volume in a Batch

GDAL is talkative under some conditions, and a batch that opens ten thousand objects can produce more CPL messages than application records. Two settings bound it.

Raising the bridge’s effective level to WARNING in production keeps the debug chatter out of the stored stream while leaving it available under a verbose flag. And setting CPL_DEBUG to a specific subsystem rather than ON — CPL_DEBUG=GTiff, say — narrows the firehose to the driver you are actually investigating.

The combination that works day to day is: bridge installed always, level at WARNING, CPL_DEBUG unset. When something needs investigating, one run with --verbose and CPL_DEBUG set to the relevant subsystem produces a detailed record of exactly that run, and nothing else changes.