Article

Classifying Recoverable and Fatal GDAL Errors

Sort every failure into retry, skip or abort before any retry logic runs, keyed on the exception type and GDAL’s numeric error code rather than on message text. Message wording changes between versions; codes and types do not. It builds on the Error Handling in Spatial Pipelines guide, part of the broader Spatial Batch Processing & Async Workflows reference.

Prerequisites

Three Responses, Not Two

Every failure is one of three things A retryable failure is one where the same call may succeed later: a reset connection, a throttle, a timeout. A skippable failure is permanent for this input but harmless to the batch: a corrupt file, an invalid geometry. A fatal failure invalidates the whole run: a missing driver, exhausted credentials, a full destination. retry the same call may succeed shortly transient network and I/O skip permanent for this input, harmless to the rest corrupt or invalid data abort the whole run is invalid from here environment or credentials

Collapsing this to two — retry or fail — is what produces the two familiar bad behaviours: a batch that dies on its first corrupt tile, or one that retries a missing driver four times per input for ten thousand inputs.

Complete Working Implementation

# gistools/error_policy.py
from __future__ import annotations

import enum
import errno
import socket
from dataclasses import dataclass

import rasterio.errors as rio_errors


class Disposition(enum.Enum):
    RETRY = "retry"
    SKIP = "skip"
    ABORT = "abort"


# GDAL CPLE_ codes worth naming. The numbers are stable across versions.
CPLE_AppDefined = 1
CPLE_OutOfMemory = 2
CPLE_FileIO = 3
CPLE_OpenFailed = 4
CPLE_NotSupported = 6
CPLE_NoWriteAccess = 7
CPLE_HttpResponse = 11

_RETRY_CODES = {CPLE_FileIO, CPLE_HttpResponse}
_ABORT_CODES = {CPLE_OutOfMemory, CPLE_NotSupported, CPLE_NoWriteAccess}

_RETRY_ERRNOS = {errno.EAGAIN, errno.EBUSY, errno.ECONNRESET,
                 errno.ETIMEDOUT, errno.EPIPE, errno.EHOSTUNREACH}


@dataclass(frozen=True)
class Decision:
    disposition: Disposition
    reason: str


def classify(exc: BaseException) -> Decision:
    """Map an exception onto one of three responses."""
    # 1. Transport-level failures: almost always worth another attempt.
    if isinstance(exc, (socket.timeout, TimeoutError, ConnectionError)):
        return Decision(Disposition.RETRY, "transport failure")

    if isinstance(exc, OSError) and exc.errno in _RETRY_ERRNOS:
        return Decision(Disposition.RETRY, f"errno {exc.errno}")

    # 2. Environment problems: nothing about this run will improve.
    if isinstance(exc, (MemoryError, rio_errors.DriverRegistrationError)):
        return Decision(Disposition.ABORT, "environment cannot support the run")

    if isinstance(exc, FileNotFoundError):
        return Decision(Disposition.SKIP, "source is missing")

    if isinstance(exc, PermissionError):
        return Decision(Disposition.ABORT, "credentials or permissions")

    # 3. GDAL's own codes, where available.
    code = getattr(exc, "code", None) or getattr(exc, "err_no", None)
    if isinstance(code, int):
        if code in _RETRY_CODES:
            return Decision(Disposition.RETRY, f"CPLE code {code}")
        if code in _ABORT_CODES:
            return Decision(Disposition.ABORT, f"CPLE code {code}")

    # 4. Data problems: this input is bad, the batch is fine.
    if isinstance(exc, (rio_errors.RasterioIOError, rio_errors.CRSError,
                        ValueError)):
        return Decision(Disposition.SKIP, type(exc).__name__)

    # 5. Anything unrecognised aborts. An unknown failure is not a safe retry.
    return Decision(Disposition.ABORT, f"unclassified {type(exc).__name__}")

Applying it inside a unit:

from gistools.error_policy import Disposition, classify


def process(unit, log, dead_letters, attempt: int = 0) -> bool:
    try:
        do_work(unit)
        return True
    except BaseException as exc:                   # noqa: BLE001 — classified below
        decision = classify(exc)
        log.warning("unit_failed", extra={"unit": unit.key,
                                          "disposition": decision.disposition.value,
                                          "reason": decision.reason})
        if decision.disposition is Disposition.RETRY and attempt < 4:
            raise                                  # let the retry wrapper see it
        if decision.disposition is Disposition.SKIP:
            dead_letters.record(unit, exc)
            return False
        raise                                      # abort: propagate

Step Annotations

  1. Transport failures are checked first. ConnectionError is also an OSError, and checking the broader class first would swallow the specific case.
  2. FileNotFoundError skips rather than retries. A missing source will not appear on the fourth attempt, and retrying it multiplies a common failure by five.
  3. PermissionError aborts. Expired credentials affect every remaining unit, so continuing wastes the rest of the run producing identical failures.
  4. Unrecognised exceptions abort. The safe default for something you have not classified is to stop, because retrying an unknown failure can duplicate side effects.
  5. except BaseException with a re-raise. It catches KeyboardInterrupt long enough to log and then re-raises, so an interrupted batch still records where it stopped.

Reading the Decision Back Out

What the log says when the policy is wrong A high retry rate concentrated on one error class usually means something permanent is classified as transient. A skip set that grows through the run means a data problem is being tolerated rather than reported. An abort late in a long run means a condition that should have been checked at start-up. many retries, one class something permanent is classified as transient move it to skip or abort wasted time skips grow steadily a data problem is being tolerated silently alert on the skip rate hidden quality issue abort at 90% a precondition was not checked at start-up move the check earlier wasted compute

All three are visible from the structured records the classification writes, which is the argument for logging the disposition rather than just the exception.

One Named Gotcha: Message Matching Breaks on Upgrade

Classifying on "Access denied" in str(exc) works until GDAL rewords the message, at which point every such failure falls through to the default. Because the default aborts, the symptom is a batch that suddenly stops on inputs it used to skip — after a dependency bump nobody connected to it.

Codes and types are stable; strings are not. Where a code genuinely is unavailable, prefer matching on the exception type alone and accept the coarser decision rather than pinning behaviour to prose.

Verification

# Exercise each branch deliberately with a fault-injection fixture.
pytest tests/test_error_policy.py -q

# In a real run, the disposition mix should look sane.
jq -r 'select(.event=="unit_failed") | .disposition' run.jsonl | sort | uniq -c

Expect retries to be a small share, skips to be a known and stable share, and aborts to be zero on a healthy run. An abort count above zero on a run that completed means something was classified as fatal and then swallowed somewhere, which is worth finding.

Testing a classification policy means producing each failure deliberately, and each one has a cheap way to induce it.

Inducing each failure class on purpose A missing file needs only a wrong path. A corrupt raster is a real file truncated to half its length. A refused connection is a URL pointing at a closed port. An exhausted destination is a small tmpfs mount filled before the write. a missing source point at a path that does not exist expect SKIP a corrupt raster truncate a real file to half its bytes expect SKIP a refused connection a URL on a closed local port expect RETRY a full destination a tiny tmpfs mount, filled first expect ABORT

Each of those is a few lines in a test and pins one branch of the policy, which is what stops a refactor quietly turning a skip into an abort.

Where the Policy Lives

A classification function is only useful if every failure path goes through it, and the temptation is to handle some failures locally because they seem obvious.

Keeping one policy function, called from one place, is what makes the behaviour predictable and the log honest. A try/except inside a helper that swallows a RasterioIOError and returns None means that failure never reaches the classifier, never appears in the disposition counts, and never lands in the dead-letter store — and the unit reports success with no output.

The rule that works is that helpers raise and only the unit boundary catches. It is a small discipline and it keeps the whole error story in one readable function.

Evolving the Policy Safely

The classification will need changing — a new error class appears, or something classified as fatal turns out to be transient. Two habits make those changes safe.

Add a test alongside the change, inducing the failure and asserting the disposition. The test is usually four lines and it pins the decision against a future refactor.

And log the disposition on every failure, not just the interesting ones. The counts are what tell you whether a change had the intended effect, and a policy change deployed without them is a change whose outcome nobody can see. A week of counts before and after is enough to know whether the retry rate moved in the direction you wanted.

Codes Worth Knowing

A handful of GDAL error codes account for most of what a batch pipeline sees, and knowing them by number rather than by message makes the classification durable.

CPLE_OpenFailed covers a dataset that could not be opened, which is almost always permanent for that input: a missing file, an unsupported format, a corrupt header. Retrying it wastes time.

CPLE_FileIO covers a read or write that failed partway, which is frequently transient — a network blip, a full disk that was subsequently freed, a truncated response. It is the code most worth retrying.

CPLE_OutOfMemory covers a native allocation failure. It is not retryable in the ordinary sense, because the same allocation will fail again, but it is recoverable in a different sense: a smaller window would succeed. A pipeline that halves its window and retries once on this code specifically converts a hard failure into a slower success.

CPLE_NotSupported covers a driver or a capability that this build does not have. It affects every input of that type, so it belongs in the abort column rather than the skip one — continuing produces ten thousand identical failures.

CPLE_AppDefined is the catch-all and carries no information by itself. Anything classified from it has to fall back on the exception type, which is why the policy above checks types before codes.

Recording the Classification

A disposition that is decided and not recorded is a decision nobody can review. Logging it alongside the error class turns the policy into something measurable: a week of counts shows whether the retry rate is where you expected, whether skips are concentrated on one source, and whether anything is falling through to the unclassified default.

That last count is the one to watch. An unclassified failure aborts the run by design, and a non-zero count means a class of error the policy has not seen — which is exactly the moment to add a branch and a test rather than to widen the default.

A final practical point: keep the classifier free of I/O. A policy function that opens a file to decide whether a failure is retryable can itself fail, and a failure inside the error handler is the hardest kind to diagnose. Everything the decision needs should be in the exception already.