Retrying Transient GDAL I/O Errors with Exponential Backoff

Wrap the remote read in a tenacity retry that fires only on transient failures — RasterioIOError carrying an HTTP 5xx or throttling signal — using wait_exponential plus wait_random jitter and stop_after_attempt, and let permanent failures like a missing key, a CRS mismatch, or an unsupported format raise straight through with exit code 10 or 11. That single distinction keeps a flaky S3 backend from killing an otherwise healthy batch. This page is part of the Error Handling in Spatial Pipelines guide inside the broader Spatial Batch Processing & Async Workflows reference.

Prerequisites

• Python 3.10 or later
• pip install tenacity rasterio (rasterio 1.3+ bundles GDAL 3.4+)
• GDAL virtual filesystem access configured for your backend: AWS credentials for /vsis3/, or a plain /vsicurl/ URL for public HTTPS objects

When a network read fails inside a worker, the exception surfaces from GDAL’s C layer as a rasterio RasterioIOError. The retry logic here lives entirely around the read call, so it composes cleanly with the concurrency model described in Async I/O for Raster Processing and with the recovery path in Error Handling in Spatial Pipelines.

Transient vs Permanent: the decision that drives everything

Retrying is only useful when a repeat of the exact same call could succeed. A throttled S3 request or a momentary 503 will clear on its own; a 404 or a corrupt file never will. Blindly retrying every exception turns a fast, deterministic failure into a slow one — five attempts with backoff against a missing key wastes 30-plus seconds per file and multiplied across a batch that is hours of dead time. The SVG below is the classifier that the code implements.

Complete Working Implementation

The script reads a raster window from a cloud object over /vsis3/ or /vsicurl/. The retry wraps only the read. Classification lives in is_transient, and permanent conditions raise dedicated exceptions that map to exit codes. Copy it, set the URL, and run:

#!/usr/bin/env python3
"""
Retry transient cloud GDAL reads with exponential backoff and jitter.
Usage: python read_with_retry.py /vsis3/my-bucket/scenes/scene_001.tif
"""
import sys
import logging
import rasterio
from rasterio.windows import Window
from rasterio.errors import RasterioIOError
from tenacity import (
    retry,
    retry_if_exception,
    stop_after_attempt,
    wait_exponential,
    wait_random,
    before_sleep_log,
)

logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s [%(levelname)s] %(message)s",
    handlers=[logging.StreamHandler(sys.stdout)],
)
log = logging.getLogger("gdal-retry")

# Substrings GDAL/CURL surface for a temporary backend condition.
TRANSIENT_MARKERS = (
    "http error 500", "http error 502", "http error 503", "http error 504",
    "timed out", "timeout", "connection reset", "connection refused",
    "temporarily", "slowdown", "throttl", "please try again",
)
# Substrings that mean retrying can never help.
PERMANENT_MARKERS = (
    "http error 403", "http error 404", "no such file",
    "not recognized as", "not a supported", "does not exist",
)


class UnsupportedFormat(Exception):
    """Raised for a permanent format error -> exit 11."""


class CrsMismatch(Exception):
    """Raised when the raster CRS is not the expected one -> exit 10."""


def is_transient(exc: BaseException) -> bool:
    """Return True only for errors a bit-for-bit retry could clear.

    We match on the message because GDAL flattens HTTP status and CURL
    conditions into the RasterioIOError string; there is no status attribute.
    """
    if not isinstance(exc, RasterioIOError):
        return False
    msg = str(exc).lower()
    if any(p in msg for p in PERMANENT_MARKERS):
        return False
    return any(t in msg for t in TRANSIENT_MARKERS)


@retry(
    retry=retry_if_exception(is_transient),   # ONLY transient reads are retried
    wait=wait_exponential(multiplier=1, max=30) + wait_random(0, 2),
    stop=stop_after_attempt(5),               # bound worst-case per file
    before_sleep=before_sleep_log(log, logging.WARNING),
    reraise=True,                             # surface the real error, not RetryError
)
def read_window(url: str, expected_epsg: int = 4326) -> "tuple":
    """Read the top-left 512x512 window from a remote raster.

    This function is a PURE READ. It is safe to run any number of times,
    which is exactly why the retry decorator belongs here and nowhere near
    a write. Permanent conditions are converted to typed exceptions that
    escape the retry predicate untouched.
    """
    with rasterio.open(url) as ds:
        if ds.crs is None or ds.crs.to_epsg() != expected_epsg:
            # Permanent: no number of retries fixes a wrong projection.
            raise CrsMismatch(
                f"{url} is EPSG:{ds.crs.to_epsg() if ds.crs else 'unknown'}, "
                f"expected EPSG:{expected_epsg}"
            )
        window = Window(col_off=0, row_off=0, width=512, height=512)
        data = ds.read(1, window=window)
        return data, ds.profile


def main() -> None:
    if len(sys.argv) != 2:
        log.error("usage: read_with_retry.py <vsis3-or-vsicurl-url>")
        sys.exit(2)
    url = sys.argv[1]
    try:
        data, profile = read_window(url, expected_epsg=4326)
    except CrsMismatch as exc:
        log.error("CRS mismatch: %s", exc)
        sys.exit(10)
    except UnsupportedFormat as exc:
        log.error("unsupported format: %s", exc)
        sys.exit(11)
    except RasterioIOError as exc:
        # Reached only when the error was permanent OR retries were exhausted.
        log.error("read failed permanently after retries: %s", exc)
        sys.exit(1)
    log.info("read %s block: shape=%s dtype=%s", url, data.shape, data.dtype)
    sys.exit(0)


if __name__ == "__main__":
    main()

Step Annotations

1. retry_if_exception(is_transient) — This is the whole design. Tenacity calls is_transient on every raised exception; only a True result triggers another attempt. A CrsMismatch, an UnsupportedFormat, or a RasterioIOError carrying 404 returns False and propagates on the first try.

2. wait_exponential(multiplier=1, max=30) + wait_random(0, 2) — Tenacity lets you add wait strategies. The exponential term grows the base delay (roughly 1s, 2s, 4s, 8s) while max=30 caps it; the random term adds up to 2 seconds of jitter so a fleet of workers does not retry in lockstep after a shared outage.

3. stop_after_attempt(5) — Bounds the worst case. Five attempts against a capped delay keeps a single stubborn object from stalling the batch. Pair the attempt cap with the delay cap; neither alone is sufficient.

4. reraise=True — Without this, tenacity wraps the final failure in a RetryError, and your except RasterioIOError clause never fires. Re-raising the original exception keeps the exit-code mapping in main working.

5. Typed permanent exceptions — CrsMismatch and UnsupportedFormat exist so a permanent condition carries its own exit code (10 and 11) instead of collapsing into a generic runtime failure. They are raised inside the retried function but are invisible to the predicate, so they escape immediately.

6. before_sleep_log — Emits a WARNING before each backoff sleep, so the log makes the retry sequence auditable: you see attempt 2 of 5, attempt 3 of 5, and finally either a success line or the propagated error.

Named Gotcha: Retrying a write duplicates output

The most damaging mistake is putting the retry around code that also writes. Reads are idempotent — running read_window five times reads the same bytes five times and changes nothing. A write is not: if gdal.Warp or an upload succeeds on the object store but the response times out on the way back, the retry runs the write a second time, leaving a duplicated or half-flushed file. Blanket-retrying is doubly wrong when the underlying error is a 404 you should have failed on in a few milliseconds; instead you burn the full backoff budget writing garbage.

The fix has two parts. First, decorate only the pure read, exactly as above — never the function that produces output. Second, if you must retry a write, make it atomic: write to a temporary key such as scene_001.tif.tmp, verify the byte count, then issue a single rename to the final path so a partial object is never visible. Permanent errors still bypass the retry entirely, so a missing source never triggers a wasteful write attempt in the first place.

Verification

Run against a real object and read the log. A healthy transient recovery shows the retry warnings followed by a success line; an exhausted or permanent failure shows the give-up path and a non-zero exit code:

# Success after transient blips: expect WARNING retry lines then an OK read.
python read_with_retry.py /vsis3/my-bucket/scenes/scene_001.tif
echo "exit=$?"    # 0

# Permanent 404: expect ZERO retry lines and an immediate failure.
python read_with_retry.py /vsis3/my-bucket/scenes/missing.tif
echo "exit=$?"    # 1, and no "before sleep" WARNING appears

A correct run against a flaky object prints lines like Retrying read_window in 2.3 seconds as it raised RasterioIOError up to four times, then read ... block: shape=(512, 512). A permanent 404 prints no retry warnings at all — that absence is the proof that classification worked. To capture the exhausted-retry case in a structured feed for a dead-letter queue, catch the re-raised RasterioIOError in main and serialize the URL and message before exiting.

FAQ

Related

• Error Handling in Spatial Pipelines — parent guide covering failure capture, exit-code conventions, and recovery patterns for batch raster and vector workflows
• Building a Dead-Letter Queue for Failed Geometry Transforms — where to route reads that exhaust their retry budget instead of losing them