Article

Mocking Cloud Storage I/O in Geospatial CLI Tests

Serve your fixture over a throwaway local HTTP server that honours Range headers, point GDAL at it with /vsicurl/, and your cloud-reading code path runs unmodified with no network and no credentials. The server also records which byte ranges were requested, which turns does it stream? from an assumption into an assertion. It builds on the Testing Geospatial CLI Tools guide, part of the broader CLI Architecture & Design Patterns reference.

Prerequisites

Three Places to Cut the Wire

There is more than one boundary you could fake, and they buy different things. Choosing deliberately matters, because the cheapest one to set up is also the one that tests the least.

Where to cut the connection to the cloud Patching the Python function is instant to set up but skips GDAL's virtual file system entirely. A local HTTP server exercises the real GDAL code path including range requests. A full object-store emulator additionally exercises authentication and listing, at a much higher setup cost. patch the function monkeypatch a reader to return local bytes setup: seconds skips GDAL entirely use for: orchestration local HTTP server GDAL fetches over the loopback with Range setup: one fixture real vsicurl path use for: most tests object-store emulator signing, listing, ACLs as well as reads setup: a container highest fidelity use for: auth paths only

The middle column is the default for a reason: it is the only one of the three where a failure to send range requests — the bug that turns a windowed read into a full download — actually shows up.

Complete Working Implementation

# tests/conftest.py
import http.server
import socketserver
import threading
from pathlib import Path

import pytest


class RangeHandler(http.server.SimpleHTTPRequestHandler):
    """Serve files from a directory, honouring single-range requests,
    and record every range served so tests can assert on access patterns."""

    requested_ranges: list[tuple[int, int]] = []

    def log_message(self, *args):        # keep pytest output clean
        pass

    def do_GET(self):
        path = Path(self.directory) / self.path.lstrip("/")
        if not path.is_file():
            self.send_error(404)
            return
        data = path.read_bytes()
        rng = self.headers.get("Range")

        if not rng:
            self.send_response(200)
            self.send_header("Content-Length", str(len(data)))
            self.send_header("Accept-Ranges", "bytes")
            self.end_headers()
            self.wfile.write(data)
            return

        # "bytes=start-end", end inclusive and optional
        spec = rng.split("=", 1)[1]
        start_s, _, end_s = spec.partition("-")
        start = int(start_s)
        end = int(end_s) if end_s else len(data) - 1
        end = min(end, len(data) - 1)
        type(self).requested_ranges.append((start, end))

        chunk = data[start:end + 1]
        self.send_response(206)
        self.send_header("Content-Range", f"bytes {start}-{end}/{len(data)}")
        self.send_header("Content-Length", str(len(chunk)))
        self.send_header("Accept-Ranges", "bytes")
        self.end_headers()
        self.wfile.write(chunk)


@pytest.fixture
def range_server(tmp_path):
    """Serve tmp_path over loopback HTTP; yields (base_url, requested_ranges)."""
    RangeHandler.requested_ranges = []

    def handler(*args, **kwargs):
        return RangeHandler(*args, directory=str(tmp_path), **kwargs)

    with socketserver.TCPServer(("127.0.0.1", 0), handler) as httpd:
        port = httpd.server_address[1]
        thread = threading.Thread(target=httpd.serve_forever, daemon=True)
        thread.start()
        try:
            yield f"http://127.0.0.1:{port}", RangeHandler.requested_ranges
        finally:
            httpd.shutdown()
            thread.join(timeout=5)

The test then reads a COG through GDAL’s virtual file system and asserts on what was fetched:

# tests/test_cog_streaming.py
import rasterio
from rasterio.windows import Window
from rasterio.env import Env


def test_windowed_read_fetches_ranges_not_the_whole_file(cog_path, range_server, tmp_path):
    base_url, ranges = range_server
    # cog_path fixture has already written a tiled, overview-bearing COG into tmp_path
    url = f"/vsicurl/{base_url}/{cog_path.name}"
    total_bytes = cog_path.stat().st_size

    with Env(GDAL_DISABLE_READDIR_ON_OPEN="EMPTY_DIR",
             CPL_VSIL_CURL_ALLOWED_EXTENSIONS=".tif"):
        with rasterio.open(url) as src:
            src.read(1, window=Window(0, 0, 128, 128))

    fetched = sum(end - start + 1 for start, end in ranges)
    assert ranges, "no Range requests were made — the read was not streamed"
    assert fetched < total_bytes * 0.5, (
        f"fetched {fetched} of {total_bytes} bytes; expected a partial read"
    )

What the recorded ranges look like is itself informative. A correct windowed read produces a short, recognisable sequence; a misconfigured one produces either a single enormous fetch or a storm of tiny ones.

Reading the recorded range pattern A healthy windowed read fetches a small header range and then one range per intersecting tile. A striped input produces one range covering the whole file. A read with directory probing enabled produces many tiny requests before the useful ones. healthy — header plus tiles 3 requests, about 6% of the object striped input — one full fetch 1 request, 100% of the object — the assertion catches this probing enabled — a storm of tiny reads

Asserting on total bytes rather than request count is what lets one test cover the first two rows. Catching the third needs a separate assertion on request count, and it is worth having on any tool that reads from an object store in a loop.

Faking the Write Path Too

Reads are the half people remember. A batch that writes results back to object storage has the same problem in reverse, and it is harder to fake convincingly because the write path involves multipart uploads, retries and eventual visibility.

For most tests the honest answer is to keep the write local. Point the tool at a file:// destination in the test and cover the object-store write with a much smaller number of tests against a real emulator. The reason is that what usually breaks in a write path is not the transfer — that is the library’s job — but the surrounding logic: whether a temporary key is used, whether the final rename happens only after a successful flush, and whether a retry after a partial upload leaves debris. All three are testable against a local filesystem, because they are decisions your code makes.

Where a fake genuinely helps is in forcing failures. A write target that raises on the third call, or that accepts the upload and then reports the object as missing, exercises recovery paths that a working backend never will. A small class implementing just the two methods your writer calls is usually enough, and it is far easier to reason about than an emulator configured to misbehave.

class FlakyStore:
    """Accepts writes, failing the nth one, so retry logic can be exercised."""

    def __init__(self, fail_on: int):
        self.fail_on = fail_on
        self.calls = 0
        self.objects: dict[str, bytes] = {}

    def put(self, key: str, data: bytes) -> None:
        self.calls += 1
        if self.calls == self.fail_on:
            raise ConnectionResetError("simulated reset mid-upload")
        self.objects[key] = data

The assertion that matters against it is not that the retry happened — that is easy — but that after the retry there is exactly one object under the final key and nothing under a temporary one. That is the property Retrying Transient GDAL I/O Errors with Exponential Backoff describes from the production side.

Step Annotations

  1. Port zero, then read it back. Binding to port 0 lets the operating system choose a free port, which is what makes the fixture safe under pytest-xdist — two workers never collide.
  2. daemon=True on the thread. If a test fails and the fixture teardown is skipped, a daemon thread does not keep the interpreter alive, so a broken test cannot hang the suite.
  3. GDAL_DISABLE_READDIR_ON_OPEN="EMPTY_DIR". Without it, GDAL probes for sidecar files next to the object, which produces a burst of 404s and slows the test. In production it is the same setting that stops a cloud read making eight useless requests before the useful one.
  4. The assertion is a fraction, not a count. Asserting exactly three ranges couples the test to GDAL’s internal read-ahead heuristics, which change between versions. Asserting that less than half the file was fetched captures the property that matters and survives upgrades.
  5. requested_ranges is reset in the fixture, not the handler. Class attributes persist between tests, and a stale list from a previous test is a false pass waiting to happen.

One Named Gotcha: GDAL Caches Across Tests in the Same Process

GDAL keeps a virtual-file-system cache of already-fetched blocks, keyed by URL. Because the fixture reuses a port and often the same filename, a second test can read entirely from cache, make zero range requests, and fail the assert ranges check for a reason that has nothing to do with the code.

Two fixes, and it is worth applying both:

from osgeo import gdal

# 1. give each test a distinct URL
url = f"/vsicurl/{base_url}/{cog_path.name}?t={uuid4().hex}"

# 2. clear the cache between tests
@pytest.fixture(autouse=True)
def clear_vsi_cache():
    yield
    gdal.VSICurlClearCache()

The query-string trick alone is enough for most suites and avoids importing osgeo directly, which matters if your project depends only on rasterio.

The three GDAL settings below account for most of the difference between a cloud read that behaves in a test and one that behaves in production, so pin them in the fixture rather than inheriting whatever the machine has.

Settings that change what a cloud read requests Directory probing adds a burst of 404 requests before the useful one. The allowed-extension list stops GDAL probing for sidecar files. The virtual file system chunk size sets the minimum bytes fetched per range, which determines how many requests a windowed read needs. setting pin it to if you do not GDAL_DISABLE_READDIR_ON_OPEN EMPTY_DIR a burst of 404s first CPL_VSIL_CURL_ALLOWED_EXTENSIONS .tif sidecar probing CPL_VSIL_CURL_CHUNK_SIZE a fixed value version-dependent counts

The third row is why an assertion on request count is brittle and an assertion on bytes fetched is not: the chunk size decides how many requests a given window needs, and its default has moved between GDAL releases.

Verification

Run the test with GDAL’s HTTP tracing on to see the exchange for yourself:

CPL_DEBUG=ON CPL_CURL_VERBOSE=YES pytest tests/test_cog_streaming.py -s 2>&1 | grep -i "range\|VSICURL"

You should see a small number of Range: bytes= lines — one for the header, one or more for the tiles that intersect the window — and no line fetching from byte zero to the end. If the trace shows a single full-file fetch, the input is not tiled, and the assertion is doing its job.

Keeping the Fake Honest

Any fake drifts away from the thing it replaces, and a cloud-storage fake drifts in two specific directions worth watching for.

The first is latency. A loopback server answers in microseconds, so a pipeline whose concurrency is tuned for a hundred-millisecond round trip behaves nothing like it does in production. Tests written against the fake will happily pass with a semaphore of one, because there is nothing to overlap. If concurrency behaviour is what you are testing, add a deliberate delay to the handler — a few milliseconds is enough to make the difference between serial and parallel visible in the recorded timestamps.

The second is error behaviour. A real object store returns 503 under load, 403 when a token expires mid-run, and occasionally a truncated body with a 200 status. None of those appear in a fixture that always succeeds, so the recovery code described in the section on error handling in spatial pipelines is never exercised. Adding an injectable failure to the handler — a query parameter the test can set to make the next request fail with a chosen status — costs a dozen lines and turns a whole category of untested code into tested code.

Neither drift is a reason to prefer a heavier emulator. Both are reasons to be explicit about what a given test is checking. A test named test_windowed_read_is_partial should use the fast, always- succeeding fixture; a test named test_retries_on_503 should use the one that fails on demand. When the fixture’s behaviour is visible in the test’s name, nobody later assumes the first test proves something about resilience.