Cloud Storage I/O for Spatial Batches

Object storage behaves nothing like a filesystem, and a batch pipeline that treats it as one will be slow, expensive, or wrong — often all three. This guide covers the access paths GDAL offers, the settings that decide how many HTTP requests a windowed read costs, and the write pattern that keeps a partially-uploaded raster from ever being visible. It is part of the Spatial Batch Processing & Async Workflows guide.

Prerequisites

• Python 3.10 or later with pip install rasterio pyogrio fsspec s3fs — rasterio’s wheels include the GDAL virtual file system drivers.
• Credentials available to the process, through the provider’s standard environment variables. Where those come from is covered in Environment Variable Sync.
• A grasp of why layout matters, from Streaming Cloud-Optimized GeoTIFFs with Async Range Requests.

Problem framing

Four assumptions carried over from filesystems cause almost all the trouble. Listing a directory is cheap — on object storage it is a paginated API call costing money per thousand keys. A file exists or it does not — object stores may answer a read for a key that was just written with a 404. Opening a file is cheap — every open is at least one HTTP round trip, and often three. And a partial write leaves a partial file — on an object store it usually leaves no object at all, which is better, but a multipart upload that fails halfway leaves billable fragments.

None of these makes object storage a poor choice. They make it a different one, and the patterns below are what the difference costs.

The three ways to reach an object

The rule of thumb: if the batch reads a fraction of each object, use the virtual file system; if it reads all of each object more than once, download; if it needs to work identically against several providers including a local directory, use fsspec.

Step-by-step implementation

1. Read a window without downloading the object

import rasterio
from rasterio.env import Env
from rasterio.windows import from_bounds

CLOUD_TUNING = {
    "GDAL_DISABLE_READDIR_ON_OPEN": "EMPTY_DIR",   # do not probe for sidecars
    "CPL_VSIL_CURL_ALLOWED_EXTENSIONS": ".tif,.tiff",
    "GDAL_HTTP_MULTIPLEX": "YES",
    "GDAL_HTTP_VERSION": "2",
    "VSI_CACHE": "TRUE",
    "VSI_CACHE_SIZE": str(64 << 20),
}


def read_bbox(uri: str, bounds, dst_crs=None):
    with Env(**CLOUD_TUNING):
        with rasterio.open(uri) as src:
            window = from_bounds(*bounds, transform=src.transform)
            return src.read(window=window), src.window_transform(window)

GDAL_DISABLE_READDIR_ON_OPEN is the single highest-value setting. Without it, every open probes the “directory” for sidecar files, which on an object store is a list request plus several 404s — often more requests than the actual read.

2. Write through a temporary key and promote

import os
import uuid

import rasterio


def write_atomic(uri: str, data, profile) -> str:
    """Write to a unique temporary key, then promote it to `uri`."""
    tmp = f"{uri}.{uuid.uuid4().hex}.tmp"
    with rasterio.open(tmp, "w", **profile) as dst:
        dst.write(data)
    _promote(tmp, uri)
    return uri

On a POSIX filesystem _promote is os.replace. On an object store it is a server-side copy followed by a delete of the temporary key — object stores have no rename, and doing the copy server-side avoids pulling the bytes back through your process.

3. Enumerate inputs without listing everything

Listing dominates the cost of a large batch far more often than reading does.

Writing a manifest at the end of the job that produced the data is the cheapest possible fix, and it makes the consuming batch reproducible — the same manifest replays the same set of inputs.

Configuration & state management

Cloud settings divide the same way the section on environment variables describes. Credentials belong to the environment and must never appear in a task payload or a log record. Tuning belongs to the machine, because the right cache size depends on the node. Endpoints and bucket names belong to the configuration file, because they change per deployment and are not secret.

The one addition specific to object storage is the requester-pays flag. A bucket configured that way rejects unauthenticated-style reads with a 403 that says nothing about billing, and the fix — AWS_REQUEST_PAYER=requester — is impossible to guess from the error. Setting it explicitly per bucket in configuration saves the hour it otherwise takes.

Error handling & gotchas

A 403 can mean expired credentials, a missing requester-pays flag, or a wrong region. All three produce the same status. Logging the endpoint and region alongside the failure narrows it to one guess instead of three.

Read-after-write is not immediate everywhere. A batch that writes an object and immediately lists for it may not see it. Do not use listing to confirm a write; use the write’s own response.

Retries multiply cost as well as time. A failing range request retried five times bills five times. Cap retries and route the failure to the dead-letter path described in Error Handling in Spatial Pipelines.

Cross-region reads are slow in a way that looks like a code problem. A pipeline running in one region against a bucket in another pays tens of milliseconds per request. Check the region before optimising anything else.

Cost on object storage has three components, and geospatial batches usually get surprised by the one they did not budget for.

The middle row is why request count, not bandwidth, is the number to watch. A batch that reads efficiently but makes four unnecessary probes per open pays for those probes on every one of ten million objects.

Placing compute next to data

The single largest performance variable in a cloud-backed pipeline is not any setting in this guide — it is whether the compute is in the same region as the bucket. Cross-region reads add tens of milliseconds to every request, and a windowed read makes several requests per tile, so the penalty multiplies through the batch in a way that no amount of caching recovers.

The check is quick and worth making before any tuning:

import boto3

s3 = boto3.client("s3")
bucket_region = s3.get_bucket_location(Bucket="my-bucket")["LocationConstraint"] or "us-east-1"
print("bucket:", bucket_region, "compute:", boto3.session.Session().region_name)

When they differ, the options are to move the compute, replicate the data, or accept the cost with open eyes. What does not work is optimising around it: a pipeline that reads a hundred tiles per minute cross-region will not reach a thousand by tuning caches, because the limit is round trips.

The related trap is a bucket that is regional and a pipeline that is not — a scheduled job that runs wherever the platform happens to place it. Pinning the region in the job definition removes a source of variance that otherwise makes benchmark results irreproducible.

Credentials that expire mid-batch

Short-lived credentials are the norm now, and a batch that runs longer than the credential lifetime will fail partway through with a 403 that looks like a permissions problem.

Two arrangements avoid it. The first is to use a credential provider that refreshes — an instance role, or a web-identity token — rather than static keys, so the SDK renews transparently. GDAL understands instance metadata and will refresh from it; it does not refresh a static key pair, because there is nothing to refresh.

The second, for the case where static credentials are unavoidable, is to make the batch resumable so a credential failure is a restart rather than a loss. That is the same checkpointing the section recommends for every other reason, and it means an expired credential costs the time since the last checkpoint rather than the whole run.

Detecting the case is worth doing explicitly, because the error is indistinguishable from a genuine permissions problem. A 403 arriving after a long period of successful reads is almost always expiry; a 403 on the first read is almost always configuration. Recording the elapsed time and the success count alongside the failure makes the distinction visible in the log without anyone having to reason about it.

Structuring keys for the access pattern

Key layout is decided once and lived with for years, and the two decisions that matter are both about listing rather than reading.

Prefixes should encode whatever the consuming batches filter on — usually date, sometimes tile identifier, occasionally product type. A layout of product/date/tile.tif lets a batch processing one day list a few dozen keys; a flat layout of tile-date.tif forces it to list everything and filter in the client.

Cardinality at each level should be moderate. A prefix with ten million keys under it is slow to list and can attract per-prefix rate limiting; a prefix with three is a level that buys nothing. Aiming for hundreds to low thousands per prefix works well in practice.

The one thing not to encode in the key is anything that changes. A key containing a processing version means every reprocess writes to a new location and consumers must be told; a key containing only identity, with the version in the object’s metadata, means a reprocess overwrites in place and consumers need no change.

Verification

# Count requests for a single windowed read, with GDAL tracing on
CPL_CURL_VERBOSE=YES CPL_DEBUG=ON python -c "
from gistools.cloud import read_bbox
read_bbox('/vsis3/bucket/tile.tif', (-2.1, 51.3, -1.8, 51.6))
" 2>&1 | grep -c 'Range: bytes='

# Confirm no temporary keys survive a successful batch
aws s3 ls s3://bucket/out/ --recursive | grep -c '\.tmp$'   # expect 0

A single windowed read against a well-formed COG should produce a handful of range requests, not dozens. A high count almost always means GDAL_DISABLE_READDIR_ON_OPEN is unset or the object is not tiled.

Performance notes

Concurrency against an object store is limited by the provider’s per-prefix request rate rather than by your bandwidth. Spreading keys across prefixes helps; hammering one prefix with fifty workers produces throttling responses that look like transient failures.

HTTP/2 multiplexing is worth enabling — GDAL_HTTP_MULTIPLEX=YES with GDAL_HTTP_VERSION=2 — because a windowed read issues several small requests and multiplexing removes most of the per-request overhead. The gain is largest exactly where it matters: many small reads rather than few large ones.

FAQ

Designing for the pipeline you will have

Two decisions about cloud storage are effectively permanent once data exists, and both are worth making deliberately at the start rather than discovering later.

The first is the format written. An archive of striped GeoTIFFs cannot be read partially, and the remedy — rewriting every object as a COG — is a full pass over the data that grows more expensive every month the decision is deferred. Writing cloud-optimised output from the first job costs nothing extra at write time and determines what every future consumer can do cheaply.

The second is whether a manifest exists. A producing job that records what it wrote gives every consumer a cheap enumeration; one that does not forces each consumer to list. Adding a manifest later is possible and requires the expensive listing you were trying to avoid, once per existing dataset.

Both decisions are made by whoever writes the first pipeline, usually without realising they are decisions. Stating them explicitly in a project’s conventions — cloud-optimised outputs, and a manifest per run — costs a paragraph and saves a migration.

Testing cloud paths without a cloud account

A pipeline whose cloud behaviour can only be exercised against a real bucket will have that behaviour tested rarely, which is where most of the surprises come from. Two arrangements remove the dependency.

For reads, a local HTTP endpoint honouring range requests exercises the whole GDAL path — virtual file system, range requests, caching — with no credentials and no network. The fixture is about forty lines, and it also records what was requested, which turns does it stream? into an assertion.

For writes, keep the object-store call behind a small interface with two methods and provide a filesystem-backed implementation for tests. The logic worth testing is the temporary-key-then-promote sequence and the cleanup, none of which needs a real store; what a real store adds is transfer mechanics, which is the library’s responsibility rather than yours.

What remains genuinely untestable locally is authentication, and that is the right thing to cover with a small number of tests against a real bucket, run on a schedule rather than on every commit. Separating those from the rest by a marker keeps the fast suite fast and keeps the slow, credentialed tests from being quietly disabled the first time they flake.

A minimum viable configuration

For a team adding cloud reads to an existing pipeline, four settings and one habit cover most of the distance. Disable directory probing. Set an extension allow-list. Enable HTTP/2 with multiplexing. Enable the virtual file system cache with a size you chose. And run everything inside one long-lived environment block rather than creating a new one per object.

Together those turn a read that issues a dozen requests into one that issues three, without any change to the pipeline’s structure. Everything else in this guide is refinement on top of that.

Where the cost actually accrues

Cloud storage bills on three axes and geospatial batches are unusual in which one dominates.

Storage itself is rarely the issue: an archive of rasters is large but the per-gigabyte cost is small, and it is a number someone budgeted deliberately. Egress matters only when results leave the region, which a pipeline running beside its data avoids entirely.

Requests are what surprise people. A batch reading ten thousand objects with a poorly configured open — directory probing on, no extension allow-list — issues perhaps six requests per object where two would do, and the difference is forty thousand requests per run. At a nightly cadence that is more than a million a month, for no benefit at all.

The reason this is worth stating plainly is that request cost is invisible in every ordinary measurement. Wall clock barely moves, memory does not move, and the pipeline appears healthy. The only place it shows up is the bill, usually a month later, attributed to a service rather than to a job. Counting requests during development — with GDAL’s own tracing, as the verification section does — turns an invisible cost into a number someone can act on.

Related

• Streaming Cloud-Optimized GeoTIFFs with Async Range Requests — the layout that makes windowed cloud reads cheap.
• Distributed Task Queues for Spatial Jobs — the shared storage a distributed batch needs.
• Error Handling in Spatial Pipelines — classifying the failures a network introduces.