Article

Reading COGs from S3 with rasterio and /vsis3/

Prefix the key with /vsis3/ and open it as you would a local path. Inside a rasterio.Env carrying four settings — directory probing off, an allowed-extension list, HTTP/2 multiplexing and a virtual file system cache — a windowed read costs a handful of range requests instead of a full download. It builds on the Cloud Storage I/O for Spatial Batches guide, part of the broader Spatial Batch Processing & Async Workflows reference.

Prerequisites

  • Python 3.10 or later
  • pip install "rasterio>=1.3" — the wheels include the S3 virtual file system driver
  • Credentials the process can find: an instance role, a profile, or the standard access-key variables
  • A source object that is genuinely a COG; a striped GeoTIFF cannot be read partially at all

What Happens on rasterio.open

The open is not free, and knowing what it costs explains why some settings matter so much.

The requests one windowed read actually makes Credentials resolve locally with no request. Directory probing, if left on, issues a list and several 404s. The header range request fetches the tile offsets. Then one range request per intersecting tile returns the pixels. resolve credentials no request directory probe 1 list + n 404s header range tile offsets tile ranges the pixels you asked for with probing left on with probing off

The red band is pure waste and is on by default. Turning it off is the single change that most improves cloud read performance, and it costs nothing because sidecar files are irrelevant for a self-contained COG.

Complete Working Implementation

# gistools/cloud_read.py
from __future__ import annotations

import rasterio
from rasterio.env import Env
from rasterio.warp import transform_bounds
from rasterio.windows import from_bounds

CLOUD_ENV = {
    # Do not probe the "directory" for sidecar files on every open.
    "GDAL_DISABLE_READDIR_ON_OPEN": "EMPTY_DIR",
    # Only consider these extensions when a probe does happen.
    "CPL_VSIL_CURL_ALLOWED_EXTENSIONS": ".tif,.tiff,.ovr",
    # Reuse connections and pipeline the small tile requests.
    "GDAL_HTTP_MULTIPLEX": "YES",
    "GDAL_HTTP_VERSION": "2",
    # Keep fetched chunks around for the life of the Env block.
    "VSI_CACHE": "TRUE",
    "VSI_CACHE_SIZE": str(64 * 1024 * 1024),
    # Bound the damage from a hung connection.
    "GDAL_HTTP_TIMEOUT": "30",
    "GDAL_HTTP_CONNECTTIMEOUT": "10",
}


def read_bbox(key: str, bbox, bbox_crs: str = "EPSG:4326"):
    """Read the pixels covering `bbox` from an S3-hosted COG.

    `key` is a bucket-relative path such as "tiles/2026/08/scene.tif".
    Returns (array, transform) for the window that was actually read.
    """
    uri = f"/vsis3/{key}"
    with Env(**CLOUD_ENV):
        with rasterio.open(uri) as src:
            left, bottom, right, top = transform_bounds(
                bbox_crs, src.crs, *bbox, densify_pts=21)
            window = from_bounds(left, bottom, right, top,
                                 transform=src.transform)
            window = window.round_offsets().round_lengths()
            data = src.read(window=window, boundless=False)
            return data, src.window_transform(window)

Step Annotations

  1. densify_pts=21 on the bounds transform. Reprojecting only the four corners of a bounding box understates the extent whenever the projection curves, and the resulting window clips the edges of what the caller asked for. Densifying the edges before transforming fixes it.
  2. round_offsets().round_lengths() before reading. A window with fractional offsets forces GDAL to resample; rounding to whole pixels keeps the read aligned and the values untouched.
  3. VSI_CACHE_SIZE inside the Env block, not globally. Scoping it means a long-running worker does not hold sixty-four megabytes of chunks between tasks.
  4. Timeouts are set explicitly. GDAL’s defaults are generous, and a hung connection in a batch of ten thousand tiles turns into a stalled worker that nothing notices.
  5. boundless=False is the default and is deliberate here. A boundless read silently pads with nodata when the window extends past the raster, which hides a bounding box that does not actually intersect the scene.

One Named Gotcha: Credentials Resolve Differently for GDAL and boto3

A process where boto3 works and /vsis3/ returns 403 is common and confusing. GDAL does not use boto3’s credential chain. It reads AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN, AWS_PROFILE and the instance metadata service — but the order and the supported profile features differ, and it does not understand every arrangement an AWS config file can express.

The reliable diagnosis is to ask GDAL what it resolved:

CPL_DEBUG=ON python -c "
import rasterio
from rasterio.env import Env
with Env(CPL_DEBUG='ON'):
    rasterio.open('/vsis3/bucket/tile.tif')
" 2>&1 | grep -i 'aws\|credential\|region'

Two settings fix most cases: AWS_REGION set explicitly, because an unset region resolves to a default that will not match your bucket; and AWS_S3_ENDPOINT when the store is S3-compatible rather than S3 itself. For a requester-pays bucket, add AWS_REQUEST_PAYER=requester — its absence also produces a bare 403.

The settings divide into two groups, and only one of them changes what the read costs.

Which settings change cost and which change safety Disabling directory probing and enabling HTTP/2 multiplexing reduce the number of requests. The extension allow-list narrows what a probe considers. Timeouts do not reduce cost but bound how long a hung connection can stall a worker. GDAL_DISABLE_READDIR_ON_OPEN removes the probe entirely — the biggest single win fewer requests GDAL_HTTP_MULTIPLEX with VERSION 2 pipelines the small tile requests over one connection less latency CPL_VSIL_CURL_ALLOWED_EXTENSIONS narrows what any remaining probe will look for fewer 404s GDAL_HTTP_TIMEOUT and CONNECTTIMEOUT bound a hung connection so a worker cannot stall safety, not cost

Only the first three affect the bill. The last is the one that keeps a batch of ten thousand tiles from parking on a single unreachable object for the rest of the afternoon.

Verification

# 1. The read stayed partial.
CPL_CURL_VERBOSE=YES python -c "
from gistools.cloud_read import read_bbox
read_bbox('tiles/scene.tif', (-2.1, 51.3, -1.8, 51.6))
" 2>&1 | grep -c 'Range: bytes='

# 2. Nothing probed for sidecars.
CPL_CURL_VERBOSE=YES python -c "..." 2>&1 | grep -c '404'    # expect 0

# 3. The source really is tiled — otherwise none of this helps.
python -c "
import rasterio
with rasterio.open('/vsis3/tiles/scene.tif') as s:
    print('tiled:', s.profile.get('tiled'), 'blocks:', s.block_shapes[:1],
          'overviews:', s.overviews(1))
"

A range count in single figures with no 404s is the healthy signature. An overview list that comes back empty means every read at reduced resolution will fetch full-resolution tiles, which is the other half of what makes a COG a COG.

When the Source Is Not a COG

Three properties make a GeoTIFF cloud-readable, and a file can miss any one of them while still opening perfectly well from local disk.

What makes a GeoTIFF readable from object storage Internal tiling lets a window map to a small set of byte ranges. Embedded overviews let a zoomed-out read fetch far less data. A header written at the front of the file means one request finds the tile offsets rather than a seek to the end. internal tiling a window maps to a few byte ranges without it: the whole row spans the image missing → full reads embedded overviews a zoomed-out read fetches a reduced level without it: full-resolution tiles for a thumbnail missing → slow previews header at the front one request finds the tile offsets without it: a seek to the end of the object missing → an extra round trip

rio cogeo validate reports all three in one command, and running it over a sample of the inputs before a large batch is the cheapest hour you will spend. A source failing any of them is worth rewriting once rather than paying for on every read.

Reading Many Objects Efficiently

A single windowed read is the unit this guide covers; a batch reads thousands, and three things change at that scale.

Connection reuse becomes significant. Each rasterio.open on a new URL establishes a connection unless the underlying HTTP layer keeps one alive, and TLS setup is a measurable fraction of a small read. Keeping the reads inside one long-lived Env block, rather than opening and closing an environment per object, lets GDAL reuse connections across them.

Per-prefix rate limits become visible. Object stores throttle per key prefix rather than per bucket, so a batch hammering tiles/2026/08/ with fifty concurrent readers can see throttling responses while the bucket as a whole is far from any limit. Spreading work across prefixes, or bounding concurrency per prefix rather than globally, avoids it.

Header re-fetching becomes a real cost. Every open reads the tile directory, and for a batch that reads several windows from the same object that is a repeated expense. Opening once and reading several windows inside the same context is the fix, and it is worth restructuring a loop for.

# Instead of: for window in windows: with rasterio.open(uri) as src: ...
with Env(**CLOUD_ENV):
    with rasterio.open(uri) as src:
        for window in windows:
            yield src.read(1, window=window)

That single change frequently halves the request count for a tiled workload, because the header is fetched once rather than once per window.

Overviews and Reduced-Resolution Reads

A read at reduced resolution should fetch a reduced-resolution level, and it only does so if the object has overviews and the read asks for them. out_shape is what asks:

with rasterio.open(uri) as src:
    thumb = src.read(1, out_shape=(1, 512, 512))     # picks a suitable overview

Without out_shape, a read of the full extent fetches every full-resolution tile and resamples in memory — which for a large scene is a transfer of hundreds of megabytes to produce a thumbnail.

Checking that overviews exist is one line, and worth doing before a batch that will produce previews: src.overviews(1) returns the decimation factors, and an empty list means every preview read will be expensive. Building them is a one-off cost against a source that will be read many times, and it is the other half of what makes a COG worth the name.

A final note on paths: keep the /vsis3/ prefix confined to the boundary that talks to GDAL. Passing virtual file system paths through the rest of the pipeline means every function has to know about them, and a path that works for rasterio will not work for pathlib, fsspec or a test fixture on disk. Carrying a plain URI internally and translating at the point of use keeps everything else portable, and makes the local-filesystem test path a one-line substitution rather than a special case.

A note on other stores

The same pattern applies to Google Cloud Storage through /vsigs/ and to Azure through /vsiaz/, with the credential variables changing and everything else staying the same. For an S3-compatible store, /vsis3/ works with AWS_S3_ENDPOINT set and AWS_VIRTUAL_HOSTING usually set to FALSE, since most compatible stores use path-style addressing.

It is also worth setting GDAL_HTTP_MAX_RETRY and GDAL_HTTP_RETRY_DELAY explicitly rather than inheriting whatever the build defaults to. GDAL retries some HTTP failures internally, and a batch that also retries at the task level multiplies the two — five internal retries inside four task retries is twenty attempts against an object that is not coming back.