Article

Tuning GDAL VSI Cache Settings for Cloud Reads

Three separate caches sit between your read() and an HTTP request, and they are configured independently: the virtual file system chunk cache holds raw byte ranges, the block cache holds decoded tiles, and a small header cache holds the file directory. Sizing them deliberately is what turns a chatty batch into a quiet one. It builds on the Cloud Storage I/O for Spatial Batches guide, part of the broader Spatial Batch Processing & Async Workflows reference.

Prerequisites

  • Python 3.10 or later with pip install rasterio
  • A COG on object storage you can read repeatedly
  • Somewhere to observe request counts — GDAL’s own tracing is enough

The Three Caches

They serve different purposes and are missed at different points in a read.

Where a read can be satisfied without touching the network A read first checks the block cache of decoded tiles. A miss there checks the chunk cache of raw byte ranges. A miss there issues an HTTP range request. Separately, a header cache avoids re-fetching the tile directory when the same object is reopened. src.read() a window block cache decoded tiles GDAL_CACHEMAX chunk cache raw byte ranges VSI_CACHE_SIZE HTTP the expensive bit header cache — CPL_VSIL_CURL_CACHE_SIZE saves re-fetching the tile directory on reopen

The block cache is the one people know about and the least useful for cloud reads, because a batch that visits each tile once never gets a hit. The chunk cache and the header cache are where the savings are.

Complete Working Implementation

# gistools/vsi_tuning.py
from __future__ import annotations

import os

MB = 1024 * 1024


def cloud_read_env(*, workers: int = 1, memory_budget_mb: int = 4096) -> dict[str, str]:
    """Cache settings sized to a per-process budget.

    The three caches are per process, so a pool of `workers` multiplies all of
    them. Dividing the budget first is what keeps a pool inside its limit.
    """
    per_worker = memory_budget_mb // max(workers, 1)

    # Decoded tiles: useful only when tiles are revisited. Keep it modest.
    block_mb = max(64, min(256, per_worker // 8))
    # Raw ranges: the one that saves requests. Give it the larger share.
    chunk_mb = max(32, min(512, per_worker // 4))
    # File headers: tiny per object, but a batch touches many objects.
    header_mb = 16

    return {
        "GDAL_CACHEMAX": str(block_mb),                    # in MB
        "VSI_CACHE": "TRUE",
        "VSI_CACHE_SIZE": str(chunk_mb * MB),              # in bytes
        "CPL_VSIL_CURL_CACHE_SIZE": str(header_mb * MB),   # in bytes
        "GDAL_DISABLE_READDIR_ON_OPEN": "EMPTY_DIR",
        "GDAL_HTTP_MULTIPLEX": "YES",
        "GDAL_HTTP_VERSION": "2",
    }


def apply_to_environment(**kwargs) -> None:
    """Set the values before any GDAL import, for a worker entry point."""
    for key, value in cloud_read_env(**kwargs).items():
        os.environ.setdefault(key, value)

Using it inside a read, scoped rather than global:

import rasterio
from rasterio.env import Env

from gistools.vsi_tuning import cloud_read_env

with Env(**cloud_read_env(workers=8, memory_budget_mb=16384)):
    with rasterio.open("/vsis3/bucket/scene.tif") as src:
        arr = src.read(1, window=window)

Step Annotations

  1. GDAL_CACHEMAX is in megabytes, the others in bytes. The units differ, and setting GDAL_CACHEMAX to a byte count silently gives a cache of several terabytes, which GDAL clamps in ways that vary by version.
  2. The chunk cache gets twice the block cache. For a batch that reads each tile once, a raw-range hit saves an HTTP request while a decoded-tile hit saves only a decode. Requests are the expensive resource.
  3. os.environ.setdefault, not assignment. An operator who has set a value deliberately should keep it; the helper supplies a floor, not an override.
  4. The budget is divided by worker count first. All three caches are per process, so a pool of eight multiplies the total by eight — the mistake described in Sizing Worker Pools for GDAL-Bound Queues.
  5. The header cache is fixed rather than scaled. It holds a few kilobytes per object, so sixteen megabytes covers thousands of objects and there is no reason to make it a function of anything.

One Named Gotcha: VSI_CACHE Is Off by Default and Silently Does Nothing

VSI_CACHE_SIZE has no effect unless VSI_CACHE is also set to TRUE. Setting the size alone is a very easy mistake — the name suggests it enables the thing it sizes — and it produces a batch that appears tuned and makes exactly as many requests as an untuned one.

The check is direct:

from osgeo import gdal
print("VSI_CACHE:", gdal.GetConfigOption("VSI_CACHE"))
print("VSI_CACHE_SIZE:", gdal.GetConfigOption("VSI_CACHE_SIZE"))

If the first prints None, the second is inert regardless of its value. The same pattern applies to GDAL_HTTP_MULTIPLEX, which requires GDAL_HTTP_VERSION=2 to have any effect — multiplexing is an HTTP/2 feature and is quietly ignored over HTTP/1.1.

Access patterns decide which cache earns its memory, and the three that appear in geospatial batches want quite different settings.

Matching cache sizes to the access pattern A single pass over many distinct tiles gets no reuse, so a large block cache is wasted. Repeated reads of the same scene benefit from both caches. Overlapping windows across a mosaic benefit most from the raw chunk cache, because neighbouring windows share byte ranges. one pass, many distinct tiles no tile is read twice, so nothing is reused small block cache many reads of one scene the same tiles are decoded repeatedly large block cache overlapping windows on a mosaic neighbouring windows share underlying byte ranges large chunk cache

Most batch pipelines are the first row and are configured as though they were the second, which reserves gigabytes across a worker pool for hits that never happen.

Verification

Measure rather than assume, by reading the same window twice and counting requests each time:

python - <<'PY'
import rasterio
from rasterio.env import Env
from rasterio.windows import Window
from gistools.vsi_tuning import cloud_read_env

with Env(**cloud_read_env(workers=1, memory_budget_mb=4096), CPL_CURL_VERBOSE="YES"):
    with rasterio.open("/vsis3/bucket/scene.tif") as src:
        src.read(1, window=Window(0, 0, 512, 512))   # cold
        src.read(1, window=Window(0, 0, 512, 512))   # should be warm
PY

Run it with stderr piped through grep -c 'Range: bytes='. A correctly configured cache produces requests on the first read and none on the second. If the count doubles, one of the two caches is disabled — almost always VSI_CACHE.

Caches Interact With the Worker Pool

Because every cache is per process, tuning them in isolation and then choosing a worker count undoes the tuning.

Cache size and worker count trade against each other On a sixteen gigabyte budget, four workers can afford a substantial cache each, eight workers a modest one, and sixteen workers barely more than the minimum. The product, not either number alone, is what the node must hold. 4 workers 3.2 GB each cache can be generous best for large windows roomy 8 workers 1.6 GB each cache stays modest the usual compromise balanced 16 workers 0.8 GB each cache near the minimum throughput may drop cramped

The right order is to fix the per-task working set first, then choose the worker count from the memory budget, then give whatever is left to the caches. Doing it the other way round produces a configuration that is internally consistent and exceeds the limit.

Measuring the Hit Rate Rather Than Guessing

Every recommendation above can be replaced by a measurement, and the measurement is cheap enough that it should be.

GDAL reports cache activity under CPL_DEBUG, and the ratio of reads served from cache to reads that went to the network is the number that decides whether the cache is earning its memory. A run showing almost no hits is a run whose cache should be shrunk to its floor and the memory given to the arrays.

CPL_DEBUG=ON python -m gistools warp-batch inputs.txt out/ --dst-crs EPSG:3857 2>&1 \
  | grep -ciE 'VSICURL.*(hit|cache)'

The absolute count matters less than how it changes when you halve or double the cache. A cache whose hit count is insensitive to its size is oversized; one whose hits scale with size is undersized and worth raising until they stop scaling.

Caches and Worker Restarts

One property of these caches is easy to forget and occasionally decisive: they are per process and therefore lost when a worker restarts.

For a pool using maxtasksperchild to contain leaks, every restart discards the accumulated cache, and a low value turns the cache into pure overhead — it is populated and thrown away repeatedly without ever being read. Either raise the value substantially or drop the cache to its floor; the combination of a small maxtasksperchild and a large cache is the worst of both.

The same applies to a queue whose workers are recycled on a schedule. A worker living for minutes gets little from a warm cache; one living for hours gets a great deal. Sizing the cache from the worker’s expected lifetime, rather than from the machine, is a small refinement that removes the mismatch.

There is a related effect on the header cache specifically. Headers are small and re-fetching them is cheap individually, but a batch reading ten thousand objects with a worker that restarts every hundred tasks re-fetches a hundred sets of headers it already had. Keeping that cache modest and the worker lifetime long is the arrangement that avoids it.

Defaults worth adopting

For a pipeline that reads each object once — which is most batch work — the settings that serve well are a block cache at its floor, a chunk cache of a few hundred megabytes, and a header cache of sixteen. That combination spends memory where it is used and leaves the rest for the arrays, which is where a windowed read actually needs it.

Departing from those defaults is worth doing when a measurement says so, and not before.

A Word on Measurement Order

Tuning these caches before checking the two settings that remove requests entirely — directory probing and the extension allow-list — measures the wrong thing. A read making six requests where two would do will show a cache improvement, because there is more to cache, and the improvement is smaller than simply not making the four unnecessary requests.

The order that works is to remove waste first, then size the caches against what is left. It also makes the cache measurement interpretable: once every request is one the read genuinely needed, a cache hit is unambiguously a saved round trip rather than a saved mistake.

Finally, remember that these settings are per process and are read at initialisation, so a change made after the first dataset opens has no effect on that process. In a worker pool the place to set them is the initializer, before any geospatial import — the same ordering constraint that applies to every other GDAL configuration variable.