Article

Tuning GDAL_CACHEMAX for Batch Raster Workloads

Set GDAL_CACHEMAX explicitly, in megabytes, before any GDAL import, and size it from whether tiles are revisited rather than from the machine. The default is a percentage of physical memory taken per process, so eight workers on a large node can reserve tens of gigabytes for a cache that never gets a hit. It builds on the Memory Management for Large Datasets guide, part of the broader Spatial Batch Processing & Async Workflows reference.

Prerequisites

What the Cache Actually Holds

The block cache is one of several pools, not all of them The block cache holds decoded tiles from datasets currently open. It does not hold arrays you allocated with a read call, raw byte ranges from remote objects, or driver scratch space. Confusing them leads to tuning the wrong number. decoded tiles blocks read from open datasets, keyed by dataset and offset this is GDAL_CACHEMAX your numpy arrays whatever read() returned to you not in the cache raw remote byte ranges held by the virtual file system cache VSI_CACHE_SIZE driver scratch space temporary buffers inside a warp or a compressor not tunable

The distinction matters because the most common memory problem in a batch — the arrays for a large window — is in the second row, and no amount of cache tuning touches it.

Complete Working Implementation

# gistools/cache_tuning.py
from __future__ import annotations

import os
from pathlib import Path

MB = 1 << 20
CGROUP_V2 = Path("/sys/fs/cgroup/memory.max")


def container_memory_bytes() -> int:
    try:
        raw = CGROUP_V2.read_text().strip()
        if raw != "max":
            return int(raw)
    except OSError:
        pass
    return os.sysconf("SC_PAGE_SIZE") * os.sysconf("SC_PHYS_PAGES")


def cachemax_mb(*, workers: int, revisits_tiles: bool,
                per_worker_arrays_mb: int) -> int:
    """A block-cache size in megabytes, sized from the access pattern.

    revisits_tiles: True when the same source tiles are read more than once —
    overlapping windows, multi-band passes, or an overview build.
    """
    budget_mb = int(container_memory_bytes() * 0.8) // MB
    per_worker = budget_mb // max(workers, 1)
    spare = max(per_worker - per_worker_arrays_mb, 0)

    if not revisits_tiles:
        # No reuse: a big cache buys nothing. Keep it small enough to hold a
        # few blocks so a partial window read is not re-decoded.
        return max(64, min(128, spare))
    return max(128, min(1024, spare // 2))


def apply() -> None:
    os.environ.setdefault("GDAL_CACHEMAX", str(cachemax_mb(
        workers=int(os.environ.get("GIS_WORKERS", "8")),
        revisits_tiles=os.environ.get("GIS_REVISITS", "0") == "1",
        per_worker_arrays_mb=int(os.environ.get("GIS_ARRAYS_MB", "1024")),
    )))
    os.environ.setdefault("GDAL_NUM_THREADS", "1")

Call apply() at the top of the worker entry point, before importing rasterio — the value is read at library initialisation and setting it afterwards is silently ignored.

Step Annotations

  1. Megabytes, not bytes. GDAL interprets a bare number as megabytes; a byte count like 536870912 asks for half a petabyte and is clamped in version-dependent ways.
  2. The budget is divided by worker count first. The cache is per process, so a pool multiplies it — the arithmetic in Sizing Worker Pools for GDAL-Bound Queues.
  3. The no-reuse branch caps at 128 MB. A single-pass batch gets essentially no cache hits, so memory spent there is memory taken from the arrays that actually need it.
  4. setdefault, not assignment. An operator who set the value deliberately keeps it.
  5. A floor of 64 MB. Below that, GDAL re-decodes blocks within a single windowed read, which is slower than having no cache at all would be.

When a Large Cache Pays

Cache size follows the access pattern A single pass over distinct tiles gets no hits and wants a small cache. Overlapping windows re-read shared tiles and want a moderate one. Building overviews reads every tile several times at successive levels and is the case where a large cache genuinely pays. single pass, distinct tiles each tile read once hit rate near zero memory better spent on the arrays 64–128 MB overlapping windows neighbouring windows share boundary tiles a moderate cache turns re-reads into hits 256–512 MB building overviews every tile is read again at each level the one case where a large cache clearly wins 512 MB–1 GB

Measuring the hit rate settles it rather than guessing: CPL_DEBUG=ON reports cache activity, and a run showing almost no hits is a run whose cache is wasted memory.

One Named Gotcha: The Default Is a Percentage, Not a Number

Historically GDAL_CACHEMAX defaulted to a fixed number of megabytes; modern GDAL defaults to a percentage of physical memory. On a 256 GB node that is a substantially larger cache than anyone intended, per process, and a pool of sixteen workers can reserve more than the machine has before any array is allocated.

# What each running worker actually has
for pid in $(pgrep -f 'gistools'); do
  tr '\0' '\n' < /proc/$pid/environ | grep -c GDAL_CACHEMAX
done | sort | uniq -c

A count of zero means the default is in force and the total is whatever the percentage works out to on that machine — which is why the same code can be fine on a laptop and get killed on a large node.

Verification

# Observe hits and misses for a representative unit.
CPL_DEBUG=ON python -m gistools warp in.tif out.tif --dst-crs EPSG:3857 2>&1 \
  | grep -ci 'GDAL: GDALDatasetRasterIO'

# Confirm the pool total stays inside the limit at the peak.
ps -o rss= -p $(pgrep -d, -f gistools) | awk '{s+=$1} END {print s/1024" MB"}'

Run the second command repeatedly during a batch and keep the maximum. That number, not the per-worker figure, is what the container limit is compared against.

Three symptoms distinguish a cache that is too small from one that is too large, and they look nothing alike.

Reading the symptom back to the cache Repeated decodes of the same block with plenty of memory free means the cache is too small. A pool killed for memory with a near-zero hit rate means the cache is too large for a single-pass workload. Good throughput with a high hit rate means the size is right. the same block decoded repeatedly with memory free — the cache is too small raise it killed for memory, near-zero hits a single-pass workload does not need a cache lower it high hit rate, stable memory nothing to do leave it alone

The middle row is the common one in batch work, and the instinct it provokes — reduce worker count — fixes the symptom while leaving gigabytes reserved for a cache that never gets a hit.

Interaction With Everything Else That Uses Memory

The block cache is one term in a sum, and tuning it in isolation moves the problem rather than solving it. Three other terms usually dominate.

The arrays a read returns are the largest, and they scale with window size and band count. A pipeline whose windows grew from 512 to 2048 pixels square increased that term sixteenfold, which no cache setting compensates for.

The worker count multiplies everything. A per-worker figure that fits comfortably becomes a total that does not, and the arithmetic is the same one the section applies to worker sizing.

The virtual file system cache, for remote sources, is a separate pool with its own setting. Sizing the block cache carefully and leaving that one at its default undoes the work.

The order that avoids circular tuning: measure the arrays first, because they are the largest and the least adjustable; choose a worker count from the remaining budget; then divide what is left between the two caches according to whether tiles are revisited.

When the Answer Is Not a Cache at All

A batch struggling with memory frequently has a structural problem that no setting fixes.

Reading whole files rather than windows is the most common. The cache cannot help, because nothing is being reused — the memory is going into one enormous array.

Holding results in a list until the end is the second. Memory grows monotonically through the run, which looks like a leak and is a design choice; writing each result as it completes fixes it.

Datasets left open in a loop is the third, and it is the one that most looks like a cache problem because the memory is held by GDAL. Closing them — or using a context manager — releases both the handles and the blocks they pinned, and often removes the pressure entirely.

Reading the Cache Statistics

GDAL exposes what its cache is doing, and two numbers turn the tuning from guesswork into a measurement.

gdal.GetCacheUsed() reports the bytes currently held, and comparing it against GetCacheMax() answers whether the cache is even being filled. A cache sitting at ten percent of its limit is oversized by a factor of ten, and shrinking it is free.

The hit rate is not exposed directly, but CPL_DEBUG=ON reports block reads, and comparing the count of block reads against the number of distinct blocks the run should have touched approximates it. A ratio near one means each block was read once and the cache is not helping; a ratio well above one means blocks are being re-read and a larger cache would.

from osgeo import gdal

print("cache max:", gdal.GetCacheMax() / 2**20, "MB")
print("cache used:", gdal.GetCacheUsed() / 2**20, "MB")

Sampling those two at the end of a representative unit, and logging them, gives a running record of whether the setting still fits the workload — which matters because the workload changes and the setting does not.

The Interaction With Overviews

One case genuinely rewards a large cache and is easy to miss: building overviews. The operation reads every block at full resolution to produce the first level, then reads that level to produce the next, and so on. Blocks are read several times each, in a pattern with good locality, which is exactly what a block cache is for.

A pipeline that builds overviews as a separate step can raise the cache for that step alone, through a scoped rasterio.Env, and leave it small for the streaming work either side. That is a better arrangement than a single value chosen as a compromise between two workloads with opposite needs.

One last note on units: GDAL accepts a suffix, so GDAL_CACHEMAX=512MB and GDAL_CACHEMAX=512 mean the same thing while GDAL_CACHEMAX=536870912 means half a petabyte. Writing the suffix explicitly removes the ambiguity for the next reader, and costs two characters.

Two further variables shape the same budget and are worth setting alongside it. GDAL_SWATH_SIZE bounds the buffer GDAL uses internally for a warp, and its default is derived from the cache size — so a large cache silently enlarges it too. GDAL_MAX_DATASET_POOL_SIZE bounds how many datasets a VRT keeps open at once, which for a mosaic over thousands of sources is a real amount of memory held outside both the cache and your arrays. Neither needs tuning often, and both belong in the same place as the cache setting so that a future reader finds the whole memory story together rather than one term of it.