Article

Sizing Worker Pools for GDAL-Bound Queues

Compute worker count as the container’s memory limit divided by the measured per-task footprint, capped at core count — then set GDAL_CACHEMAX explicitly so that footprint is a number you chose rather than a default multiplied by however many workers you started. Sizing on cores alone is the most common cause of a batch that dies at eighty percent on a bigger input. It builds on the Distributed Task Queues for Spatial Jobs guide, part of the broader Spatial Batch Processing & Async Workflows reference.

Prerequisites

  • Python 3.10 or later
  • pip install psutil rasterio for the measurement helper
  • A container or host whose memory limit you can read — on cgroup v2 that is /sys/fs/cgroup/memory.max

The Four Terms That Set the Limit

Every worker costs the same four things, and only one of them is what people usually count.

What one worker costs before it does anything Interpreter and imports are a fixed cost per worker. The GDAL block cache is whatever GDAL_CACHEMAX says, per process. Array buffers scale with the window size and the number alive at once. Driver scratch space is small but not zero. interpreter + imports ~180 MB, fixed GDAL block cache GDAL_CACHEMAX, per process array buffers scales with window size driver scratch small, opaque, real the arithmetic workers = min(cores, floor((memory_limit x 0.8) / per_worker_footprint)) The 0.8 leaves room for the page cache, allocator fragmentation and the spike at the peak moment.

The third box is the only one that changes with the work, which is why the measurement below takes a representative task rather than a synthetic one — the number is meaningless without a real window size and band count.

Complete Working Implementation

# gistools/sizing.py
from __future__ import annotations

import multiprocessing
import os
import resource
from pathlib import Path

CGROUP_V2 = Path("/sys/fs/cgroup/memory.max")
CGROUP_V1 = Path("/sys/fs/cgroup/memory/memory.limit_in_bytes")
HEADROOM = 0.8


def memory_limit_bytes() -> int:
    """The memory this process is actually allowed, container-aware."""
    for path in (CGROUP_V2, CGROUP_V1):
        try:
            raw = path.read_text().strip()
        except OSError:
            continue
        if raw and raw != "max":
            value = int(raw)
            if value < (1 << 62):          # v1 uses a huge sentinel for "unlimited"
                return value
    # Fall back to physical memory.
    return os.sysconf("SC_PAGE_SIZE") * os.sysconf("SC_PHYS_PAGES")


def measure_task_footprint(run_once) -> int:
    """Peak RSS delta, in bytes, of one representative task."""
    before = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss * 1024
    run_once()
    after = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss * 1024
    return max(after - before, 1)


def worker_count(per_worker_bytes: int, cores: int | None = None) -> int:
    cores = cores or multiprocessing.cpu_count()
    budget = int(memory_limit_bytes() * HEADROOM)
    by_memory = max(1, budget // per_worker_bytes)
    return max(1, min(cores, by_memory))


def gdal_cache_bytes(per_worker_bytes: int, workers: int) -> int:
    """A cache size that fits inside the per-worker budget, floor 64 MB."""
    budget = int(memory_limit_bytes() * HEADROOM) // workers
    return max(64 << 20, min(512 << 20, budget - per_worker_bytes))

Wiring it into the worker start-up:

# gistools/worker_entry.py
import os

from gistools.sizing import gdal_cache_bytes, worker_count

BASELINE = 180 << 20          # interpreter + imports, measured once
TASK_PEAK = 2 << 30           # measured with measure_task_footprint

workers = worker_count(BASELINE + TASK_PEAK)
os.environ.setdefault("GDAL_CACHEMAX",
                      str(gdal_cache_bytes(BASELINE + TASK_PEAK, workers) >> 20))
os.environ.setdefault("GDAL_NUM_THREADS", "1")

print(f"starting {workers} workers, GDAL_CACHEMAX={os.environ['GDAL_CACHEMAX']} MB")

Step Annotations

  1. GDAL_CACHEMAX is set before any GDAL import. The library reads it at initialisation; setting it afterwards has no effect, which is the same ordering trap described in Loading dotenv Files in a Geospatial CLI.
  2. GDAL_NUM_THREADS=1 because the queue already supplies the parallelism. Letting each worker spawn its own threads multiplies concurrency by a factor nobody counted and reliably makes things slower once the count exceeds the cores.
  3. The cgroup v1 sentinel check. An unlimited v1 cgroup reports a number close to 2⁶³, and using it as a budget yields a worker count in the millions.
  4. ru_maxrss is in kilobytes on Linux and bytes on macOS. The multiplication above is correct for Linux, which is where workers run; a cross-platform helper needs a branch.
  5. The floor of one worker. On a small container the arithmetic can yield zero, and a pool of zero workers consumes the queue silently without doing anything.

One Named Gotcha: The Cache Is Per Process, Not Per Node

GDAL_CACHEMAX is often read as a global budget. It is not — each process gets its own cache of that size. Eight workers with the default 5% of physical memory on a 64 GB node reserve over 25 GB between them before a single array is allocated, and the symptom is a batch that dies with plenty of apparent headroom in the per-worker measurements.

# What each worker actually reserved
for pid in $(pgrep -f 'celery worker'); do
  tr '\0' '\n' < /proc/$pid/environ | grep GDAL_CACHEMAX
done | sort | uniq -c

If that command prints nothing, the default is in force and the total is invisible. Setting the value explicitly — even to the same number the default would have chosen — makes it auditable, which is most of the battle.

Two knobs interact in a way that is easy to miss: worker count and cache size multiply, so raising one silently shrinks the budget available to the other.

Worker count against cache size, on a 32 GB node With a two-gigabyte per-task working set, four workers fit at any cache size, eight fit up to a 512 megabyte cache, and sixteen exceed the budget even with the minimum cache. cache 128 MB cache 512 MB cache 2 GB 4 workers 9 GB 10 GB 16 GB 8 workers 18 GB 21 GB 33 GB 16 workers 36 GB 42 GB 65 GB green fits the 26 GB working budget; amber is at the edge; red exceeds it

The amber cell is the interesting one, because it is where most teams end up: within budget on today’s tiles and over it on a slightly larger input. Sizing to the green column leaves the margin that makes next month’s data a non-event.

Verification

# 1. The pool sized itself as expected.
python -c "from gistools.sizing import worker_count, memory_limit_bytes;            print('limit GB:', memory_limit_bytes()/2**30,                  'workers:', worker_count((180<<20)+(2<<30)))"

# 2. Peak usage across the whole pool stays under the limit during a real batch.
while pgrep -f 'celery worker' > /dev/null; do
  ps -o rss= -p $(pgrep -d, -f 'celery worker') | awk '{s+=$1} END {print s/1024" MB"}'
  sleep 5
done | sort -n | tail -1

# 3. Nothing was killed.
dmesg -T | grep -i 'killed process' | tail -5

The second command’s maximum is the number that matters, and it should land below the limit with the headroom you budgeted. If it lands at ninety-five percent, the batch will survive today’s input and not next month’s.

Signals That the Pool Is the Wrong Size

Four observable symptoms map cleanly onto sizing mistakes, which makes tuning a matter of reading rather than guessing.

Reading the symptom back to the cause Idle CPU with a full queue means too few workers or a task that waits. Kills under load mean too many workers for the memory. High context switching means oversubscription, often GDAL threads on top of workers. Steady memory growth across a long run means a leak rather than a sizing error. CPU idle, queue full workers are waiting on I/O, or there are too few raise the count kills under load the pool exceeds the memory limit at the peak lower the count high context switching GDAL threads multiplying the worker count pin GDAL_NUM_THREADS memory grows over hours not a sizing problem — a leak look for open datasets

The last row is worth separating out explicitly, because it is routinely mistaken for the second and answered by reducing workers. That makes the batch slower without fixing anything.

Revisiting the Numbers

A worker count derived once stays correct until something underneath changes, and three things routinely do.

The node changes. A move to a different instance family alters both cores and memory, usually not in the same ratio, and a count derived from the old machine is wrong on the new one. Deriving it at start-up from the container limit rather than pinning it in configuration makes the move a non-event.

The data changes. A corpus whose tiles have grown means a larger working set per task, so the same worker count now exceeds the budget. Recording the observed per-task peak in the structured records makes this visible as a trend rather than as an incident.

The work changes. Adding a band, changing the resampling method, or switching to a dtype twice the width all move the working set. A change to the processing code deserves a re-measurement of the footprint for the same reason it deserves a test.

The check that catches all three is comparing the observed pool peak against the limit at the end of every batch, and logging the ratio. A ratio that has crept from sixty percent to ninety over a few months is a warning with weeks of notice; a kill is the same information with none.

Heterogeneous Workers

A single pool sized for the largest unit is wasteful when most units are small, and the queue makes a better arrangement available: two pools, on two queues, with different sizing.

The heavy queue takes the large windows and runs few workers with generous memory. The light queue takes metadata reads, manifest writes and small tiles, and runs many workers with small limits. Routing is by task name, which the parent guide covers, and the sizing arithmetic is applied separately to each.

The gain is not subtle. A machine running eight uniform workers sized for the worst case is frequently running eight workers doing trivial work; the same machine running three heavy and sixteen light workers keeps both kinds of work moving and uses the memory that the uniform arrangement left idle.

The cost is one more deployment unit and the discipline of routing tasks correctly. For a pipeline whose unit sizes span an order of magnitude — which most geospatial batches do — it repays that quickly.

Autoscaling and the Memory Bound

A pool that scales on queue depth alone will happily add workers a node cannot hold, because depth says nothing about what each worker will consume.

Scaling on two signals fixes it: add workers while the queue is deep and the memory headroom is above a threshold; stop adding when either condition fails. The second condition is what turns a scaling rule into a safe one, and it needs the same per-task footprint figure the fixed sizing uses.

Where the platform only supports scaling on one metric, the practical alternative is to cap the maximum at the number the memory arithmetic allows and let it scale freely below that. It gives most of the benefit of autoscaling with none of the risk, and the cap is a number you already have.