Article

Dask vs Multiprocessing for Geospatial Workloads

On a single machine, ProcessPoolExecutor is the right default for independent per-file work: it has no scheduler, no serialisation of a task graph, and nothing to tune. Dask earns its overhead only when output chunks share input chunks, because then its scheduler can load a chunk once and use it several times. It builds on the Multiprocessing Geospatial Tasks guide, part of the broader Spatial Batch Processing & Async Workflows reference.

Prerequisites

  • Python 3.10 or later
  • pip install "dask[distributed]" rasterio rioxarray for the Dask examples
  • A workload you can characterise as either independent units or a chunked array computation

The Question, and What Follows From It

A list or a graph Independent units read their own input and write their own output with no sharing, so a scheduler has nothing to optimise. A chunked computation has output chunks depending on several input chunks, and several output chunks depending on the same input, which is exactly what a task graph exists to exploit. a list of units convert 10,000 shapefiles reproject 10,000 tiles no unit needs anything another unit produced process pool a graph of chunks median over an overlapping stack zonal stats across a mosaic one input chunk feeds several output chunks Dask

Almost every geospatial batch that people reach for Dask to solve is the left-hand column. The tell is whether you could shuffle the units into any order and get the same result — if you could, there is no graph.

Side by Side on the Same Work

# A list of units: process pool, no scheduler
from concurrent.futures import ProcessPoolExecutor, as_completed
from multiprocessing import get_context

from gistools.warp import warp_one


def run_pool(paths, dst_dir, dst_crs, workers=8):
    ctx = get_context("spawn")          # never fork with GDAL loaded
    with ProcessPoolExecutor(max_workers=workers, mp_context=ctx) as pool:
        futures = {pool.submit(warp_one, p, dst_dir / p.name, dst_crs): p
                   for p in paths}
        for future in as_completed(futures):
            future.result()             # re-raises in the parent
# A graph of chunks: Dask, because the median needs the whole stack
import rioxarray
import xarray as xr
from dask.distributed import Client, LocalCluster


def run_dask(paths, dst, chunk=2048):
    cluster = LocalCluster(n_workers=4, threads_per_worker=1,
                           memory_limit="6GB",
                           env={"GDAL_NUM_THREADS": "1"})
    with Client(cluster):
        stack = xr.concat(
            [rioxarray.open_rasterio(p, chunks={"x": chunk, "y": chunk})
             for p in paths],
            dim="scene",
        )
        median = stack.median(dim="scene", skipna=True)
        median.rio.to_raster(dst, tiled=True, blockxsize=512, blockysize=512)

The second example is where a scheduler pays: each output chunk needs the same spatial chunk from every scene, and Dask loads each one once. A process pool expressing the same computation would read every input chunk once per output chunk.

Where the Overheads Are

What each approach costs before it does any work A process pool starts workers once and dispatches through a pipe. Dask additionally builds and serialises a task graph, runs a scheduler process, and requires thread and memory limits to be tuned to avoid oversubscribing GDAL. worker start-up both pay it once per process, under spawn a wash task dispatch a pipe write against a graph node in a scheduler pool is cheaper moving results a pool returns values; Dask may move chunks between workers pool is cheaper tuning burden a pool needs a worker count; Dask needs threads and memory too pool is simpler

None of those overheads is large in absolute terms. They matter because for a list of units they buy nothing — you pay them to get scheduling you do not need.

One Named Gotcha: Dask Threads Multiply GDAL Threads

LocalCluster defaults to several threads per worker, and GDAL will use its own threads if allowed. The product oversubscribes the machine, and the symptom is a job that gets slower as you add workers. Setting threads_per_worker=1 alongside GDAL_NUM_THREADS=1 is the correct configuration for GDAL-heavy work, because the library’s locking around dataset handles undoes most of what extra threads would give you.

The same trap catches a process pool the moment someone sets GDAL_NUM_THREADS=ALL_CPUS in the environment. Pinning it explicitly in the worker entry point — rather than inheriting whatever the machine has — removes the whole category, and it is the same discipline Environment Variable Sync recommends for every tuning variable.

Verification

# Scaling should be roughly linear for a list of units.
for w in 1 2 4 8; do
  /usr/bin/time -f "$w workers: %e s" python -m gistools.bench --pool --workers $w
done

# For the graph case, compare Dask against a naive pool implementation.
python -m gistools.bench --dask   --workers 4
python -m gistools.bench --pool-naive --workers 4

If the pool scales linearly and Dask does not beat it, the work is a list and the scheduler is pure overhead. If the naive pool re-reads inputs and Dask is several times faster, the work is a graph and you have your answer.

Four questions settle the choice faster than a benchmark does, and three of them can be answered without running anything.

Four questions, three of them answerable on paper If the units can be shuffled into any order without changing the result, there is no graph. If several outputs read the same input, there is. If a single unit exceeds memory, Dask's chunking helps. A dashboard is a real reason to choose Dask even for a list. can units be shuffled freely? yes means there is no graph to schedule process pool do outputs share inputs? yes means a scheduler can reuse loaded chunks Dask does one unit exceed memory? yes means chunking within a unit is needed Dask do you need live visibility? the dashboard is a legitimate reason on its own Dask

The last row is worth stating because it is usually left unsaid. Dask’s dashboard is genuinely good, and choosing it for observability rather than for scheduling is a defensible decision.

What a Process Pool Does Not Give You

Being fair about the trade means naming what you give up by staying with the standard library.

There is no visibility. A ProcessPoolExecutor has no dashboard, no per-task timing, and no view of what each worker is doing. Everything you want to know has to come from your own logging, which is another argument for the structured records the section recommends.

There is no spilling. When a worker’s memory grows beyond the budget it is killed; Dask can spill intermediate results to disk and continue, more slowly. For a pipeline whose units are independent that rarely matters, because each unit fits or the unit is too big. For a computation over a stack it matters a great deal.

There is no adaptive scaling. A pool has the workers it was created with. Dask can add and remove them as the graph’s demands change, which is valuable for a workload whose parallelism varies through its phases and irrelevant for one that does not.

And there is no resilience. A worker killed mid-task loses that task, and the pool raises. Dask reschedules. For a batch that checkpoints — as Implementing Checkpointing for Interrupted Spatial Batches describes — a rerun recovers, so the difference is one of convenience rather than correctness.

A Middle Path Worth Knowing

For a workload that is mostly a list with a small graph-shaped step at the end — convert ten thousand files, then build one mosaic over the results — the right answer is often both, in sequence rather than in combination.

Run the conversion with a process pool, because it is a list. Then run the mosaic with Dask, because it is a graph. The two phases share nothing but the intermediate files, which are on disk anyway, and each uses the tool suited to it.

Trying to express both phases in one framework is where the awkwardness comes from. A Dask graph over ten thousand independent conversions carries scheduler overhead for no benefit; a process pool building a median over an overlapping stack re-reads every input once per output chunk. Splitting at the natural boundary avoids both.

Trying Both Before Deciding

For a workload that is genuinely ambiguous — some sharing, but not much — the cheapest way to decide is to implement it both ways against a small corpus. The process-pool version is usually under thirty lines, and the Dask version not much more, so the experiment costs an afternoon and settles a question that otherwise gets argued about for weeks.

Three measurements decide it: wall clock, peak memory across all processes, and how much the answer changes when the corpus doubles. The last is the important one, because a workload with sharing scales differently from one without, and a comparison at one size can point the wrong way.

Run both against the same corpus, in randomised order, at least three times each. The variance between repeats is usually larger than people expect, and a single pair of runs frequently shows a difference that disappears under repetition.

What Not to Optimise

Whichever you choose, two things are usually not the bottleneck and absorb effort anyway.

Task dispatch overhead matters only when tasks are very short. For units taking seconds, the difference between a pipe write and a graph node is unmeasurable, and time spent choosing between them is time not spent on the access pattern.

Serialisation of results matters only when results are large. A pipeline returning paths rather than arrays — which is what the reference-payload discipline produces — moves a few hundred bytes per unit, and no serialiser is meaningfully faster than another at that size.

The thing that usually is the bottleneck is how the data is read: block alignment, overviews, and how many times each input is opened. Those are the same for both frameworks, which is another way of saying the choice matters less than it feels like it should.

A Note on rioxarray

Much of Dask’s appeal for raster work comes through rioxarray, which presents a raster as a chunked labelled array and makes operations across a stack read naturally. It is genuinely good, and it carries one assumption worth being explicit about: chunk boundaries should align with the file’s internal block layout.

A chunk size chosen for convenience — a round thousand, say — that does not divide the block size means every chunk read touches partial blocks, and GDAL decodes neighbours it will discard. Passing chunks="auto" lets rioxarray read the block shape and choose accordingly, and passing an explicit size means taking responsibility for the alignment yourself.

That is the same block-alignment concern the section raises for windowed reads generally, appearing in a different guise. It is worth checking early, because the symptom — a computation that is slower than it should be with no obvious cause — looks like a scheduler problem and is not.