Article

Choosing Between Celery, RQ and Dask for Geospatial Batches

Choose by the shape of the work, not by popularity. Independent per-file units with retries and scheduling point at Celery; the same units without scheduling point at RQ, which is a tenth of the configuration; a computation that is genuinely a graph over chunked arrays points at Dask, whose scheduler exists for exactly that. It builds on the Distributed Task Queues for Spatial Jobs guide, part of the broader Spatial Batch Processing & Async Workflows reference.

Prerequisites

  • Python 3.10 or later
  • A batch already working on one machine, per Multiprocessing Geospatial Tasks
  • Redis available, for either queue option; Dask needs no broker

The Question That Decides It

Everything else follows from whether your units depend on each other.

Which distribution layer fits the work If units depend on each other through shared intermediate arrays, Dask. If they are independent and you need scheduling, routing or chained workflows, Celery. If they are independent and you do not, RQ. do units share intermediate results? yes Dask a graph, not a list mosaics, zonal stats, resampling stacks no need scheduling, routing or chains? yes Celery more moving parts no RQ a tenth of the config

Most geospatial batches land in the bottom-right leaf and are built on the middle one, because Celery is what people have heard of. The cost of that mistake is not throughput — it is the four extra components in the deployment that nobody needed.

The Same Batch, Three Ways

Converting ten thousand shapefiles to GeoPackage is the canonical independent-unit workload. Here it is in each tool, at the same level of completeness.

# RQ — the whole thing
from redis import Redis
from rq import Queue

from gistools.convert import shp_to_gpkg

q = Queue("convert", connection=Redis())
for path in shapefiles:
    q.enqueue(shp_to_gpkg, str(path), str(path.with_suffix(".gpkg")),
              job_timeout=600, retry=Retry(max=3, interval=[10, 60, 300]))
# Celery — the task, plus a config module and a worker deployment
from celery import Celery

app = Celery("gistools", broker="redis://localhost:6379/0")
app.conf.update(task_acks_late=True, worker_prefetch_multiplier=1,
                task_serializer="json", result_expires=3600)


@app.task(bind=True, max_retries=3, retry_backoff=True)
def convert(self, src: str, dst: str) -> str:
    from gistools.convert import shp_to_gpkg
    return shp_to_gpkg(src, dst)


for path in shapefiles:
    convert.delay(str(path), str(path.with_suffix(".gpkg")))
# Dask — correct, but the scheduler has nothing to optimise
from dask.distributed import Client

client = Client("tcp://scheduler:8786")
futures = [client.submit(shp_to_gpkg, str(p), str(p.with_suffix(".gpkg")))
           for p in shapefiles]
client.gather(futures)

All three work. The RQ version is complete as written; the Celery version needs a worker command and a config module; the Dask version needs a scheduler and workers running, and gains nothing from the graph machinery because there is no graph.

Where Dask Earns Its Place

The picture inverts when units share intermediate results. A mosaic that resamples a stack of overlapping rasters, computes a per-pixel median and writes one output has a real dependency graph: each output chunk depends on several input chunks, and several output chunks depend on the same input.

A list of units versus a graph of chunks On the left, four independent units each read one input and write one output with no sharing. On the right, three output chunks each depend on two or three overlapping input chunks, so a scheduler that reuses a loaded chunk does strictly less work. independent units — a queue is enough nothing is shared, so nothing can be reused a dependency graph — Dask pays off the middle input is read once and used three times

That reuse is the whole value proposition. A queue would read the shared input once per consuming task, because a queue has no idea the tasks are related.

One Named Gotcha: Dask Workers and GDAL Threads Multiply

Dask defaults to several threads per worker, and GDAL happily uses several threads of its own if GDAL_NUM_THREADS is set to ALL_CPUS. The product oversubscribes the machine badly: four workers times four threads times GDAL’s own pool is far more concurrency than the hardware has, and throughput drops while memory rises.

Pin both explicitly:

from dask.distributed import LocalCluster

cluster = LocalCluster(
    n_workers=4,
    threads_per_worker=1,          # one GDAL context per worker
    memory_limit="6GB",
    env={"GDAL_NUM_THREADS": "1", "GDAL_CACHEMAX": "512"},
)

One thread per worker is the right default for GDAL work specifically, because the library’s own locking around dataset handles undoes most of what extra threads would buy.

Verification

Whichever you choose, the same three measurements tell you whether it is working:

# throughput — units finished per minute, from the log stream
jq -r 'select(.event=="unit_finished") | .ts' run.jsonl | cut -c1-16 | uniq -c

# balance — units per worker; a wide spread means dispatch, not the data
jq -r 'select(.event=="unit_finished") | .worker' run.jsonl | sort | uniq -c

# waste — retried units as a share of the total
jq -r 'select(.retry_count > 0) | .key' run.jsonl | wc -l

If the second command shows an even spread and the third is near zero, the distribution layer is not your bottleneck, and any further tuning belongs in the task body rather than in the scheduler.

What Each One Costs to Operate

Throughput is rarely the deciding factor, because all three saturate the same hardware. Operational surface is, and it differs by a lot.

Components you have to run and watch RQ needs Redis and a worker process. Celery needs a broker, optionally a result backend, and benefits from a monitoring component. Dask needs a scheduler process and workers whose thread and memory limits must be tuned to avoid oversubscription. RQ Redis worker processes two things to run jobs visible with rq info Celery broker worker processes result backend, monitoring four, and each can misbehave Dask scheduler workers, thread-tuned no broker at all but tuning is mandatory

Dask’s lack of a broker is genuinely attractive, and its dashboard is the best of the three for understanding where time goes. What it does not give you is durability: a scheduler restart loses the graph, whereas a queue’s messages survive. For a batch that must resume after an interruption, that difference outweighs everything else on this diagram.

The Cost Nobody Estimates

The comparison people make is about throughput, and the cost that actually decides is operational: how many components must be running, monitored and upgraded for the pipeline to work.

RQ needs Redis and a worker process. Both are things most teams already run or can run in an afternoon, and the failure modes are few enough to hold in your head. Celery needs a broker, workers, usually a result backend, and benefits from a monitoring component — four things, each with its own configuration and its own way of going wrong. Dask needs a scheduler and workers, with no broker, but the scheduler is stateful and its restart loses the graph.

None of that argues against Celery or Dask. It argues for choosing them because you need what they add, rather than because they are what people write blog posts about. A pipeline that outgrows RQ will tell you clearly — you will find yourself wanting routing, or scheduling, or chained workflows — and migrating at that point is a smaller job than operating a component you did not need for a year.

A Practical Sequence

For a team starting from a working single-machine batch, the sequence that tends to work is: make the units idempotent first, then move to RQ, then move to Celery only if a specific need appears.

Idempotency first, because it is required by all three and is the change most likely to surface bugs in the existing code. Doing it while everything is still in one process means those bugs are easy to find.

RQ next, because it proves the distribution model — reference payloads, shared storage, recovery on worker loss — with the smallest possible amount of new infrastructure. If the pipeline works on RQ across two machines, the hard part is done.

Celery only when a named requirement appears: scheduled runs, several queues with different worker profiles, or a workflow with a fan-in step. Each of those is a real reason. We might need it later is not, and it is what puts four components into a deployment that needed two.

Migration Paths Between Them

Because the three are not equally interchangeable, it is worth knowing which moves are cheap before committing to one.

RQ to Celery is straightforward. Both take a function and arguments, both run it in a worker, and the task bodies transfer unchanged. What changes is the configuration and the deployment, which is real work but bounded and does not touch the geospatial code at all.

Celery to RQ is equally easy in the mechanical sense and usually motivated by removing components rather than adding them. The features that will be missed are routing and scheduled tasks, and it is worth checking whether anything actually uses them before assuming they are needed.

Either queue to Dask is a rewrite, because the model is different: instead of submitting independent callables you describe a computation over chunked arrays and let the scheduler decide the order. Code written as a task body does not become a graph node without restructuring.

Dask to a queue is the same rewrite in reverse and is the one people are most often surprised by. A pipeline expressed as xarray operations over a chunked stack has no per-unit function to submit; extracting one means deciding what a unit is, which is a design question rather than a translation.

The practical consequence: choosing between the two queues is a low-stakes decision that can be revisited, and choosing between a queue and Dask is a high-stakes one that should follow from the shape of the work rather than from preference.

Observability, Compared

The three differ most in what you can see while a job runs, and for a long batch that matters more than the throughput difference between them.

Dask’s dashboard is genuinely excellent: a live task graph, per-worker memory, a profile of where time is going, and a view of what is spilling to disk. For diagnosing a slow computation it is the best of the three by a wide margin, and it is a legitimate reason to choose Dask even for a workload that is only a list.

Celery’s introspection is functional and command-line shaped: active tasks, reserved tasks, worker statistics. Third-party dashboards exist and add a component. It answers what is running well and why is it slow poorly.

RQ’s is minimal — queue length, job status, failed jobs — and is often enough, because a pipeline whose units are uniform rarely needs more than a count and a failure list.

The gap is closed in every case by the structured records the pipeline emits itself, which is the argument for treating structured logging as part of the distribution work rather than as something to add later. A dashboard tells you about tasks; your own records tell you about tiles, CRS pairs and windows, which is what a geospatial question is actually about.