Article

Listing Remote Geospatial Datasets with fsspec

Use fsspec to enumerate inputs through one interface regardless of where they live, and prefer a manifest or a prefix-scoped glob over a recursive scan. Discovery is where a large batch spends its first hour if nobody planned it, and the fix costs a few lines. It builds on the Cloud Storage I/O for Spatial Batches guide, part of the broader Spatial Batch Processing & Async Workflows reference.

Prerequisites

  • Python 3.10 or later
  • pip install fsspec s3fs gcsfs aiohttp — install only the backends you need
  • Read access to whichever store holds the inputs

What a Listing Costs

A recursive listing is a paginated API call, and the page size is the unit of cost.

Requests before the first unit of work A recursive scan of ten million keys is ten thousand paginated requests. A listing scoped to one date prefix is a few dozen. Reading a manifest object is one request and returns exactly the keys the batch needs. recursive scan of the bucket ~10,000 requests, minutes of wall clock, most keys discarded listing under one date prefix ~40 requests — if the key layout allows it reading a manifest 1 request, exact contents, and reproducible

The bottom row is not always available — you can only read a manifest if something wrote one — but it is worth arranging, because it is the only option whose cost does not grow with the bucket.

Complete Working Implementation

# gistools/discovery.py
from __future__ import annotations

import json
from collections.abc import Iterator

import fsspec


def iter_inputs(uri: str, pattern: str = "**/*.tif",
                manifest: str | None = None) -> Iterator[str]:
    """Yield input URIs, cheapest strategy first.

    `uri` is a directory-like location: s3://bucket/prefix, gs://..., file://...
    `manifest` is an optional object holding a JSON list of keys.
    """
    fs, root = fsspec.core.url_to_fs(uri)

    if manifest:
        with fs.open(manifest, "r") as handle:
            for key in json.load(handle):
                yield fs.unstrip_protocol(key)
        return

    # find() streams pages rather than materialising the whole listing.
    for path in fs.glob(f"{root.rstrip('/')}/{pattern}"):
        yield fs.unstrip_protocol(path)


def write_manifest(uri: str, keys: list[str], manifest: str) -> None:
    """Record exactly what a producing job wrote, for consumers to read."""
    fs, _ = fsspec.core.url_to_fs(uri)
    with fs.open(manifest, "w") as handle:
        json.dump(sorted(keys), handle)

And opening whatever it yields, without caring which store it came from:

import fsspec
import rasterio


def open_any(uri: str):
    """Open a raster from any fsspec-supported location."""
    if uri.startswith(("s3://", "/vsis3/")):
        # GDAL's own driver is faster for windowed raster reads.
        return rasterio.open(uri.replace("s3://", "/vsis3/"))
    with fsspec.open(uri, "rb") as handle:
        return rasterio.open(handle)

Step Annotations

  1. url_to_fs rather than a hardcoded backend. It returns the right filesystem for the scheme, which is what lets the same discovery code run against a local directory in tests and an object store in production.
  2. unstrip_protocol on the way out. glob returns bare paths without the scheme, and a consumer given bucket/key instead of s3://bucket/key will try to open a local file.
  3. The manifest branch returns early. Falling through to a glob after reading a manifest would defeat the point; the manifest is authoritative or it is not used at all.
  4. glob over find for the pattern case. find returns everything and filters in Python, which for a large prefix means transferring metadata for keys you discard.
  5. GDAL’s driver is preferred for raster reads even when fsspec could do it. An fsspec file object works, but GDAL’s virtual file system issues better range requests, which matters for the windowed reads described in Reading COGs from S3 with rasterio and /vsis3/.

One Named Gotcha: fsspec Caches Listings for the Process Lifetime

fsspec caches directory listings aggressively, which is usually helpful and is wrong for a long-running worker. A batch that lists a prefix, waits for a producer to add files, and lists again sees the original listing and processes nothing new.

fs, root = fsspec.core.url_to_fs(uri)
fs.invalidate_cache(root)      # before a re-listing that must see new keys

The same cache also causes a confusing failure in tests: a fixture that writes a file after the filesystem object was created is invisible to it. Creating the filesystem after the fixture, or invalidating explicitly, avoids an hour of confusion over a file that demonstrably exists.

Choosing an enumeration strategy is mostly a question of what wrote the data and whether you control the key layout.

Which strategy your situation allows If you control the producing job, write a manifest and read it. If the key layout encodes a date or tile partition, scope the listing to a prefix. If neither, a recursive scan is the only option and should be cached rather than repeated. you control the producer write a manifest at the end of the producing job one request, exact the key layout partitions the data scope the glob to the prefix you need tens of requests neither applies scan once, cache the result, and reuse it amortise the scan

The third row is a stopgap that becomes permanent surprisingly often. Writing the cached scan to an object turns it into the manifest of the first row, and the next batch gets the cheap path.

Verification

# 1. Count requests for a discovery pass, with the backend's own logging on.
python - <<'PY'
import logging, fsspec
logging.basicConfig(level=logging.DEBUG)
logging.getLogger("s3fs").setLevel(logging.DEBUG)
from gistools.discovery import iter_inputs
print(sum(1 for _ in iter_inputs("s3://bucket/scenes/2026/08")))
PY

# 2. The manifest and the listing agree.
python -c "
from gistools.discovery import iter_inputs
a = set(iter_inputs('s3://bucket/out'))
b = set(iter_inputs('s3://bucket/out', manifest='s3://bucket/out/manifest.json'))
print('only in listing:', len(a - b), 'only in manifest:', len(b - a))
"

The second check is the one to run after a batch finishes. A key in the listing but not the manifest means something wrote an output the job did not record; the reverse means a recorded output never landed, which is a genuine failure the batch did not notice.

What a Manifest Should Contain

A manifest that lists only keys works. A manifest that carries a little more turns the consuming batch from possible into cheap.

How much a manifest should carry Keys alone remove the listing cost. Adding size and checksum lets a consumer verify without reading and estimate the work before starting. Adding a bounding box per key lets a consumer select spatially without opening a single object. keys only removes the listing cost the consumer still opens every object to find out anything about it the minimum plus size and checksum verify without reading estimate total work before the batch starts worth the bytes plus a bounding box select spatially with no opens at all the consuming batch reads only what it needs the useful version

The third level is what makes a manifest a spatial index in miniature. For a batch that processes one region out of a continental archive, it is the difference between opening ten thousand objects to find forty and opening forty.

Turning a Listing Into a Work List

Discovery produces paths; a batch needs units of work, and the step between them is where two avoidable problems live.

The first is ordering. An object store returns keys in lexicographic order, which for a tiled archive means all of one row before any of the next. Processing in that order concentrates reads on one prefix at a time, which is exactly the access pattern that attracts per-prefix throttling. Shuffling the work list with a fixed seed spreads the load and keeps the run reproducible.

The second is grouping. Where several units read the same object — overlapping windows, or several bands — grouping them so one worker handles them together turns repeated remote reads into one read and several windows. That is the same reasoning behind the copy-first threshold, applied at the level of scheduling rather than storage.

from collections import defaultdict
import random

def build_work_list(uris, windows_by_uri, seed: int = 0):
    grouped = [(uri, windows_by_uri[uri]) for uri in uris]
    random.Random(seed).shuffle(grouped)      # spread across prefixes, reproducibly
    return grouped

A fixed seed matters more than it looks: an unshuffled list is reproducible and badly distributed, a randomly shuffled one is well distributed and irreproducible, and a seeded shuffle is both.

Keeping the Manifest Current

A manifest is only useful while it matches reality, and three things put it out of date.

A partial run that writes some outputs and fails leaves objects the manifest does not mention. Writing the manifest only after a successful completion — as the last step, from the collected results — avoids it, at the cost of having no manifest for a failed run.

A manual deletion removes an object the manifest still lists. Consumers should treat a missing object named in a manifest as a warning rather than a fatal error, and record it, so the discrepancy is visible without stopping the batch.

A re-run that overwrites some objects and not others leaves a manifest that is correct about identity and stale about size or checksum. Recording a version or timestamp per entry lets a consumer notice, and is another argument for a manifest carrying more than keys.

The reconciliation check — listing once and comparing against the manifest — catches all three and costs one listing. Running it after each batch rather than never turns a class of silent divergence into a line in the log.

Falling back gracefully

A pipeline that requires a manifest and finds none should fall back to a listing with a warning rather than failing. The warning matters: a run that silently takes the expensive path is a run whose cost nobody attributes correctly, and the first sign is a bill rather than a log line.

Recording which path was taken, in the same structured record as everything else, makes the question answerable afterwards. It also makes the improvement measurable — a pipeline that moves from listing to manifest can show the change in request count rather than asserting it.

One further note on caching: fsspec can also cache the objects themselves, not just listings, through its simplecache and blockcache layers. For a pipeline that reads each object once those add nothing; for one that reads repeatedly they are a simpler alternative to managing a local copy by hand, and they respect the same threshold logic as an explicit copy would.

A last note on protocols: fsspec treats a plain path as local, so a work list mixing local and remote entries works without any branching in the pipeline. That is genuinely useful for testing, where the same discovery code can run against a temporary directory, and it is worth preserving by carrying full URIs rather than stripping the scheme for tidiness.