Article

GeoPackage vs FlatGeobuf for Streaming Vector Reads

Both formats let a reader jump to an arbitrary feature without parsing everything before it, which is what makes chunked reading viable. They differ in how: GeoPackage is a SQLite database with an R-tree index, FlatGeobuf is a flat buffer with a packed R-tree in its header. That difference decides which one suits remote reads, which suits appends, and which suits attribute filters. It builds on the Chunked Vector Data Reading guide, part of the broader Spatial Batch Processing & Async Workflows reference.

Prerequisites

  • Python 3.10 or later with pip install pyogrio geopandas
  • GDAL 3.1 or later for FlatGeobuf write support
  • A layer large enough that reading it whole is not an option

How Each One Finds a Feature

Two ways to reach feature number 400,000 GeoPackage resolves through SQLite: an index lookup gives a row id, and the database reads the page holding it. FlatGeobuf reads a packed R-tree from the file header, which yields a byte offset that is read directly with no intermediate structure. GeoPackage SQLite b-tree plus an R-tree virtual table several page reads per feature lookup attribute filters use SQL richest querying FlatGeobuf a packed R-tree in the file header one header read, then a direct byte offset no attribute index at all cheapest seeking

The consequence for remote reads is decisive. FlatGeobuf’s header can be fetched with one range request and then every feature is a direct offset, which is exactly the access pattern object storage rewards. A GeoPackage read issues many small seeks that each become a request.

Complete Working Implementation

# gistools/vector_chunks.py
from __future__ import annotations

from collections.abc import Iterator

import geopandas as gpd
import pyogrio


def iter_chunks(path: str, size: int = 50_000,
                bbox: tuple | None = None) -> Iterator[gpd.GeoDataFrame]:
    """Yield successive chunks from a seekable vector format."""
    info = pyogrio.read_info(path)
    total = info["features"]

    offset = 0
    while offset < total:
        frame = pyogrio.read_dataframe(
            path,
            skip_features=offset,
            max_features=size,
            bbox=bbox,          # pushed down to the R-tree in both formats
        )
        if frame.empty:
            break
        yield frame
        offset += size


def convert_for_streaming(src: str, dst: str) -> None:
    """Rewrite a layer as FlatGeobuf, which is the cheaper format to stream."""
    frame = pyogrio.read_dataframe(src)
    pyogrio.write_dataframe(frame, dst, driver="FlatGeobuf",
                            spatial_index=True)

Step Annotations

  1. read_info rather than reading the layer. It parses the header only, so the feature count costs one small read regardless of layer size.
  2. bbox is pushed down, not filtered afterwards. Both drivers use their spatial index for it, so a regional extract never materialises features outside the box.
  3. The empty-frame break. A layer whose count is stale — which happens after an append to a GeoPackage — would otherwise loop past the end returning nothing.
  4. spatial_index=True on write. FlatGeobuf without its packed R-tree is a flat file with no seeking at all, which silently removes the property the format was chosen for.
  5. Chunk size at fifty thousand. Large enough to amortise the per-call overhead, small enough that a chunk of typical polygons stays well inside a worker’s memory budget.

Which to Choose

Choosing between the two by what the pipeline needs Remote streaming favours FlatGeobuf because one header read enables direct offsets. Attribute filtering favours GeoPackage because SQL indexes apply. Appending favours GeoPackage because FlatGeobuf must be rewritten. Multiple layers in one file favour GeoPackage. streaming from object storage one header read, then direct offsets FlatGeobuf filtering on attributes SQL indexes apply; FlatGeobuf has none GeoPackage appending to an existing layer FlatGeobuf must be rewritten wholesale GeoPackage several layers in one file FlatGeobuf is one layer per file, by design GeoPackage

For a read-only intermediate in a batch pipeline — which is what most chunked reads are — FlatGeobuf is usually the better fit. For anything a person will open, query and edit, GeoPackage is.

One Named Gotcha: Appending to FlatGeobuf Invalidates the Index

FlatGeobuf’s R-tree is written once, in the header, over the features that existed at write time. Appending features afterwards produces a file whose index does not cover them, and a spatial query silently returns the original subset with no error and no warning.

The safe pattern is to treat FlatGeobuf outputs as immutable: write once, and if more features arrive, write a new file and read the set. For a batch pipeline that is natural — each unit produces its own output — and it is another reason the format suits intermediates better than it suits a working dataset.

Verification

# The index exists and covers every feature.
python -c "
import pyogrio
info = pyogrio.read_info('out.fgb')
print('features:', info['features'], 'capabilities:', info.get('capabilities'))
"

# A bbox read touches far fewer features than a full read.
python -c "
import pyogrio, time
t = time.perf_counter(); a = pyogrio.read_dataframe('out.fgb'); full = time.perf_counter() - t
t = time.perf_counter(); b = pyogrio.read_dataframe('out.fgb', bbox=(-2.1,51.3,-1.8,51.6)); part = time.perf_counter() - t
print(f'{len(a)} in {full:.2f}s vs {len(b)} in {part:.2f}s')
"

If the bbox read takes about as long as the full read, the spatial index is missing or was invalidated by an append, and every chunked read is quietly scanning.

Conversion is cheap and one-directional in practice, so it is worth knowing what each conversion keeps and what it drops.

What a conversion keeps and drops Geometry and attributes survive both ways. The spatial index is rebuilt rather than copied. Styling and metadata tables stored in a GeoPackage have no FlatGeobuf equivalent. Multiple layers become multiple files. geometry and attributes survive both directions unchanged safe the spatial index rebuilt on write, not copied set spatial_index=True styling and metadata tables no FlatGeobuf equivalent — dropped lost going out multiple layers one FlatGeobuf file per layer becomes several files

The third row is why a GeoPackage a person curated should not be replaced by its FlatGeobuf conversion — keep both, and use the flat one as the pipeline’s input.

Reading Either One Remotely

Both formats can be read over HTTP, and they behave very differently when you do.

FlatGeobuf is well suited to it. The packed R-tree sits in the header, so one range request gives a reader everything it needs to compute byte offsets for the features it wants, and each feature is then a direct range read. A bounding-box query against a remote FlatGeobuf touches a handful of ranges regardless of how large the file is.

GeoPackage is not. SQLite performs many small seeks — through the b-tree, then the R-tree, then the data pages — and each one becomes an HTTP request. A query that is instant locally can take minutes remotely, and the cause is invisible from the query itself.

The practical rule: if the layer will be read over a network, write FlatGeobuf. If it will be read from local disk or a mounted volume, either works and GeoPackage’s richer querying may decide it.

Converting as a Pipeline Step

Because the conversion is cheap relative to repeated reads, it is usually worth doing as an explicit first stage rather than as a manual preparation.

def ensure_streamable(src: str, cache_dir: Path) -> str:
    """Return a seekable path for `src`, converting once if necessary."""
    import pyogrio

    if src.lower().endswith((".fgb", ".gpkg")):
        return src
    dst = cache_dir / (Path(src).stem + ".fgb")
    if not dst.exists():
        pyogrio.write_dataframe(pyogrio.read_dataframe(src), dst,
                                driver="FlatGeobuf", spatial_index=True)
    return str(dst)

Making it a step rather than a prerequisite means the pipeline works on whatever it is given, pays the conversion once, and reuses it on subsequent runs. The if not dst.exists() check is what makes the second run cheap, and pairing it with the source’s modification time is a small refinement worth adding if the inputs are ever regenerated.

The one caveat is disk: a converted copy of a large archive is a second copy, and the cache directory needs a retention policy or it will fill the volume. Treating it as a cache — safe to delete, rebuilt on demand — rather than as an output is what keeps that manageable.

Schema and Type Fidelity

The two formats differ in what they can express about attributes, and the differences bite on conversion rather than on read.

GeoPackage inherits SQLite’s type system, which is permissive: a column can hold values of several types and the declared type is advisory. That flexibility is convenient when writing and a hazard when reading, because a column declared as integer can contain a string that arrived from a badly formed source.

FlatGeobuf declares its schema in the header and enforces it, which means a conversion from a GeoPackage with a mixed column fails at write time rather than producing a file whose schema lies. That failure is a feature — it surfaces a data problem at the point where it can be fixed — and it surprises people who expected the conversion to be lossless.

Two column types need particular attention. Dates and datetimes round-trip cleanly between the two, but a GeoPackage column holding date strings rather than proper date values converts to text, and the consumer downstream then sorts lexicographically. And a nullable integer column with nulls becomes a float column in some conversion paths, because that is how a missing integer is represented in the intermediate — which silently changes join behaviour.

Checking the schema after conversion, rather than assuming it, catches both:

import pyogrio

before = pyogrio.read_info(src)
after = pyogrio.read_info(dst)
for name in before["fields"]:
    if name not in after["fields"]:
        print("dropped:", name)
before_types = dict(zip(before["fields"], before["dtypes"]))
after_types = dict(zip(after["fields"], after["dtypes"]))
for name, dtype in before_types.items():
    if after_types.get(name) != dtype:
        print(f"type changed: {name}: {dtype} -> {after_types.get(name)}")

Running that once against a representative layer tells you whether the conversion is safe for your schema, and the answer usually holds for every layer from the same source.

Writing Either One From a Batch

Both formats are reasonable outputs and they impose different constraints on how a batch writes them.

A GeoPackage can be appended to, so several units can add to one layer — provided they coordinate, because SQLite locks the database during a write. In practice that means either one writer, or one file per unit merged afterwards, and the second is usually simpler and faster.

FlatGeobuf cannot be appended to meaningfully, because its spatial index is written once over the features present. A batch producing FlatGeobuf therefore writes one file per unit by construction, which suits a distributed pipeline: no coordination, no locking, and the set of files is the output.

Either way, writing one file per unit and treating the set as the product is the arrangement that scales. Merging into a single file is then an optional final step rather than a constraint on the work, and it can be skipped entirely for consumers that accept a directory.