Article

pyogrio vs Fiona for Large Vector Datasets

For reading large Shapefiles and GeoPackages, choose pyogrio when you want the whole layer as a GeoDataFrame fast — it moves data through GDAL’s Arrow interface and is commonly 5–20x quicker than Fiona. Choose Fiona when you need true record-by-record streaming at constant memory. This page is part of the Chunked Vector Data Reading for Spatial Pipelines guide inside the broader Spatial Batch Processing & Async Workflows reference.

Prerequisites

  • Python 3.10 or later
  • pip install pyogrio fiona geopandas pyproj shapely
  • A GDAL 3.6+ build; pyogrio’s fastest path needs GDAL compiled with the Arrow (columnar) read API

Both libraries wrap the same GDAL/OGR drivers, so they read identical formats — the difference is entirely in how each hands features back to Python. For memory strategy across the whole pipeline, read Memory Management for Large GIS Datasets; for a windowed streaming pattern in depth, see Reading Large GeoJSON in Chunks with pyogrio.

How the Two Readers Differ

pyogrio reads a layer columnarly: GDAL fills Arrow record batches (one contiguous buffer per attribute column plus a WKB geometry column), and pyogrio hands those directly to geopandas with almost no per-feature Python overhead. Fiona reads row-wise: for every feature it constructs a Python dictionary shaped like GeoJSON ({"geometry": ..., "properties": ...}). That dict is convenient and stable, but allocating millions of them dominates runtime and creates garbage-collector pressure.

The consequence is a throughput-versus-memory trade. pyogrio wins throughput but, by default, holds the entire layer resident. Fiona holds one feature at a time, so its peak memory is flat no matter how large the source grows.

pyogrio vs Fiona decision matrix A decision flow on the left routes a reading need to pyogrio or Fiona, and a comparison matrix on the right contrasts the two on throughput, peak memory, streaming, and dependency footprint. Which reader? Need every feature at once? yes pyogrio bulk GeoDataFrame no Layer larger than RAM? stream one at a time yes Fiona constant memory no pyogrio skip/max window Comparison matrix Throughput pyogrio Fiona Peak memory high flat True streaming windowed native Output shape GeoDataFrame dict Dependency Arrow + GDAL GDAL Rule of thumb: pyogrio for analytics-scale bulk loads; Fiona for unbounded, one-at- a-time streaming pipelines.

Complete Working Implementation

The script below reads the same GeoPackage layer twice — once in bulk with pyogrio, once record-by-record with Fiona — reprojects both results to EPSG:4326, and prints a timing and feature-count comparison so the trade-off is measurable on your own data:

#!/usr/bin/env python3
"""
Benchmark pyogrio (bulk Arrow read) against Fiona (record streaming) on one layer.
Usage: python compare_readers.py ./data/parcels.gpkg --layer parcels
"""
import sys
import time
import argparse
from pathlib import Path

import pyogrio
import fiona
from fiona.transform import transform_geom
from pyproj import CRS

TARGET_CRS = CRS.from_epsg(4326)   # canonical WGS84 lon/lat


def read_with_pyogrio(path: Path, layer: str) -> tuple[int, float]:
    """Bulk read the whole layer into a GeoDataFrame, then reproject.

    read_dataframe pulls Arrow record batches from GDAL and builds one
    columnar GeoDataFrame. This is the fast path but holds the full layer
    in memory, so peak RSS scales with the source size.
    """
    start = time.perf_counter()
    gdf = pyogrio.read_dataframe(path, layer=layer)
    if gdf.crs is not None and gdf.crs != TARGET_CRS:
        gdf = gdf.to_crs(TARGET_CRS)          # vectorised reprojection
    count = len(gdf)
    elapsed = time.perf_counter() - start
    return count, elapsed


def read_with_fiona(path: Path, layer: str) -> tuple[int, float]:
    """Stream the layer one feature at a time, reprojecting each geometry.

    fiona.open yields a GeoJSON-like dict per feature. Nothing is
    materialised for the whole layer, so peak memory stays flat regardless
    of feature count — at the cost of one Python dict allocation per record.
    """
    start = time.perf_counter()
    count = 0
    with fiona.open(path, layer=layer) as src:
        src_crs = CRS.from_user_input(src.crs) if src.crs else None  # e.g. EPSG:25832
        needs_reproj = src_crs is not None and src_crs != TARGET_CRS
        for feature in src:                   # constant-memory iterator
            geom = feature["geometry"]
            if needs_reproj:
                geom = transform_geom(
                    src_crs.to_string(), "EPSG:4326", geom, precision=7,
                )
            _ = geom                          # hand off to downstream consumer
            count += 1
    elapsed = time.perf_counter() - start
    return count, elapsed


def main() -> None:
    parser = argparse.ArgumentParser(
        description="Compare pyogrio bulk reads against Fiona streaming"
    )
    parser.add_argument("source", type=Path, help="Path to a Shapefile or GeoPackage")
    parser.add_argument("--layer", default=None, help="Layer name (GeoPackage only)")
    args = parser.parse_args()

    if not args.source.exists():
        print(f"source not found: {args.source}", file=sys.stderr)
        sys.exit(2)                            # usage/argument error

    layer = args.layer or pyogrio.list_layers(args.source)[0][0]

    p_count, p_time = read_with_pyogrio(args.source, layer)
    f_count, f_time = read_with_fiona(args.source, layer)

    print(f"pyogrio: {p_count:>8,} features in {p_time:6.3f}s")
    print(f"fiona:   {f_count:>8,} features in {f_time:6.3f}s")
    print(f"speedup: {f_time / p_time:5.1f}x  (pyogrio vs fiona)")

    if p_count != f_count:
        print("MISMATCH: feature counts differ", file=sys.stderr)
        sys.exit(1)                            # runtime error
    print("counts match — readers agree")
    sys.exit(0)


if __name__ == "__main__":
    main()

Step Annotations

  1. pyogrio.read_dataframe(path, layer=layer) — the single call that does all the work. GDAL fills Arrow batches and pyogrio assembles them into a GeoDataFrame in one pass, skipping the per-feature Python dict that dominates Fiona’s runtime.

  2. gdf.to_crs(TARGET_CRS) — reprojection happens on the whole geometry column at once through pyproj, so N features cost one vectorised transform rather than N individual calls. Guarding on gdf.crs != TARGET_CRS avoids a needless no-op transform.

  3. with fiona.open(path, layer=layer) as src — Fiona’s iterator yields exactly one feature dict at a time. The layer is never fully resident, which is what keeps peak memory flat; this is the property to reach for when the source is larger than RAM.

  4. transform_geom(src_crs, "EPSG:4326", geom, precision=7) — reprojects each geometry individually since there is no column to batch. precision=7 caps coordinate decimals at roughly centimetre resolution for EPSG:4326, keeping downstream output stable.

  5. pyogrio.list_layers(...)[0][0] — Shapefiles have a single implicit layer; GeoPackages can carry many. Defaulting to the first layer name lets the same script handle both formats without a required --layer flag.

  6. Exit codes — the script returns 2 for a missing source (usage error), 1 when the two readers disagree on feature count (runtime error), and 0 on success, matching the domain convention used across the batch pipeline error-handling patterns.

Named Gotcha: pyogrio Loads the Whole Layer by Default

pyogrio.read_dataframe() reads the entire layer into one GeoDataFrame. On a 5 GB GeoPackage this can push peak resident memory past available RAM and trigger an OOM kill long before your reprojection runs — the exact opposite of the constant-memory behaviour people assume they get from a “chunked” reader.

The fix is to read bounded windows instead of the whole layer. Pass skip_features and max_features to slide a fixed-size window across the source, or use pyogrio.open_arrow(..., batch_size=...) to pull record batches without materialising everything:

import pyogrio

CHUNK = 100_000
info = pyogrio.read_info("./data/parcels.gpkg", layer="parcels")
total = info["features"]

for offset in range(0, total, CHUNK):
    gdf = pyogrio.read_dataframe(
        "./data/parcels.gpkg",
        layer="parcels",
        skip_features=offset,     # window start
        max_features=CHUNK,       # window size — bounds peak memory
    ).to_crs("EPSG:4326")
    # process this window, then let it fall out of scope before the next

This gives pyogrio’s throughput with Fiona-like bounded memory. If you cannot know the record count cheaply or need strict one-at-a-time semantics, Fiona remains the simpler correct choice — it is slower per feature but its memory ceiling never moves.

Verification

Confirm the two readers agree and see the timing gap on your own file:

python compare_readers.py ./data/parcels.gpkg --layer parcels
# pyogrio:  1,204,318 features in  1.812s
# fiona:    1,204,318 features in 24.905s
# speedup:  13.7x  (pyogrio vs fiona)
# counts match — readers agree
echo "exit: $?"   # 0 means both readers returned the same feature count

A matching feature count and a printed speedup confirm both paths read the same layer correctly. If the counts differ, the most common cause is passing the wrong --layer to one reader — verify layer names with pyogrio.list_layers(path) and fiona.listlayers(path).

Performance Notes

pyogrio’s advantage scales with attribute width: the more columns a layer carries, the more the columnar Arrow transfer beats building a dict per feature. On a narrow, geometry-only layer the gap shrinks. Fiona’s cost is dominated by Python object allocation, so pushing filtering into the OGR layer with fiona.open(..., where="...") or a bounding-box bbox= reduces the number of dicts it ever builds. For pipelines that read many files, combine either reader with the strategies in Memory Management for Large GIS Datasets to keep aggregate RSS bounded.

FAQ

Is pyogrio always faster than Fiona?

For bulk reads of a full layer into a GeoDataFrame, pyogrio is typically 5 to 20 times faster because it moves columns through GDAL’s Arrow interface instead of building one Python dict per feature. For genuine record-by-record streaming where you never materialise the whole layer, the throughput gap narrows and Fiona’s constant memory becomes the deciding factor.

Does pyogrio load the entire file into memory?

By default pyogrio.read_dataframe reads the whole layer into a single in-memory GeoDataFrame, so peak memory scales with layer size. To bound it, pass skip_features and max_features to read fixed-size windows, or use pyogrio.open_arrow with a batch_size to pull record batches without materialising everything at once.

When should I still choose Fiona?

Choose Fiona when you need true streaming over a layer larger than RAM, when you process one feature at a time and discard it, or when you depend on its stable per-feature GeoJSON-like mapping for schema introspection. Fiona is slower per feature but holds memory constant regardless of layer size.

Do pyogrio and Fiona return the same feature count and geometry?

Yes, both wrap the same GDAL/OGR drivers, so a correctly written read returns identical feature counts and identical geometries for the same source layer. Verify by comparing len(gdf) against the Fiona iteration count and by reprojecting both to EPSG:4326 before comparison.