Managing GDAL and PROJ Environment Variables Across Shells

Your GDAL and PROJ variables differ between shells because each shell sources a different startup file, and a Docker or CI environment sources none of them. The fix is to stop hardcoding paths in .bashrc and instead detect the correct directories at runtime with pyproj.datadir.get_data_dir() and rasterio, then set os.environ defaults before osgeo.gdal initialises. This page is part of the Environment Variable Sync for Python GIS CLI Tools guide within the broader CLI Architecture & Design Patterns reference.

Prerequisites

• Python 3.10 or later
• pip install pyproj rasterio (both bundle a matching PROJ/GDAL build)
• A system or conda GDAL 3.4+ install; the helper works with either

If your paths already live in a config file, pair this with Configuration File Management so the detected values become documented defaults rather than hidden shell state.

What Each Variable Controls

Five variables decide whether a geospatial CLI behaves identically everywhere. When any of them drift between shells, the tool produces different numbers from the same input:

• GDAL_DATA points at GDAL’s support files: EPSG CSV tables, driver templates, and coordinate-system definitions. If it is wrong, gdal.Warp fails to look up authority codes.
• PROJ_DATA (formerly PROJ_LIB) points at the directory holding proj.db, the SQLite database of CRS definitions and datum-shift grids. A wrong value is the direct cause of PROJ: proj_create: Cannot find proj.db.
• GDAL_CACHEMAX sizes the raster block cache. A bare 512 means megabytes on modern builds but percent-of-RAM on older ones, so always write 512MB.
• GDAL_NUM_THREADS caps GDAL’s internal worker threads. Left at ALL_CPUS, it oversubscribes the CPU when a CLI already runs its own worker pool.
• CPL_DEBUG toggles verbose driver and VSI logging. Setting it to ON prints exactly which proj.db and data directory the library resolved, which is the fastest way to diagnose a mismatch.

A mismatch on PROJ_DATA is the most damaging: if two shells resolve two different proj.db files, a transform from EPSG:4326 to EPSG:32633 can apply a different datum-shift grid on each machine, so identical inputs yield coordinates that disagree by metres. The library rarely errors; it silently returns the wrong answer.

Complete Working Implementation

The module below is the centerpiece. It detects the correct data directories from the installed Python packages, sets os.environ defaults before GDAL is imported, and can emit the equivalent shell exports. Save it as geo_env.py and import it as the first line of your CLI entry point:

"""geo_env.py — detect and pin GDAL/PROJ paths before GDAL initialises.

Import this module BEFORE any `from osgeo import gdal` or `import rasterio`,
so os.environ is authoritative when the C libraries read their config.
"""
from __future__ import annotations

import os
from pathlib import Path


def _detect_proj_data() -> Path | None:
    """Return the directory that actually contains proj.db.

    pyproj bundles a matching PROJ build and knows where its data lives,
    so this is the single most reliable source of the correct path.
    """
    try:
        from pyproj.datadir import get_data_dir
    except ImportError:
        return None
    candidate = Path(get_data_dir())
    return candidate if (candidate / "proj.db").is_file() else candidate


def _detect_gdal_data() -> Path | None:
    """Return GDAL's support-file directory as rasterio resolves it."""
    try:
        from rasterio._env import GDALDataFinder
    except ImportError:
        return None
    found = GDALDataFinder().search()
    return Path(found) if found else None


def sync_geo_env(cache_mb: int = 512, num_threads: int = 1) -> dict[str, str]:
    """Set GDAL/PROJ env defaults from detected paths. Returns what was set.

    Uses setdefault so an operator-provided value (from a flag, .env file,
    or CI secret) always wins over auto-detection.
    """
    resolved: dict[str, str] = {}

    proj_dir = _detect_proj_data()
    if proj_dir is not None:
        # PROJ 9 reads PROJ_DATA; PROJ_LIB is the legacy alias. Set both so
        # mixed fleets (PROJ 6-8 and PROJ 9) all find proj.db.
        for key in ("PROJ_DATA", "PROJ_LIB"):
            os.environ.setdefault(key, str(proj_dir))
            resolved[key] = os.environ[key]

    gdal_dir = _detect_gdal_data()
    if gdal_dir is not None:
        os.environ.setdefault("GDAL_DATA", str(gdal_dir))
        resolved["GDAL_DATA"] = os.environ["GDAL_DATA"]

    # Always carry an explicit unit — a bare "512" is MB on new builds but
    # percent-of-RAM on old ones. "512MB" is unambiguous everywhere.
    os.environ.setdefault("GDAL_CACHEMAX", f"{cache_mb}MB")
    os.environ.setdefault("GDAL_NUM_THREADS", str(num_threads))
    resolved["GDAL_CACHEMAX"] = os.environ["GDAL_CACHEMAX"]
    resolved["GDAL_NUM_THREADS"] = os.environ["GDAL_NUM_THREADS"]
    return resolved


def as_shell_exports(resolved: dict[str, str]) -> str:
    """Render the resolved env as bash/zsh `export` lines for `eval`."""
    return "\n".join(f'export {k}="{v}"' for k, v in resolved.items())


if __name__ == "__main__":
    # `eval "$(python geo_env.py)"` pins the same paths in an interactive shell.
    print(as_shell_exports(sync_geo_env()))

The equivalent shell snippet, for a .bashrc/.zshrc or a Docker ENV layer, derives the paths from the same Python source of truth rather than hardcoding a guess:

# Pin GDAL/PROJ paths in any POSIX shell from the installed Python packages.
# Works in bash and zsh; drop into a Dockerfile RUN or a CI setup step.
eval "$(python -m geo_env)"

# Or resolve directly without the module, e.g. in a minimal container:
export PROJ_DATA="$(python -c 'import pyproj.datadir as d; print(d.get_data_dir())')"
export PROJ_LIB="$PROJ_DATA"
export GDAL_DATA="$(python -c 'from rasterio._env import GDALDataFinder as G; print(G().search())')"
export GDAL_CACHEMAX="512MB"
export GDAL_NUM_THREADS="1"

Step Annotations

1. _detect_proj_data() reads pyproj first — pyproj ships with a PROJ build and exposes get_data_dir(), which returns the directory PROJ itself would use. This is more reliable than probing filesystem locations because it reflects the exact package that will run the transforms.

2. _detect_gdal_data() uses rasterio’s finder — rasterio’s GDALDataFinder().search() walks the same resolution order GDAL uses internally, so the returned directory matches the driver that your CLI actually loads, whether it came from conda or the system package.

3. setdefault, not assignment — every write uses os.environ.setdefault so an explicitly provided value always wins. An operator can still override any path with a --gdal-data flag or an entry loaded from a .env file, keeping detection as the fallback rather than a hard override.

4. Both PROJ_DATA and PROJ_LIB — PROJ 9 reads PROJ_DATA and treats PROJ_LIB as a deprecated alias. Writing both means a machine still on PROJ 6–8 and a machine on PROJ 9 resolve the identical proj.db, which is the whole point of syncing across environments.

5. GDAL_CACHEMAX carries a unit — the helper writes 512MB, never a bare number, so the raster block cache is sized identically on every build and you avoid the percent-versus-megabytes ambiguity that silently changes memory behaviour.

6. Import order is load-bearing — sync_geo_env() must run before from osgeo import gdal. GDAL reads these variables when its driver registry initialises on first use; setting them afterward has no effect for the current process.

Named Gotcha: conda and system GDAL put proj.db in different places

The single most common cause of PROJ: proj_create: Cannot find proj.db on a machine that “worked yesterday” is two GDAL builds fighting over the path. A conda or mamba environment installs proj.db under $CONDA_PREFIX/share/proj, while a system apt/yum GDAL installs it under /usr/share/proj. If your .bashrc hardcodes export PROJ_LIB=/usr/share/proj but you launch the CLI inside an activated conda environment running PROJ 9, the library loads the conda binary yet is pointed at the system database — a version mismatch that either fails outright or applies the wrong datum-shift grid.

The fix is exactly what _detect_proj_data() does: never hardcode the directory. Let pyproj.datadir.get_data_dir() report the path for the PROJ build that is actually loaded, so activating or deactivating conda automatically moves the path with it. If you must keep a static export for a container, generate it from the same command at build time rather than copying a literal path between machines.

Verification

Confirm every environment resolves the same database and that a known transform is correct. Run this in each shell — bash, zsh, the Docker container, and the CI job — and compare:

# 1. Print the paths Python actually resolved
python -c 'import geo_env, json; print(json.dumps(geo_env.sync_geo_env(), indent=2))'

# 2. Confirm PROJ finds proj.db and can describe a CRS
projinfo EPSG:4326 | head -n 3

# 3. Confirm a datum-shifted transform returns identical numbers everywhere
python - <<'PY'
from pyproj import Transformer
t = Transformer.from_crs("EPSG:4326", "EPSG:32633", always_xy=True)
print("%.4f %.4f" % t.transform(15.0, 50.0))   # deterministic across machines
PY

If projinfo EPSG:4326 prints a CRS block instead of a “Cannot find proj.db” error, and the transform prints the same two numbers in every environment, the sync is correct. To see which files the C library chose, run any command with CPL_DEBUG=ON and grep for PROJ and GDAL_DATA in the output. For turning this into a build-time check, wire the same commands into your Packaging & CI/CD pipeline so a drifted image fails the job before release.

FAQ

Related

• Environment Variable Sync for Python GIS CLI Tools — parent guide covering how GDAL, PROJ, and CLI settings stay consistent across shells, containers, and CI
• Loading .env Files in a Geospatial CLI — layer operator-supplied overrides on top of the detected GDAL and PROJ defaults