Article

Benchmarking Cloud Versus Local Raster Reads

Measure both paths against the same corpus with caches in a known state, and record request count alongside elapsed time. The comparison produces one actionable number: the reuse threshold above which copying an object locally beats reading it remotely again. It builds on the Benchmarking Spatial Batch Pipelines guide, part of the broader Spatial Batch Processing & Async Workflows reference.

Prerequisites

  • Python 3.10 or later with pip install rasterio boto3
  • The same COG available both locally and on object storage
  • Compute in the same region as the bucket, or the comparison measures the distance rather than the storage

What Makes the Comparison Unfair by Default

Four differences will bias the result unless they are pinned deliberately.

Pinning the four things that skew the comparison The operating system page cache makes a repeated local read almost free. GDAL's virtual file system cache does the same for a repeated remote read. Cross-region placement adds latency unrelated to storage. The first remote request pays TLS setup that later ones do not. page cache state a repeated local read never touches the disk drop caches between runs virtual file system cache a repeated remote read never touches the network clear it between runs region placement a cross-region read adds tens of milliseconds pin the region first connection setup the first request pays TLS and DNS discard a warm-up read

Getting these wrong produces the two classic wrong conclusions: that cloud reads are catastrophically slow, measured cross-region against a warm local cache, or that they are free, measured warm against a cold disk.

Complete Working Implementation

# bench/cloud_vs_local.py
from __future__ import annotations

import json
import statistics
import subprocess
import time
from dataclasses import dataclass

import rasterio
from rasterio.env import Env
from rasterio.windows import Window

CLOUD_ENV = {
    "GDAL_DISABLE_READDIR_ON_OPEN": "EMPTY_DIR",
    "CPL_VSIL_CURL_ALLOWED_EXTENSIONS": ".tif",
    "GDAL_HTTP_MULTIPLEX": "YES",
    "GDAL_HTTP_VERSION": "2",
    "VSI_CACHE": "TRUE",
    "VSI_CACHE_SIZE": str(64 << 20),
}


@dataclass
class Reading:
    label: str
    durations: list[float]

    def summary(self) -> dict:
        ordered = sorted(self.durations)
        return {
            "label": self.label,
            "n": len(ordered),
            "median_ms": round(statistics.median(ordered) * 1000, 1),
            "p95_ms": round(ordered[min(int(0.95 * len(ordered)), len(ordered) - 1)] * 1000, 1),
        }


def _drop_page_cache() -> None:
    try:
        subprocess.run(["sync"], check=True)
        with open("/proc/sys/vm/drop_caches", "w") as handle:
            handle.write("3")
    except (OSError, PermissionError):
        pass


def _clear_vsi_cache() -> None:
    try:
        from osgeo import gdal
        gdal.VSICurlClearCache()
    except ImportError:
        pass


def time_reads(uri: str, windows: list[Window], *, cloud: bool,
               repeats: int = 5, cold: bool = True) -> Reading:
    env = CLOUD_ENV if cloud else {}
    durations: list[float] = []

    with Env(**env):
        with rasterio.open(uri) as src:      # warm-up open, discarded
            src.read(1, window=windows[0])

    for _ in range(repeats):
        if cold:
            _drop_page_cache()
            _clear_vsi_cache()
        with Env(**env):
            with rasterio.open(uri) as src:
                for window in windows:
                    started = time.perf_counter()
                    src.read(1, window=window)
                    durations.append(time.perf_counter() - started)

    return Reading("cloud" if cloud else "local", durations)


def compare(local_path: str, cloud_uri: str, windows: list[Window]) -> dict:
    results = [
        time_reads(local_path, windows, cloud=False, cold=True).summary(),
        time_reads(local_path, windows, cloud=False, cold=False).summary(),
        time_reads(cloud_uri, windows, cloud=True, cold=True).summary(),
        time_reads(cloud_uri, windows, cloud=True, cold=False).summary(),
    ]
    print(json.dumps(results, indent=2))
    return {"readings": results}

Step Annotations

  1. The warm-up open is outside the timing loop. It absorbs TLS negotiation, DNS and driver registration, none of which recur and all of which would dominate a small sample.
  2. Both caches are cleared for the cold case. Clearing only one produces a hybrid nobody experiences in production.
  3. Windows are supplied by the caller. Their size and placement dominate the result, and hiding them inside the harness makes two benchmarks look comparable when they are not.
  4. Percentiles rather than a mean, for the same reason as elsewhere. Remote reads have a much longer tail than local ones, and the mean conceals exactly the difference you are measuring.
  5. VSICurlClearCache is guarded by an import check. It lives in the osgeo bindings, which a rasterio-only environment may not expose.

One Named Gotcha: Cold Numbers Are Not Reproducible on a Shared Machine

Dropping the page cache affects the whole host. On a shared build machine another tenant refills it between your two runs, and the cold measurement drifts by a factor of two for reasons invisible to you. Worse, in a container the write to drop_caches usually fails silently, so what you record as cold is warm.

Two mitigations, in order of preference: run the comparison on a dedicated host, or drop the cold case entirely and compare warm against warm while stating that clearly. A warm-only comparison is still useful — it isolates the network from the disk — and it is honest, which a mislabelled cold number is not.

Deriving the Copy-First Threshold

The point of the comparison is a decision, not a table. Given a per-read cost for each path and a one-off copy cost, the threshold is where repeated remote reads overtake copying once.

Turning the measurements into a rule Reading each object once or twice is cheaper remotely, because the copy pays for bytes never used. Around three reads the two are comparable. Beyond that, copying once and reading locally wins on both time and request charges. 1–2 reads per object the copy pays for bytes you never read remote reads win on both time and cost read remotely about 3 reads the two are within noise of each other choose on operational grounds, not speed borderline 4 or more reads the copy amortises local reads are faster and free of request charges thereafter copy first

The exact crossover moves with object size and window size, which is why measuring it against your own corpus beats adopting someone else’s number. What does not move is the shape: there is always a threshold, and a pipeline that reads each input several times is always on the wrong side of it.

Verification

python -m bench.cloud_vs_local --local scenes/a.tif \
       --cloud /vsis3/bucket/scenes/a.tif --window 0,0,512,512 --repeats 5

Sanity checks on the output: the warm local median should be the smallest number by a wide margin; the cold cloud p95 should be the largest; and the warm cloud median should sit between them. Any other ordering means something in the setup is not doing what it claims — usually a cache that was not cleared, or a bucket in another region.

Four variables move the crossover, and knowing which ones apply to your pipeline is most of the analysis.

What moves the threshold A larger object makes copying more expensive and pushes the threshold up. A larger window means each remote read transfers more, pushing it down. More reuse pushes it down. A cross-region bucket pushes it down sharply, because every request pays the latency. larger objects copying costs more, so read remotely for longer threshold rises larger windows each remote read transfers more of the object threshold falls more reuse the copy amortises over more reads threshold falls a cross-region bucket every request pays the latency threshold falls sharply

The last row usually dominates everything else, which is why checking region placement comes before any other measurement.

What the Numbers Do Not Capture

A time-and-request comparison is the right measurement and it leaves out three things that belong in the decision.

Storage cost differs. A local copy occupies disk for the life of the batch, and on a large corpus that is a volume someone provisions. Reading remotely occupies nothing. For a one-off batch the difference is negligible; for a nightly job over a growing archive it compounds.

Failure modes differ. A local read fails when the disk fails, which is rare and total. A remote read fails transiently and often, which is why the retry classification in Classifying Recoverable and Fatal GDAL Errors matters much more for the remote path. A pipeline that reads remotely without retry handling is faster in a benchmark and less reliable in production.

Freshness differs. A copy is a snapshot; a remote read sees whatever is there now. For an archive that never changes, that is irrelevant. For a bucket someone is still writing to, a copy can silently process yesterday’s version — which is a correctness difference rather than a performance one, and it outweighs any of the numbers above.

Applying the Result

The output of this comparison is a threshold, and the threshold is only useful if something acts on it. Two arrangements work.

The simplest is a flag: --cache-inputs copies each object locally before processing and is turned on for the pipelines whose reuse count is above the threshold. It puts the decision in the operator’s hands, documented by the measurement.

The more satisfying is automatic: the pipeline counts how many times each input appears in its work list, and copies the ones above the threshold while streaming the rest. It costs a pass over the work list and removes the decision entirely, which matters when the same pipeline runs over different work lists with different reuse patterns.

Either way, record which path each unit took in the structured log. Without it, a later question about why one run was slower than another has an invisible variable in it, and the answer is unreachable.

It is also worth recording the object size and window size in the result file alongside the timings. Both move the threshold, and a comparison from six months ago against tiles half the size is not a comparison at all. Storing them turns a number into a data point that can be plotted, which is what lets a team see the threshold shift as their data grows rather than rediscovering it each year.

Reporting the comparison

The result is most useful stated as a rule rather than a table. Below three reads per object, stream; above, copy is something a colleague can apply; a grid of medians and percentiles is something they have to interpret.

Keep the table in the result file for the record, and put the rule in the pipeline’s documentation next to the flag that implements it. Six months later the table will be stale and the rule will still be roughly right, because the shape of the trade-off changes far more slowly than the absolute numbers do.

One More Variable: Object Size Versus Window Size

The threshold this comparison produces assumes a fixed relationship between the object and the part of it you read, and that ratio is worth stating explicitly whenever the number is quoted.

Reading a two-hundred-pixel window from a scene of ten thousand pixels transfers a fraction of a percent of the object, so streaming is overwhelmingly cheaper and the threshold sits high. Reading a window covering half the scene transfers half the object, and two such reads have already cost more than a copy would have. The same pipeline can sit on either side of the line depending on which product it is generating that night.

The practical response is to record the ratio in the result rather than assuming it. A result file carrying object size, window size and reuse count describes the situation completely, and a comparison against it six months later can tell whether the conclusion still applies or whether the data has moved out from under it.