Article

Writing Batch Outputs to Object Storage Atomically

Write to a temporary key nobody else uses, promote it with a server-side copy, then delete the temporary. A reader watching the final key sees either nothing or the complete object — never a truncated GeoTIFF — and a retried task cannot corrupt the output of the attempt it duplicated. It builds on the Cloud Storage I/O for Spatial Batches guide, part of the broader Spatial Batch Processing & Async Workflows reference.

Prerequisites

  • Python 3.10 or later
  • pip install rasterio boto3 (or the equivalent client for your store)
  • Write access to a bucket, and permission to copy and delete keys within it

Why a Direct Write Is Not Safe

The failure is not theoretical, and it has two distinct shapes depending on how the write is done.

Three write patterns, one of them safe Streaming directly to the final key with a virtual file system writer can expose a partial object to readers. A multipart upload that fails leaves uncompleted parts that are billed but invisible. Writing to a temporary key and promoting with a server-side copy exposes nothing partial. stream to the final key simplest to write a reader mid-write may see a truncated file and a retry overwrites a good object with a bad one multipart, uncompleted nothing partial is visible but the parts persist invisibly, and are billed needs a lifecycle rule to clean them up temporary key, then copy the final key appears complete or not at all the copy is server-side — no bytes cross your process safe under retry

The middle column is the one that surprises people: a failed upload leaves nothing visible in a listing and still accrues storage charges, sometimes for years, until someone adds a lifecycle rule to abort incomplete uploads.

Complete Working Implementation

# gistools/cloud_write.py
from __future__ import annotations

import os
import tempfile
import uuid
from contextlib import contextmanager
from pathlib import Path

import boto3
import rasterio

s3 = boto3.client("s3")


@contextmanager
def local_staging(suffix: str = ".tif"):
    """A local path to write into, removed whether or not the block succeeds."""
    fd, path = tempfile.mkstemp(suffix=suffix)
    os.close(fd)
    path = Path(path)
    try:
        yield path
    finally:
        path.unlink(missing_ok=True)


def publish(bucket: str, key: str, data, profile) -> str:
    """Write a raster and make it visible at `key` only when complete."""
    tmp_key = f"{key}.{uuid.uuid4().hex}.tmp"

    with local_staging() as staged:
        with rasterio.open(staged, "w", **profile) as dst:
            dst.write(data)
        s3.upload_file(str(staged), bucket, tmp_key)

    # Server-side copy: the bytes never come back through this process.
    s3.copy_object(Bucket=bucket, Key=key,
                   CopySource={"Bucket": bucket, "Key": tmp_key})
    s3.delete_object(Bucket=bucket, Key=tmp_key)
    return f"s3://{bucket}/{key}"

Staging locally rather than writing straight to the temporary key matters for GeoTIFF specifically: the driver seeks backwards to update the header after writing the tiles, and seeking is exactly what an object store does not support.

Step Annotations

  1. A random suffix, not the task name. Two concurrent attempts at the same output must not share a temporary key, or one will overwrite the other’s partial upload and the copy will promote a corrupt object.
  2. local_staging cleans up in finally. A crash between the write and the upload otherwise leaves a full-size temporary file on the worker’s disk, and a batch fills the volume in an hour.
  3. copy_object rather than download-and-upload. The store does the copy internally, so the promotion costs one API call regardless of object size instead of two transfers.
  4. Delete after copy, not before. If the delete fails, a harmless temporary key is left behind; if the copy failed and the delete had already run, the data would be gone.
  5. No conditional check before the copy. Overwriting the final key is the correct behaviour for a retried unit, because both attempts produce identical bytes — which is the idempotency property described in Making Spatial Tasks Idempotent for Safe Retries.

One Named Gotcha: Copy Has a Size Limit

copy_object handles objects up to five gigabytes. Above that the call fails with an InvalidRequest mentioning the size, and the fix is a multipart copy — which is still server-side, just expressed differently.

def promote(bucket: str, tmp_key: str, key: str) -> None:
    size = s3.head_object(Bucket=bucket, Key=tmp_key)["ContentLength"]
    if size <= 5 * 1024 ** 3:
        s3.copy_object(Bucket=bucket, Key=key,
                       CopySource={"Bucket": bucket, "Key": tmp_key})
    else:
        # boto3's managed transfer picks multipart automatically.
        s3.copy({"Bucket": bucket, "Key": tmp_key}, bucket, key)
    s3.delete_object(Bucket=bucket, Key=tmp_key)

Mosaics cross five gigabytes routinely, so this branch is not an edge case for raster work. Writing it once in a helper — rather than discovering it when a large scene fails at three in the morning — is the whole point of putting the promotion behind a function.

Each stage of the publish sequence has a distinct failure, and the pattern is chosen so that none of them is visible to a reader.

What a failure at each stage leaves behind A failure while staging locally leaves a temporary file the context manager removes. A failed upload leaves an unreferenced temporary key. A failed copy leaves the temporary key and no final object. A failed delete leaves a harmless temporary key. staging locally the context manager removes the file in finally nothing left uploading to the temporary key an orphan temporary key, invisible to readers swept by lifecycle copying to the final key no final object — the unit is simply redone safe to retry deleting the temporary a harmless leftover key, already superseded cosmetic only

At no point does a reader watching the final key see a partial object, which is the property the whole sequence exists to provide.

Verification

# 1. No temporary keys survive a successful batch.
aws s3api list-objects-v2 --bucket "$BUCKET" --prefix out/ \
  --query "Contents[?ends_with(Key, '.tmp')].Key" --output text | wc -w   # expect 0

# 2. No incomplete multipart uploads are accruing charges.
aws s3api list-multipart-uploads --bucket "$BUCKET" \
  --query 'Uploads[].{Key:Key,Started:Initiated}' --output table

# 3. The published object opens and carries the CRS you meant.
python -c "
import rasterio
with rasterio.open('/vsis3/$BUCKET/out/scene.tif') as s:
    print(s.crs, s.width, s.height, s.count)
"

The second check is worth scheduling rather than running once. Incomplete uploads accumulate quietly, and a lifecycle rule that aborts them after a day removes the whole category of surprise.

Publishing More Than One Object at a Time

A raster is rarely alone. A published product is often a data object plus overviews plus a sidecar of metadata, and making the set appear atomically needs one more idea.

Making a set of objects appear together A marker object written after everything else lets a reader check one key before trusting the set. A versioned prefix with a pointer object swaps the whole set with one write. Per-object atomicity plus a reader that tolerates a partial set is simplest but pushes the problem downstream. a marker written last publish every object, then write _SUCCESS readers check the marker before using the set simple and effective versioned prefix write under out/v42/, then update a pointer one write swaps the set and rollback is trivial best for products tolerant readers each object is atomic; readers cope with gaps no coordination at all, but complexity moves pushes it downstream

The versioned-prefix arrangement is worth the extra key for anything other systems consume, because it makes rolling back a bad batch a single pointer update rather than a restore.

Concurrency and the Final Key

The pattern above is safe under retry because each attempt writes to a distinct temporary key. It is worth being precise about what it does and does not guarantee when two attempts overlap.

Both attempts upload to different temporary keys, so neither corrupts the other. Both then copy to the same final key, and the store serialises those copies — the second overwrites the first. Because both attempts produced identical bytes, the outcome is the same either way, which is the property that makes this idempotent rather than merely safe.

What it does not guarantee is that a reader between the two copies sees the same object as a reader after them. On a store with strong read-after-write consistency they are byte-identical, so it does not matter. On one with weaker guarantees a reader may briefly see the older of the two, which is still a complete, valid object — never a partial one.

The case that genuinely needs more care is two attempts producing different bytes, which happens when the computation is not deterministic. A resampling method that depends on thread scheduling, or a timestamp embedded in the output, breaks the assumption. Removing non-determinism from the write path is the fix, and embedding a timestamp is the most common cause: put it in the manifest, not in the object.

Cleaning Up What Failure Leaves

Two kinds of debris accumulate, and they need different remedies because only one is visible.

Temporary keys from failed uploads appear in a listing and can be removed by a lifecycle rule matching the suffix. A rule expiring *.tmp after one day covers every failure mode without any coordination with the pipeline, and it runs whether or not the pipeline does.

Incomplete multipart uploads do not appear in a listing at all. They accrue storage charges indefinitely and are found only by an explicit API call. The lifecycle configuration has a separate rule for aborting them, and adding it at the same time as the first is the only reliable way to remember.

Both rules are configuration rather than code, which means they belong in whatever provisions the bucket rather than in the pipeline. A bucket created by hand for a proof of concept, and then quietly promoted to production, is where this is almost always missing — and the symptom is a storage bill that grows while the visible object count does not.

Writing vector output

The same sequence applies with one change: stage the whole layer locally, then upload. A GeoPackage is a SQLite database that seeks while writing, and an object store cannot serve those seeks. Writing locally and uploading the finished file is not a workaround — it is the only arrangement that performs acceptably, and it happens to give the atomicity for free.

One further consideration for large outputs: set the multipart chunk size deliberately rather than accepting the default. A very small chunk size produces thousands of parts for a large mosaic, each of which is a request, while a very large one wastes a whole chunk’s transfer when a single part fails and has to be resent. Something in the region of sixteen to sixty-four megabytes suits raster outputs well, and it is worth setting once in the client configuration rather than per call.