Article

Streaming Batch Progress to a CI Log

Detect the non-interactive case, disable the live layout, and emit one complete line per completed unit plus a periodic rollup. A build log then shows a batch that is visibly alive without megabytes of cursor-movement escape sequences, and a failure can be located by scrolling rather than by rerunning. It builds on the Progress Tracking in Batch Jobs guide, part of the broader Spatial Batch Processing & Async Workflows reference.

Prerequisites

What a Build Log Actually Needs

Four requirements a live bar does not meet CI systems kill a job that produces no output for some minutes, so the stream must prove liveness. A failure must be locatable by scrolling rather than by rerunning. Total log size must stay bounded. A summary at the end must be readable without scrolling back. liveness output at least every minute or the job is killed a heartbeat line locatable failures the failing input named on its own line one line per failure bounded size a repainting bar produces megabytes of escapes no cursor movement a scannable summary counts and timings at the end, not the middle one final block

The first row is the one that turns a working pipeline into a mysteriously failing build. A long warp that prints nothing for twenty minutes is indistinguishable, to the CI system, from a hang.

Complete Working Implementation

# gistools/ci_progress.py
from __future__ import annotations

import os
import sys
import time
from dataclasses import dataclass, field


def is_interactive() -> bool:
    if os.environ.get("CI"):
        return False
    if os.environ.get("FORCE_TTY"):
        return True
    return sys.stderr.isatty()


@dataclass
class CiProgress:
    """Append-only progress for a captured stream."""
    total: int
    every: int = 100                 # a rollup line every N units
    heartbeat_s: float = 30.0        # never go longer than this without output
    done: int = 0
    failed: int = 0
    _started: float = field(default_factory=time.monotonic)
    _last_line: float = field(default_factory=time.monotonic)
    _durations: list[float] = field(default_factory=list)

    def _emit(self, text: str) -> None:
        print(text, file=sys.stderr, flush=True)
        self._last_line = time.monotonic()

    def start(self) -> None:
        self._emit(f"[batch] starting: {self.total} units")

    def unit_done(self, key: str, seconds: float) -> None:
        self.done += 1
        self._durations.append(seconds)
        if self.done % self.every == 0:
            self._rollup()
        elif time.monotonic() - self._last_line > self.heartbeat_s:
            self._rollup()

    def unit_failed(self, key: str, reason: str) -> None:
        self.failed += 1
        self._emit(f"[batch] FAILED {key}: {reason}")

    def _rollup(self) -> None:
        elapsed = time.monotonic() - self._started
        rate = self.done / elapsed * 60 if elapsed else 0.0
        remaining = self.total - self.done
        eta_min = remaining / rate if rate else float("inf")
        pct = self.done / self.total * 100 if self.total else 0.0
        self._emit(
            f"[batch] {self.done}/{self.total} ({pct:.1f}%) "
            f"{rate:.0f}/min  failed={self.failed}  eta={eta_min:.0f}m"
        )

    def finish(self) -> int:
        elapsed = time.monotonic() - self._started
        ordered = sorted(self._durations)
        p95 = ordered[min(int(0.95 * len(ordered)), len(ordered) - 1)] if ordered else 0.0
        self._emit("[batch] " + "-" * 48)
        self._emit(f"[batch] finished {self.done}/{self.total} in {elapsed/60:.1f}m")
        self._emit(f"[batch] failed: {self.failed}")
        self._emit(f"[batch] p95 per unit: {p95*1000:.0f} ms")
        return 0 if self.failed == 0 else 1

Choosing between the two renderers at the entry point:

def make_reporter(total: int):
    if is_interactive():
        from gistools.rich_progress import LiveProgress
        return LiveProgress(total)
    return CiProgress(total)

Step Annotations

  1. flush=True on every line. Python buffers a non-terminal stream aggressively, and without the flush a CI log shows nothing for minutes and then everything at once — which defeats the heartbeat entirely.
  2. The heartbeat is time-based, not count-based. A batch of very slow units would otherwise go silent between rollups, which is exactly the case the idle timeout catches.
  3. Failures get their own line immediately. They are what someone scrolls to find, and burying a failure in a rollup makes it invisible.
  4. Output goes to stderr. stdout belongs to the data, which is the separation Structured Logging for Geospatial CLIs sets out.
  5. finish returns an exit code. The summary and the exit status are decided in one place, so a batch that reports failures cannot accidentally exit zero.

Sizing the Rollup Interval

How often to print for a ten-thousand-unit batch A line per unit produces ten thousand lines and buries the failures. Every hundred units gives a hundred lines, which is scannable and still shows a stall within a minute. Every thousand gives ten lines and may exceed the idle timeout between them. every unit 10,000 lines failures are buried in the noise too chatty every 100 units 100 lines plus failures a stall is visible within a minute the useful default every 1,000 units 10 lines may exceed the CI idle timeout needs a heartbeat

Combining a count trigger with a time trigger — as the implementation above does — gets the benefits of the middle column regardless of how fast the units happen to be.

One Named Gotcha: Rich Still Repaints Under CI=true

Setting CI does not by itself stop Rich from emitting control sequences; the Console has to be told. A Progress created without force_terminal=False will detect the captured stream, fall back to a simplified renderer, and still emit periodic redraws that fill a build log.

from rich.console import Console
from rich.progress import Progress

console = Console(stderr=True, force_terminal=False, no_color=True, width=100)
with Progress(console=console, disable=not is_interactive()) as progress:
    ...

disable=True is the setting that matters: it turns the progress object into a no-op rather than a quieter renderer, which is what leaves the field clear for the append-only stream.

Verification

# Simulate CI and count the lines and the escape sequences.
CI=true gis raster warp-batch inputs/ out/ --dst-crs EPSG:3857 2>ci.log
wc -l ci.log
grep -c $'\x1b' ci.log        # expect 0 — no escape sequences at all

# Confirm the heartbeat: no gap longer than the idle timeout.
awk '{print $1}' ci.log | uniq -c | head

Zero escape sequences and a line at least every thirty seconds is the signature you want. A log with control characters will render as garbage in most CI viewers even though it looked fine locally.

The same detection decides more than the progress renderer, and keeping the decision in one place stops the outputs disagreeing.

One decision, four behaviours Progress becomes append-only lines. Colour is stripped. Width is fixed rather than detected. The summary becomes a block of plain lines rather than a rendered panel. Deciding each separately produces output where half is styled and half is not. progress rendering a live bar becomes append-only lines the visible one colour stripped, so a log file stays greppable NO_COLOR too terminal width fixed at a readable value rather than detected 100 columns the summary a plain block rather than a rendered panel still scannable

Threading one decision through all four is the reason to compute it once at the entry point and pass it down, rather than calling isatty wherever output happens.

Making the Summary Actionable

The final block is the part most people read, and a summary that reports only counts leaves the reader to work out whether the run was acceptable.

Three additions make it answer that directly. A failure list — the first few failing inputs by name, not just a count — turns seven failed into something someone can start on. A comparison against the previous run, if one is available, distinguishes a normal failure rate from a new problem. And an explicit verdict line, stating whether the run met whatever threshold the pipeline defines, removes the judgement call from the reader.

[batch] finished 9,993/10,000 in 47.2m
[batch] failed: 7 (0.07%) — threshold 1.0% — OK
[batch] first failures: tiles/34/12/9.tif, tiles/34/13/2.tif, tiles/35/01/7.tif

That third line is what turns a log into a starting point. Without it the next step is a search through forty thousand lines for the word FAILED, which is exactly what a summary exists to avoid.

Exit Codes Matter More Here

In an interactive session a user sees the summary and judges it. In CI nothing reads the summary — the build system branches on the exit code alone, so the code has to carry the verdict.

The scheme that works is the one the section recommends generally: zero when the run met its threshold, a distinct non-zero code when it completed but exceeded the failure threshold, and another when it could not complete at all. A build that distinguishes those three can alert differently on each, which is the difference between a useful notification and one everybody mutes.

Returning the code from the same function that prints the summary, as the implementation above does, keeps the two from disagreeing — which they will if the verdict is computed in one place and the exit in another.

Progress in a Distributed Batch

Everything above assumes one process producing the stream. A distributed batch has several, and the arrangement that works is different in one respect: no worker should attempt to render aggregate progress, because none of them knows the total.

Workers emit one line per completed unit into their own stream, carrying the batch identifier. The aggregate — counts, rate, ETA — is produced by whatever submitted the work and can see the queue, either by polling queue depth or by counting completion records. That separation keeps the workers simple and puts the summary where someone is watching.

The CI case then usually has only the submitting process in the build log at all, with the workers elsewhere. That is the right shape: a build log showing ten thousand lines from eight interleaved workers is unreadable, while one showing a rollup every thirty seconds and a failure line per failed unit is exactly what a person needs.

The one thing to keep from the single-process design is the heartbeat. A submitter waiting on a queue produces no output naturally, and a CI system will kill it for idleness long before the batch finishes.

Choosing What Counts as an Event

A log line per completed unit is the obvious design and is frequently the wrong granularity. Two alternatives suit different batches.

For a batch of uniform, fast units, a line per unit is noise: ten thousand near-identical lines that nobody reads and that push the interesting output out of the visible window. A rollup every hundred, plus a line per failure, carries the same information in a hundredth of the space.

For a batch of slow, distinct units — a handful of large mosaics, say — a line per unit is exactly right, because each one is an event someone cares about and the total volume is small.

The rule that generalises: emit a line whenever something happened that a reader would want to know about, and a rollup on a timer otherwise. That produces a log whose length tracks how interesting the run was rather than how much work it did, which is the property that makes it worth reading.