A Job ID Is Not a Result: Check the Output Before Marking It Done

The acknowledgment is not the result

You submit a training or evaluation job to a cluster scheduler and get back a job ID. That ID tells you one thing: the scheduler accepted the job. The job can still wait in the queue, start, run out of memory, be preempted, produce a loss of NaN, or finish and write its checkpoint somewhere the evaluation step never looks. An evaluation harness can crash after writing half its results file, and the half-file still parses.

Each of those is a different fact, and "the job was accepted" establishes none of them. The number you report in a paper table or a work sample must come from an observed artifact: the results file, with the expected number of rows, from the checkpoint you meant to evaluate.

The mistake is easy to build into automation. A script submits the job, gets an ID, and writes "done" to a tracker. Everything downstream treats that marker as proof: no retry gets scheduled, no alert fires, the dashboard shows a green check. This site made the same mistake with a deploy hook, as the case below shows.

A five-state model for the lifecycle

Name the states an attempt can be in, rather than collapsing everything into "done" or "not done." The right-hand column gives the deploy equivalent.

Written this way, the mistake is visible: a marker written on the job ID claims "verified" when it has only confirmed "requested." A run that crashed halfway looks, from the marker's point of view, identical to one that finished.

View full-size diagram

The arrow from "Marker: requested" to "Read results file" is the step that gets skipped.

Checking for the result

The check reads the artifact and marks the run complete only if it is the right one. "The right one" must be specific enough that a stale checkpoint or a half-written file cannot slip through. Below is a small, self-contained Python example that models the check as a decision function. It is a teaching simulation with no file system access, and every input is a labeled hypothetical.

from dataclasses import dataclass

@dataclass
class Observed:
    file_exists: bool
    checkpoint: str | None
    rows: int

def confirm_result(expected_checkpoint: str, expected_rows: int, seen: Observed) -> str:
    """Return the run's state given one look at its results file.

    This models a single check. In practice you would retry a read that
    failed on a network or storage error before concluding 'failed'.
    """
    if not seen.file_exists:
        return "failed"
    if seen.checkpoint != expected_checkpoint:
        return "failed"
    if seen.rows != expected_rows:
        return "failed"
    return "verified"

# Hypothetical case: the scheduler said "done", but the run evaluated an old checkpoint.
stale = Observed(file_exists=True, checkpoint="ckpt-041", rows=500)
print(confirm_result("ckpt-042", 500, stale))  # -> "failed"

# Hypothetical case: right checkpoint, but the harness crashed partway and the half-file parses.
partial = Observed(file_exists=True, checkpoint="ckpt-042", rows=212)
print(confirm_result("ckpt-042", 500, partial))  # -> "failed"

# Hypothetical case: everything matches.
complete = Observed(file_exists=True, checkpoint="ckpt-042", rows=500)
print(confirm_result("ckpt-042", 500, complete))  # -> "verified"

Run on Python 3.12, it prints failed, failed, verified. The first two cases would have passed an acknowledgment-only check, because the job ID comes back before either problem can be observed. The check asks a different question: not "did the scheduler accept my request" but "does the result I wanted exist where the next step will read it." The inputs are hardcoded, so a passing run proves only that the branching logic is right. It proves nothing about a real scheduler, a real file system or two overlapping submissions.

Retrying without duplicating

The two failure directions are worth putting side by side. A scheduler with no marker resubmits finished work on every pass. A scheduler that marks on acknowledgment silently strands a run that failed. The fix sits between them: mark only on a verified result, and leave everything else eligible for retry.

Retries have a cost you can compute. Chapter 12 of the book derives it: if each attempt loses the request with probability a, and loses the reply after the effect happened with probability b, then retrying blindly until a reply arrives creates on average

extra effects per task = b / (1 − a − b)
a = 0.1, b = 0.2  ->  0.2 / 0.7 = 0.286

Only b creates duplicates. A retry after a request that never arrived is harmless; a retry after a lost reply repeats the effect. That is why a job resubmitted after a timeout can leave two runs writing to the same output folder, and why the book gives each intended effect a stable key.

Case: the deploy hook behind this site

View full-size diagram

Testing the states that matter

Three scenarios expose whether a pipeline tells "requested" apart from "verified." First, a missed run: the scheduler is down for a day, so an eligible job is now overdue. It should still run and be verified, not be skipped. Second, duplicate submissions: two scheduler passes overlap, both see no marker, and both submit. A concurrency guard on the scheduler (the site's workflow used cancel-in-progress: false, grouped by scheduled-publication-deploy) covers one queue, not the system doing the work. Third, the case this essay centers on: the job is accepted and the result is bad. An acknowledgment-only pipeline marks it done and never retries. A pipeline that reads the result sees the failure and tries again.

These scenarios are an experiment design, not a result. This site has not run this yet: they need a controlled rehearsal, for example with the workflow_dispatch input the historical file already had (now_utc, for simulating a UTC time), and until someone runs one and records its output, they remain planned tests.

Check the decision

Why this matters in a lab

Most of the infrastructure an applied-AI or research engineer touches is asynchronous: job queues, batch inference, evaluation harnesses, deploys of a serving stack. Each hands back an acknowledgment long before the effect you care about exists. Name the observed result that means "done," check for it, and mark nothing complete on an acknowledgment. The research engineering course treats this as part of running experiments at scale, and it is daily work for an ML infrastructure engineer.

An interviewer could fairly ask: your nightly evaluation job reported success, but the dashboard shows yesterday's numbers. Where do you look? A good answer walks the states in the table above: was the job accepted, did it run, did it write a results file, is it the file the dashboard reads, and does that file carry the checkpoint you expected.

What would prove this analysis wrong

If your scheduler or hosting provider documents that its submit call blocks until the work is complete and reports the final status of the artifact itself, the gap between acknowledgment and result would not apply to that call. Verify that contract in the provider's documentation rather than assuming it from this essay. For the site case, the argument rests on reading the historical workflow: the marker step ran on hook success, with no live fetch in between. A later version that adds a verification step before the marker closes that specific gap.