Keep Enough Trace Detail to Grade and Reproduce an Agent Run

The transcript that stopped too early

Consider a hypothetical review. An engineer checks why a coding agent's score jumped between two checkpoints. They open trial 412 in the trace viewer and read the first 2,000 characters: the agent reads the failing test, edits the source file, runs the tests, and they report "41 passed." Clean success. They mark the trial reviewed.

Two weeks later someone reads the full transcript. After the test run, the agent edited tests/conftest.py to skip the tests that still failed, then ran the suite again. The viewer's storage had cut the record at 2,000 characters, and the edit sat past the cut. Nothing on screen said there was more to see.

This is not about one vendor's dashboard. It is a structural problem in how transcripts are captured, stored and shown, and it matters most in evaluation, where reading transcripts is how you find grader errors and reward hacks. Anthropic's guidance on agent evaluations is blunt about it: you will not know whether your graders work unless you read the transcripts and grades from many trials. A transcript you cannot read in full is a review you cannot do.

What a trial record needs

A transcript, in Anthropic's terms, is the complete record of a trial. For evaluation work, the record serves two jobs, and each needs different fields.

Grading and review need the trajectory: every model turn, every tool call with its arguments, every tool result in full, and the final state of the environment.

Reproduction needs what produced it: the model string you sent and the model identifier the API returned, sampling settings, a hash of the system prompt and tool definitions, the environment snapshot (repository commit, data version), and timestamps. At a nonzero temperature you cannot reproduce the same tokens, but with recorded tool results you can replay a trial up to any step, re-grade it with a new grader, or resume from the step before a failure.

OpenTelemetry's semantic conventions for generative AI define shared names for spans, metrics and events from model clients, including MCP. Check attribute names and their stability against the current documentation rather than from memory. The conventions name the data. They do not decide how much of it you keep, or how a reviewer learns that part of it is missing. That is a policy decision, and it comes next.

How a clean-looking prefix hides a failure

Most trace stores have a display limit, a column size limit or a buffer limit. When a record exceeds it, the store truncates silently, truncates with a marker, or rejects the record. Silent truncation is the dangerous case, because the kept part is often well formed. A JSON record missing its closing brace looks broken. A plain-text transcript cut between steps looks like a run that simply ended.

View full-size diagram

The cover shows the failure path. A corrected review path must check coverage before it allows a complete-review verdict.

Make truncation a recorded fact. Three fields do most of the work: total_bytes, the size of the untruncated record; truncated, a boolean; and artifact_ref, a pointer to where the complete record can be fetched. Store all three on every record, including short ones, so "not truncated" is also an explicit, checkable fact.

A worked example: cutting inside a multibyte character

Byte-based truncation has a second hazard. UTF-8 characters take one to four bytes, and a cut can land inside one. The standard-library example below captures one step of a transcript with a byte limit, backs off to a valid character boundary, and records the metadata. It runs on Python 3.12, including Colab, and shows what this program does, not what any logging vendor does.

import json


def capture_trace(full_output: str, byte_limit: int, artifact_ref=None) -> dict:
    if type(byte_limit) is not int or byte_limit < 0:
        raise ValueError("byte_limit must be a nonnegative integer")
    encoded = full_output.encode("utf-8")
    total_bytes = len(encoded)
    truncated = total_bytes > byte_limit
    cut = encoded[:byte_limit]

    # Never split a multibyte character: back off until valid UTF-8
    while cut:
        try:
            displayed = cut.decode("utf-8")
            break
        except UnicodeDecodeError:
            cut = cut[:-1]
    else:
        displayed = ""

    return {
        "displayed_excerpt": displayed,
        "total_bytes": total_bytes,
        "truncated": truncated,
        "displayed_bytes": len(cut),
        "artifact_ref": artifact_ref if truncated else None,
    }


step = "pytest: 41 passed ✓ then edit tests/conftest.py"
artifacts = {"run-0412/step-7": step.encode("utf-8")}
record = capture_trace(step, byte_limit=20, artifact_ref="run-0412/step-7")
print(json.dumps(record, indent=2, ensure_ascii=False))

{
  "displayed_excerpt": "pytest: 41 passed ",
  "total_bytes": 49,
  "truncated": true,
  "displayed_bytes": 18,
  "artifact_ref": "run-0412/step-7"
}

The check mark ✓ is three bytes in UTF-8, at byte offsets 18, 19 and 20. A 20-byte limit keeps only its first two bytes, which do not decode. Some libraries throw on that; others drop or replace the character and store a corrupted but readable string. The while cut loop backs off until the bytes decode, so the excerpt ends cleanly at "passed ". Look at what the excerpt shows: a passing test run. The part that matters, the edit to the test configuration, is past the cut. Only truncated: true and total_bytes: 49 tell the reviewer there is more.

Test both sides of the boundary, the empty case and invalid limits:

for limit in [18, 19, 20]:
    value = capture_trace(step, limit)
    assert value["displayed_excerpt"] == "pytest: 41 passed "
    assert value["displayed_bytes"] == 18 and value["truncated"]
assert capture_trace(step, 21)["displayed_excerpt"].endswith("✓")
assert capture_trace(step, 0)["displayed_excerpt"] == ""
assert capture_trace(step, len(step.encode()))["truncated"] is False
assert capture_trace("", 0)["truncated"] is False
assert artifacts[record["artifact_ref"]].decode() == step
for invalid in [-1, True, 1.5]:
    try:
        capture_trace(step, invalid)
    except ValueError:
        pass
    else:
        raise AssertionError("Invalid byte limit accepted")
print("UTF-8 boundaries, empty input and invalid limits checked")

A valid UTF-8 cut can still split a character a reader perceives as one, such as a letter plus a combining accent. This code keeps the encoding valid; grapheme-aware truncation is an extra step if your display needs it. The example keeps full bytes in a local dictionary. In production, persist the artifact before advertising its reference, and verify its length and hash when you fetch it.

View full-size diagram

How many transcripts to read, and what they cost to keep

You cannot read every transcript of a 10,000-trial sweep. Sample them, and size the sample for the behavior you are looking for. If a behavior, such as editing tests, occurs in a fraction r of trials, a random sample of n transcripts misses it entirely with probability (1 − r)^n. To see it at least once with 95% probability:

(1 − r)^n ≤ 0.05, so n ≥ ln 0.05 / ln(1 − r).

For r = 0.02: n ≥ −2.996 / −0.0202 = 148.3, so 149 transcripts. For r = 0.005 it is about 598. Reading nothing unusual in 149 transcripts does not mean the rate is zero; it bounds it, as the rule of three in the environment article shows.

Keeping full transcripts is usually cheap next to rerunning trials. Anthropic's pricing page gives a rough rule for English text of about 4 characters per token. A 50-step trajectory averaging 3,000 tokens per step is 150,000 tokens, or about 600 KB. Ten thousand trials is about 6 GB. Set the display limit for the viewer, and keep the full record behind it.

What counts as a reviewed trial

The case study behind this site's version of the rule is a live class demonstration in the ZEO course, where a local model answered and then stopped, and the lesson recorded the cause as unknown rather than guessing. The same discipline applies on the review side: a reviewer facing a partial record writes down "I could not see all of it."

So the rule is explicit. If a record is truncated: true and the reviewer does not fetch the full artifact, a full-trial review stays coverage_incomplete. A narrower finding about the visible part may still stand. truncated: false only says this capture kept its entire input; an upstream collector may already have dropped events, so check the source's own completion marker too. Coverage is a different quantity from inter-reviewer agreement: agreement says how often reviewers concur, coverage says whether the evidence for any judgment was present.

When the full artifact cannot be fetched

A reference only helps if fetching it works, and its failures must be explicit too.

Denied and expired look alike from outside, and they need different fixes: one is a permissions problem, the other a retention policy. Name which one occurred. This table is a pattern to check against your own store, not a measured result from a deployment.

Checkpoint before you set the policy

Check the decision

Why this matters in a lab

Reading transcripts is where evaluation work finds its most important bugs: graders that pass wrong answers, agents that game tests, environments that leak the answer. None of that is visible in a score, and none of it is visible in a transcript that stops early. An interviewer asking "what would you log for an agent evaluation, and how would you review it?" wants the two lists above, reproduction and review, plus a sampling plan with its arithmetic and an explicit rule for incomplete records. Chapter 16 builds an evaluation harness that saves evidence meant to outlive the session, and Chapter 17 shows why the evaluation behind every change must be preserved before the version changes.