Live Demo Takeaways and Dataflow Integrity

This ran once, live, with nothing faked: three scenarios, no edited video, no pre-cached traces. Under the hood: Nebius Token Factory (Nebius's hosted inference API), a planner model called MiniMax-M2.5 (80B thinking) and an executor model called Qwen3-235B-A22B-Instruct — the same planner/executor split this course has been building since Week 3. This is what the repository does right now, on this network, against real models.

make example-research-real && make example-reviewer-real && make example-pub-booking-real

What to watch for while it runs

Four things, whether you're watching this live or running it yourself afterward:

1. The planner output — "Planner produced N subgoals, K to loop, M to structured." Watch that K and M match what the scenario actually has wired in.
2. The executor's tool choices — every call is logged; does it call the tools the task actually needs?
3. The dataflow integrity check at the end of each run — ✓ real source versus ✗ FAKE source, all citations from real web_lookup versus a fabricated ID.
4. The actual artifact — every run prints workspace/*.md inline. Read it. Check it against reality, not against how confident it sounds.

Real LLMs are probabilistic — a given run might differ slightly from another. That's expected; it's what a dataflow check is for.

Here's what each scenario is expected to do. Research assistant: one subgoal in the loop half, one web_lookup call returning two papers, a write_file to workspace/report.md, complete_task fires, and the dataflow check confirms 2 arXiv IDs, both traceable to the real web_lookup call — the report references 2 of 2 real paper titles. Code reviewer: the workspace seeded with a 316-byte sample.py, one subgoal, analyze_workspace_file(path="sample.py") reading the real file, a review written to workspace/review.md, and a dataflow check confirming real source with 4 findings — the pathological sample file has exactly 4 real issues, and the review names all four. Pub booking: the loop half runs pub_search then pub_availability, identifies the Haymarket Tap, and hands off; the structured half's deterministic rule commits the booking and writes workspace/booking.md. Run the same scenario again with --oversize (party of 12 against a cap of 8) and the same code, same rule, escalates instead — nothing gets written.

Nine tool calls total, across roughly 25 seconds, two different models, three different output artifacts, zero fabrications. Six hours before this demo was recorded, none of that was true — the research assistant hallucinated citations, the code reviewer reviewed code that didn't exist, the pub-booking scenario split subgoals across a structured half that only one of three scenarios actually had wired in. The debugging walked through in the previous lesson is the reason this demo works.

The principle the clean run can hide

Here's the trap: watching three clean runs in a row makes it tempting to conclude the framework itself now prevents fabrication. It doesn't, and that's the whole point of this second half of the lesson.

Three failure modes make the distinction concrete — and all three pass every structural check the framework offers:

Every one of those would have shipped to production on a green ticket trail. That's not a hypothetical framed for effect — Failure 6 from the previous lesson is a working instance of exactly the first row.

Structural checks and the dataflow check aren't redundant layers — they're two different filters, and only the second one catches what actually ships wrong:

The dataflow check is the only stage in this picture that reads content — everything above it only confirms an operation happened, not that what it produced was true.

The pattern, and a worked example

Every scenario in sovereign-agent carries a module-level log and a post-run check built from it:

_TOOL_CALL_LOG: list[dict] = []

def my_tool(args) -> ToolResult:
    result = _do_the_real_work(args)
    _TOOL_CALL_LOG.append({
        "args_hash": hash(frozenset(args.items())),
        "result_fingerprint": _fingerprint(result),
    })
    return result

def audit():
    if not _TOOL_CALL_LOG:
        warn("tool never called — output is hallucinated")
    # cross-check the output artifact against facts from the log

Here's the actual implementation from the research-assistant scenario — the same shape, made concrete:

def _web_lookup(query: str) -> ToolResult:
    hits = _search_corpus(query)
    _TOOL_CALL_LOG.append({
        "query": query,
        "arxiv_ids": [p["arxiv"] for p in hits],
    })
    return ToolResult(output={"results": hits})

# After the run
all_real_ids = {id for call in _TOOL_CALL_LOG for id in call["arxiv_ids"]}
cited_ids = set(re.findall(r"\b(\d{4}\.\d{4,5})\b", report_md))
fabricated = cited_ids - all_real_ids
if fabricated:
    warn(f"report cites IDs web_lookup never returned: {fabricated}")

If the model invents a paper — even a plausibly-real-looking one, since arXiv IDs are date-shaped and easy to imitate — this check catches it, because it compares what was actually returned against what got written, not against how confident the writing sounds.

Why this matters past a teaching scenario

Picture an LLM writing medical reports from chart data instead of research citations. Structural checks would verify the report was generated, the file hashes to the recorded value, the state machine advanced — all green. None of that detects the model ignoring the chart and writing a boilerplate report, borrowing another patient's values that happened to be sitting in context, or fabricating lab results that read as plausible.

This isn't a stretch. The course deck puts a number on it — at least three production healthcare systems this year hit this exact failure mode — but that number is unattributed here: no named system, no incident report, no citable source, so treat it as illustrative rather than a claim to repeat to a colleague as fact. What is checkable is the mechanism: a dataflow check catches this failure mode; a system with only structural checks doesn't.

The framework gives you atomic operations, an audit trail, state consistency, recovery semantics. It cannot give you "this tool was called with the right inputs for this task" or "the output references facts from the tool's results" — the framework has no way to know what "right" means for your scenario. Only the scenario's own dataflow check can verify that, which is why the rule has no carve-out:

In sovereign-agent, this check runs about 30 lines of code per scenario. It's the one check that would have caught the previous lesson's worst failure — Failure 6, the fabricated code review, where every structural guarantee passed clean and the actual output was still silently wrong. Every scenario you ship gets one. Every one. No exceptions.

What to carry forward

You've now seen the architecture, the failures that shaped it, and the demo that proves it out. The next lesson turns to what v0.1.0 deliberately left unfixed — three named gaps in the framework — and what changes when this stops being a teaching system and starts running in production.