What a model call costs, and a deployment that survives

Draft chapter. The textbook guide covers setup and the supplied code. Practice in Exercise Book 19, then check Solutions 19 once you have tried it.

Lucy's agent can now receive work, ask for approval and recover an uncertain order. Those abilities are useful only while a host runs the program and preserves its records. Closing a terminal must not silently end the morning routine. Updating the code must not discard yesterday's purchases. Restoring a backup must not persuade the supplier that an old approval is new authority.

Part A first measures what the agent spends. It covers prefill and decode, the memory bandwidth that bounds decode, the key-value cache, batching, latency and cost per task, all derived and measured on local models. Part B then turns your Chapter 10 worker into an installed Linux service. We keep one writable state directory outside the release, use the operating system's service manager and distinguish process health from business progress. Then we construct the backup and restore boundary and recover an older local snapshot against a separate supplier account.

The deployment is deliberately modest: one Linux host, one user service for shop work, SQLite and an operator-owned environment file. A maintained service manager supplies restart behavior. We still own the decisions about work, permissions, receipts and reconciliation. The machine's availability remains a condition of the promise, not something an agent loop can manufacture.

Learning objectives

Part A treats a model call as a cost. After it you should be able to:

• explain why decode is limited by memory bandwidth and prefill by arithmetic, and estimate a decode-rate ceiling;
• derive the key-value cache's memory from a model's architecture, and explain grouped-query attention;
• explain how batching raises throughput, and apply Little's law to a server that does not batch;
• decompose a request's latency into overhead, prefill and decode, and report percentiles;
• estimate the cost of an agent loop, whose input grows with the square of its length.

Part B deploys and maintains the agent. After it you should be able to construct a systemd unit for the bounded worker; distinguish liveness from progress; inspect open work and uncertain outcomes; create a consistent, protected backup; restore without preserving obsolete authority; preflight an upgrade before stopping services; and verify that a compatible rollback preserves business history.

The portable deliverable is a maintenance checkpoint that creates real SQLite state and a separate loopback supplier process. It restores an older snapshot, rejects stale authority, reconciles later receipts and completes fresh work. The Linux deliverable is an installed service whose release, state, restart and reboot behavior and completed work are recorded in a receipt. Neither a generated unit nor a green portable checkpoint proves that the host started it.

Part A: what a model call costs

Before deploying the agent, understand what it spends. Every call to a model costs time and money, and the costs follow from a few facts about how a transformer produces text. Those facts decide what an agent can afford to do. They explain why a long transcript is expensive, why output tokens cost more than input tokens, and why serving systems batch requests. This part derives them and measures them on the local models used throughout the book.

The functions live in the chapter's learner file, and the measurements in the chapter's receipt.

import json
import runpy
import statistics

econ = runpy.run_path(
    "book/textbook/learner/profrod_sovereign_agent_ch19_inference_economics_learner.py"
)
measured = json.loads(open("docs/evidence/book-ch19/ch19-inference-receipt-v1.json").read())

Two phases: prefill and decode

Generating an answer happens in two phases.

• Prefill runs the model over the whole prompt at once. All $P$ prompt tokens pass through each layer together, as one large matrix multiplication per weight matrix, and the pass fills the key-value cache described below.
• Decode produces the answer one token at a time. Each new token needs a full forward pass, and that pass cannot start until the previous token is known.

The arithmetic per token is the same in both phases. A weight matrix of $m \times n$ parameters costs $mn$ multiplications and $mn$ additions per token, so a model with $N$ parameters costs about $2N$ floating-point operations per token. What differs is how often the weights are read. In prefill, one read of each weight serves all $P$ tokens. In decode, every step reads every weight to produce a single token.

The experiment reads each phase's duration from the server's own timing fields, with a fresh prefix on every prompt so no cached work is reused:

uv run python book/textbook/experiments/profrod_sovereign_agent_textbook_ch19_inference_v1.py \
    --out ch19-inference-receipt.json

One run, recorded on 2026-09-26 with Ollama 0.32.5 on an Apple M4 Pro, took 67 seconds.

for name, model in measured["models"].items():
    prefill = model["prefill"][1]["tokens_per_second"]
    decode = model["decode_tokens_per_second"]["short_prompt"]
    print(
        f"{name:13} {model['parameters'] / 1e9:.2f}B parameters,"
        f" {model['file_bytes'] / 1e9:.2f} GB: prefill {prefill:6.0f} tok/s,"
        f" decode {decode:5.1f} tok/s, ratio {prefill / decode:4.1f}"
    )

qwen2.5:0.5b  0.49B parameters, 0.40 GB: prefill   5590 tok/s, decode 211.1 tok/s, ratio 26.5
qwen3:0.6b    0.75B parameters, 0.52 GB: prefill   4201 tok/s, decode 217.7 tok/s, ratio 19.3
qwen2.5:1.5b  1.54B parameters, 0.99 GB: prefill   1752 tok/s, decode 129.6 tok/s, ratio 13.5

Prefill is an order of magnitude faster per token than decode on the same hardware. Reading a prompt is cheap; writing an answer is expensive. This is why API providers price output tokens several times higher than input tokens.

Decode is limited by memory, not arithmetic

A decode step must bring every weight from memory to the processor. If memory delivers $B$ bytes per second and the weights occupy $W$ bytes, no implementation can take more than

$ \text{decode rate} \le \frac{B}{W} $

steps per second at batch size one. Compare the work with the traffic. A decode step performs $2N$ operations and reads $W = bN$ bytes, where $b$ is the bytes per parameter. The arithmetic intensity is therefore $2/b$ operations per byte: about three for these 4-bit models, and one for 16-bit weights. Modern processors can do tens to hundreds of operations in the time it takes to read one byte, so at batch size one the arithmetic units mostly wait for memory.

The measurement gives one more check. If each step reads the weights once, the decode rate times the file size is the bandwidth actually achieved. Apple lists 273 GB/s for the M4 Pro.

for name, model in measured["models"].items():
    print(f"{name:13} implied weight reads {model['implied_weight_read_gb_per_second']} GB/s")
small, large = measured["models"]["qwen2.5:0.5b"], measured["models"]["qwen2.5:1.5b"]
step = {m["file_bytes"]: 1 / m["decode_tokens_per_second"]["short_prompt"] for m in (small, large)}
(w1, t1), (w2, t2) = sorted(step.items())
seconds_per_byte = (t2 - t1) / (w2 - w1)
overhead = t1 - w1 * seconds_per_byte
print(f"step time = {overhead * 1000:.2f} ms + weights / {1 / seconds_per_byte / 1e9:.0f} GB/s")

qwen2.5:0.5b  implied weight reads 84.0 GB/s
qwen3:0.6b    implied weight reads 113.8 GB/s
qwen2.5:1.5b  implied weight reads 127.8 GB/s
step time = 2.72 ms + weights / 197 GB/s

No model reaches the ceiling, and the smallest is furthest from it. The fit through the two qwen2.5 models explains why. Each step pays a fixed cost of a few milliseconds, for launching work, sampling a token and bookkeeping, plus the weight reads. For a small model the fixed cost is most of the step. The reads alone proceed at close to three-quarters of the published bandwidth. As models grow, the weight term dominates and decode approaches $B/W$. This is why, for large models, doubling the size roughly halves the decode rate.

The key-value cache

Attention lets each new token look at every earlier token. Recomputing the earlier tokens' keys and values at every step would repeat all of prefill, so the server stores them: the key-value (KV) cache. For every layer, every KV head and every token, it keeps one key vector and one value vector of the head dimension $d$:

$ \text{KV bytes} = 2 \times L \times H_{\text{kv}} \times d \times b \times T, $

for $L$ layers, $H_{\text{kv}}$ KV heads, $b$ bytes per value and $T$ tokens. Grouped-query attention (GQA) lets several query heads share one KV head, which divides the cache by the ratio of query heads to KV heads. The experiment reads $L$, the head counts and $d$ from each model file.

for name, model in measured["models"].items():
    print(
        f"{name:13} {model['layers']} layers, {model['kv_heads']} of {model['attention_heads']}"
        f" heads for KV, d = {model['head_dim']}: {model['kv_bytes_per_token_f16']:,} B/token;"
        f" full {model['context_length']:,}-token context"
        f" {model['kv_bytes_full_context_f16'] / 1e9:.2f} GB (weights {model['file_bytes'] / 1e9:.2f} GB)"
    )

qwen2.5:0.5b  24 layers, 2 of 14 heads for KV, d = 64: 12,288 B/token; full 32,768-token context 0.40 GB (weights 0.40 GB)
qwen3:0.6b    28 layers, 8 of 16 heads for KV, d = 128: 114,688 B/token; full 40,960-token context 4.70 GB (weights 0.52 GB)
qwen2.5:1.5b  28 layers, 2 of 12 heads for KV, d = 128: 28,672 B/token; full 32,768-token context 0.94 GB (weights 0.99 GB)

View full-size diagram

xychart-beta
    title "Weights against a full-context KV cache"
    x-axis "Model" ["qwen2.5:0.5b", "qwen3:0.6b", "qwen2.5:1.5b"]
    y-axis "Gigabytes" 0 --> 5
    bar "Weights" [0.40, 0.52, 0.99]
    bar "KV cache, one sequence at full context" [0.40, 4.70, 0.94]

At its full context, qwen3:0.6b's cache would be about nine times the size of its weights. Long contexts are paid for in memory, per sequence. Grouped-query attention is why qwen2.5:1.5b's cache is small: two KV heads instead of twelve divide it by six.

The cache is also read on every decode step, so a long context slows decode. Add the cache to the weights in the step-time fit, and predict the decode rate after a 2,000-token prompt.

bandwidth = 1 / seconds_per_byte
for name in ("qwen2.5:1.5b", "qwen3:0.6b"):
    model = measured["models"][name]
    tokens = statistics.median(
        run["prompt_eval_count"]
        for run in measured["runs"]
        if run["model"] == name and run["kind"] == "decode-long_prompt"
    )
    short = 1 / model["decode_tokens_per_second"]["short_prompt"]
    predicted = 1 / (short + tokens * model["kv_bytes_per_token_f16"] / bandwidth)
    observed = model["decode_tokens_per_second"]["long_prompt"]
    print(f"{name:13} after {tokens} tokens: predicted {predicted:.1f} tok/s, measured {observed}")

qwen2.5:1.5b  after 2010 tokens: predicted 124.9 tok/s, measured 118.8
qwen3:0.6b    after 1998 tokens: predicted 173.8 tok/s, measured 178.1

The prediction uses nothing but the architecture, the context length and the bandwidth fitted on other runs, and it lands within 5% of both measurements. The model with four times as many KV heads per layer slows much more.

Batching, and a server that did not batch

Decode's weakness suggests the remedy. One read of the weights can serve many sequences at once: a batch of $k$ sequences performs $2Nk$ operations for the same $W$ bytes, raising the arithmetic intensity to $2k/b$. Until the arithmetic units are busy, a batch of $k$ produces about $k$ times the tokens in the same step time. This is how serving systems reach their throughput, and why a provider can sell tokens more cheaply than a single user can generate them.

The experiment sent one, two and four simultaneous requests to the local server.

for row in measured["concurrency_qwen2.5:1.5b"]:
    print(
        f"{row['concurrent_requests']} at once: {row['wall_seconds']:.2f} s wall,"
        f" {row['aggregate_tokens_per_second']} tok/s total,"
        f" {row['per_request_decode_tokens_per_second']} tok/s per request while decoding"
    )

1 at once: 0.66 s wall, 97.1 tok/s total, 131.4 tok/s per request while decoding
2 at once: 1.23 s wall, 103.8 tok/s total, 127.8 tok/s per request while decoding
4 at once: 2.39 s wall, 107.0 tok/s total, 128.1 tok/s per request while decoding

View full-size diagram

xychart-beta
    title "Total decode throughput against concurrent requests, qwen2.5:1.5b"
    x-axis "Requests at once" [1, 2, 4]
    y-axis "Tokens per second" 0 --> 400
    line "Ideal batching" [97.1, 194.2, 388.4]
    line "This server" [97.1, 103.8, 107.0]

This server, as configured, did not batch. It served the requests one after another. Total throughput stayed near one request's decode rate, and wall time grew in proportion to the number of requests. Batching is a property of the serving software and its configuration, not of the model. The consequence for an agent is queueing. Little's law, from Chapter 8, says the average number of requests in progress equals the arrival rate times the time each spends in the system. A server that handles one request at a time, each taking $W$ seconds, can sustain at most $1/W$ requests per second. Above that rate, the queue grows without bound.

Latency: where the time goes, and the tail

A request's time is a sum of three terms:

$ T \approx T_0 + \frac{P}{\text{prefill rate}} + \frac{O}{\text{decode rate}}, $

with $P$ prompt tokens, $O$ output tokens and a fixed overhead $T_0$. Averages hide slow requests, so latency is reported by percentiles. The $q$th percentile is the smallest observed value that at least $q\%$ of observations do not exceed.

The experiment ran thirty requests of Chapter 16's request-interpretation task on qwen2.5:1.5b.

task = measured["request_task_qwen2.5:1.5b"]
print("mean tokens: prompt", task["mean_prompt_tokens"], "output", task["mean_output_tokens"])
print("seconds:", task["wall_seconds"])
print("share of time:", task["share_of_time"])
walls = [r["wall_seconds"] for r in measured["runs"] if r["kind"] == "request-task"]
print("p50 recomputed:", round(econ["percentile"](walls, 50), 3))

mean tokens: prompt 274.4 output 17.9
seconds: {'p50': 0.38, 'p90': 0.402, 'max': 0.41, 'mean': 0.386}
share of time: {'prefill': 0.423, 'decode': 0.342, 'other': 0.235}
p50 recomputed: 0.38

For this task the prompt is fifteen times longer than the answer, so reading the prompt takes more time than writing the answer, despite prefill's speed. This is the common shape of agent calls: a long transcript in, a short decision out. It is why providers offer prompt caching, which reuses the prefill of a repeated prefix. The spread is narrow on an idle local machine, where p90 is only 6% above the median. A shared service's tail is set by queueing, and must be measured under load.

Cost per task

A hosted model charges per token, with separate prices for input and output. An agent loop adds a multiplier: each call resends the whole transcript so far. Call $k$ of a loop sends the first prompt $P_0$ plus $k$ additions of about $\Delta$ tokens each, so over $n$ calls the input tokens total

$ \sum_{k=0}^{n-1} (P_0 + k\Delta) = nP_0 + \Delta,\frac{n(n-1)}{2}. $

Input grows with the square of the loop's length.

first, added = 300, 60  # tokens in the first prompt, and added by each tool round trip
input_price, output_price = 0.15, 0.60  # illustrative dollars per million tokens, not a quote
for calls in (2, 5, 10, 20):
    tokens = econ["loop_input_tokens"](calls, first, added)
    cents = econ["cost_cents"](tokens, 20 * calls, input_price, output_price)
    print(f"calls {calls:2}: {tokens:6,} input tokens, {cents:.4f} cents per task")

calls  2:    660 input tokens, 0.0123 cents per task
calls  5:  2,100 input tokens, 0.0375 cents per task
calls 10:  5,700 input tokens, 0.0975 cents per task
calls 20: 17,400 input tokens, 0.2850 cents per task

View full-size diagram

xychart-beta
    title "Input tokens over an agent loop"
    x-axis "Model calls in the loop" [2, 5, 10, 20]
    y-axis "Input tokens" 0 --> 18000
    line "Resending the transcript" [660, 2100, 5700, 17400]
    line "Growing only with the calls" [660, 1650, 3300, 6600]

Ten times the calls, from two to twenty, costs about twenty-six times the input. Three design rules follow, each already in this book:

• Keep loops short, and put computation in tools, as in Chapter 3.
• Bound every loop with a budget of calls and cost.
• Keep a stable prefix at the start of the transcript, so that prompt caching can reuse it.

A local model has no per-token price, but not zero cost: the measured task occupies the machine for about 0.4 seconds per call. At one request at a time, the machine caps the whole shop at about two and a half calls per second.

Part B: deploy and maintain the agent

Part A measured what each call costs. This part keeps the calls running: a host service, backups, restores that do not revive old authority, and upgrades that keep the day's records. It is built on your own code: the Chapter 10 worker as the service, and Chapter 11's orders, approvals and supplier as the authority a restore must not revive.

Put code, state and credentials in deliberate places

A release is a checkout of the course at a reviewed commit, with the virtual environment its lockfile describes. A state directory contains the database and operator configuration. The service names the release's interpreter explicitly rather than depending on an interactive shell's activated environment. This gives an upgrade a visible target and leaves the previous code available for a reviewed rollback.

For the following commands, choose absolute paths without spaces or systemd expansion characters. LUCY_COURSE is the course folder of the release the service runs, and LUCY_STATE is the writable state directory. The commands are a host recipe and are not executed by the portable manuscript checker.

export LUCY_STATE=/srv/lucy/state
export LUCY_COURSE=/srv/lucy/profrodai-resources/courses/sovereign-agent-book
cd "$LUCY_COURSE"
uv sync --frozen --python 3.12
mkdir -p "$LUCY_STATE"
chmod 700 "$LUCY_STATE"

Use the committed lock. A fresh dependency resolution during an urgent maintenance window changes two things at once: the application and the dependency set. The lock does not certify every dependency's behavior, but it makes the environment being installed identifiable. Record the source commit and Python version with the release rather than using a folder name as evidence of its contents.

Create agent.env locally with mode 0600. An empty file is sufficient for the offline teaching model. A live model and the phone channel require the operator's actual configuration. Do not paste credentials into the manuscript, source tree or proof bundle.

The backup function below snapshots SQLite only. It does not copy environment files, skill source files, saved evaluation reports or the supplier's ledger. A complete maintenance runbook inventories those separately. Active skill content held in SQLite survives its backup, but a record pointing to a missing report file is not a complete evidence archive.

View full-size diagram

flowchart LR
    release[Reviewed release and frozen environment] --> unit[User service]
    env[Operator environment file] --> unit
    unit --> state[Writable state directory]
    state --> snapshot[Consistent local backup]
    unit --> supplier[Independent supplier account]

Construct the host contract

The unit starts the same serve command you built in Chapter 10, your learner file run by the release's interpreter. It does not put a second scheduler inside the language model. The process receives signals, runs bounded passes and waits between them. Waiting does not call the model. A failed process is restarted by systemd, and on start serve finishes the turn its own worker label left running, with a report asking Lucy to ask again.

Paths are deliberately restricted. systemd unit files have expansion rules that differ from shell quoting. Accepting arbitrary strings and placing quotation marks around them would create an escaping language inside a teaching installer. Here, an unsupported installation path produces a refusal that the builder can resolve by selecting a supported path.

The following listings execute cumulatively from the course folder. The chapter's learner file is the completed comparison copy.

import hashlib
import json
import math
import os
import re
import runpy
import sqlite3
import subprocess
import sys
import tempfile
import time
import uuid
from pathlib import Path
from typing import Any

ORDERS = runpy.run_path("book/textbook/learner/profrod_sovereign_agent_ch11_approval_learner.py")
SUPPLIER = runpy.run_path("book/textbook/learner/profrod_sovereign_agent_ch19_supplier_learner.py")
LOOP = runpy.run_path("book/textbook/learner/profrod_sovereign_agent_ch03_agent_loop_learner.py")
STORE, TOOLS, CATALOG = ORDERS["STORE"], ORDERS["TOOLS"], ORDERS["CATALOG"]
PATH = re.compile(r"/[A-Za-z0-9_./-]+")


def unit_text(state: Path, python: Path, course: Path) -> str:
    """A systemd user unit that runs the Chapter 10 worker with an explicit interpreter.

    systemd has expansion rules unlike shell quoting, so paths are restricted rather than
    escaped: absolute, with no spaces or expansion characters."""
    state, python, course = state.absolute(), python.absolute(), course.absolute()
    if any(not PATH.fullmatch(str(path)) for path in (state, python, course)):
        raise ValueError("service paths must be absolute with no spaces or expansion characters")
    worker = course / "book/textbook/learner/profrod_sovereign_agent_ch10_wakeups_learner.py"
    return f"""[Unit]
Description=Lucy's always-on teaching agent
After=network-online.target

[Service]
Type=simple
WorkingDirectory={course}
ExecStart={python} -I {worker} serve --root {state}
EnvironmentFile={state}/agent.env
Restart=on-failure
RestartSec=10
TimeoutStopSec=90
UMask=0077
NoNewPrivileges=true
PrivateTmp=true

[Install]
WantedBy=default.target
"""


unit = unit_text(
    Path("/srv/lucy/state"),
    Path("/srv/lucy/course/.venv/bin/python"),
    Path("/srv/lucy/course"),
)
print("Restart on failure:", "Restart=on-failure" in unit)
print("Bounded shutdown:", "TimeoutStopSec=90" in unit)
print("Runs your Chapter 10 worker:", "ch10_wakeups_learner.py serve" in unit)
try:
    unit_text(Path("/srv/lucy with spaces"), Path("/usr/bin/python3"), Path("/srv/lucy/course"))
except ValueError:
    print("Unsupported path refused")

Restart on failure: True
Bounded shutdown: True
Runs your Chapter 10 worker: True
Unsupported path refused

Restart=on-failure restarts a process after a failure, not after every intentional clean stop. RestartSec=10 spaces failed starts. TimeoutStopSec=90 gives the bounded loop time to stop before systemd's final termination policy applies. That timeout does not guarantee an external request has been recalled. The supplier may have accepted a request before the process received its stop signal.

NoNewPrivileges constrains later privilege gain; it does not remove credentials the program already has. PrivateTmp gives the service its own temporary directory. Neither turns the worker into the sandbox of Chapter 15. Model-written code still goes through the report tool's separate boundary.

The installation function checks the environment file, refuses to overwrite a different unit and uses bounded subprocess calls. Its result includes systemctl's reported state and exit code. A caller must inspect those values. Returning a dictionary is not itself proof that a unit is active or that Lucy has received a result.

def service(action: str, state: Path, python: Path, course: Path) -> dict[str, Any]:
    """Install, inspect or remove the unit with `systemctl --user`. Linux only."""
    if not sys.platform.startswith("linux"):
        raise ValueError("service installation requires Linux and a user systemd manager")
    if action not in {"install", "status", "uninstall"}:
        raise ValueError("invalid service action")
    name = "lucy-agent.service"
    path = Path.home() / ".config/systemd/user" / name
    content = unit_text(state, python, course)
    if action == "install":
        env = state.absolute() / "agent.env"
        if not env.is_file() or env.stat().st_mode & 0o077:
            raise ValueError("create the environment file with mode 0600 first")
        path.parent.mkdir(parents=True, exist_ok=True)
        if path.exists() and path.read_text() != content:
            raise FileExistsError("a different service is already installed")
        path.write_text(content)
        subprocess.run(["systemctl", "--user", "daemon-reload"], check=True, timeout=20)
        subprocess.run(["systemctl", "--user", "enable", "--now", name], check=True, timeout=20)
    elif action == "uninstall":
        if path.exists() and path.read_text() != content:
            raise ValueError("refuse to remove another installation")
        subprocess.run(["systemctl", "--user", "disable", "--now", name], check=True, timeout=20)
        path.unlink(missing_ok=True)
        subprocess.run(["systemctl", "--user", "daemon-reload"], check=True, timeout=20)
    result = subprocess.run(
        ["systemctl", "--user", "show", name, "--property=ActiveState,SubState,MainPID,NRestarts"],
        capture_output=True,
        text=True,
        timeout=20,
    )
    return {
        "action": action,
        "unit": str(path),
        "status": result.stdout.strip(),
        "exit_code": result.returncode,
    }


print("Service function constructed:", callable(service))
print("Host installation requires:", "Linux user systemd manager")

Service function constructed: True
Host installation requires: Linux user systemd manager

The fixed unit name supports one teaching shop per Linux user. A second independent shop is an architectural extension, not a reason to overwrite the first unit. Uninstalling verifies that the installed unit is the one these paths would write before removing it. This ownership check makes a mistaken path visible during maintenance.

Install, then prove useful work

On the Linux host, prepare the state with your Chapter 10 functions, then install the service with the release's interpreter. The first due time is now, so the service has a brief to produce as soon as it starts.

cd "$LUCY_COURSE"
install -m 600 /dev/null "$LUCY_STATE/agent.env"
.venv/bin/python - "$LUCY_STATE" <<'EOF'
import runpy, sys, time
from pathlib import Path

wake = runpy.run_path("book/textbook/learner/profrod_sovereign_agent_ch10_wakeups_learner.py")
ops = runpy.run_path("book/textbook/learner/profrod_sovereign_agent_ch19_operations_learner.py")
state = Path(sys.argv[1])
db, queue = wake["open_shop"](state / "agent.sqlite")
wake["seed_shop"](db)
wake["CHANNEL"]["activate_opening_skill"](db)
wake["schedule"](db, "morning-v1", "lucy", "Prepare the opening brief.",
                 first_due=time.time(), interval_seconds=86_400)
db.close()
print(ops["service"]("install", state, Path(".venv/bin/python").absolute(), Path.cwd()))
EOF
systemctl --user show lucy-agent.service --property=ActiveState,SubState,MainPID,NRestarts

The first observation is an active process. The decisive application observation is that the scheduled work becomes finished, with a grounded report, while the service, rather than the invoking terminal, does the work. Inspect the work's source; an older finished item does not prove the newly installed process handled anything.

User services also depend on the lifetime of the user manager. Configure lingering for the teaching user through the host's administration path and check it with loginctl show-user "$USER" -p Linger. A service running during an open login does not prove it will start unattended after the next boot.

We ran the chapter's service experiment on Ubuntu 26.04 in a virtual machine, with systemd 259, lingering enabled and Python 3.12. The service finished a scheduled brief and a vanilla stock episode. We killed its process with SIGKILL; systemd restarted it ten seconds later, as RestartSec=10 says, and the new process finished work scheduled afterward. We then rebooted the machine. The service came back with a new boot identity and finished work scheduled after the reboot. The receipt records the unit, each process identity, the restart count and every report. It is one short run on one machine, not a month of uptime.

Observe open work and uncertain outcomes

Process liveness is a narrow fact. A live process may be waiting for a supplier, unable to reach a model, repeatedly failing a tool or holding an approval that Lucy has not answered. We therefore summarize durable business state as well as the host's process state. The health function is read-only and uses the database as its source.

The ordering store is Chapter 11's: its stock, work queue, leases and orders. This chapter adds its own line of migrations, owned by operations, with the authority epoch that a restore replaces and a paused flag. Claiming work checks that flag in the same transaction that takes the lease.

OPERATIONS_MIGRATIONS = {
    1: (
        # The authority epoch: a restore replaces it, so anything holding the old one is stale.
        "CREATE TABLE operations ("
        " id INTEGER PRIMARY KEY CHECK (id = 1), epoch TEXT NOT NULL,"
        " paused INTEGER NOT NULL DEFAULT 0)",
        "CREATE TABLE recoveries ("
        " digest TEXT PRIMARY KEY, plan TEXT NOT NULL, created REAL NOT NULL)",
    ),
}


def open_operations(path: str | Path):
    """Chapter 11's store and queue, with the operations tables and a first epoch."""
    db, queue = ORDERS["open_shop"](path)
    db.migrate("operations", OPERATIONS_MIGRATIONS)
    with db.immediate() as connection:
        connection.execute(
            "INSERT OR IGNORE INTO operations(id, epoch) VALUES (1, ?)", (uuid.uuid4().hex,)
        )
    return db, queue


def control(connection) -> tuple[str, bool]:
    epoch, paused = connection.execute("SELECT epoch, paused FROM operations WHERE id=1").fetchone()
    return epoch, bool(paused)


def configured_supplier(db, endpoint: str):
    """The supplier client for the current epoch, registered with the account. Refused while
    paused: a restored database must be reconciled before it may order again."""
    epoch, paused = control(db.connection)
    if paused:
        raise PermissionError("operations are paused; inspect and recover the account first")
    client = SUPPLIER["EpochClient"](endpoint, epoch)
    client.set_epoch(epoch)
    return client


def claim(db, worker_id: str, *, seconds: float = 300):
    """Chapter 8's claim and Chapter 11's lease in one transaction; nothing while paused."""
    with db.immediate() as connection:
        if control(connection)[1]:
            return None
        row = connection.execute(
            "SELECT work_id, session_id, text FROM work WHERE state='pending'"
            " ORDER BY work_id LIMIT 1"
        ).fetchone()
        if row is None:
            return None
        connection.execute(
            "UPDATE work SET state='running', worker_id=? WHERE work_id=?", (worker_id, row[0])
        )
        connection.execute(
            "INSERT INTO assignment_leases VALUES (?, ?, ?) ON CONFLICT (work_id) DO UPDATE"
            " SET worker_id=excluded.worker_id, expires=excluded.expires",
            (row[0], worker_id, time.time() + seconds),
        )
    return ORDERS["QUEUE"]["Assignment"](row[0], row[1], row[2], worker_id)


def health(db) -> dict[str, Any]:
    """Durable business state, read-only: what process liveness cannot say."""
    connection = db.connection
    epoch, paused = control(connection)
    return {
        "paused": paused,
        "epoch": epoch,
        "work": dict(connection.execute("SELECT state, count(*) FROM work GROUP BY state")),
        "uncertain_orders": connection.execute(
            "SELECT count(*) FROM assistant_orders WHERE status IN ('SENDING','UNKNOWN')"
        ).fetchone()[0],
        "uncertain_reports": connection.execute(
            "SELECT count(*) FROM reports WHERE delivery IN ('sending','unknown')"
        ).fetchone()[0],
    }


temporary = tempfile.TemporaryDirectory(prefix="lucy-ch19-")
root = Path(temporary.name)
db, queue = open_operations(root / "agent.sqlite")
queue.admit("morning", "lucy", "Prepare a stock brief.")
observation = health(db)
print("Paused:", observation["paused"])
print("Pending work:", observation["work"]["pending"])
print("Uncertain orders:", observation["uncertain_orders"])
print("Uncertain reports:", observation["uncertain_reports"])

Paused: False
Pending work: 1
Uncertain orders: 0
Uncertain reports: 0

Chapter 8's queue records what is open, not when it arrived, so this summary cannot report an age. Adding the admission time is a small migration; until then, pair the open count with each item's history before deciding that a worker is broken. A pending approval may be correctly waiting for Lucy.

Read the service's journal with journalctl --user -u lucy-agent.service. It logs work identity and status rather than prompts or credentials, so correlate a work ID with its records.

Construct a consistent backup

SQLite can keep committed changes in its write-ahead log. Copying only the main database file while the service runs can omit those changes. The backup API asks SQLite to produce a consistent snapshot across its own storage mechanism. It avoids inventing a file-copy protocol for a live database.

The destination is created exclusively with restrictive permissions. A repeated command refuses to replace the earlier snapshot. After copying, we check SQLite integrity and synchronize the file. This is evidence about the resulting database file, not a promise that every storage device or filesystem will survive every power failure. The host backup plan must still account for its own durability and retention requirements.

def backup(db, destination: Path) -> Path:
    """A consistent SQLite snapshot, created exclusively and privately, checked and synced."""
    if destination.exists() or destination.is_symlink():
        raise FileExistsError("backup destination already exists")
    destination.parent.mkdir(parents=True, exist_ok=True)
    # Exclusive creation makes a repeated run a refusal, not a replacement of evidence.
    descriptor = os.open(destination, os.O_CREAT | os.O_EXCL | os.O_WRONLY, 0o600)
    os.close(descriptor)
    try:
        with sqlite3.connect(destination) as snapshot:
            db.connection.backup(snapshot)
            if snapshot.execute("PRAGMA integrity_check").fetchone()[0] != "ok":
                raise ValueError("backup integrity check failed")
        with destination.open("rb") as stream:
            os.fsync(stream.fileno())
    except BaseException:
        destination.unlink(missing_ok=True)
        raise
    return destination


snapshot = backup(db, root / "before.sqlite")
with sqlite3.connect(snapshot) as saved:
    print("Snapshot integrity:", saved.execute("PRAGMA integrity_check").fetchone()[0])
    print("Snapshot work:", saved.execute("SELECT count(*) FROM work").fetchone()[0])
try:
    backup(db, snapshot)
except FileExistsError:
    print("Repeated destination refused")
print("Private snapshot permissions:", snapshot.stat().st_mode & 0o077 == 0)

Snapshot integrity: ok
Snapshot work: 1
Repeated destination refused
Private snapshot permissions: True

Keep the snapshot's release and schema versions beside it. Retain a copy outside the host whose failure you are preparing for, using an operator-selected storage and access policy. This chapter does not implement a cloud backup service. Its concrete contract is the consistent file, its protected handling and a rehearsed restore path.

A successful backup is incomplete operational evidence until someone has restored it into a controlled environment and inspected its meaning. Byte integrity can coexist with obsolete business facts. Our failure experiment deliberately uses such a snapshot: it is internally consistent and still predates accepted supplier orders.

Restore records without restoring obsolete authority

Suppose the morning snapshot contains an approved vanilla order. During the day, the supplier accepts it, the delivery arrives and a strawberry order is accepted too. Restoring the morning snapshot loses those later local records. It does not remove the supplier's accepted orders or take the delivered tubs back out of the freezer.

Restore therefore starts paused. It replaces the authority epoch, drops every lease so that no old holder passes Chapter 11's ownership check, and revokes old approvals. It preserves the database file, using SQLite's backup machinery to replace the contents. Replacing the pathname with a different database file would leave existing connections attached to the old file and create competing writable histories.

View full-size diagram

sequenceDiagram
    participant O as Operator
    participant D as Active database
    participant B as Checked snapshot
    participant S as Supplier account
    O->>B: Validate schema and integrity
    O->>D: Pause
    B->>D: Copy prepared paused image with a new epoch through SQLite
    O->>S: Fence account and inspect retained receipts
    O->>D: Apply exact plan with current physical counts
    D-->>O: Reconciled state, resumed

The restore function prepares the image before disturbing active state. It requires the snapshot's schema versions, Chapter 4's per-component meta rows, to equal the current database's. An older snapshot must be migrated as a copy with reviewed current code; preserve the original as evidence. If the final copy fails, the active database stays paused with its old epoch: repair the underlying problem and restore again from the same snapshot. Never unpause by hand to get past it.

def schema(connection) -> dict[str, int]:
    """Each component's schema version, from Chapter 4's `meta` table."""
    return dict(connection.execute("SELECT key, value FROM meta"))


def restore(db, source: Path) -> str:
    """Pause the active database, then copy a checked snapshot into it with new authority.

    The database file is kept: replacing the path would leave open connections writing to the
    old file. The copied image carries a new epoch and is paused; every lease is dropped, so no
    old holder passes Chapter 11's ownership check; every approval is revoked. A request the
    supplier already accepted is not recalled; recovery finds it. Returns the new epoch."""
    if source.resolve() == db.path.resolve() or not source.is_file():
        raise ValueError("a separate existing backup is required")
    with sqlite3.connect(f"{source.resolve().as_uri()}?mode=ro", uri=True) as snapshot:
        if snapshot.execute("PRAGMA integrity_check").fetchone()[0] != "ok":
            raise ValueError("restore source is corrupt")
        if schema(snapshot) != schema(db.connection):
            raise ValueError("restore requires the same schema versions; migrate a copy first")
        # Prepare the restored image before disturbing active state.
        with tempfile.TemporaryDirectory(prefix="lucy-restore-") as temporary:
            image = Path(temporary) / "restored.sqlite"
            epoch = uuid.uuid4().hex
            with sqlite3.connect(image) as prepared:
                snapshot.backup(prepared)
                prepared.execute("UPDATE operations SET epoch=?, paused=1 WHERE id=1", (epoch,))
                prepared.execute("DELETE FROM assignment_leases")
                # Old approvals can be reconciled, but cannot authorize a new send.
                prepared.execute(
                    "UPDATE assistant_orders SET revoked=1, approved_until=0,"
                    " status=CASE WHEN status IN ('DRAFT','APPROVED') THEN 'REVOKED'"
                    " ELSE status END"
                )
                prepared.execute("UPDATE assistant_spending SET reserved_cents=0")
                prepared.commit()
                with db.immediate() as connection:
                    connection.execute("UPDATE operations SET paused=1 WHERE id=1")
                prepared.backup(db.connection)
    return epoch


holder = claim(db, "before-restore")
old_connection = ORDERS["StateStore"](db.path)
inode = db.path.stat().st_ino
restore(db, snapshot)
print("Database file preserved:", db.path.stat().st_ino == inode)
print("Restored state paused:", health(db)["paused"])
print("New worker withheld:", claim(db, "replacement") is None)
try:
    ORDERS["assert_current"](old_connection.connection, holder)
except PermissionError:
    print("Old open connection refused")
old_connection.close()

Database file preserved: True
Restored state paused: True
New worker withheld: True
Old open connection refused

There is deliberately no automatic resume line. A process that already passed an external write boundary may still complete remotely. Dropping leases prevents newly admitted local writes by obsolete holders; it cannot travel backward through a supplier's network request. Account recovery needs the supplier's independent boundary and history.

Reconcile the account with current observations

The chapter's supplier is Chapter 11's with two additions: an account epoch that every order must carry, and an export of every receipt the account holds. Inspection sets the account's epoch to the restored one. A client still holding the previous epoch can no longer submit new orders. A real supplier would authenticate who may rotate an epoch; the teaching supplier on loopback does not.

Inspection returns a plan template with unknown physical counts. That is intentional. A receipt can establish that the supplier accepted six tubs; it cannot establish that the delivery arrived, that no tubs were sold or that the freezer count matches the old snapshot. The operator supplies a current count for every product and an explicit delivery observation for every accepted order.

The plan is bound to exact bytes by a digest and must be recent. Recovery checks the digest, the operator, the paused authority, the supplier's current receipts, the product set and the delivery observations before changing anything. Then, in one transaction, it records every accepted order as delivered or confirmed, sets the spending to what the supplier accepted, corrects stock with Chapter 4 events, finishes restored running work with a report that says it was interrupted, and resumes. A changed plan needs its own digest. Repeating an applied plan changes nothing.

def inspect_account(db, client, *, actor: str, policy) -> dict[str, Any]:
    """Fence the supplier account with the restored epoch, then list every receipt it holds.

    The plan template leaves every physical count and delivery unknown: receipts say what the
    supplier accepted, not what arrived or what was sold since."""
    if actor not in policy.operators:
        raise PermissionError("operator is not allowlisted")
    epoch, paused = control(db.connection)
    if not paused:
        raise PermissionError("account inspection is for a paused, restored database")
    client.set_epoch(epoch)  # an old process with the previous epoch can no longer order
    receipts = client.receipts()
    accepted = sorted(r["operation"] for r in receipts if r.get("status") == "ACCEPTED")
    return {
        "receipts": receipts,
        "plan_template": {
            "authority": epoch,
            "observed_at": None,
            "receipts": accepted,
            "inventory": {sku: {"on_hand": None} for sku in sorted(CATALOG)},
            "deliveries": {
                operation: {"received": None, "reference": ""} for operation in accepted
            },
        },
    }


def recover(
    db,
    client,
    raw: bytes,
    digest: str,
    *,
    actor: str,
    policy,
    now: float | None = None,
    freshness: float = 900,
) -> dict[str, Any]:
    """Apply an operator's exact plan: the supplier's accepted orders, today's counts and
    deliveries. Checks everything first, changes everything in one transaction, then resumes.
    The same plan again is a duplicate and changes nothing."""
    if hashlib.sha256(raw).hexdigest() != digest:
        raise ValueError("plan bytes do not match the reviewed digest")
    if actor not in policy.operators:
        raise PermissionError("operator is not allowlisted")
    now = time.time() if now is None else now
    done = db.connection.execute("SELECT plan FROM recoveries WHERE digest=?", (digest,)).fetchone()
    if done is not None:
        return {"status": "ACTIVE", "duplicate": True}
    plan = json.loads(raw)
    epoch, paused = control(db.connection)
    if not paused or plan.get("authority") != epoch:
        raise PermissionError("the plan must belong to the current paused restore")
    observed = plan.get("observed_at")
    if not isinstance(observed, (int, float)) or not math.isfinite(observed):
        raise ValueError("the plan needs the time its observations were made")
    if not now - freshness <= observed <= now:
        raise ValueError("the plan's observations are not current")
    receipts = {
        r["operation"]: r
        for r in client.receipts()
        if r.get("status") == "ACCEPTED"
        and SUPPLIER["CHAPTER11"]["valid_proposal"](r.get("proposal"))
    }
    if sorted(receipts) != plan.get("receipts"):
        raise ValueError("the supplier's receipts changed since inspection")
    inventory, deliveries = plan.get("inventory", {}), plan.get("deliveries", {})
    if sorted(inventory) != sorted(CATALOG) or any(
        type(count.get("on_hand")) is not int or count["on_hand"] < 0
        for count in inventory.values()
    ):
        raise ValueError("a current count for every product is required")
    if sorted(deliveries) != sorted(receipts) or any(
        type(d.get("received")) is not bool for d in deliveries.values()
    ):
        raise ValueError("an explicit delivery observation for every accepted order is required")
    spent = 0
    with db.immediate() as connection:
        for operation, receipt in receipts.items():
            proposal = receipt["proposal"]
            encoded = json.dumps(proposal, sort_keys=True, separators=(",", ":"))
            amount = proposal["quantity"] * proposal["unit_cost_cents"]
            status = "DELIVERED" if deliveries[operation]["received"] else "CONFIRMED"
            spent += amount
            if connection.execute(
                "SELECT 1 FROM assistant_orders WHERE id=?", (operation,)
            ).fetchone():
                connection.execute(
                    "UPDATE assistant_orders SET status=?, receipt=? WHERE id=?",
                    (status, json.dumps(receipt), operation),
                )
            else:
                # An order the snapshot never saw: imported from the supplier's receipt.
                connection.execute(
                    "INSERT INTO assistant_orders(id,work_id,proposal,digest,amount,target,"
                    "status,revoked,receipt,created) VALUES (?,0,?,?,?,?,?,1,?,?)",
                    (
                        operation,
                        encoded,
                        hashlib.sha256((client.identity + "\n" + encoded).encode()).hexdigest(),
                        amount,
                        client.identity,
                        status,
                        json.dumps(receipt),
                        now,
                    ),
                )
        # Work the snapshot restored as running has no lease and no worker: finish it with a
        # report that says so, as Chapter 9 did for a stopped cell.
        for (work_id,) in connection.execute(
            "SELECT work_id FROM work WHERE state='running' ORDER BY work_id"
        ).fetchall():
            connection.execute("UPDATE work SET state='finished' WHERE work_id=?", (work_id,))
            (generation,) = connection.execute(
                "SELECT count(*) + 1 FROM reports WHERE work_id=?", (work_id,)
            ).fetchone()
            connection.execute(
                "INSERT INTO reports (report_id, work_id, generation, body, delivery)"
                " VALUES (?, ?, ?, ?, 'pending')",
                (
                    f"r{work_id}.{generation}",
                    work_id,
                    generation,
                    "This request was interrupted by a restore. Please send it again.",
                ),
            )
        # A local send the supplier never accepted cannot be accepted now: the account is fenced.
        connection.execute(
            "UPDATE assistant_orders SET status='REVOKED' WHERE status IN ('SENDING','UNKNOWN')"
        )
        connection.execute(
            "INSERT OR IGNORE INTO assistant_spending(id, limit_cents) VALUES (1, ?)",
            (policy.total_cents,),
        )
        connection.execute(
            "UPDATE assistant_spending SET reserved_cents=0, spent_cents=? WHERE id=1", (spent,)
        )
        for sku, count in inventory.items():
            (tubs,) = connection.execute("SELECT tubs FROM stock WHERE sku=?", (sku,)).fetchone()
            if count["on_hand"] != tubs:
                # A counted correction, as a Chapter 4 stock event with its own identity.
                event = STORE["StockEvent"](
                    f"recovery:{digest[:16]}:{sku}", sku, count["on_hand"] - tubs, "recovery count"
                )
                connection.execute(
                    "INSERT INTO events (event_id, sku, delta, payload, reason)"
                    " VALUES (?, ?, ?, ?, ?)",
                    (event.event_id, sku, event.delta, event.payload(), event.reason),
                )
                connection.execute("UPDATE stock SET tubs=? WHERE sku=?", (count["on_hand"], sku))
        connection.execute("INSERT INTO recoveries VALUES (?, ?, ?)", (digest, raw.decode(), now))
        connection.execute("UPDATE operations SET paused=0 WHERE id=1")
        ORDERS["record_event"](
            connection,
            "operations.recovered",
            {"digest": digest, "orders": len(receipts), "spent_cents": spent, "actor": actor},
        )
    return {"status": "ACTIVE", "duplicate": False, "orders": len(receipts), "spent_cents": spent}


def stock_position(db) -> list[dict[str, Any]]:
    """Stock on hand, tubs ordered but not yet delivered, and the remaining need."""
    on_order = dict(
        db.connection.execute(
            "SELECT json_extract(proposal,'$.sku'), sum(json_extract(proposal,'$.quantity'))"
            " FROM assistant_orders WHERE status='CONFIRMED' GROUP BY 1"
        )
    )
    rows = []
    for sku, tubs in sorted(db.stock().items()):
        point, ordered = CATALOG[sku]["reorder_point"], on_order.get(sku, 0)
        rows.append(
            {
                "sku": sku,
                "on_hand": tubs,
                "on_order": ordered,
                "reorder_point": point,
                "needed": max(0, point - tubs - ordered),
            }
        )
    return rows


policy = ORDERS["SpendingPolicy"](frozenset({"lucy"}))
try:
    recover(db, None, b"{}", "0" * 64, actor="lucy", policy=policy)
except ValueError:
    print("Changed plan bytes: refused before anything else")
db.close()
temporary.cleanup()

Changed plan bytes: refused before anything else

In the chapter checkpoint, the snapshot contains one order, and the independent supplier holds two accepted orders. Vanilla has been received, bringing physical stock to eight; four strawberry tubs remain pending. Recovery imports both receipts and records 2,600 cents once. The old supplier client is then used for an attempted order and is refused. A new worker claims fresh work and completes it, while a request admitted during the pause waited rather than running.

View full-size diagram

flowchart TD
    snapshot[One order in morning snapshot] --> paused[Paused restore]
    receipts[Two retained supplier receipts] --> plan[Exact recovery plan]
    counts[Current physical counts and delivery observations] --> plan
    paused --> plan
    plan --> result[Two reconciled orders and 2600 cents spent]
    result --> fresh[Resumed: new work claimed and completed]
    result --> interrupted[Restored running work reported as interrupted]

Your agent does not yet keep a durable account of model calls, so a restore loses no model allowance: there is none to lose. Chapter 3's per-loop limits still bound each turn. A runtime that does keep such an account must say what a restore did to it, rather than letting lost usage read as zero.

A supplier without complete discovery or an equivalent write fence needs a different recovery contract. An empty search result does not prove that a previously admitted request cannot arrive later. The teaching supplier makes those requirements inspectable; it does not establish that every commercial supplier offers them.

Upgrade and roll back without erasing the day

Install the target release's frozen environment before stopping the current service. Then read the existing database's schema versions with a read-only connection, sqlite3.connect("file:...?mode=ro", uri=True) and schema(), and compare them with the migrations the target code declares. Do this without opening the store, so the preflight cannot migrate state while deciding whether the target understands it. Your Chapter 4 store also refuses, before changing anything, to open a database whose schema is newer than its code.

For a compatible upgrade, stop the service and coordinate any other writers. Take a new backup, open the state with the target code to apply its reviewed forward migrations, install its unit and schedule a unique request. Compare orders, stock and spending before and after the switch. A new finished request plus unchanged retained business facts is stronger evidence than installation output alone. We did not run an upgrade experiment for this edition.

systemctl --user stop lucy-agent.service
cd "$LUCY_COURSE" && git fetch && git checkout <reviewed-commit> && uv sync --frozen --python 3.12
systemctl --user start lucy-agent.service
systemctl --user show lucy-agent.service --property=ActiveState,MainPID,NRestarts

Take the backup and run the preflight between the first two lines. The unit names the course folder, so a release switch in place needs no new unit; a release in a new folder needs service("install", ...) with its paths, after service("uninstall", ...) with the old ones. The Linux maintenance appendix collects the host checks.

Code rollback keeps the business state. A prior release must understand the current schema and preserve the authority contract. Matching schema versions are necessary, but they do not prove that older code enforces a later security repair. Review the actual change before choosing a fallback release.

Restoring an old database to make old code start is a different operation. It can erase local knowledge of orders that still exist remotely and therefore enters the paused reconciliation path. Keep these two operations distinct in the runbook and in the evidence: code replacement preserves history; state recovery reconstructs an authorized present from incomplete local history.

In a coding agent

Operating Claude Code is this chapter's cost model at team scale. A long session resends its transcript every turn, so input grows with the conversation, and prompt caching is what keeps that affordable. Claude Code's cost page reports an average across enterprise deployments of about USD 13 per developer per active day, and recommends a pilot group to set a baseline before a wider rollout.

For measurement, /usage breaks down a session's input, output and cache tokens, and OpenTelemetry export sends usage, cost and tool activity across an organization as metrics, events and optional traces. Both dollar figures are estimates from token counts. The bill of record is the provider's.

Try it in Claude Code: work through one task, then compare the cache reads in /usage with the input tokens, and estimate what the same session would cost without the cache.

Failure experiment and learner verification

Exercise 1: restore the day against retained receipts

Run the portable checkpoint from the course folder. It uses temporary local state, an actual supplier process and authored physical observations. It does not alter installed services or contact a real supplier. Its systemd line is an explicit limit on what this command proves.

uv run --python 3.12 python book/textbook/checkpoints/profrod_sovereign_agent_ch19_deployment_restoration_checkpoint.py

Expected observations

Expected observations include a preserved database file, refusal of the old connection and supplier epoch, one local order after restore followed by two reconciled orders, 2,600 cents expenditure, eight vanilla tubs on hand, four strawberry tubs pending, an interrupted report for the work the snapshot had running, and fresh work completing. The course gate runs it with the supplied packages refused on import, so every function it reaches is yours; we also changed the learner files in 24 small ways, from keeping old leases to letting the supplier ignore the epoch, and every change fails it.

Exercise 2: prove work after the terminal closes

For the host experiment, record the exact release and unit paths, main process identities, restart counters and a unique completed request. Stop the terminal session and verify another scheduled request. Reboot the isolated host and verify the service and fresh work again. Inspect retained business records after the reboot; old successful logs alone cannot establish that the new process handled anything.

The committed Linux receipt distinguishes these observations from the portable checkpoint: a served brief and stock episode, a restart after SIGKILL with new work finished, and a real reboot with new work finished. That finite experiment does not establish a month of uptime or an uninterrupted-service guarantee.

Exercise 3: refuse unsafe maintenance inputs

For an adversarial exercise, attempt to restore the active database onto itself, reuse a backup destination, use an unsupported unit path and supply a changed recovery digest. Each should refuse before it can silently replace evidence or regain authority. Then attempt a downgrade whose code does not know the newest migration; the current service must remain untouched by the preflight refusal.

For an operating exercise, define a useful alert threshold for work age in Lucy's shop. A morning brief waiting five minutes may warrant attention; a large order awaiting an explicit approval may be correctly blocked for hours. State the business reason for the threshold and the record you would inspect before restarting anything. Restarting a healthy process cannot supply missing approval.

Exercise 4: predict a model you have not measured

Pick a model you have not run, and read its layer count, KV heads and head dimension from its model card or file. Predict its KV cache per token and its decode rate on your machine, from this chapter's fitted step time. Then measure it with the chapter's experiment, and explain the difference.

Exercise 5: price a loop with caching

A provider charges a tenth of the input price for a cached prefix. Extend loop_input_tokens to split each call's input into a cached prefix, the transcript before this call, and new tokens. Compare the cost of the 20-call loop with and without caching.

Summary

A model call has two phases. Prefill reads the prompt quickly; decode writes the answer one token at a time, limited by how fast the weights can be read. The key-value cache costs memory in proportion to the context, and grouped-query attention reduces it. Batching amortizes each weight read across many sequences, if the server does it. An agent loop that resends its transcript pays for input quadratically. Measure all of these, because the model file and the server's own timings are enough to predict them.

We constructed the host unit and maintenance boundary around your own bounded worker, and ran it on Linux through a kill and a reboot. Releases make the selected code visible, while persistent state survives process replacement. Health observations distinguish liveness from unfinished work and uncertain effects. SQLite backup produces a consistent local snapshot, and restore invalidates old authority rather than treating old records as permission to repeat the day.

Account recovery requires more than file integrity. The supplier's epoch fence and retained receipts combine with current physical observations and an exact plan. Work the snapshot had running is reported as interrupted, not quietly resumed. A compatible code rollback preserves these records instead of replacing them with a convenient older database.

Active recall

Why does a decode step read every weight, and a prefill step read each weight once for the whole prompt? Derive the KV cache's size from the architecture. Why does a loop of ten calls cost far more than ten times one call?

Why can copying only the live SQLite file miss committed work? Why preserve the active database file during restore? What can an accepted supplier receipt establish about physical stock? Why does a matching schema not fully justify a code rollback? Which observation proves the service handled a new request after reboot? Why must replaying a recovery plan change nothing?

Vocabulary

Prefill processes the prompt; decode generates one token per step. Arithmetic intensity is operations per byte read. The KV cache stores each earlier token's keys and values; grouped-query attention shares KV heads across query heads. Batching serves several sequences with one weight read. A percentile is the value below which a given share of observations fall.

A release identifies reviewed code and its environment. A user service is managed by the host's user-level service manager. A consistent snapshot reflects SQLite's committed state at a valid point. An authority epoch marks which restore is current, and an account epoch fences a supplier account against clients from before it. Account reconciliation combines external receipts with present observations. A code rollback changes the executable while retaining compatible business history.

The next chapter combines the mechanisms into Lucy's accelerated business day. Its final report must distinguish orders, deliveries, approvals, work and expenditure, and every reported result must lead back to the records that support it.

Keep building with Prof Rod

Found this material through a colleague, classroom or shared download? Get the complete book at profrod.ai/book and join the Prof Rod learner community. Bring one result, one question or one failure you learned from. Share this resource with another learner and keep its source links with it so they can find the full course and future updates.