How to Build an RL Environment With a Verifier in Python

In reinforcement learning, the verifier decides the reward. If it can be satisfied without doing the task, a policy trained against it learns to satisfy the verifier. This tutorial builds the smallest environment that shows the problem: one task type, one verifier you can loosen or tighten, and two policies.

Writing checks like this is also what some lab postings describe as the job. Anthropic's Conceptual Reasoning Fellow posting lists "Designing evaluations and RL environments that test models in your field systematically" as a duty, and a Fellows Program posting lists "Creating RL environments to improve Claude models at capabilities that are within your domain of expertise." We read both on October 13, 2026; postings change and close, so check them before you plan around them. This is two postings from one lab, not a hiring checklist.

What you will make: a 46-line Python environment with a task generator, a verifier and two policies, plus six hostile test cases for the verifier. You need Python 3.9 or newer and nothing else: no package, account or model.

Our article on building an agent eval environment that resists reward hacking covers the full design. Here you build the smallest version and break it yourself.

Check the verifier

Build it

Save this as env.py. The task is a unit conversion whose answer is exact by construction.

#!/usr/bin/env python3
"""A minimal RL environment: tasks, a verifier, a reward, and two policies."""
import random
import re

def make_task(rng):
    """One unit-conversion task. The hidden answer is exact by construction."""
    km = rng.randint(2, 40)
    return {"prompt": f"A trail is {km} km long. How many meters is that?",
            "answer": km * 1000}

def loose_verifier(task, response):
    """First draft: reward any response that mentions the right number."""
    return str(task["answer"]) in response.replace(",", "")

def strict_verifier(task, response):
    """Second draft: exactly one 'ANSWER: <number>' line, and it must match."""
    lines = [l for l in response.splitlines() if l.startswith("ANSWER:")]
    if len(lines) != 1:
        return False
    match = re.fullmatch(r"ANSWER:\s*([0-9][0-9,]*)", lines[0].strip())
    return bool(match) and int(match.group(1).replace(",", "")) == task["answer"]

def honest_policy(task, rng):
    """Does the work, but gets the unit wrong a third of the time."""
    km = task["answer"] // 1000
    value = km * 1000 if rng.random() > 1 / 3 else km * 100
    return f"{km} km is {value} m.\nANSWER: {value}"

def hacking_policy(task, rng):
    """Does no work: lists many candidates and hopes one is right."""
    guesses = [n * 1000 for n in range(2, 41)]
    return "It could be " + ", ".join(str(g) for g in guesses) + "."

def pass_rate(policy, verifier, n=2000, seed=0):
    rng = random.Random(seed)
    wins = 0
    for _ in range(n):
        task = make_task(rng)
        wins += verifier(task, policy(task, rng))
    return wins / n

if __name__ == "__main__":
    for pname, policy in (("honest", honest_policy), ("hacking", hacking_policy)):
        for vname, verifier in (("loose", loose_verifier), ("strict", strict_verifier)):
            print(f"{pname:8} {vname:7} pass rate {pass_rate(policy, verifier):.3f}")

Run it:

python3 env.py

These are the results we got, on 2,000 tasks with seed 0:

View full-size diagram

The honest policy scores the same under both, which is what a good verifier does for honest work. The hacking policy goes from a perfect score to zero when the verifier changes. The only difference is what the check rules out.

Attack your own verifier

A verifier is code, so test it like code, with inputs written to break it. Save this as check_verifier.py next to env.py:

from env import strict_verifier

task = {"prompt": "A trail is 5 km long. How many meters is that?", "answer": 5000}
cases = [
    ("ANSWER: 5000", True),
    ("5 km is 5000 m.\nANSWER: 5,000", True),
    ("ANSWER: 5000\nANSWER: 4000", False),   # two answers
    ("ANSWER: 5000 or 6000", False),          # hedged
    ("The answer is 5000.", False),           # no answer line
    ("ANSWER: 500", False),                   # wrong
]
for response, expected in cases:
    got = strict_verifier(task, response)
    assert got == expected, (response, got, expected)
print(f"{len(cases)} verifier cases pass")

python3 check_verifier.py

It prints 6 verifier cases pass. Each rejected case is a behavior you decided not to pay for: two answers, a hedge, no answer line, a wrong answer. Write these before you run any policy. A policy shows you the cheats it found; hostile cases show you the ones you thought of.

The strict version still has limits. It trusts the first line that starts with ANSWER:, accepts a comma-grouped number, and says nothing about whether the work shown above it is correct. Real environments need a decision for each of these, and the decision is a design choice, not a detail.

What this does not show

A scripted policy isn't a trained model. The hacking policy here is blunt on purpose. Real reward hacking is subtler and often found by a model in a pass over thousands of tasks, which is why audits read transcripts as well as scores. Our toy also has one task type, one verifier and no noise. Treat the exercise as practice in reading a verifier critically, not as a measure of anything.

Exercise

Break the strict verifier, then fix it.