Preview an Agent Change Without Handing It Production Keys

The preview that looked isolated

You maintain an agent that manages evaluation jobs for your team. It reads the nightly eval results, and when a case looks flaky it submits a re-run job to the shared GPU cluster through the scheduler's API. You change the agent's prompt and a tool, push a branch, and your platform builds a preview at an address like pr-482-eval-agent.example.dev. It has its own URL, its own logs and its own deployment ID. You point a test script at it and ask it to handle a flaky case.

Before you run that test: does a different hostname mean the preview cannot submit a real job?

Check the credentials first. In LangChain's description of LangSmith preview builds, parent secrets are copied into a preview at creation unless overridden. That is a documented default for one product; check your own platform rather than assuming it behaves the same. If the preview received the production scheduler token, it can submit jobs to the shared cluster even though its URL is temporary.

The cost is easy to compute. At an illustrative price of 2.50 USD per GPU-hour, not a quote from any provider, one job on 8 GPUs for 3 hours costs 8 × 3 × 2.50 = $60. A test loop that submits ten of them costs $600, and it occupies GPUs a colleague's training run was waiting for.

Why the hostname is a distraction

The intuition comes from places where it holds: a staging database is usually a different database, and a test endpoint is usually a different endpoint. A credential is not scoped by URL. In this example the scheduler token is a bearer credential with permission to submit jobs and no network restriction, so any request carrying it acts as its account. Real services may add restrictions; inspect the permissions and network rules yours actually enforces. If the preview's code reads SCHEDULER_TOKEN from its environment and that variable holds the production value, the scheduler cannot tell the preview from production.

Start with an inventory for the preview: destination URL, credential identity, allowed operations, and outbound network rules. Each entry answers a different question about what happens after a request arrives.

View full-size diagram

The branch and hostname sit at A and B. What decides whether a real job runs sits at E and F, and nothing about the branch name touches it. To interrupt the chain safely, interrupt it at E.

Test the denial path locally

Use the scheduler's sandbox, or replace it with a local test double while you develop the denial path. The standard-library test below checks how the application handles a denied action. Both counters belong to the mock; neither observes a real service.

import unittest


class MockScheduler:
    """Stands in for a real job scheduler during a preview test."""

    def __init__(self):
        self.validation_calls = 0
        self.real_submissions = 0  # accepted submissions inside this mock only

    def validate_job(self, job):
        self.validation_calls += 1
        return job.get("eval_case") is not None

    def submit_job(self, job, credential):
        if credential != "PRODUCTION_ONLY_TOKEN":
            raise PermissionError("denied: preview credential cannot submit jobs")
        self.real_submissions += 1
        return {"status": "submitted"}


class PreviewAgent:
    def __init__(self, scheduler, credential):
        self.scheduler = scheduler
        self.credential = credential

    def rerun_flaky_case(self, job):
        if not self.scheduler.validate_job(job):
            return "invalid"
        try:
            self.scheduler.submit_job(job, self.credential)
            return "submitted"
        except PermissionError:
            return "denied"


class PreviewIsolationTest(unittest.TestCase):
    def test_preview_credential_cannot_submit_job(self):
        scheduler = MockScheduler()
        agent = PreviewAgent(scheduler, credential="PREVIEW_SCOPED_TOKEN")
        result = agent.rerun_flaky_case({"eval_case": "c-417", "gpus": 8})

        self.assertEqual(result, "denied")
        self.assertEqual(scheduler.validation_calls, 1)
        self.assertEqual(scheduler.real_submissions, 0)


if __name__ == "__main__":
    unittest.main(argv=["preview-test"], exit=False)

Save it as test_preview.py and run python test_preview.py, or paste it into a notebook cell. The expected result is one passing test. rerun_flaky_case validates the job first, so validation_calls becomes 1. Then it tries submit_job with the credential it was given. The test built the agent with PREVIEW_SCOPED_TOKEN, so the mock raises PermissionError, the agent returns denied, and real_submissions stays at 0.

That zero describes the mock only. It cannot tell you which token a deployed process holds, whether another code path sends HTTP requests, or whether a real scheduler accepted a job. The toy's value is that it fixes the shape of the evidence before you wire up a real deployment: a validation count above zero and a real-submission count of zero for any forbidden action.

Keep three kinds of evidence apart

Use the scheduler's sandbox or a dedicated test queue with a zero GPU quota for integration tests. Do not submit a live job just to see whether the preview can. If the service offers no safe validation operation, set the restriction through its administrative controls and write down the remaining uncertainty where a reviewer will see it.

Record what you verified

A preview test should leave a small record a reviewer can check, not just a green line in a terminal. Here is the record for a labeled hypothetical run:

The two SHA fields exist because a preview that deploys a different commit from the one reviewed defeats review, whatever its credentials. The credential owner field exists because "preview" is not a security boundary; a named, scoped service account is. The scheduler record bounds the observation to one run and one window. Absence of a job in one test does not prove no future request can succeed, so pair it with the credential policy.

Record the service's revocation behavior with the rotation test. Some systems take time to stop accepting an old credential, and a test run inside that window answers a different question. Keep credential values out of the report; record an identity or version that cannot authorize anything.

Check the decision

Why this matters in a lab

In a research or applied team, agents hold real authority: they submit jobs to shared clusters, write to results tables and model registries, and open pull requests. Every change to such an agent is a release of its model, prompt, tools and configuration, and preview is the first place that release runs outside a laptop. A team will expect you to test it there without letting it spend GPU hours or overwrite results. An interviewer might ask, "How do you test a change to an agent that can launch training jobs?" A strong answer separates the local denial test from the deployed credential's scope and the receiving service's log, names what each proves, and retires the credential afterward. The next stages are shadow runs and rollback, and Chapter 19 covers where credentials belong in a deployed agent.

What would prove this wrong

If the mock does not mirror the real service's authentication, for example if the real scheduler accepts a broader set of tokens than the mock checks, a passing test proves nothing about the live system. If teardown fails silently and the scoped token lingers, the isolation you verified no longer describes the running preview an hour later. And if the preview never deploys, as in the blocked case above, the failure tells you about deployment permissions, not about whether secrets would have been isolated.