AI Agent Preview Deployment: Test Without Real Side Effects

The preview that looked isolated but wasn't

Suppose you build an agent that places purchase orders through a supplier's API. You cut a branch, push it, and your deployment platform spins up a preview at some address like pr-482-yourapp.vercel.app. It looks separate. It has its own URL, its own build logs, its own deployment ID. You point your test script at it, send a fake order, and watch to see what happens.

Here is the question worth sitting with before you run that test: does a different hostname mean the preview cannot place a real order?

Check the credentials before sending the order. In LangChain's description of LangSmith preview builds, parent secrets are copied at creation unless overridden. That is a documented default for this product; check your own platform's configuration rather than assuming it behaves the same way. If the preview receives a production supplier key, it may have permission to place orders even though its URL is temporary.

The example below separates two questions: does the application handle a denied write correctly, and does the deployed preview actually lack permission to write? A local unit test can answer the first. The second requires checking the environment and the receiving service.

Why the hostname is a distraction

The mistake is understandable because it maps onto an intuition that works almost everywhere else in software. A staging database is usually a different database. A test API is usually a different endpoint. So when a URL changes, we expect the resources behind it to have changed too. But a credential is not scoped by URL. In this example, the supplier uses a bearer credential with order-creation permission and no network restriction. A request presenting it can act as its account. Real suppliers may impose additional restrictions, so inspect the permissions and network rules your supplier actually enforces. If your preview's code reads SUPPLIER_API_KEY from its environment and that variable holds the same value as production, the supplier's system cannot tell the difference between an order placed by your real checkout flow and an order placed by a test script hitting a pull-request preview. A branch label has no protective effect unless an enforcement rule uses it. Adding a preview header to the request also provides no protection if the supplier ignores that header.

Start with a short inventory: destination URL, credential identity, allowed operations and outbound network rules. A separate hostname is useful for routing test traffic. Each entry in that inventory answers a different question about what happens after the traffic arrives.

Look at where the branch and the hostname sit in that chain. They are early, at steps A and B. The thing that determines whether a real order gets placed sits at E and F, several steps downstream, and nothing about the branch name touches it. If you want to interrupt this chain safely, you have to interrupt it at E, not at A.

Testing AI agent preview deployment permissions

Use a supplier sandbox or replace the supplier with a local test double while developing the denial path. The following Python standard-library example checks how the application handles a denied action. Both counters belong to the mock. Neither observes a network service.

import unittest


class MockSupplier:
    """Stands in for a real supplier API during a preview test."""

    def __init__(self):
        self.validation_calls = 0
        self.real_writes = 0  # accepted writes inside this mock only

    def validate_draft(self, order):
        self.validation_calls += 1
        return order.get("sku") is not None

    def send_order(self, order, credential):
        if credential != "PRODUCTION_ONLY_KEY":
            raise PermissionError("denied: preview credential cannot place orders")
        self.real_writes += 1
        return {"status": "placed"}


class PreviewAgent:
    def __init__(self, supplier, credential):
        self.supplier = supplier
        self.credential = credential

    def run_preview_task(self, order):
        if not self.supplier.validate_draft(order):
            return "invalid"
        try:
            self.supplier.send_order(order, self.credential)
            return "placed"
        except PermissionError:
            return "denied"


class PreviewIsolationTest(unittest.TestCase):
    def test_preview_credential_cannot_place_order(self):
        supplier = MockSupplier()
        agent = PreviewAgent(supplier, credential="PREVIEW_SCOPED_KEY")
        result = agent.run_preview_task({"sku": "AX-100"})

        self.assertEqual(result, "denied")
        self.assertEqual(supplier.validation_calls, 1)
        self.assertEqual(supplier.real_writes, 0)


if __name__ == "__main__":
    unittest.main()

Save this as test_preview.py and run python -m unittest test_preview.py. The expected result is one passing test. Walk through why. PreviewAgent.run_preview_task always validates the draft first, so validation_calls becomes 1: the preview is allowed to check that an order is well-formed. Then it attempts send_order with whatever credential it was given. Because the test constructs the agent with credential="PREVIEW_SCOPED_KEY" rather than the string "PRODUCTION_ONLY_KEY", the mock supplier raises PermissionError, the agent catches it and returns "denied", and critically, real_writes stays at 0. That zero describes the mock only: its accepted-write branch was not reached in this test. It cannot tell us which keys a deployed process holds, whether another code path sends HTTP requests, or whether an external supplier accepted an order.

This is a toy in-memory example, worth naming as such directly. MockSupplier keeps its counters in ordinary Python attributes with no persistence, no network call and no concurrency guarantee; a production preview pipeline would need a real deployment, a real scoped credential system, and a way to inspect the actual supplier's logs, not just an in-process counter. The value of the toy version is that it makes the acceptance rule concrete before you spend time wiring up an actual cloud deployment: whatever your real preview does, it should produce the same shape of evidence, a validation count greater than zero and a real-write count that stays at zero for any forbidden action.

Separate the three acceptance checks

Use the supplier's sandbox for integration tests where possible. Do not attempt a live purchase merely to see whether a preview credential can complete it. If the supplier offers no safe validation operation, establish permission restrictions through its administrative controls and document the remaining behavioral uncertainty. A reviewer should be able to see that gap without reverse-engineering the test.

Recording what you actually verified

A useful preview test produces a small, checkable record, not just a pass or fail in your terminal. For the worked case above, that record should include the reviewed commit SHA, the SHA the preview actually deployed, who or what owns the credential the preview process holds, and the teardown result once the preview is retired. Here is why each column matters and what it would look like filled in for a labeled hypothetical run.

The reviewed SHA and preview SHA columns exist because a preview that silently deploys a different commit than the one a reviewer approved defeats the point of review entirely, independent of credentials. The credential owner column exists because "preview" is not itself a security boundary; a named, scoped service account is. The supplier audit record bounds the observation to a named run and time window. Pair it with the credential policy: absence of an order in one test does not prove that no future request can succeed.

Record the provider's revocation semantics alongside the rotation test. Some systems may take time to stop accepting an old credential. A test performed before that interval expires answers a different question from a test performed afterward. Keep credential values out of the report; record an identity or version identifier that cannot authorize a request.

Check the decision

What would break this argument

It is worth being precise about what would falsify the claim that a mock-and-counter approach solves the problem, because it does not solve everything. If your mock supplier does not faithfully mirror the real supplier's authentication behavior, for instance if the real API accepts a broader set of credentials than your mock checks for, then a passing test proves nothing about the live system. If your preview's teardown step fails silently and the fixture or scoped credential lingers past review, the isolation you verified at test time no longer describes the running preview an hour later. And if a deployment blocks entirely, the way the existing course record shows a Vercel preview blocked on commit-author permission, that failure tells you about your team's deployment permissions, not about whether secrets would have been isolated had the build succeeded. None of these are edge cases to wave away; they are the actual boundary of what "zero real writes in this test" is allowed to claim.

Before the next preview, name the forbidden operation and its receiving service. Test the application's denial path locally, then assign a separate deployment check for credential scope, network access and receiver-side activity. Keep those results in separate columns so a green unit test cannot accidentally stand in for permission evidence.