Validate Tool Arguments in Code, Even With Strict Mode
A model's tool call is a request for your code to act. OpenAI's function calling guide lays out the flow: the model returns a tool call, and you "execute code on the application side with input from the tool call." Everything that decides whether the action is allowed belongs in that code.
Strict mode helps with shape. With strict: true, the guide requires additionalProperties: false on each object and every field marked required, and it recommends enums "to prevent invalid states." A schema can guarantee that split is one of the strings you listed. It cannot know that this is a development run that must never read the test split.
That second rule is the one that protects a research result. An agent that can read held-out data while you tune will leak it into your choices, and the number you report stops meaning what you say it means.
The check. Take a tool such as load_examples(dataset, split, limit). In the tool implementation, not in the prompt, enforce three things: the dataset is on an allow-list, split is train or dev unless the run is flagged as the final evaluation, and limit has an upper bound. Then write four tests that call the tool function directly:
- a missing argument;
- a malformed value, such as
limit: -5; split: "test"during a development run;- a valid call.
Expected: the first three return a contained error message the model can read, and none of them touches the data. The fourth returns rows. Calling the function directly matters: it tests your code, not the model's tendency to send clean arguments.
Because the rules live in code, they survive a changed prompt, a different model and a second caller that never went through the schema. Structured output and typed tools covers what a schema guarantees and what it leaves to you.
