Give a Research Agent's Tool Failures a Shape It Can Act On

Your research agent has a tool that fetches an eval run from the experiment tracker. When the call fails, the model sees whatever string you return. "Something went wrong" leaves it to guess, and a guessing agent tends to retry without end or fill in plausible numbers.

OpenAI's function-calling guide sets the division of labor. Your application, not the model, executes the function, then returns a function_call_output carrying the call_id and an output string. That string can hold JSON, plain text or error codes. So give failures a shape:

{"error": {"code": "not_found", "retryable": false,
           "ask_user_for": "a run id from the current project"}}

Four codes cover most tracker calls: not_found (bad run id), forbidden (no access to that project), unavailable (the service is down) and invalid_argument (a malformed filter). Keep tokens, internal hostnames and other users' data out of the message.

Keep the retry policy in code. If a transient failure happens with probability q and attempts are independent, three tries all fail with probability q³, so q = 0.1 becomes 0.001. An outage is not independent, though: every retry fails together. Cap the attempts in your application and record the final outcome, instead of asking the model to decide when to stop.

Test each shape with a fake tool. Make a stub that returns each code in turn, and for each one count the tool calls and read the final reply:

• not_found should produce a request for the right run id;
• forbidden should stop without retrying;
• unavailable should retry up to your cap, then say the tracker is down;
• invalid_argument should produce one corrected call.

Chapter 2 builds typed tools that validate their own inputs, and chapter 12 derives what retries cost when you cannot tell whether a call went through.

OpenAI documentation.