Measure Whether One Tool Schema Works Across Model Providers

Same schema, different behavior

You have an agent that records benchmark results into your team's comparison table through one tool, record_result. Its arguments are a benchmark name, a metric, a numeric value and a source reference. It works on one provider. A teammate wants to try another, for cost or latency, and says, "Both take JSON Schema for tools, so switching is a config change."

Half of that is true. The schema itself travels. The book's Chapter 14 shows the conversion: an MCP tool's inputSchema becomes the function's parameters unchanged, and on Claude's API the same schema goes under input_schema. What does not travel is how each model uses the tool: which tool it picks, what argument values it produces, when it declines to call a tool at all, and what the tool text costs in tokens.

So portability is not a property you can read off the schema. It is a claim you test with an experiment.

What Chapter 14 measured

Chapter 14 offered the same tools, in the same function-calling shape, to three small local models through Ollama and to Claude Haiku 4.5 and Claude Sonnet 5.5 through the Messages API, with 24 requests per condition. Three of its findings bear on portability.

The same tool costs different tokens. Each native tool added about 39 prompt tokens on qwen2.5, about 55 on Claude Haiku 4.5 and about 74 on Claude Sonnet 5.5, whose newer tokenizer splits text more finely. For a 20-tool catalog, that is roughly 780, 1,100 and 1,480 tokens before the request says anything. Your per-attempt cost, and your prefill latency, depend on the provider even when the schema is identical.

The same catalog routes differently. With 64 tools, qwen2.5:1.5b routed 14 of 24 requests to the right tool, while both Claude models held between 22 and 24 of 24 at every catalog size. The schema was the same. The behavior was not.

Every model sometimes answers in words instead of calling a tool. Across 216 native answers per model, it happened between 5 and 33 times. In an agent loop, a text answer ends the turn, so a declined call looks like a reply, not an error.

The chapter also notes a practical difference: Claude Sonnet 5.5 accepted no temperature setting in that run, so "same sampling configuration" was not available across all five models. A portability experiment has to record such differences rather than assume them away.

Keep one strict contract, and thin adapters

Put the stable part in one place: a strict contract for a result row. The book's Chapter 2 builds this kind of validator with Pydantic, configured with extra="forbid" and strict=True, so a numeric string like "0.713" is rejected rather than converted and an unexpected key is an error rather than silently dropped.

On each side of that contract sits a provider adapter: code that turns the provider's response into a plain dictionary for the contract to validate. The adapter is supposed to be provider-specific. It must not make decisions for the contract.

Here is a designed teaching case, not a run against any live API. Four raw payloads go through two adapters that make different choices:

Rows two and three show the problem. Adapter A's coercion means its pipeline never exercises the rejection path Adapter B's does, so the two providers look equally clean while one of them sends strings. Adapter A's dropped field was a model asserting that it verified the number, a claim the application should see and reject. Row four is worse: an adapter that turns a refusal into an empty success destroys the one signal that says nothing was recorded.

A runnable version of the table, standard library only, makes the rule testable:

REQUIRED = {"benchmark": str, "metric": str, "value": float, "source_ref": str}


def strict_contract(row):
    if not isinstance(row, dict) or set(row) != set(REQUIRED):
        return "rejected"
    for key, kind in REQUIRED.items():
        if type(row[key]) is not kind:
            return "rejected"
    return "accepted"


def adapter_a(raw):  # coerces and drops, which hides information
    if raw.get("refused"):
        return "provider_refused"
    row = {k: raw[k] for k in REQUIRED if k in raw}
    if isinstance(row.get("value"), str):
        row["value"] = float(row["value"])
    return row


def adapter_b(raw):  # passes values through unchanged, but hides refusals
    if raw.get("refused"):
        return {}
    return dict(raw)


base = {"benchmark": "long-context-qa", "metric": "accuracy", "source_ref": "Table 3"}
cases = {
    "clean number": {**base, "value": 0.713},
    "numeric string": {**base, "value": "0.713"},
    "extra field": {**base, "value": 0.713, "verified": True},
    "refused": {"refused": True},
}
for name, raw in cases.items():
    verdicts = []
    for adapter in (adapter_a, adapter_b):
        out = adapter(raw)
        verdicts.append(out if isinstance(out, str) else strict_contract(out))
    print(f"{name:15} A: {verdicts[0]:17} B: {verdicts[1]}")

It prints accepted for both on the clean number, accepted against rejected for the numeric string and the extra field, and provider_refused against rejected for the refusal. Correct adapters keep the raw value, surface refusals as their own outcome, and leave every accept-or-reject decision to the contract.

View full-size diagram

View full-size diagram

Three failure kinds an adapter must keep apart

These three codes are categories your application defines; no provider guarantees those names. Before writing a real adapter, map each provider's actual error fields and exception types to them from its current documentation.

The portability experiment

Here is the protocol. It is written for you to run; this article reports no results from it.

Size the experiment honestly. Chapter 14 used 24 requests per condition, and notes that a 95% Wilson interval around 0.75 then runs from 0.55 to 0.88: differences smaller than about 0.2 are within the noise of one run. To compare two providers that are both near 0.9, you need hundreds of requests, not dozens. Chapter 16's table gives the numbers.

For the related question of where state lives when the tool is served over MCP, see state and identity behind a stateless MCP server. For turning per-provider acceptance rates into a routing decision, see routing by cost per accepted outcome.

Check the decision

Why this matters in a lab

Research and applied teams move between models constantly: a new internal checkpoint, an external provider for comparison, a smaller model for cost. Tool-using agents are only as portable as their tool contract, and the contract is only as portable as the evidence behind it. A team will expect you to separate what the schema guarantees from what a model does with it, and to turn "it should work on the other provider" into a paired experiment with intervals. An interviewer might ask, "We want to evaluate our agent on three model providers. How do you make the comparison fair?" A strong answer freezes the schema and requests, records configuration differences that cannot be matched, keeps adapters from hiding information, and reports each rate with its interval.

What would prove this wrong

If two providers produce the same tool choices, argument values and refusal behavior on a large, varied request set, the adapters are thin, and the token costs are acceptable, the tool is portable for that request set, and the claim should be stated that narrowly. If your numeric rules change with the provider, as in if provider == "b": allow a string value, the provider has leaked into the contract. And providers change their APIs and documentation on their own schedule: this article's account reflects the book's recorded runs and pages checked for it, and a migration decision should recheck both.