# TOOLCALL-300 A conformance suite for the code between your model and your tool: the layer that has to turn what the model actually emitted into a call your schema will accept. 300 labelled cases of tool calls that do not match the declared schema — wrong tool name, missing required argument, undeclared arguments, `"3"` where an integer was declared, enum values that are nearly right, nested objects flattened, a scalar where an array was declared, `arguments` delivered as a JSON string, the same call emitted three times, a tool that was never declared, and a stream that stopped mid-call — each with the declared tools, ground truth, and a one-command scorer that exits 2 on a regression so it can gate CI. **50 of the 300 have no correct call in them at all.** On those, the only passing answer is a refusal. That is the number most parsers are not ready for. TOOLCALL-300 grades the **schema** layer. Getting JSON out of prose, code fences and trailing commas is a different job — every input here that is meant to parse, parses. **Free forever, public domain (CC0):** 30 cases (`sample30.jsonl`), the scorer (`score.py`), the generator (`generate.py`) and the reference normaliser (`toolshim.py`). Run it against your own code right now; the number it prints is about your code, not mine. ``` python3 score.py --corpus sample30.jsonl --adapter yourmodule:normalise python3 score.py --corpus sample30.jsonl --adapter naive # the control python3 score.py --spec # the grading contract python3 score.py --selftest # 22/22 ``` ## The control comes first `naive` is what you get if you trust the model: parse the output, pass it on. ``` TOOLCALL-300 adapter: naive cases 300 exact match 10 / 300 (3.3%) refused correctly 10 / 50 invented calls 40 <- calls returned where there was none to make false refusals 50 <- gave up on a call that was repairable schema-invalid returns 240 <- returned a call the declared schema still rejects ``` Every point it scores comes from output so broken it would not parse at all. It passes **0 of 250** repairable cases, and on the 50 cases where the honest answer is "there is no call here" it hands your server a call **40 times** — including all 25 calls to a tool that was never declared. That is the shape of the problem. ## What a normaliser written from the spec scores `toolshim.py` (included, CC0) implements the grading spec. One run, no tuning afterwards. ``` TOOLCALL-300 adapter: toolshim:normalise cases 300 exact match 293 / 300 (97.7%) refused correctly 49 / 50 invented calls 1 false refusals 0 schema-invalid returns 0 ``` | category | n | naive | toolshim | |---|---|---|---| | wrong_tool_name | 25 | 0 | 25 | | missing_required_arg | 25 | 0 | 25 | | extra_undeclared_arg | 25 | 0 | 25 | | type_coercion | 25 | 0 | 25 | | enum_violation | 25 | 0 | 25 | | nested_flattened | 25 | 0 | 25 | | array_vs_scalar | 25 | 0 | 25 | | args_as_string | 25 | 0 | 25 | | multiple_calls | 25 | 0 | 25 | | hallucinated_tool | 25 | 0 | 25 | | **truncated** | 25 | 0 | **19** | | **unrecoverable** | 25 | 10 | **24** | **That 97.7% is an in-sample number and is worthless as a claim.** `toolshim.py` and this corpus were written by the same author from the same rulebook, so the figure measures agreement with the rulebook and nothing else. It is published because hiding it would be worse, and because the interesting part is that a normaliser written from the spec *still* failed 7 cases. Your own number, on code that never met this rulebook, is the one worth having — and it is the one the free 30 cases will give you in about a minute. ## The seven failures, named and left unfixed Repairing them after seeing the score would turn a measurement into a claim, so they stay. - **4 cases — an empty container invented from an open bracket** (`tc300-0254`, `0266`, `0267`, `0274`). The stream stopped at `"tags": [` or inside the first element. `toolshim` dropped the incomplete element and closed the array, emitting `"tags": []`. The model never wrote a tag. An empty array is not "no value", it is a value, and a server told to clear a field will clear it. - **2 cases — a complete element kept out of an unclosed array** (`tc300-0258`, `0273`): `"tags": [ "regression"` became `"tags": ["regression"]`. This one is a real ambiguity in rule 11 and it is worth stating plainly: the corpus takes the strict reading — a property whose container was never closed was not completely written, so it is dropped — and `toolshim` takes the generous one. Both are defensible. The corpus is consistent about it across all 25 truncated cases, and the label was fixed before anything was scored. If you disagree with that reading, you now know exactly which 2 of 300 cases you are disagreeing with, which is more than most suites will tell you. - **1 case — a number read as complete when it was cut mid-digits** (`tc300-0288`). `"days": 1` was the tail of a truncated stream. The original could have been 1, 12 or 14; the digits after the cut are gone. `toolshim` returned `days: 1` and produced a call the schema happily accepts. This is the worst failure in the set, because nothing downstream can detect it: the call is valid, plausible and wrong. ## Grading spec (summary — `python3 score.py --spec` prints the whole contract) 1. `expected_kind: "value"` — pass only by returning exactly that call, compared as `json.dumps(v, sort_keys=True, separators=(",",":"))`. Extra top-level keys on the returned object (an id, a type) are ignored; `name` and `arguments` are not. 2. `expected_kind: "unrecoverable"` — pass only by **refusing**. Returning `{"name": ..., "arguments": {}}` fails. Turning "the model produced no usable call" into "the model called a tool with no arguments" is the failure this suite exists to measure. 3. **Tool identity:** namespace prefixes, surrounding whitespace, a trailing `()`, letter case, and `-` `_` and space as separators are noise. A name resolves only if it matches exactly one declared tool after that normalisation. Otherwise, refuse. 4. **Enums** are matched the same way — trim, case-fold, `-` and space as `_`. Matches none: refuse. Never the closest-looking member. 5. **Coercion** only where it is lossless and reversible: `"3"`→`3`, `"12.5"`→`12.5`, `3.0`→`3`, `"true"`/`"True"`→`true`. Anything else: refuse. 6. **A missing required property is filled from the schema's own `default` and from nowhere else.** No default: refuse. 7. Undeclared properties are dropped. A flattened nested object is re-nested only when each key belongs to exactly one declared nested property and to no top-level one. 8. array-vs-scalar is repaired in the one-element direction only. 9. `arguments` as a JSON string is decoded, repeatedly if it was encoded more than once. 10. Several calls collapse to one **only** if every copy canonicalises to the same value. 11. Truncation: keep what was completely written, drop the incomplete tail, close the open containers, invent nothing. If that leaves a required property missing with no default: refuse. 12. No case expects `null`, so `None` is an unambiguous refusal signal. ## Adapter protocol - `--adapter module:callable` — takes `(text, tools)`, returns `{"name", "arguments"}`, or returns `None` / raises to refuse. - `--adapter-cmd "..."` — subprocess: `{"text": ..., "tools": [...]}` on stdin, the call on stdout; non-zero exit or empty stdout is a refusal. - `--adapter naive` — the control. - `--baseline base.json` — exit **2** if exact matches fell, if any category fell, if invented calls rose, or if schema-invalid returns rose. That is the CI gate. ## Provenance Every case is synthesised by `generate.py` from declared tool schemas and named mutation rules, and the ground truth is produced **by construction**: the correct call is built from the schema first, and the malformed text is derived from it. No parser, validator or model is ever asked what the answer is, so the labels cannot inherit anyone's bug. Nothing is scraped; none of it came out of anyone's production traffic or user data. The generator is deterministic — same seed, byte-identical corpus — so the numbers above can be reproduced and the corpus can be checked for quiet edits after the fact: ``` python3 generate.py --out . --seed 20260819 # rebuilds the corpus python3 generate.py --check-only --out . # re-runs every integrity check python3 generate.py --list-checks # prints all 23 classes of check ``` `toolcall300.jsonl` sha256 `96f7cfb4d85844a0bd25e8f403260ff6b6e4e793df62b30ec8cfcb95157e748a` · `sample30.jsonl` sha256 `464eebacfb8168a3a328ef3126848c5847244ea29f553a722dcc3c831943c2bc` **The generator refuses to write a corpus that would flatter a parser.** 23 classes of integrity assertion block the write; the last build ran **2730** of them with 0 failures. They include: exactly 25 cases per category · all 300 ids and all 300 input texts unique · every expected call names a declared tool and validates against its schema · every mangled name resolves to exactly one tool and every hallucinated one to none · every enum violation maps to exactly one member and every unrecoverable enum value to none · every truncated input genuinely fails to parse · no undeclared key is also the name of a nested property (or "drop it" and "re-nest it" would both be defensible) · and, the one that matters most, **no recoverable case is already schema-valid as sent.** A suite with freebies in it inflates every score ever run on it. ## The other 270 `sample30.jsonl` is a stratified slice of the same corpus — 2 or 3 cases from each of the 12 categories. The full 300 cases with the label rationale for each one — why that ground truth and not another, written per case — are the paid product: **€29 single developer · €99 team/CI licence.** The scorer, the generator, the reference normaliser and the 30 free cases stay CC0 forever whether you buy or not. ## Licence `score.py`, `generate.py`, `toolshim.py`, `sample30.jsonl` and this README: **CC0 1.0 Universal** — public domain, no attribution required, no warranty. The 270 remaining cases and their rationale are licensed per purchase (see `LICENSE-COMMERCIAL.txt`).