# Toolkit Labs > Small, dependency-free Python tools, labelled test corpora and reproducible parser benchmarks for engineers shipping LLM-based agents. Every artefact ships with its own benchmark and publishes the number it scored. Standard library only, Python 3.8+, no install and no network calls. Free artefacts are CC0 1.0 (public domain, no attribution required); one paid dataset is licensed per seat. Numbers below are the raw output of the command named beside them, re-run when this file was written. In-sample numbers are labelled and are not performance claims. A machine-readable version of this page, with per-artefact metrics, known failures and sha256 sums, is at /.well-known/toolkit-catalog.json. ## Datasets - [MALFORMED-300 free tier, CC0](https://toolkitlabs.org/malformed300/malformed300-free.zip): zip of 30 stratified cases of malformed LLM JSON output with ground truth, plus the scorer. The stdlib control `json.loads` scores 8.3% exact match on the full 300 cases (`python3 score.py --corpus malformed300.jsonl --parser json`) - every point it scores comes from refusing everything, correct on the 25 unrecoverable cases and wrong on the other 275. - [MALFORMED-300 scorer, CC0](https://toolkitlabs.org/malformed300/score.py): grades any parser against the corpus, prints the grading contract with `--spec`, passes 14/14 hand-derived selftest checks, and exits 2 on a regression against a baseline so it can gate CI. - [MALFORMED-300 sample cases, CC0](https://toolkitlabs.org/malformed300/sample30.jsonl): the 30 free cases as JSONL, 2-3 from each of the 12 categories. - [MALFORMED-300 full corpus, paid](https://toolkitlabs.org/#malformed300): all 300 labelled cases with the rationale for each label. EUR 29 single developer, EUR 99 team/CI licence. `jsonshim` scores 94.0% exact match on it (`python3 score.py --corpus malformed300.jsonl --parser jsonshim:loads`), inventing a value on 5 of the 25 unrecoverable cases; those 5 are named in the README and left unfixed. Delivery is an unlisted download URL shown by Stripe after payment - it is shareable, and there is no per-buyer access control. Details and the buy links are on the product page. - [TOOLCALL-300 free tier, CC0](https://toolkitlabs.org/toolcall300/toolcall300-free.zip): zip of 30 stratified cases of model tool calls that do not match the declared schema, with ground truth, the scorer, the generator and the reference normaliser. The control `naive` - parse the model output and pass it on - scores 3.3% exact match on the full 300 cases (`python3 score.py --corpus toolcall300.jsonl --adapter naive`), hands back an invented call on 40 of the 50 cases that contain no call at all, and returns 240 calls the declared schema still rejects. - [TOOLCALL-300 scorer, CC0](https://toolkitlabs.org/toolcall300/score.py): grades any normaliser against the corpus, prints the grading contract with `--spec`, passes 22/22 hand-derived selftest checks, and exits 2 on a regression - fewer exact matches, a fallen category, more invented calls or more schema-invalid returns - so it can gate CI. - [TOOLCALL-300 sample cases, CC0](https://toolkitlabs.org/toolcall300/sample30.jsonl): the 30 free cases as JSONL, 2-3 from each of the 12 categories, rationale stripped. - [TOOLCALL-300 declared tool schemas, CC0](https://toolkitlabs.org/toolcall300/tools.json): the 10 tool schemas the corpus declares, regenerated from the corpus itself and checked case by case for consistency; every case also carries its own `tools` inline, so the scorer never needs this file. - [TOOLCALL-300 full corpus, paid](https://toolkitlabs.org/#toolcall300): all 300 labelled cases with the rationale for each label. EUR 29 single developer, EUR 99 team/CI licence. The free `toolshim.py` scores 97.7% exact match on it (`python3 score.py --corpus toolcall300.jsonl --adapter toolshim:normalise`) - that number is in-sample, since normaliser and corpus were built beside each other, and is not a performance claim; no held-out number for it exists. Its 7 failures are named in the README and left unfixed. Delivery is an unlisted download URL shown by Stripe after payment - it is shareable, and there is no per-buyer access control. - [Monthly add-on, paid](https://toolkitlabs.org/monthly/): 60 new JSON cases and 60 new tool-call cases each month, seeded from the month, scored in the same run against the frozen reference lines of both suites, with a changelog of what moved. EUR 9.00 per month. Delivery is one unlisted page that gains an issue each month; cancelling stops future issues and does not revoke what was already downloaded. - [MALFORMED-300 product page](https://toolkitlabs.org/malformed300/): the corpus directory - what the 300 cases cover, the stdlib control number, the free files with their byte sizes, and the buy links for the paid 270. - [TOOLCALL-300 product page](https://toolkitlabs.org/toolcall300/): the same for the tool-call suite - the naive control, the reference normaliser's in-sample score, the free files, and the buy links. ## Tools - [agentguard, CC0](https://toolkitlabs.org/tools/agentguard/agentguard.zip): prompt-injection screening for untrusted text entering an LLM agent. Held-out F1 0.400 on 12 never-tuned-on cases (`python3 bench.py holdout.jsonl`: TP 2 FP 2 TN 4 FN 4) - it caught 2 of 6 injections and fired on 2 of 6 benign strings. Its F1 1.000 on `corpus.jsonl` is in-sample, because the rules were tuned against that file, and is worthless as a claim. A `pass` verdict means nothing obvious, never safe. - [jsonshim, CC0](https://toolkitlabs.org/tools/jsonshim/jsonshim.zip): pulls the JSON out of a model reply, reports every repair it made, and returns an error with no value when it cannot recover one. 94.0% exact match on MALFORMED-300, 37/37 selftest cases pass, and it still invents a value on 5 unrecoverable cases - documented, not fixed. - [evaldiff, CC0](https://toolkitlabs.org/tools/evaldiff/evaldiff.zip): tells you whether the difference between two eval runs is real or noise - Wilson intervals, McNemar's exact test, seeded bootstrap, and the ids that flipped. No held-out benchmark exists for it; `python3 evaldiff.py --selftest` checks it against hand-derived statistics and prints SELFTEST PASS. - [toolshim, CC0](https://toolkitlabs.org/toolcall300/toolcall300-free.zip): the reference tool-call normaliser, shipped inside the TOOLCALL-300 free zip. Repairs the declared-schema mismatches the suite covers and refuses instead of inventing a call. 293/300 on TOOLCALL-300 with 0 false refusals and 1 invented call - in-sample, worthless as a claim, and it still fails 6 truncated cases. - [jsonshim-mcp, CC0](https://toolkitlabs.org/tools/jsonshim-mcp/jsonshim_mcp.py): a one-file MCP stdio server exposing `recover_json` and `classify_json_failure` to an agent, standard library only, no install and no network call. Scored on the same 300 cases as the leaderboard: 218/300 exact, 193/275 recoverable, **0 of 25 unrecoverable cases invented**, 55 false refusals; truncation and raw control characters are its documented blind spots at 0/25 each. - [Tools index](https://toolkitlabs.org/tools/): the four free CC0 tools on one page with what each scored, including where each loses to a library published by somebody else. ## Docs - [MALFORMED-300 README](https://toolkitlabs.org/malformed300/README.md): what the suite covers, the grading spec, the provenance of every case, and the failures it found. - [agentguard README](https://toolkitlabs.org/tools/agentguard/README.md): the rule set, the normaliser, and the held-out misses listed one by one. - [jsonshim README](https://toolkitlabs.org/tools/jsonshim/README.md): what it repairs, what it refuses, and the API. - [evaldiff README](https://toolkitlabs.org/tools/evaldiff/README.md): why McNemar's exact test and not the chi-square approximation. - [Machine-readable catalog](https://toolkitlabs.org/.well-known/toolkit-catalog.json): one JSON object per artefact with licence, price, download URL, metric, known failures and sha256. - [TOOLCALL-300 README](https://toolkitlabs.org/toolcall300/README.md): the 12 categories, the grading spec, the provenance of every case, and the seven failures of the reference normaliser named one by one. - [jsonshim-mcp page](https://toolkitlabs.org/tools/jsonshim-mcp/): the MCP server's install snippet, its per-category recovery table, and the two categories it recovers none of. ## Benchmarks - [The JSON recovery leaderboard](https://toolkitlabs.org/leaderboard/): eleven Python and ten JavaScript JSON parsers measured on all 300 MALFORMED-300 cases, one run each, nothing tuned afterwards. Best Python result 265/300 exact (`json-repair` 3.15.0) against 282/300 for `jsonshim`, which leads only by refusing 20 of the 25 unrecoverable cases where `json-repair` refuses 1; on the 275 recoverable cases `json-repair` recovers 264 and `jsonshim` 262, so the tool published here loses that column. Best JavaScript result 205/300 and 196/275 (`jsonrepair` 3.15.0, npm, a different project from the PyPI one); the weakest, `untruncate-json` 0.0.1, recovers 25 of 275 because closing a truncated document is all it claims to do. - [Truncated / streaming JSON, one category read properly](https://toolkitlabs.org/leaderboard/truncated/): the 21 libraries on the 25 truncated cases of MALFORMED-300, split three ways per parser - recovered exactly, refused, or returned a wrong value without refusing. Best Python 23/25 (`json-repair` 0.63.2, tied with `jsonshim`), best JavaScript 20/25 (`jsonc-parser` 3.3.1). 8 Python and 4 JavaScript parsers recover none of them, which is dialect parsers refusing a document nobody closed rather than guessing; across both languages a wrong value came back 43 times, most often from `dirty-json` at 15 of 25. Derived counts published as [truncated.json](https://toolkitlabs.org/leaderboard/truncated/truncated.json), cross-checked against both raw runs. - [In-browser JSON recovery tool](https://toolkitlabs.org/fix/): paste one malformed model reply and see what `jsonrepair` 3.15.0, `JSON5` 2.2.3 and `JSON.parse` each return for it, plus which of the 12 failure categories the text matches. Static page, both libraries vendored unmodified, no network call and no upload. - [Leaderboard raw run, Python, CC0](https://toolkitlabs.org/leaderboard/leaderboard.json): every number on the leaderboard page comes out of this file - per-parser totals, per-category counts, invented values, false refusals, library versions and the corpus sha256. - [Leaderboard raw run, JavaScript, CC0](https://toolkitlabs.org/leaderboard/leaderboard_js.json): the same fields for the Node run, scored against the identical corpus sha256. - [Leaderboard harness, Python, CC0](https://toolkitlabs.org/leaderboard/run_leaderboard.py): the runner, with [adapters.py](https://toolkitlabs.org/leaderboard/adapters.py) holding one short adapter per library so the fairness of each call is checkable in a minute. - [Leaderboard harness, JavaScript, CC0](https://toolkitlabs.org/leaderboard/run_js.mjs): the whole Node harness in one file, same grading contract, canonical comparison done inside its own language. - [JSON inside a code fence or prose](https://toolkitlabs.org/leaderboard/fenced/): the 21 measured parsers on the 50 `fenced` and `prose_wrapped` cases of MALFORMED-300, split per parser into recovered, refused and wrong-value-returned. Both Python libraries that extract before parsing take 25/25 and 25/25; 9 of the 11 Python parsers recover none of the 50 and refuse every one of them, because a code fence is not a looser grammar. Best JavaScript: `jsonc-parser` 18/25 fenced and 8/25 prose. Derived counts published as [fenced.json](https://toolkitlabs.org/leaderboard/fenced/fenced.json). - [The 11 Python parsers head to head](https://toolkitlabs.org/leaderboard/python/): every Python library of the run on all 300 cases, with the 275 recoverable ones, the 25 unrecoverable ones it correctly refused, the values it invented instead and its false refusals. Top of the table `jsonshim` 282/300, then `json-repair` 265/300; the remaining eight sit between 25 and 123 because they are dialect parsers. Derived table published as [python.json](https://toolkitlabs.org/leaderboard/python/python.json). - [json-repair vs jsonrepair](https://toolkitlabs.org/leaderboard/name-collision/): the Python and the JavaScript library with the confusable names, category by category on one corpus - 265/300 and 205/300 overall. Caveat stated on the page: the runtime is not a controlled variable (Python 3.12.3 vs node v25.8.2); the corpus, its sha256 and the grading rule are. The JavaScript one returns repaired text, so the harness parses its output before comparing values. Derived table published as [name-collision.json](https://toolkitlabs.org/leaderboard/name-collision/name-collision.json). - [MALFORMED-300 conformance API](https://toolkitlabs.org/api/): the leaderboard as machine-readable JSON, CC0-1.0, no account and no attribution required. [leaderboard.json](https://toolkitlabs.org/api/leaderboard.json) carries all 21 parsers (11 Python, 10 JavaScript) with per-category summary counts, the corpus sha256 `ded36c275f9ff481a9e264f9e900ccbb16da9cf430ff6fe8ffc23b9c02acb88a` and the runtime versions; [schema.json](https://toolkitlabs.org/api/schema.json) documents the fields. A shields.io endpoint per parser lives at `https://toolkitlabs.org/api/badge/{language}/{slug}.json`, e.g. `badge/python/jsonshim.json` (282/300) and `badge/javascript/jsonrepair.json` (205/300). Summaries only: per-case rows are not published. Nobody pays to be listed, ranked or removed. ## Conformance API - [schema.json](https://toolkitlabs.org/api/schema.json): field documentation for leaderboard.json — what `exact`, `invented_values`, `false_refusals` and `by_category` mean, and the badge contract. CC0-1.0, no account, no attribution required. - Badge endpoints: `https://toolkitlabs.org/api/badge/{language}/{slug}.json`, 22 files (one per scored parser plus `corpus.json`), each a shields.io endpoint object with `schemaVersion: 1`. Render one with `https://img.shields.io/endpoint?url=https%3A%2F%2Ftoolkitlabs.org%2Fapi%2Fbadge%2Fpython%2Fjsonshim.json`. Free, no signup; placement is never for sale and no ranking can be bought. - [The human leaderboard page](https://toolkitlabs.org/leaderboard/) carries a schema.org `Dataset` block in `application/ld+json` naming the CC0 licence, the corpus sha256 `ded36c275f9ff481a9e264f9e900ccbb16da9cf430ff6fe8ffc23b9c02acb88a` and both JSON distributions, so a dataset crawler can take the results without parsing the tables. ## Optional - [Site index](https://toolkitlabs.org/): the same catalog as a web page, with the purchase links and the seller of record. - [MALFORMED-300 CC0 licence text](https://toolkitlabs.org/malformed300/LICENSE-CC0.txt): the public domain dedication covering the free tier. ## Services - [Parser audit](https://toolkitlabs.org/audit/): paid, one-off. We run your JSON or tool-call parser over MALFORMED-300 with the harness that produced the leaderboard and publish a private report: score per failure category, every case id it fails, whether it invents values on the 25 unrecoverable cases, and where it lands against the parsers already measured.