Toolkit Labs

Best JSON repair library for LLM output — 21 measured

When your LLM returned invalid JSON, the cause is usually mundane: a code fence around the object, a trailing comma inside it, single quotes instead of double, or a reply truncated mid-string. This page measures which repair library actually recovers that malformed output — 11 Python and 10 JavaScript parsers over 300 labelled cases, one run each, nothing tuned afterwards.

11 Python parsers and 10 JavaScript parsers, 300 labelled cases of malformed language-model output, one run each, nothing tuned afterwards. Measured 2026-08-19T12:31:18Z on Python 3.12.3. Corpus MALFORMED-300, sha256 ded36c275f9ff481a9e264f9e900ccbb16da9cf430ff6fe8ffc23b9c02acb88a.

One broken reply rather than three hundred? The in-browser tool runs two of these libraries and the strict parser on your own text, in the page, uploading nothing.

The JSON recovery leaderboard — Python results

Two columns, because these libraries do not share a design goal. All 300 grades by the suite’s spec: the 25 unrecoverable cases pass only by refusing, so returning {} where the model produced nothing is a failure. 275 recoverable drops the refusal question entirely and asks only what a library recovers — a tool designed never to refuse is not penalised there for its design.

What the numbers say

Two of the 11 Python libraries clear half marks. The rest are not bad at their job; they are doing a different job. json5, pyjson5, demjson3 and dirtyjson are dialect parsers — they read a looser grammar than strict JSON, so they handle comments, trailing commas, unquoted keys and single quotes, and they score zero on code fences, prose wrappers and truncation, because a model wrapping its answer in prose is not a JSON dialect. It is not malformed JSON at all until something finds the JSON inside it.

The honest headline is not the one that flatters the tool published here. On the 275 recoverable cases json-repair recovers more than jsonshim does. jsonshim leads the all-300 column on refusal policy alone: it declines 20 of the 25 unrecoverable cases, json-repair declines 1, because json-repair is built to always return something. If your pipeline can tell an empty object from an absent one, that difference costs you nothing. If it cannot, it is the whole ballgame — an invented value does not raise, does not log, and reaches your database looking exactly like data.

Nobody wins every category. Read the grid, not the total.

By failure category — Python

Cases correct out of 25 in each category. unrecoverable is the refusal column: 25 means the parser declined every case where the model produced nothing recoverable.

JavaScript / TypeScript

The same 300 cases, the same grading contract, 10 libraries from the Node ecosystem — because an agent that parses model output in TypeScript has exactly the same problem and a different set of libraries to solve it with. Measured 2026-08-19T12:33:03Z on node v25.8.2, same corpus, same sha256, one run each.

The gap between the two ecosystems on this corpus is the finding, and it is larger than expected: the best JavaScript result recovers 196 of the 275 recoverable cases, where the best Python result recovers 264. Note that jsonrepair (npm, josdejong) and json-repair (PyPI, mangiucugna) are different projects with similar names, not two builds of one thing — so this is a comparison of ecosystems, not of one library against itself.

Two results are cross-checks rather than findings, and they are the reason to trust the rest of the grid. The stdlib control scores identically in both languages — JSON.parse 25/300 and 0/275, json.loads 25/300 and 0/275 — and the two independent implementations of the JSON5 grammar also land on the same pair of numbers, 118/300 and 93/275. The harnesses were written separately and compare canonical values inside their own language, so agreement there says the grading, not the language, is doing the work.

cd js && npm install
node run_js.mjs jsonrepair ../../malformed300/malformed300.jsonl .

leaderboard_js.json — the raw JavaScript run · run_js.mjs, the whole harness, one file

Same format, two implementations

Totals can agree by accident; per-case verdicts cannot. Two of these libraries exist in both languages under the same format name, so the harnesses were asked a narrower question: case by case, do the two implementations reach the same verdict?

JSON5 — 300 of 300 identical. json5 0.15.0 (PyPI) and json5 2.2.3 (npm) do not merely land on the same totals; they agree on every single case, in both directions. Two independent implementations of one written grammar, behaving as one thing.

Hjson — 282 of 300 identical, so 18 cases where the two disagree. hjson 3.1.0 (PyPI) passes 4 that hjson 3.2.2 (npm) fails (wrappers (4)), and the npm one passes 14 that the PyPI one fails (raw control (8), unquoted keys (3), comments (2), trailing comma (1)). Same format name, different behaviour, and the difference is concentrated where a format has to decide what to do with characters a spec left open. If your pipeline reads Hjson in one language and writes it in another, that gap is yours to know about.

Counts and categories only. Case ids and returned values are deliberately not published here: they would reconstruct ground truth that is part of the paid corpus. The comparison is reproducible from the row files each harness writes locally — crosslang.py is the whole of it.

Put the number in your README

Every parser in the tables above has a live badge endpoint. It is free, it is CC0, it needs no signup and no API key, and it changes only when the run does. Placement is never for sale: this page publishes measurements, it does not rank anyone for money, and there is nothing you can buy that moves a number on it.

Markdown — jsonshim (Toolkit Labs), 282/300:

[![MALFORMED-300](https://img.shields.io/endpoint?url=https%3A%2F%2Ftoolkitlabs.org%2Fapi%2Fbadge%2Fpython%2Fjsonshim.json)](https://toolkitlabs.org/leaderboard/)

HTML:

<a href="https://toolkitlabs.org/leaderboard/"><img alt="MALFORMED-300 282/300" src="https://img.shields.io/endpoint?url=https%3A%2F%2Ftoolkitlabs.org%2Fapi%2Fbadge%2Fpython%2Fjsonshim.json"></a>

Any row works: https://toolkitlabs.org/api/badge/{language}/{slug}.json. The JavaScript leader is jsonrepair — https://toolkitlabs.org/api/badge/javascript/jsonrepair.json. Every slug, every per-category count and the corpus sha256 are in /api/leaderboard.json, documented by /api/schema.json; the API page lists all 22 badge URLs. CC0-1.0 — copy it, re-host it, no attribution required.

Reproduce it

The harness is public domain and the whole point of it is that you do not take this page’s word for anything. Thirty of the cases and the scorer are CC0 too, so a first run costs nothing and needs no account.

curl -O https://toolkitlabs.org/malformed300/sample30.jsonl
curl -O https://toolkitlabs.org/malformed300/score.py
python3 score.py --corpus sample30.jsonl --parser json          # the control
python3 score.py --corpus sample30.jsonl --parser yourmodule:recover

leaderboard.json — the raw run, every number on this page comes out of it · run_leaderboard.py · adapters.py · suite README

How the comparison was kept fair

• Every library is called through its own documented entry point, with its own documented lenient mode where it has one. adapters.py is short on purpose, so you can check that in a minute.
• No input is pre-cleaned. Each parser sees the identical raw string.
• The only post-processing is normalising library-specific container and sentinel types to plain dict/list/None, so the comparison is about values, not about which class wrapped them.
• One run each, and then both halves were re-run afterwards purely as determinism checks. The six original JavaScript parsers reproduced every figure exactly. The seven original Python parsers were re-run in a freshly installed set of packages, on a machine where none of them was importable ten minutes earlier, and also reproduced every figure exactly — totals, invented values, false refusals and the whole category grid. No parser was tried, adjusted and re-run. The harness was validated by reproducing an earlier independently-published score for jsonshim exactly — 282/300, 5 invented, 4 false refusals — before any other library was scored.
• Versions are printed above, read from each installed package. The runner refuses to score anything unless every declared library imports first — a missing package aborts the run with exit 3 instead of quietly scoring as a refusal, which is exactly what one run of this harness did before the check existed.
• Ground truth in the corpus is produced by construction: the expected value exists before the malformed text does, so no parser was ever consulted about the right answer.

Where a library and this suite disagree about what a case should return, the suite is a stated grading contract, not a law — python3 score.py --spec prints it in full. Disagreements are welcome and checkable; that is why the ground truth ships.

The full corpus

The 30 free cases and the scorer are CC0 forever, bought or not. The remaining 270 cases, with the label rationale for each — why that ground truth and not another — are the paid product, and they are what let you run this comparison on your own parser at full resolution instead of on a tenth of it.

Single developer — €29 · Team / CI licence — €99

Instant download after payment; digital goods, delivered immediately. Payments processed by Stripe. Prices include VAT where applicable.