LLM Determinism Benchmark

What it does

LLM Determinism Benchmark (also called SOB, or Structured Output Benchmark) evaluates how accurately large language models extract and structure data into JSON. It tests whether a model not only produces valid JSON that matches a schema, but also whether the values inside that JSON are correct. For example, it checks whether an invoice date is accurate or an array is ordered correctly, not just whether the JSON parses.

Who it is for

Developers and organizations building deterministic workflows that rely on LLMs to convert unstructured data (invoices, transcripts, PDFs, medical records) into structured formats. Teams that feed LLM output directly into downstream systems need confidence that hallucinated or incorrect values won't silently break their pipelines.

Pricing

The site does not show prices.

How it stands out

Existing benchmarks like JSONSchemaBench focus only on schema compliance—whether the JSON is valid and conforms to shape. SOB separates structure from accuracy. It measures value correctness per field using seven distinct metrics: Value Accuracy, JSON Pass Rate, Path Recall, Structure Coverage, Type Safety, and others. The benchmark uses three modalities (text, image, and audio) to reflect real-world extraction tasks, though images and audio are normalized to text before scoring to isolate schema-handling capability. Ground truth is human-authored and LLM-verified, so errors are unambiguous. Most importantly, SOB does not conflate reasoning ability with extraction ability—it isolates how well a model grounds values in source material and handles nested schemas under different content distributions.

What a founder should check

First, verify whether practitioners actually run benchmarks before deploying structured output systems into production, or if most teams skip formal validation. Second, examine whether existing tools (Anthropic's native structured output, OpenAI's JSON mode, or simpler schema validators) are sufficient for most use cases, and how much additional correctness SOB claims to unlock. Third, understand the switching cost—whether results from SOB would actually change which model an engineering team selects, or if cost and latency dominate the decision over extraction accuracy metrics.