Benchmarks
Eval harnesses and open replicas for engineers who need reproducible typed Jev numbers before they pick a route.
Use this page when you are choosing between hosted Jev, a gateway, and a local replica. The listings document calibration studies, latency harnesses, and head-to-head tables, not chat leaderboards. Read the methodology, then open the author's scripts. Star counts are hints, not votes.
Who it is for: Engineers choosing between hosted Jev, gateway routes, and local replicas who need reproducible numbers before they bake routing policy into production.
How to choose: Match the benchmark to your question: latency on laptop hardware (parallel decoding replicas), agreement with TypeSafe workflow evals (head-to-head harnesses), or calibration error (daf-jev). Read methodology sections; ignore star counts. Prefer repos that ship scripts and raw JSON logs.
91 listings in this bucket · Learn Jev · Laptop benchmark results
See Jev alternatives compared, Liquid d1 vs Jev and SemIf vs Jev.
Guide
What Jev benchmarks actually measure
Unlike chat benchmarks, these harnesses score typed answers: did the model pick the right option, score level, or noul label for structured state? Good listings document datasets, baselines, hardware, and whether numbers compare to naive JSON generation, frontier models, or TypeSafe's published workflow evals.
Featured results on this directory
jev-on-a-laptop publishes M5 Air latency and TypeSafe workflow agreement tables scraped into this site. daf-jev documents batching efficiency against sequential System One calls. The parallel constrained decoding Space reports M4 Max milliseconds per scenario. Each page links to the primary source so you can re-run scripts locally.
Highlights
Hand-picked listings with extra detail on this directory. Star counts and demos still live on each card.
- jev-on-a-laptop
Unofficial parallel decoding study with 7.9x speedups and public workflow head-to-head tables (position 1 in search for laptop Jev benchmarks).
- daf-jev
Python toolkit plus calibration benchmark scripts and manuscript pipeline around live System One calls.
- Parallel Constrained Decoding (Qwen2.5-1B-RLCD)
Hugging Face Space with M4 Max latency table comparing autoregressive JSON to parallel constrained decoding.
- SemIf
SemIf routing gates with structured Jev questions and documented eval posture.
- typesafe-ai-benchmark
Gateway-shaped rig to reproduce structured evaluate latency on your machine.
FAQ
- What is a Jev benchmark?
A reproducible harness that scores typed System One answers (Choice, Score, Noul) on fixed state, often reporting latency, agreement, or calibration error. It is not the same as MMLU-style chat benchmarks.
- Are unofficial replicas comparable to evals.typesafe.ai?
Only when the repo explicitly runs the same public questions and explains label differences. jev-on-a-laptop includes a head-to-head table with caveats; always read the methodology footnotes.
- Where do I see result tables on this site?
Hot benchmark detail pages include a Results section sourced from the project's README or Space. Numbers are not invented here; if a source is unreadable, the page falls back to FAQ and methodology text only.
- Do benchmarks prove Jev beats GPT on everything?
No. Each listing measures a narrow claim (schema validity, approval latency, batching tokens). Treat marketing speedups as hypotheses until you run the packaged scripts on your hardware.
Start here
Start here: daseinlabs/open-jev for an open replica writeup, the qwen2.5-1b-rlcd parallel-decoding benchmark for speedup numbers, and rorshopping-jev-on-a-laptop for a real hardware run. Each links to its own numbers below.
All listings
by @hhkkmon
by @iamMrDuncan
by @beni_il_
by @yuzxfred
by @Teyhouse