jev-on-a-laptop
Unofficial study: Jev-style parallel typed decisions on stock 1.
by @hhkkmon
What it does
Unofficial study: Jev-style parallel typed decisions on stock 1.5B-8B models on an Apple Silicon laptop. Benchmarks, research notes, and a Hugging Face Space demo. Numbers, harnesses, and comparisons you can rerun on your hardware instead of trusting a screenshot.
Tags in the wild: benchmarks, eval. The repo has install steps, env vars, and recent commits if you are deciding whether to depend on it.
Results
Sourced from https://github.com/rorshopping/jev-on-a-laptop on 2026-09-20. Numbers are copied as published; re-run upstream scripts to refresh.
Unofficial parallel constrained decoding study on Apple Silicon. Tables below are copied from the repository README measured on a 16 GB M5 MacBook Air unless noted.
Methodology: 28-field fraud preset: naive JSON generation vs parallel field evaluation. Quality eval: 24 labeled cases, 3 fields, 72 decisions per model (quality-eval/). Head-to-head: TypeSafe public workflow examples, 343 question-pairs answered by every model (evals/RESULTS.md).
Latency and schema validity (28-field fraud preset)
| Model (4-bit) | Naive JSON | Parallel decisions | Speedup | Naive schema-valid? | Parallel schema-valid? |
|---|---|---|---|---|---|
| Qwen2.5-1.5B | 3.3 s | 0.41 s | 7.9x | No | Yes |
| Qwen2.5-7B | 11.9 s | 1.52 s | 7.9x | No | Yes |
| Qwen3-8B | 14.3 s | 2.03 s | 7.0x | No | Yes |
Quality eval (24 cases × 3 fields per model)
| Model | Primary field accuracy | All fields exact | Latency per case |
|---|---|---|---|
| 1.5B | 58% | 50% | 147 ms |
| 7B | 96% | 72% | 611 ms |
| 8B | 92% | 85% | 646 ms |
Agreement on TypeSafe public workflow questions (343 pairs)
| Model | Agreement | Pairs |
|---|---|---|
| Opus (published) | 89.8% | 308/343 |
| DeepSeek v4.1 Flash (max) | 89.5% | 307/343 |
| Sol (published) | 89.2% | 306/343 |
| Jev / TypeSafe (published) | 86.6% | 297/343 |
| local Qwen2.5-7B (M5 Air) | 73.8% | 253/343 |
| local Qwen3-8B (3 of 4 workflows) | 71.2% | 114/160 |
Benchmark FAQ
- Is this the official Jev model?
- No. The README states this repo is unofficial research reproducing parallel constrained decoding on a stock small model (Qwen-2.5-1B-RLCD engine), not TypeSafe's hosted Jev weights.
- What hardware were the main latency tables run on?
- A 16 GB M5 MacBook Air, per the README section titled Measured results (M5 MacBook Air, 16 GB).
- How do I reproduce the benchmarks?
- Clone the repo, run ./setup.sh then ./run_benchmark.sh (default Qwen2.5-1.5B-Instruct-4bit). Full JSON logs live under results/ in the repository.
- Can I compare these accuracy numbers to evals.typesafe.ai?
- Carefully. The README explicitly warns the small rule-constructed quality eval is not comparable to TypeSafe's published workflow evals. The head-to-head table uses TypeSafe's public question set but still runs local Qwen weights, not hosted Jev.
Tags
Related products
Hand-picked neighbors with rich profiles or overlapping tags.