daf-jev
Daf-jev: composable Python toolkit for TypeSafe's Jev (System One) decision API, question builders, confidence gates, evaluator, calibration, CLI, MCP s
What it does
Daf-jev: composable Python toolkit for TypeSafe's Jev (System One) decision API, question builders, confidence gates, evaluator, calibration, CLI, MCP server, agent skill. Numbers, harnesses, and comparisons you can rerun on your hardware instead of trusting a screenshot.
Tags in the wild: benchmarks, mcp, agents. The repo has install steps, env vars, and recent commits if you are deciding whether to depend on it.
Results
Sourced from https://github.com/docxology/daf-jev on 2026-09-20. Numbers are copied as published; re-run upstream scripts to refresh.
daf-jev ships evaluation and calibration tooling around the live System One API. The README documents batching efficiency from project benchmarks; run benchmarks/bench_calibration.py locally for live ECE and Brier numbers.
Methodology: Batch vs sequential calls documented in the README quickstart section. bench_calibration.py repeats choice and noul questions across states and writes output/benchmarks/calibration_<date>.json when JEV_API_KEY is set.
Documented batching efficiency (README)
| Strategy | Relative speed | Relative tokens |
|---|---|---|
| Batched questions in one ask | Up to ~18× faster vs sequential | Sequential uses ~4× more tokens |
Benchmark FAQ
- What does daf-jev measure in its calibration benchmark?
- bench_calibration.py repeats the same choice and noul questions per state, uses modal self-consistency as a correctness proxy, and computes expected calibration error, Brier score, and a reliability table via daf_jev.calibration.
- Do I need an API key to run the benchmarks?
- bench_calibration.py skips gracefully when JEV_API_KEY is not set. Set JEV_API_KEY or TYPESAFE_API_KEY (or .env) for live model numbers.
- Is daf-jev only a benchmark repo?
- No. It is a full Python client, composition patterns (confidence_gate, composite_score), evaluator harness, MCP server, and manuscript pipeline. Benchmarks are one slice of the toolkit.
Related products
Hand-picked neighbors with rich profiles or overlapping tags.