jevals
PyPI jevals: Ragas-style and agent evals in one Jev request per trace, plus YAML gates. Openlayer ships bench tables versus Ragas and LLM judges.
Overview
jevals (github.com/openlayer-ai/jevals, PyPI jevals, MIT) from Openlayer is an eval and guardrail library that packs many checks into one System One request per trace. You pass OpenAI-style messages plus tool schemas; each eval class builds a state blob and typed Noul, Choice, or Score questions, then reduce() turns probabilities into pass scores. evaluate() is synchronous; aevaluate(), Gate.check(), and host integrations cover async loops. Backends resolve from TYPESAFE_API_KEY, AI_GATEWAY_API_KEY, KEV_BASE_URL, EIKOS_BASE_URL, JEVALS_BACKEND=laya, or OpenRouter decision models. README quickstart on examples/quickstart.py reports 1 request, 1,388 tokens, $0.00006, and 0.33s for eight evals on a weather trace.
Problem: Teams sample agent and RAG evals because LLM judges need many slow, pricey completions per trace, so production gates drift from offline metrics.
Built for: Python engineers shipping agents or RAG who want one calibrated Jev request per trace, optional YAML gates in the request path, and Ragas-named quality metrics without six to eleven LLM round trips per row.
First indexed on Jev Directory: 2026-10-18
Creator and team
- Name
- Openlayer
- Handle
- @openlayer-ai
How Jev is used
- Role in the product flow
- Merge eval states and question maps from many eval classes into one backend request; plain code handles splits, tool matching, and Presidio or regex entity passes before Jev questions
- Primitives
- ChoiceScoreNoul
- State in
- Trace dict with messages and optional tools; per-eval state() extracts evidence, claims, tool_call, or retrieved chunks.
- Decision out
- Per-eval Result with scores, pass flags, and evidence strings; Gate policies map to allow, escalate, or block with Decision reasons.
- pip install jevals plus optional [pii], [mcp], [laya], or framework extras
- evaluate(sample, [ToolChoice(), Grounded(), AnswerRelevancy(), ...])
- Backend packs parallel questions; retries on 429 and 5xx with backoff
- jevals run traces.jsonl for dataset means and --show-failures
- jevals calibrate fits thresholds to your labels; jevals bench --ragas compares Ragas
jevals is the library layer for trace scoring and gates, not a demo courtroom or a single-host hook. kylejeong-jev-as-judge is a legal what-if web app; leepokai-jev-guard risk-scores every coding-agent tool call and scans results for injection. jevals reuses the same eval class offline, in monitoring, and inside Gate.check() before refunds or SQL run. jaredpalmer-kev and receptron-laya are backends you point at with KEV_BASE_URL or JEVALS_BACKEND=laya; jkudish-jev-mcp exposes generic MCP judgment tools without packaged Ragas metrics. README is explicit that decision models trail frontier LLMs on some classification sets, so calibrate and route hard rows to llm: backends when docstrings say so.
Sourced performance claims
- README top example: 1 request, 1,388 tokens, $0.00006, 0.33s for eight evals on the quickstart trace.Source: github.com/openlayer-ai/jevals README
- README jevals bench --ragas table September 2026, jevals 0.1.4, 20 rows: Ragas gpt-4.1-mini about 6.0 LLM calls plus embeddings per sample, $2.60 per 1k samples, 22 to 35s wall; jevals with Jev 1.0 request, $0.03 per 1k samples, 0.8s; measured verdict agreement across rows.Source: github.com/openlayer-ai/jevals README Numbers
- README: Jev measured p50 244ms and p95 371ms per request through Vercel AI Gateway; pricing cited at $0.042 per million input tokens with no output billing.Source: github.com/openlayer-ai/jevals README What changed
- README accuracy section: JevBench puts Jev around smallest LLM accuracy on Banking77 and CLINC150 (83 to 87%) with task-dependent calibration; jevals calibrate fits deploy thresholds.Source: github.com/openlayer-ai/jevals README On accuracy
- Public GitHub repo openlayer-ai/jevals had 105 stars and 11 forks as of October 18, 2026.Source: GitHub API, 2026-10-18
Features and stack
Features
- Agent evals: ToolChoice, UsedToolResult, Grounded, StayedInScope, LoopDetection, ToolCallRisk, and more
- Quality: Faithfulness, AnswerRelevancy, ContextPrecision, ContextRecall, Hallucination, Correctness, Completeness
- Security: IndirectInjection, PHI with Presidio optional, PromptInjection, SecretsExposure, and related gates
- YAML Gate definitions with allow_if, block_if, escalate policies and framework hooks
- jevals bench --ragas, jevals run, jevals calibrate, and optional MCP server extra
- Backend matrix: TypeSafe, Vercel gateway, OpenRouter decision models, Kev, Laya, Eikos, emulated chat LLM
Stack
- Python 3.10+
- PyPI jevals
- TypeSafe or Vercel AI Gateway Jev
- Optional Presidio for PII and PHI
Pricing: MIT library; Jev bills per README gateway or api.typesafe.ai input pricing. Local Kev, Laya, or Eikos rows in the bench table show $0 token cost with your own hardware.
Links
FAQ
- How is jevals different from Ragas?
- README Numbers table: Ragas faithfulness and relevancy metrics need multiple gpt-4.1-mini calls plus embeddings per sample; jevals rebuilds the same metric names with typed Jev questions in one request per trace.
- Can I run evals without a TypeSafe account?
- README backends table: AI_GATEWAY_API_KEY routes Jev through Vercel AI Gateway; KEV_BASE_URL, EIKOS_BASE_URL, JEVALS_BACKEND=laya, or OpenRouter decision models are supported. Chat LLM emulation works but README warns probabilities collapse toward 0 or 1.
- Does jevals replace jev-guard?
- No. jev-guard hooks coding agents at tool boundaries. jevals defines metrics and optional Gate policies you can call yourself or wire into OpenAI Agents, LangGraph, or your loop.
- What about accuracy versus an LLM judge?
- README cites JevBench Banking77 and CLINC150 ranges and recommends jevals calibrate on your labels. It also notes LangChain agent eval agreement on five traces; treat broad classification as a known limitation.
- Where do I start?
- pip install jevals, set AI_GATEWAY_API_KEY or TYPESAFE_API_KEY, then run examples/quickstart.py or the README weather agent block with evaluate().
Related learn guides
Original Jev guidance that pairs with this product pattern.
- Jev vs LLM classification
When to gate with System One probabilities instead of asking a chat model to label things.
- System One model
The model family behind Jev: parallel typed questions, one forward pass, probabilities you can threshold.
Related products
Hand-picked neighbors with rich profiles or overlapping tags.