Jevstiller
A drop-in proxy that learns a local student from repeated Jev Choice calls and answers the sure ones on your machine.
Overview
Jevstiller (github.com/tomerglick57/Jevstiller, jevstiller.pages.dev) is a drop-in proxy in front of repeated Jev Choice calls. Point the SDK at it with TYPESAFE_BASE_URL. At first every request still goes to Jev. From Jev's answers, including the full probability distribution, it trains a small local student on a frozen bge encoder and checks agreement against a budget you set. It then answers many repeats locally, up to about 80% on the stable label sets in the README table. TweetEval sentiment and offensive were 22.2% and 24.1%. Uncertain or novel input, and a permanent random audit slice, keep going to Jev. If the audit shows the contract broke, everything falls back to Jev. The README says agreement with Jev is not accuracy. PyPI jevstiller 0.4.0. Apache-2.0.
Problem: A repeated Jev Choice is a network call every time, about 300 ms at the loads the Jevstiller README measured, even when the question never changes.
Built for: Teams with a stable Choice task (support intents, news sections, a fixed label set) who want many of the repeats answered on their own CPU after a short warm-up.
First indexed on Jev Directory: 2026-09-21
Creator and team
- Name
- tomerglick57
- Handle
- @tomerglick57
How Jev is used
- Role in the product flow
- Teacher for a local Choice student: Jev labels the traffic, the audit slice, and anything the student is not allowed to answer
- Primitives
- Choice
- State in
- The caller's Choice request: instructions, criteria, and model. The proxy keys a task by that exact content.
- Decision out
- Jev's own response until a student is promoted. After that, a local label in Jev's response shape when the threshold allows it, with header x-jevstiller-source: local.
- Services keep their own TYPESAFE_API_KEY. Jevstiller forwards it and the README says it never stores the key.
- Until the student passes its checks, requests are forwarded and Jev's responses are returned unchanged.
- The threshold uses an exact Clopper-Pearson bound at 95% confidence so disagreement stays inside the budget. target_agreement 0.98 means a 2% disagreement budget.
- Non-Choice questions and other endpoints are always forwarded. A fixed random audit slice always goes to Jev.
Jevstiller learns a task, it does not memorize one input. The README groups jevcache with semantic caches that reuse answers to near-identical inputs, and says those caches do not learn to answer new inputs. Here a frozen sentence encoder (bge-small, base, or large) stores embeddings, and a numpy logistic regression fits Jev's distribution. An out-of-distribution gate sends unfamiliar text to Jev whatever the head says. The routing threshold is chosen on a held-out set with a Clopper-Pearson finite-sample bound at 95% confidence, strictest threshold first. The audit channel is the ongoing check. If it breaks the budget, the proxy falls back to Jev on its own. The README's closest cousin is stuntd, which also learns a local head and checks a slice of live traffic. Jevstiller's own comparison says it picks the threshold with a finite-sample bound, trains and promotes by itself, and identifies a question by its exact content. Distil Labs, named in that same section, trains a replacement model as a separate job. That is a different project from Distill, the Rust coding agent harness by Samuel Fajreldines. Non-Choice questions never get a local answer. Ops in the README: TOML config, jevstiller admin, Prometheus /metrics, a Docker image at ghcr.io/tomerglick57/jevstiller, and a Kubernetes manifest. The README marks the project Alpha.
Sourced performance claims
- README replay table, bge-small on CPU, target agreement 98%, 2,000 held-out rows: Banking77 71.9% local at 99.50% agreement (accuracy 78.5% Jev and 78.5% system). CLINC150 69.0% at 99.65% (90.1% / 90.1%). AG News 80.2% at 99.40% (88.7% / 88.7%). TweetEval sentiment 22.2% at 98.70% (64.2% / 64.5%). TweetEval offensive 24.1% at 98.80% (73.8% / 74.2%).Source: github.com/tomerglick57/Jevstiller README Benchmark
- README: a local Banking77 answer is about 15 ms p50 on CPU, about 20 times faster than Jev at about 300 ms p50. The proxy section, 16 vCPU and bge-small, says local answers are about 16 ms p50 and about 50 ms p99. A late September 2026 live run reached 70.7% local at 99.45% agreement.Source: github.com/tomerglick57/Jevstiller README
- README: over 100 random splits across the 5 tasks, the calibrated threshold exceeded the 2% budget once. The status example shows an audit rate of 2%. The README states plainly that agreement with the teacher is not accuracy.Source: github.com/tomerglick57/Jevstiller README
- Public GitHub repo tomerglick57/Jevstiller had 61 stars and 8 forks on 2026-10-05. PyPI jevstiller version is 0.4.0. Created 2026-09-21. Apache-2.0.Source: GitHub and PyPI, 2026-10-05
Features and stack
Features
- Drop-in proxy: TYPESAFE_BASE_URL=http://localhost:8080, callers keep their own API keys
- Student on bge embeddings, trained from Jev's full probability distribution
- target_agreement budget, Clopper-Pearson threshold, 2% audit, automatic fallback
- PyPI jevstiller 0.4.0, Docker tag 0.4.0, docs at jevstiller.pages.dev
- TOML config, admin CLI, Prometheus /metrics, backup, Kubernetes manifest
Stack
- Python 3.10+
- bge-small encoder
- TypeSafe Jev Choice
- Docker
- Prometheus
Pricing: Apache-2.0. PyPI package jevstiller 0.4.0. Forwarded calls still spend your TypeSafe key. Local answers run on your CPU or GPU. The README publishes no hosted price.
Links
FAQ
- Can I distill Jev Choice calls into a local model?
- Yes, for a repeated Choice with a fixed label set. Jevstiller trains a student from Jev's answers, then serves the labels it is sure about. The README says this is a poor fit for changing class lists, non-text input, or volumes too small to collect a few thousand examples.
- How is Jevstiller different from jevcache?
- The Jevstiller README puts jevcache with semantic caches that reuse an answer when the input is near-identical. Jevstiller fits a student so a new sentence can be answered locally. Identical-input replay is the job jevcache documents on its own page.
- Does agreement with Jev mean the label is correct?
- Agreement with Jev means the student picked Jev's label. The README says that is separate from accuracy. If Jev is wrong, the student is wrong the same way. On Banking77 both sit at 78.5% against the dataset labels. The status report prints that note next to the agreement number.
- Does Jevstiller store my TypeSafe API key?
- The README says each service keeps its own TYPESAFE_API_KEY. Jevstiller forwards the key and does not store it. Local answers are returned only for keys Jev has already accepted.
Related learn guides
Original Jev guidance that pairs with this product pattern.
- Where to run Jev
Compare official TypeSafe, Vercel AI Gateway, OpenRouter, Cloudflare Workers AI, and classifier.dev with a fact table.
- System One model
The model family behind Jev: parallel typed questions, one forward pass, probabilities you can threshold.
Related products
Hand-picked neighbors with rich profiles or overlapping tags.