jevcrawl
PyPI MCP and CLI: Jev scores every link and passage on a site crawl so the agent LLM only reads verbatim blocks that answer the goal.
Overview
jevcrawl (github.com/RayenMlayeh/jevcrawl, Apache-2.0, PyPI jevcrawl) from Rayen Mlayeh is an MCP server and CLI for AI agents. It fetches up to four pages in parallel, segments HTML into passages, strips boilerplate, and sends each page's link labels and passage blocks to TypeSafe Jev in one batch per page. Jev scores relevance and answers fixed yes or no questions; nothing on the page is treated as instructions, so injected text can only bias a score, not steer the crawler. Results return verbatim text with heading paths, URL, and char_start or char_end into a sha256-indexed page blob. Install with uvx jevcrawl or pip; optional jevcrawl[js] extra renders JavaScript shells via Playwright. skills/jevcrawl/SKILL.md documents agent prompts; jevcrawl doctor checks keys and connectivity.
Problem: Agents that browse the web either ingest entire crawled pages and drown in boilerplate, or let a chat model pick every link and pay dollars per question.
Built for: Claude Code, Cursor, Codex, or custom MCP clients that need goal-directed site crawls with verbatim passages, URLs, and character offsets instead of model paraphrases.
First indexed on Jev Directory: 2026-10-15
Creator and team
- Name
- Rayen Mlayeh
- Handle
- @RayenMlayeh
How Jev is used
- Role in the product flow
- Per-page batch scoring of passages and outbound links plus yes or no gates for goal satisfaction and near-duplicate judgment; cached locally at ~/.jevcrawl/cache
- Primitives
- ScoreNoul
- State in
- Fixed question strings from jevcrawl plus page passages, link labels, and URLs in data fields only; query goal string from the agent or CLI --goal.
- Decision out
- Ranked passages and next links above calibrated thresholds; stopped_because answered, frontier_empty, max_pages, max_seconds, budget, or auth_error; degraded breadth-first mode with uncertain blocks after three failed Jev calls.
- Seed from sitemap or entry URL with robots.txt respect (JEVCRAWL_RESPECT_ROBOTS default 1)
- Fetch up to four concurrent pages; optional JS render for detected shells (max five per call with js extra)
- Segment and filter boilerplate; one Jev request scores every passage and link on the page
- Keep links below the priority floor in reserve; follow best links until goal Noul passes or budget hits
- Return blocks with sources; report jev_cost_usd, tokens_raw vs tokens_returned, and stats.degraded when applicable
jevcrawl spends hosted Jev on crawl steering, not on writing answers back to the user. dorkitude-webctl ranks live SERP hits and optional scrape chunks for coding agents; jevcrawl walks inward from a site you name. superagents-lab-jev-search plans Search1API queries inside a Worker. browser-use-jev-ultrafast picks DOM operations in Chromium. jkudish-jev-mcp exposes generic verify and rerank tools without a site walker. Page text goes to TypeSafe for scoring per README cost and privacy section; TYPESAFE_API_KEY is required. Default JEVCRAWL_MODEL is jev-1.13.0 with thresholds calibrated per version in jevcrawl/config.py.
Sourced performance claims
- README WebWalkerQA scoreboard on 50 tasks: 40% correct answers and 54% found the answer page for about $0.025 per task versus 30% and 33% with about $0.194 when Claude picks every link, and 8% and 11% with about $0.228 for breadth-first crawl.Source: github.com/RayenMlayeh/jevcrawl README benchmarks
- README Crawl4AI comparison on 20 tasks: jevcrawl found an answer page 10 times versus 5 for Crawl4AI BestFirstCrawlingStrategy; median crawl time 29 s versus 43 s.Source: github.com/RayenMlayeh/jevcrawl README jevcrawl vs Crawl4AI
- README docs.python.org example: 6 pages, 80,034 raw tokens reduced to 1,873 returned for about $0.0069 in 8.3 s.Source: github.com/RayenMlayeh/jevcrawl README demo
- Public GitHub repo RayenMlayeh/jevcrawl had 64 stars and 6 forks as of October 15, 2026; created 2026-10-11.Source: GitHub API, 2026-10-15
Features and stack
Features
- MCP server via uvx jevcrawl plus crawl, scrape, map, and doctor CLI
- Verbatim passages with URL, heading path, and character offsets into hashed page text
- WebWalkerQA benchmark tables versus Claude link picking, breadth-first, and Crawl4AI
- Local Jev answer cache; max_decision_usd hard stop before overspend
- Optional Playwright js extra for empty HTML shells; stats.js_rendered_urls lists them
- Fail-open degraded mode: breadth-first with uncertain blocks after repeated Jev errors
Stack
- Python 3.11+
- TypeSafe Jev (TYPESAFE_API_KEY)
- MCP
- Optional Playwright Chromium (jevcrawl[js])
- uv or pip install from PyPI
Pricing: Apache-2.0 open source; Jev input tokens bill to your TypeSafe key (README cites about $0.042 per million for jev-1.13.0 and about $0.011 average crawl cost in benchmarks).
Links
FAQ
- Does jevcrawl paraphrase answers?
- README How it works: Jev returns probabilities for fixed yes or no questions and scores; it cannot produce text. Every returned block is copied from the page with offsets for verification.
- How do I install the MCP server?
- README Quick start: claude mcp add jevcrawl -e TYPESAFE_API_KEY=<key> -- uvx jevcrawl, or equivalent mcpServers JSON with command uvx and args [jevcrawl]. Run uvx jevcrawl doctor to validate setup.
- How is jevcrawl different from webctl?
- webctl fuses multi-provider web search and optional scrape chunk filtering. jevcrawl crawls from a seed URL on one site (or map mode) with Jev picking links and passages toward a --goal.
- What happens if Jev API calls fail?
- README: after three failed Jev requests the crawl continues breadth-first, marks blocks uncertain, and sets stats.degraded instead of failing silently.
- Which Jev primitives does the crawler use?
- README describes scoring every passage and link and yes or no probability per question; jevUsage maps that to Score for relevance ranking and Noul for goal and duplicate style gates.
Related learn guides
Original Jev guidance that pairs with this product pattern.
- System One model
The model family behind Jev: parallel typed questions, one forward pass, probabilities you can threshold.
- Jev use cases
The patterns builders actually search for: moderation, routing, triage, RAG verify, and agent gates.
- Jev vs LLM classification
When to gate with System One probabilities instead of asking a chat model to label things.
Related products
Hand-picked neighbors with rich profiles or overlapping tags.