agent-desktop
agent-desktop (github.
Overview
agent-desktop (github.com/lahfir/agent-desktop) is a Rust-native CLI distributed on npm. It snapshots any app's accessibility tree, returns compact skeleton overviews with drill-down refs, and executes headless-safe clicks, typing, scrolling, and window management. Your harness calls TypeSafe Jev to choose the next ref action from structured JSON, the same pattern Muhammad Aayan (@socialwithaayan) summarized as Jev picks the next button or input.
Problem: Desktop agents that read pixels or brittle DOM dumps waste tokens and mis-click when UI refs shift between snapshots.
Built for: Agent builders on macOS who want native accessibility-tree computer use with typed next-action routing.
First indexed on Jev Directory: 2026-09-21
Creator and team
- Name
- lahfir
- Handle
- @lahfir
“Desktop automation on the accessibility tree. Jev picks the next button or input.”
How Jev is used
- Role in the product flow
- Choice over the next desktop ref action in the observe-act loop
- Primitives
- Choice
- State in
- agent-desktop snapshot JSON: skeleton regions, qualified refs (@snapshot:eN), roles, names, and values from macOS accessibility APIs.
- Decision out
- Selected command family (click, type, scroll, etc.) and target ref from the current finite action menu.
- agent-desktop snapshot --skeleton (or find) returns refs and snapshot_id
- Harness builds allowed actions from refs and safety policy
- Jev Choice picks operation and ref in one System One call
- agent-desktop executes via accessibility APIs; loop until goal or budget
agent-desktop deliberately does not embed an LLM. It is the hands and eyes: structured observation in, deterministic CLI commands out. Jev sits in the calling agent exactly like Browser Use Ultrafast: rebuild the action space from fresh state, ask one typed question, threshold confidence, then act. Refs stay stable across steps because they are accessibility identities, not coordinates. Pair with Hermes, Claude Code, or custom harnesses via npm global install, npx, or the C-ABI cdylib for in-process calls. Chromium apps can mix CDP for web content with native AX for menus and dialogs. This profile does not claim a bundled Jev binary inside agent-desktop; wire console.typesafe.ai credentials in your agent layer.
Sourced performance claims
- Progressive skeleton traversal reports seventy-eight to ninety-six percent token reduction on dense apps versus flat snapshots in README examples.Source: github.com/lahfir/agent-desktop README
Features and stack
Features
- Rust CLI with fifty-eight command names and structured JSON errors
- Skeleton snapshots with drill-down refs for Slack, Finder, Xcode, and more
- npm and npx install with prebuilt macOS binaries
- ClawHub and skills.sh agent-desktop skill docs
- Featured in Aayan top-ten Jev repos thread
Stack
- Rust
- macOS Accessibility APIs
- TypeSafe System One (in agent harness)
Pricing: Open source Apache-2.0; TypeSafe API usage is bring-your-own when your agent calls Jev.
Links
FAQ
- Does agent-desktop include Jev?
- No. It exposes desktop actions. Your agent or skill harness calls Jev separately to pick the next safe action from snapshot output.
- Which platforms are supported?
- macOS thirteen plus is production-ready per README. Windows and Linux adapters are planned with the same core contracts.
Why accessibility beats pixels
Native AX trees expose roles, names, and stable refs that survive window moves. agent-desktop compresses dense apps with skeleton overview plus drill-down so Jev sees a small action menu instead of thirty thousand tokens of Slack chrome.
Related learn guides
Original Jev guidance that pairs with this product pattern.
- Jev use cases
The patterns builders actually search for: moderation, routing, triage, RAG verify, and agent gates.
- Jev vs LLM classification
When to gate with System One probabilities instead of asking a chat model to label things.
Related products
Hand-picked neighbors with rich profiles or overlapping tags.