AI Process Manager (AIPM)

AI Process Manager — MCP Server

Structured Windows state for AI agents — no screenshots.

This MCP server exposes the AI Process Manager local HTTP API as tools for Claude Desktop, Cursor, and any MCP client. Instead of capturing pixels (~2,765 tokens per 1080p screenshot), agents read JSON and text (~15–150 tokens per query) from processes, windows, consoles, and UI Automation trees.

Requires: AIProcessManager.exe running on Windows (system tray). Node.js ≥ 14. Zero npm dependencies.

Why this exists

Computer-use agents often "look" at the desktop via screenshots. That is slow (3–5 s), expensive in tokens, and sends pixel data through the model. AIPM answers structured questions on loopback:

~94–98% fewer perception tokens per action vs screenshots (2,765 → ~60 tokens per query; the two inputs are the assumptions block of GET /analytics/summary, not a benchmark).

Quick start

1. Start the backend

Ensure AIProcessManager.exe is running in your system tray.

2. Configure your MCP client

Claude Desktop — claude_desktop_config.json:

{
  "mcpServers": {
    "ai-process-manager": {
      "command": "node",
      "args": ["C:\\path\\to\\ai-process-manager\\mcp\\server.js"]
    }
  }
}

Cursor and other stdio MCP clients take the same command / args pair. Optional env var AIPM_API overrides the API base URL (default: read from %LOCALAPPDATA%\AIProcessManager\endpoint.txt, else http://127.0.0.1:9147).

Optional env var AIPM_AGENT_NAME names this agent in the API's activity log (default mcp). Every request is sent with User-Agent: AIPM-Agent/<name> (<client>) mcp:<client>, and the API extracts <name> into the agent field of /audit, /activity and /notes. Use it when several agents share one MCP client, so GET /activity can tell them apart:

"env": { "AIPM_AGENT_NAME": "revisor" }

The name is normalized to [a-z0-9._-], max 40 chars. It is an identity the caller declares about itself — observability and coordination only, never authorization. Anyone can write any name here; nothing in the product grants or denies permission based on it. The only barrier remains the deny-by-default action allowlist, which is per app.

3. Restart the MCP client

Tools appear as ai-process-manager. Call health_check first.

Tools (28)

The 15 perception tools and the two UI probes are annotated readOnlyHint: true — clients may auto-approve them. The seven UI actions are readOnlyHint: false and refuse to run unless the user opts in (see Privacy).

UI tools — state first, then opt-in actions:

Local telemetry (writes to the local store, metadata only):

Local agent coordination (stored only on this machine):

Reading deep UI trees (Chromium/Electron)

get_ui_tree defaults to depth=4, which is enough for native Win32 apps. Chromium/Electron apps (VS Code, Slack, Discord, Claude Desktop, Teams) bury content under ~10 levels of Pane/Group wrappers — at low depth the response is only empty panes. Ask for depth=15-20 there (max 30) and cap cost with max_nodes (default 200, max 1000): max_nodes is the cost brake, not depth. When the response has truncated: true, the tree was cut — repeat with a higher depth/max_nodes before concluding anything about the window.

Acting: two families, and picking the wrong one is the classic failure

This is the part agents get wrong by default, because every other computer-use tool they have ever seen needs the window in front. Here, most of them do not:

ui_act picks between the two families below for you — try it first for click / type / pick. It attempts UI Automation and drops to real input by itself only when UIA reports the pattern is missing on that element (never for intent=escolher, where a physical click would trigger the element instead of picking an option inside it), and tells you which one ran in mecanismo (uia/fisico). It also refuses an ambiguous target with 409 alvo_ambiguo and hands back the candidates with their rect, instead of silently taking the first name match the way the tools below do. Read on for what each family means and when to call one of them directly instead.

Why: the first family talks to the app's accessibility provider and ignores z-order; the second emits real keyboard/mouse events, which Windows delivers to whatever is in the foreground — not to the hwnd you passed.

Calling the families directly: start with by-element, every time. Do not call focus_window "just in case" before it: that steals the user's foreground and buys nothing. Reach for ui_send_keys only after ui_set_value actually returned valor_nao_aplicado on that field (a contenteditable in a controlled framework), and for ui_drag only for what genuinely only moves by dragging — there is no UIA pattern for dragging.

If focus_window itself fails with foco_nao_aplicado, do not loop on the second family: it cannot succeed without the foreground. health_check reports the real-input arm (ForgePilot) under executor/forgepilot, and warns when it is installed but stopped.

Privacy

No data about you leaves the machine. There is no remote telemetry, no cloud service, and no account. There is exactly one outbound request, it carries nothing about you, and you can turn it off:

Update check. Once a day the engine asks GitHub's public Releases API whether a newer version exists. The request goes to GitHub, not to us: no identifier, no telemetry, nothing about your machine or your apps. It can be switched off in the tray menu, and when off no outbound request is made at all. Everything else in AIPM remains local.

Network

• The backend listens on the loopback interface only (127.0.0.1:9147) — not on 0.0.0.0, so nothing on the LAN can reach it.
• Every request must carry a Host header of localhost or 127.0.0.1; anything else is rejected with 403 forbidden_host (anti DNS-rebinding, so a web page you visit cannot drive the API).
• This MCP server makes exactly one kind of outbound call: HTTP to that local address. It sends a User-Agent of mcp:<your MCP client name> so the local audit log shows which agent asked.

Read-only by default

• The 15 perception tools plus ui_find are plain reads. ui_find_at is also semantically read-only, but its pixel probe uses a guarded POST; all 17 carry readOnlyHint: true.
• The 7 UI action tools (ui_act, ui_invoke, ui_set_value, ui_send_keys, ui_drag, ui_select_option, focus_window) return 403 action_denied until the user does both: enable Agent actions in the tray menu and add the target process to a per-app allowlist. Neither is on by default, and the setting is per app — allowing Notepad does not allow the browser.

What the telemetry stores — metadata only

Recorded: app/process name, element role (Button, Edit…), the element name the agent asked for, the action (invoke/set_value/focus), success or failure, duration in ms, and a failure reason. Two mechanical invariants make "metadata only" verifiable rather than a promise:

1. An action record stores what the agent requested, not what the app displayed. The role and name come from the agent's own role=/name_contains= arguments. The name of the element actually resolved on screen is never written, so what the tool saw never becomes telemetry.
2. The failure reason is a closed vocabulary (elemento_nao_encontrado, ui_timeout, erro_uia, outro, …). Any other string is stored as outro. An exception message — which could carry a file path or on-screen text — therefore cannot reach the disk.

Never stored: screen contents, window text read by read_window, the text typed by ui_set_value (value=), file contents, keystrokes, screenshots. The passively learned UI shape (ui_shape) holds only counts, UIA role names, depth and booleans: exposes_text says whether text exists, named_controls says how many elements have a name — never the text itself.

Masking: before any request is recorded, value= and api_key= are replaced with ***, so neither get_audit_log nor the on-disk query log can reveal typed text or a secret.

One honest nuance: the audit/query log stores the request line, so other query arguments stay readable — e.g. check_file(path=D:/videos/out.mp4) is logged as that path, and check_process(title_contains=...) keeps that fragment. It is a local log of what the agent asked for. Only value= and api_key= are masked.

Where the data lives, and how to delete it

Everything is under %LOCALAPPDATA%\AIProcessManager\:

To erase: quit the app from the tray, then delete the folder (or just db\ to reset learning and the economy counters; deleting actions.cfg turns the action tier back off). Nothing is written anywhere else, and the /audit ring buffer lives in memory only — it disappears when the app closes. You can inspect everything the store holds with get_audit_log, get_economy_stats, and GET /analytics/actions.

Troubleshooting

Every error payload carries a next_action field with the same guidance.

Coverage (measured, honest)

We publish what does not work yet on purpose — you should know the edges before relying on it.

Behaviour note: the first read of a Chromium/Electron window asks it to activate its accessibility tree — the same standard request a screen reader makes. That app then keeps computing accessibility data (a CPU cost in that app) and does not go back to sleep on its own. Native Win32 apps are unaffected.

Free vs paid

• Free & open (MIT): this MCP server.
• Free (closed): the AIProcessManager.exe backend — the sensor. Yours to run at no cost.
• Paid: AIPM Pilot, the autonomous computer-use agent that drives apps end-to-end using AIPM's structured perception.

Support the project

If AIPM saves you tokens, consider sponsoring.

License

MIT for the MCP server. The backend and AIPM Pilot are separate products.