dsh-computer-use-macos

@wangyong1972/dsh-computer-use-macos

English | 中文

A DeepSeek Harness (DSH) plugin that gives
models a native macOS implementation of Anthropic's "Computer Use"
capability: one computer tool that can screenshot the real display and
drive the real mouse/keyboard, using only binaries that ship with macOS
(screencapture, sips, osascript) — zero installed dependencies, zero
network calls, zero compiled helper binaries.

What it does

The plugin registers a single tool, computer, that lets the model:

• take a screenshot of the real screen and see it inline in the conversation;
• move the mouse, left-/right-/double-click at a pixel coordinate;
• type text and send key combos (Return, cmd+c, ctrl+shift+t, ...);
• scroll up/down/left/right;
• read the current cursor position (best-effort);
• wait for a short, bounded duration.

Every OS interaction goes through execFile('screencapture', [...]),
execFile('sips', [...]), or execFile('osascript', ['-l', 'JavaScript', ...])
with argv arrays — never a shell string built from model input. See
"Security posture" below.

Screenshots are automatically downscaled before being sent to the model:
any capture whose longest edge exceeds 1568px is resized in place (via the
macOS-native sips -Z, preserving aspect ratio) so it stays comfortably
under common vision-model per-image pixel limits — this is what makes
screenshot work at all on a Retina or 6K display, whose raw capture would
otherwise be rejected outright by many hosts with
"Image exceeds the configured per-side pixel limit." No setting changes
this target size. If on-screen text or controls become too small to read
reliably in the downscaled image, increase the size of that text/UI in the
app being controlled (zoom in, increase font size, etc.) rather than trying
to change the display's resolution or this plugin's downscale target —
matching Claude Code's own documented
behavior for its native
macOS computer-use tool. Every coordinate the model reads or supplies is
always in this downscaled screenshot's own pixel space, never the
display's raw native pixel space — the tool converts internally.

Claude-Code-parity UX features (all individually toggleable, see the
config table below): a machine-wide lock so a second DSH process on the same
Mac can't drive the mouse/keyboard concurrently; optionally hiding other
visible apps while acting; optionally excluding the host/terminal from the
screenshot it takes of itself; macOS notifications when the tool starts and
stops acting; and extra warning text in the approval prompt when the
frontmost app is a terminal/IDE, Finder, or a system-settings app. A global
Esc-to-abort hotkey was investigated and found infeasible under this
plugin's architecture — see DESIGN.md §10.5 for the full analysis.

Requirements

• macOS only ("os": ["darwin"] — installation fails loudly on any other
  platform).
• A DSH host with the tools and attachments services mounted (both are
  standard in a DSH web/desktop profile).
• A vision-capable model route for the screenshot action specifically (a
  text-only route gets a clear error instead of a wasted screenshot).

One-time macOS permission setup

macOS requires two manual, one-time permission grants before this plugin can
do anything. This plugin cannot grant these to itself — macOS
deliberately makes TCC (privacy) grants un-scriptable from the requesting
process. This is the same kind of one-time dance you already went through for
apps needing Full Disk Access or Screen Recording.

1. Accessibility — required for every mouse/keyboard action
   (mouse_move, left_click, right_click, double_click, type, key,
   scroll, and best-effort for cursor_position).
   Open System Settings → Privacy & Security → Accessibility and enable
   the relevant app.
2. Screen Recording — required for screenshot to return real pixels
   instead of a blank image (recent macOS versions silently return a blank
   capture, with no error, when this permission is missing).
   Open System Settings → Privacy & Security → Screen Recording and
   enable the relevant app.

Which app shows up in those lists depends on how DSH was launched. Every
mouse/keyboard/screenshot action in this plugin runs through the osascript
binary (or screencapture for screenshots), so TCC attributes the
permission to whichever process actually executed it — in practice, this is
either osascript itself, or your terminal/host app in the responsible-
process chain, depending on macOS version and how DSH was started. Trigger
one computer action first (it will fail with a clear message naming the
missing permission), then check System Settings — the correct entry only
appears in that list after the first attempt. Look for osascript,
Terminal, or your terminal app.

Once both permissions are granted, retry the failed action.

Installing into a DSH profile

npm (recommended):

dsh plugin --profile <name> add @wangyong1972/dsh-computer-use-macos

Published package: @wangyong1972/dsh-computer-use-macos

Local checkout (development):

dsh plugin --profile <name> add /path/to/dsh-computer-use-macos

This plugin ships its own cordis.patch.yml (referenced from
package.json's dsh.bundle.patch field), so adding it registers the
computer-use-macos plugin with sensible defaults automatically.

Config reference

field	default	description
enabled	true	Master switch. When false, the computer tool is not registered at all — zero prompt-token cost.
requireConfirmation	true	Route every mutating action (click/move/type/key/scroll) through the approval seam before executing. screenshot/cursor_position/wait are never gated. Fails closed (denies) when no approval service is mounted.
allowedDisplayIndex	0	Backward-compatible default 0-based display index used whenever an action omits display. Per-action selection can override it without changing config.
screenshotFormat	'png'	Reserved for future formats; only png is supported in v1.
actionTimeoutMs	10000	Hard timeout for every screencapture/sips/osascript/ps subprocess call.
maxTypeTextLength	4096	Reject action=type calls whose text exceeds this many UTF-16 code units (rejected, never truncated).
enableMachineLock	true	Hold a machine-wide lockfile while a mutating action runs, so a second DSH process on this Mac can't drive the real mouse/keyboard concurrently. A stale lock (dead holder, or held too long) is reclaimed automatically.
hideOtherAppsWhileActing	false	Hide every other visible app while a mutating action runs, keeping only this plugin's own host process visible, then restore exactly what was hidden. Off by default — visibly intrusive, opt in deliberately.
excludeHostFromScreenshots	false	Momentarily hide the host process for just the screencapture call so screenshot never shows the terminal instead of the real target app. Off by default for the same reason as above.
enableSessionNotifications	true	Post a macOS notification when the tool starts/stops acting. Best-effort; never fails the underlying action.
enableAppRiskWarnings	true	Add extra warning text to the approval prompt when the frontmost app is a terminal/IDE, Finder, or a system-settings app. Purely additive — never changes what gets approved, no effect when requireConfirmation is false.

Click diagnostics and logging

Every mouse_move/left_click/right_click/double_click call emits one
structured, privacy-safe diagnostic record via DSH's own ctx.logger()
facade (never a bespoke log file, never console.*) — named
computer-use-macos, so a host's own log exporter can filter on it. Example
line (real format, live-verified against a real @deepseek-ai/cordis
Context):

computer.left_click display=0 pixel=[500,400] point=(500,400) pid=69234 exit=0 elapsed=112ms outcome=ok cursorVerified=true

Fields: the action, allowedDisplayIndex, the requested pixel coordinate,
the resolved Quartz point, the osascript subprocess's pid/exit code,
elapsed time, a coarse outcome classification (ok /
accessibility-denied / timeout / aborted / other-error), and whether
a post-action cursor-position self-check confirmed the cursor actually
arrived at the requested point. Never logged: raw stderr/stdout text,
typed text, or key-combo content — see DESIGN.md §11.2 for the full
privacy contract. warn-level for anything other than a clean, verified
outcome; info-level otherwise.

What this can and cannot prove: the cursor self-check confirms a click's
underlying CGEventPost call reached the OS and the cursor warped to the
right screen point — it cannot confirm the target application actually
received or reacted to the click as a click (that would need per-app
accessibility-tree introspection, out of scope; see DESIGN.md §11.3). Take
a follow-up screenshot to confirm a click's actual on-screen effect.

Supported actions (v1)

list_displays, screenshot, left_click, right_click, double_click,
mouse_move, type, key, scroll, wait, cursor_position (best-effort).

Selecting a display

Call list_displays to obtain each active display's 0-based index, numeric
CGDirectDisplayID (id), main-display flag, Quartz origin/size, backing
scale, and explicit 1-based screenshotOrdinal. screenshot, cursor_position, mouse move/click actions, and scroll
accept an optional display field:

• positional strings: main, leftmost, rightmost, topmost, bottommost;
• a JSON number for a 0-based active-display index;
• a decimal string for a CGDirectDisplayID, optionally prefixed with id:.

If display is omitted, the existing allowedDisplayIndex behavior is
preserved. The inventory is refreshed for every call, invalid/out-of-range
selectors fail before any OS input event, and coordinates remain local to the
selected display's screenshot space. main must resolve to exactly one display;
a positional extreme shared by two displays is ambiguous and fails closed.
For scroll without coordinate, the display selector is accepted but does
not move the pointer; scrolling occurs at the current cursor position as before.

Screenshot identity limitation: macOS screencapture -D accepts only a
1-based ordinal, not a CGDirectDisplayID. screenshotOrdinal is the best-known
mapping derived from the same NSScreen inventory and is passed explicitly; it
is not an identity-safe guarantee that Apple will keep NSScreen and
screencapture ordering identical. Mouse/cursor geometry is ID-backed through
CGDisplayBounds, but callers must not treat CG-ID-selected screenshots as
absolutely identity-safe until the backend is replaced with direct CoreGraphics
capture.

Not yet supported (deferred to v2): left_click_drag, middle_click,
triple_click, hold_key, zoom, compensating for the macOS "natural
scrolling" preference, and per-window/per-app (accessibility-tree) targeting.
Please don't file bugs for these — they're known, intentional v1 scope cuts,
not oversights.

Security posture

• No shell-string execution anywhere. Every OS command is invoked via
  Node's execFile with an argv array, and every mouse/keyboard action's
  untrusted input (text, coordinate) crosses the JXA subprocess boundary
  as one JSON string in argv, never as interpolated AppleScript or shell
  source text.
• No network calls of any kind, anywhere in this plugin.
• All numeric input is validated and clamped against the real,
  freshly-queried screen bounds before any mouse/keyboard event is
  synthesized — never trusted from the model, even if the model is only
  echoing a coordinate it was previously shown.
• No arbitrary file path acceptance. The only filesystem paths this
  plugin ever touches are its own randomly-named temp screenshot file
  (created, optionally downscaled in place via sips, and deleted within the
  same tool call) and a single fixed-path machine-wide lockfile (§10.1).
• The machine-wide lock fails closed on contention — a second process
  holding it is a hard denial, never a silent proceed; stale-lock reclaim is
  bounded by both PID-liveness and an absolute age ceiling.
• App-hiding always restores exactly what it itself hid, never a blanket
  "show everything," so an app the user had already hidden stays hidden.
• Per-app risk warnings are purely additive to the existing approval
  prompt — they can never turn an approval "ask" into an automatic "allow,"
  and the risk table is fixed, not model- or config-editable.
• Every mutating action is gate-able via requireConfirmation, which
  fails closed (denies) when no approval service is mounted — never fails
  open.
• enabled: false fully disables the plugin: no tool registration, zero
  token cost.
• No postinstall/preinstall scripts, ever.
• peerDependencies, not dependencies, for every @deepseek-ai/*
  package this plugin composes against.

Development

pnpm install
pnpm run build      # tsc -> lib/, then copy the JXA helper verbatim
pnpm run typecheck   # tsc --noEmit
pnpm run verify      # static checks, no macOS side effects, safe for CI
pnpm test            # unit tests over lib/ (key-spec, validate, permission-errors, screenshot-scale, lock, app-tiers, notify)

Everything requiring real Accessibility/Screen Recording permissions is
covered by the manual checklist in tests/e2e.manual.md.

License

MIT