← Pace

On-device field guide

Screen-aware AI assistant for Mac

Understand how a screen-aware Mac assistant combines accessibility, local vision, interaction state, approval, and post-action verification.

A screen-aware AI assistant uses the state of the active Mac application to answer questions, guide the user, or perform an approved action. It should not treat a screenshot as unlimited authority. Reliable screen assistance combines the least invasive observation available with accessibility structure, visual grounding when necessary, and verification after each action.

Three kinds of screen context

  1. Accessibility structure: roles, labels, values, focus, and actions exposed by macOS. This is usually the best source for semantic controls.
  2. Visual context: pixels, OCR, or detected marks for applications where accessibility information is incomplete.
  3. Interaction state: the active window, recent user input, and the visible change after an action.

Each source has gaps. Accessibility can be incomplete, visual models can misread coordinates, and a visible change may not prove the intended result. A robust assistant cross-checks them and fails visibly when confidence is insufficient.

The action loop

A useful loop is observe, plan, request approval when required, act, and verify. For a click, verification might confirm a changed accessibility value or a new window state. For a workflow guide, the assistant should advance only after the user's click matches the expected state. A coordinate alone is not a durable outcome.

Pace's screen model

Pace can use local accessibility context, OCR, and a local vision model. Its click executor scores candidates, verifies the result, and has recovery paths for ambiguous targets. Watch mode records bounded local observations when enabled. Pace can also draw guidance on screen and wait for the user to complete the expected step. Screen and accessibility permissions remain controlled through macOS and Pace's own settings.

Privacy and limits

In local mode, screen analysis stays on the Mac. If a user selects an off-device planner tier, Pace shows the external state and records it in the local audit log; capability-specific privacy pins can still require local processing. Users should disable screen or journal sources they do not want and clear local history through settings.

CTA: See how approved Mac actions work, review privacy controls, or download Pace.