Pace — architecture

Pace — architecture

Doctrine

1. Tinygpt’s job is to provide Pace the BEST model for each role. This is a mix of (a) training new specialists where no off-the-shelf model fits — LoRAs, distilled SLMs, fine-tunes; (b) vetting + qualifying external open-source models when they’re already good enough — WhisperKit, embedding models, possibly VLMs. Tinygpt is NOT mandated to train everything itself. Sometimes the right call is “this external model passes our eval, ship it as-is.” Pace owns its own model runner at runtime — embeds MLX-Swift / WhisperKit / CoreML directly, loads bundled artifacts in-process. The current LM Studio development bridge is loopback-only HTTP and is guarded against non-local hosts; the shipping target is in-process model runners. The factory’s serve is for development + eval only; Pace ships self-contained, easy to install.
2. Steal from anywhere that runs local. Apple frameworks (WhisperKit, AX, EventKit, MessageUI, Shortcuts CLI, NSWorkspace, Speech, FoundationModels for short answer turns), open-source models (Qwen3, UI-Venus, mxbai-embed, BGE, Whisper, Kokoro, Piper), open-source runtimes (MLX, CoreML, llama.cpp, Outlines), open-source datasets (xLAM, ToolBench, FineWeb). The constraint is local-only, not vendor.
3. Fastest AND most precise. Both required.
4. 100 ms is the END-TO-END completion budget. User-stops-talking → action perceived. Total. Today’s path is ~500 ms on the lightest action — not good enough. Parallelize aggressively (ASR partials drive planner prefill; planner emits intent; executor dispatches; all overlapped). Never accept a “good enough” win — keep shaving.
5. All data stays local. No cloud calls, ever. No “fallback to cloud for hard cases.”
6. English speakers + Mac only. Narrow focus is the speed advantage.
7. Timelines are bullshit. Ship the next correct thing now.

The constellation

                ┌──────────────────────────────────────────────┐
                │  Voice model — WhisperKit-on-ANE             │
USER VOICE ───►│  streaming partials + LocalAgreement +       │
                │  initial_prompt biasing (vocab from repos)   │
                └─────────────────────┬────────────────────────┘
                                      │ transcript stream
                                      ▼
                ┌──────────────────────────────────────────────┐
                │  Intent disambiguator (planner emits intent) │
                │  dictate │ edit │ action │ answer            │
                │  + reference resolver via AXSelectedTextRange│
                └──┬──────────────┬─────────────────┬──────────┘
                   │              │                 │
       ┌───────────┘              │                 └───────────┐
       ▼                          ▼                             ▼
  ┌──────────┐         ┌────────────────────┐          ┌────────────┐
  │  RAG     │         │  Planner — v10     │          │ Vision     │
  │  mxbai-  │◄───────►│  parameterized     │◄────────►│ UI-Venus + │
  │  embed + │  ctx    │  action emitter    │  context │ ANE chunk  │
  │  SQLite- │         │  {spokenText,      │          │ (#266/#275)│
  │  vec on  │         │   intent, payload} │          │            │
  │  Mail,   │         │  grammar-          │          │            │
  │  Notes,  │         │  constrained       │          │            │
  │  Files,  │         │  119ms TTFW        │          │            │
  │  past    │         │                    │          │            │
  │  Pace    │         │                    │          │            │
  └──────────┘         └─────────┬──────────┘          └────────────┘
                                 │
                                 ▼
                ┌──────────────────────────────────────────────┐
                │  Executor — AX dispatch primary              │
                │  AX setValue + AXPress, then EventKit,       │
                │  MessageUI, MapKit, Contacts, shortcuts run  │
                │  CGEvent keyboard fallback (never pasteboard)│
                └──────────────┬───────────────────────────────┘
                               │
                               ▼
                ┌──────────────────────────────────────────────┐
                │  Responder                                   │
                │  streaming TTS via Apple SpeechSynth +       │
                │  bodyText streamed into AX setValue +        │
                │  HUD overlay                                 │
                └──────────────────────────────────────────────┘

Stolen from Apple

Every box above leans on something Apple already shipped. Pace writes glue, not models from scratch.

Models — tinygpt picks the best per role; Pace runs them

Tinygpt’s deliverable per pillar is “the BEST model for this role, whether we trained it or vetted it.” Each model goes through tinygpt’s eval gate before Pace bundles it. Two paths:

• Train new specialist — when no off-the-shelf model fits the role well enough (e.g., Pace planner v9/v10, voice-edit, dictation post-proc). Factory loop produces LoRA + dataset + eval.
• Vet external — when an open-source model is already good enough (e.g., WhisperKit-large-v3-turbo for ASR, mxbai-embed-large for embeddings, UI-Venus for VLM). Tinygpt verifies on Pace’s eval suite, possibly fine-tunes lightly, then qualifies for shipping.

Once approved, the artifact (LoRA file + base reference, or external model dir, or quantized CoreML bundle) is bundled into Pace and loaded in-process via Pace’s embedded MLX-Swift / WhisperKit / CoreML runner. All models load at app boot (eager), live resident in memory, fire conditionally per intent.

Every baby is English-only + Mac-only. No localization. No cross-platform. Narrower training corpora → faster convergence → higher precision per param.

100 ms per step — measured + targeted

Pace doesn’t always run all steps. Intent decides which sub-pipeline fires.

Critical paths and the 100 ms end-to-end push

The target is 100 ms cradle-to-grave for the lightest action. Today we’re at ~400-500 ms. Get there by:

1. Parallelize, don’t serialize. The classical “ASR finishes → planner starts → executor starts” pipeline is dead. Run all three in overlap.
2. Speculative dispatch on ASR partials. If partial transcript matches a high-confidence trigger (“draft mail to …”), kick off mailto: BEFORE the planner finishes. If planner disagrees, retract (close empty window). Cheap retract, big perceived win.
3. Pre-baked prompt cache at boot. System-prompt KV cache resident in memory; every PTT press skips 100% of prefill.
4. ANE for planner, not GPU. We have M8 chained for Qwen3-0.6B (#269). Run the planner on ANE so GPU is free for vision; ANE first-token is sub-30 ms.
5. MLX compile + spec decode (#262). Pending. Cuts decode latency 2-3×.
6. Skip initial TTS for routine actions. “Click save” doesn’t need spoken confirmation — the click IS the feedback. Pace now suppresses initial narration for routine local plans, gates stream-time narration once a routine plan is visible, and still speaks post-action results/failures.
7. Streaming partial-JSON → executor. Already shipped (#274). Body chunks dispatch as they emit; don’t wait for closing brace.
8. Smaller planner for routing-only cases. If a 200M-param model can do 95% of routing at 5× speed, ship it as the fast-path and fall back to v10 only when 200M is uncertain.

Path-specific targets (PERCEIVED-FEEDBACK = first-thing-user-sees-or-hears):

Perceived ≤ 100 ms is the bar for the lightest cases. Everything heavier should still START in < 200 ms even if completion takes longer. Never let user wait without feedback.

Text injection rules

1. AX setValue on AXValue / AXSelectedTextRange is primary. No clipboard pollution. Works in most AX-compliant apps.
2. CGEventKeyboardSetUnicodeString (typewriter mode) is fallback for Electron/web/non-AX-compliant. Slower but reversible.
3. NSPasteboard + ⌘V is BANNED. It pollutes user’s clipboard and Wispr users report this as a pain point. We do not bank a moat then give it away with one paste call.
4. Secure-text-entry fields (passwords) are explicitly out of scope. We refuse, with a spoken explanation.

Constraint enforcement

• All planner outputs are grammar-constrained JSON (tinygpt serve --grammar).
• All action calls are validated against the action’s args schema before dispatch.
• The model cannot emit a malformed JSON or an unknown action name. This is a hard guarantee from the decode layer, not a runtime check.

Zero-cloud rule

No network egress. Loopback-only HTTP is allowed for local development runtimes such as LM Studio, and PaceLocalEndpointGuard refuses planner/VLM endpoints that are not localhost, 127.0.0.0/8, or ::1. No telemetry “for analytics”. Crash logs are local-only or opt-in plain-text. The product positioning is unambiguous and we will not undermine it for a feature shortcut. If a capability requires cloud, we don’t ship it.

What we are NOT doing

• Multilingual support (English-only)
• Windows / Linux / iPhone (Mac-only)
• Marketplace / SDK (closed-source consumer product)
• Cloud anything (see above)
• BYOK to OpenAI/Anthropic (violates positioning)
• Generic chat (Pace acts, doesn’t ramble)
• ReAct multi-turn agentic loops (latency-killing; single-shot tool dispatch only)
• Reinventing what Apple already shipped

The ordering

Built in this sequence because each baby unblocks the next. As of v0.3.12 items 9-14 (the entire Her-arc) have shipped; items 1-8 are partial (the queued slices are either model-blocked or actionable code work — see docs/product/prds/README.md for the split).

1. v9 (LoRA path parked; runtime planner is LM Studio qwen3-30b-a3b) — body streaming demo [PARTIAL — Pace-side wiring shipped; latency demo queued]
2. Executor surface (Pace-side Swift, AX-first dispatcher with first-party fallbacks) [PARTIAL — dispatcher shipped; real-app smoke flow queued]
3. v10 (parameterized actions) — depends on executor surface existing [PARTIAL — parser/validation shipped; grammar gate + runtime-default switch queued]
4. WhisperKit integration — replace Apple Speech for code-mode + vocab biasing [PARTIAL — model-blocked]
5. RAG layer (embedding model + index over Mail/Notes/files/past sessions) [PARTIAL — lexical + best-effort re-ranker shipped; bundled embedding + SQLite-vec queued]
6. VLM port + ANE chunked [PARTIAL — model-blocked]
7. Dictation post-processor + voice-edit specialist [PARTIAL — model-blocked]
8. HUD overlay + intent disambiguator carve-out [PARTIAL — visual target ambiguity queued]
9. Restraint policy for every proactive source [SHIPPED v0.3.12]
10. Episodic memory [SHIPPED v0.3.12]
11. Always-listening mode [SHIPPED v0.3.12]
12. Proactive nudges [SHIPPED v0.3.12]
13. Barge-in TTS interrupt [SHIPPED v0.3.12]
14. Demonstration replay [SHIPPED v0.3.12]
15. Polish + ship — in flight on the v0.3.x release cadence

Plus the v0.3.11 quality-of-life batch, which is not in the original ordering but landed before the Her-arc: chat interface, conversational thread memory, planner tier picker, first-run experience, morning triage, recipe library, trust surfaces, inclusivity surface, cloud-bridge toggle. See docs/product/prds/README.md for the full PRD index.

There are no months. There are next-correct-things. Do the next one.

Source PRDs

Most of Pace’s per-pillar PRDs have been consolidated after shipping — their durable record now lives in PROJECT_STATUS.md (full history) and STATUS.md (at-a-glance). The still-open PRDs are indexed in docs/product/prds/README.md:

• on-device-meeting-notes.md — meeting mode.
• premium-chat-panel.md — Claude-Desktop-style chat surface.
• teachable-skills.md — natural-language .skill.md authoring.

For per-system implementation detail see systems.md; for the per-file map see ../development/key-files.md.

This file is the canonical architecture map. When in doubt, this doc wins.