Automation evidence matrix
Automation evidence matrix
Privacy-safe build, release, activation, failure, distribution, and Foundry
evidence contracts for HeyPace (Pace). Implements the
heypace-automation-readiness capability from the
automate-heypace OpenSpec change.
Why this exists: Pace’s product contract is fully on-device — no cloud LLM/STT/TTS/telemetry. Automation must prove the app builds, releases, activates, and fails safely without routing screen, voice, transcript, or on-device context to fleet analytics. Where proof requires a physical device or signing material that automation cannot reach, the matrix records the gap as blocked rather than inheriting a simulator/build pass as a release pass.
Companion Robot remains a frozen input — it is not a separately automated
product. Companion-mode privacy boundaries are documented in
companion-mode-privacy.md and are
covered by the same privacy-boundary assertions as the rest of the app.
Layered evidence model
Each layer is independently reportable. A pass at an earlier layer does NOT satisfy a later layer.
| Layer | What it proves | Source of truth | Automation |
| — | — | — | scripts/automation-readiness.sh flag |
| landing | Canonical surface is live, build/indexable, exposes the latest release tied to a source revision | website/ Astro build, website/src/config/release-info.json, https://heypace.app | --landing |
| build | The Swift target compiles for macOS | leanring-buddy.xcodeproj (scheme leanring-buddy) | --build |
| tests | The Swift Testing suite executes a non-zero count and passes | leanring-buddyTests via scripts/test-pace.sh | --tests |
| simulator | The app launches in a macOS destination and the test host runs | xcodebuild test result bundle | --simulator |
| signing | A Developer ID / signing identity is reachable for release packaging | Keychain (read-only presence check, never exports key material) | --signing |
| device | A physical Apple Silicon Mac ran the release smoke checklist | docs/operations/release-smoke-checklist.md (manual) | --device (always reports blocked from automation) |
| distribution | A release artifact is published and the Sparkle appcast is consistent | appcast.xml, GitHub Releases | --distribution (read-only live artifact check; never publishes) |
| activation | A first non-empty spoken reply completed locally | PaceTelemetryLog.recordFirstSuccessfulLocalActivation (local OSLog only) | --activation (central collection intentionally N/A — see below) |
| failure | A crash/failure path emits version/build + aggregate failure class without user content | PaceTelemetryLog.recordFailure (local OSLog only) | --failure (covered by unit tests, not by automation runs) |
| release-readiness | A receipt aggregating the above layers without signing or publishing | scripts/release-readiness.sh → releases/readiness-receipt.json | --release-readiness |
Current evidence state (2026-07-19)
| Layer | Status | Evidence | Notes |
|---|---|---|---|
landing |
pass | website/ builds; release-info.json carries latest version, download URL, EdDSA signature, generatedAt, and sourceRevision |
Acquisition/download intent is observable via the /download route + GitHub Releases page; no cloud analytics SDK is linked by design. generate-release-info.sh now also records the source git revision. scripts/check-landing-health.sh verifies manifest consistency and live HTTP reachability. |
build |
pass | scripts/test-pace.sh compiles and runs the full suite on Xcode 27.0 Beta 3 (1606/1606 passed, 2026-07-19). The earlier Xcode 27 beta compile block (TestCompanionScreenAnalysisClient actor isolation) is resolved. |
CI still runs the suite on the pinned macos-26 runner with a zero-tests-executed guard. |
tests |
pass | 1606/1606 passed via scripts/test-pace.sh (isolated DerivedData, 2026-07-19). Includes the new PaceTelemetryLogFailureTests and PaceTelemetryLogPrivacyBoundaryTests. |
Swift Testing (import Testing) suite; xcresulttool summary is the source of truth for executed-count. |
simulator |
pass | The xcodebuild test result bundle is the simulator evidence (macOS destination). 1606/1606 tests executed in the test host. |
The macOS destination is the only destination — there is no iOS-adjacent target. |
signing |
blocked | scripts/release-pace.sh extracts the team ID from Keychain and attempts a team-signed build, falling back to ad-hoc. Automation does not read or export the key material; it records only presence/absence of a Developer ID. |
Signing material lives in Keychain; automation never reads key bytes. |
device |
blocked | docs/operations/release-smoke-checklist.md is a manual hardware checklist. The 2026-07-13 companion milestone explicitly risk-accepted the missing hardware measurements. |
Device proof is not remotely automatable — this is a durable blocker, not a regression. |
distribution |
blocked | appcast.xml and GitHub’s release API list build 19 (v0.3.19), but the anonymous public asset URL currently returns 404. The live check now catches this instead of accepting local manifest agreement. |
Restore or republish the release asset through the explicit release workflow, then rerun scripts/check-landing-health.sh; automation does not publish or sign it. |
activation |
pass locally / N/A centrally | A privacy-safe first-local-activation signal is defined (PaceTelemetryLog.recordFirstSuccessfulLocalActivation) and emitted to local OSLog after the first non-empty spoken reply completes. There is no fleet-bound event path by design. |
Foundry records the local contract and central N/A explicitly; it must not infer a centrally observed activation. |
failure |
pass (local) | PaceTelemetryLog.recordFailure emits FAIL kind=<class> ver=<version> build=<build> to the local unified log; CompanionManager+TrustSurfacesRuntime.speakPlainLanguageFailure calls it for every documented PaceFailureKind. No user content (transcript, screen, action target) is included. |
Covered by PaceTelemetryLogFailureTests and the privacy-boundary assertion in PaceTelemetryLogPrivacyBoundaryTests. |
release-readiness |
blocked until manual gates pass | scripts/release-readiness.sh aggregates the layers above into releases/readiness-receipt.json without signing or publishing and exits 2 while signing, device, or other required evidence is blocked. |
Receipt generation is not itself a readiness pass; Foundry preserves the blocked state and may prepare diagnostics or PRs only. |
Privacy boundary (invariant)
The following MUST NOT appear in any automation artifact, log line, receipt, or Foundry-bound payload:
- raw audio buffers or audio file paths under
~/Library/Application Support/Pace/meetings/ - transcript text or transcript-derived strings
- screenshots, screen-context frames, or VLM element maps
- local action targets (click coordinates, AX labels, typed key chords)
- signing key material, Keychain values, or
PaceKeychainStoresecrets companion-observations.jsoncontentspace-tuned-turns.jsonlcontents (redacted export is opt-in and stays local)
PaceTelemetryLog.recordFailure and recordFirstSuccessfulLocalActivation
accept only closed telemetry enums and emit aggregate, privacy: .public
fields: failure class, outcome, activation kind, app version, build number,
and counts. The privacy-boundary tests
(PaceTelemetryLogPrivacyBoundaryTests) assert this at compile time by
calling every recording function with representative inputs and confirming
the API surface accepts only public-annotated scalars.
Foundry integration
Automation emits Foundry-readable receipts to releases/readiness-receipt.json.
Foundry may:
- read the receipt and the evidence matrix above,
- prepare diagnostics, PRs, or bounded tasks,
- surface blockers (e.g. the Xcode 27 beta compile block) to the dashboard.
Foundry MUST NOT:
- sign, notarize, or publish a release,
- enroll a physical device,
- push to TestFlight or the App Store (no such path exists today),
- deploy production (landing deploys remain manual
workflow_dispatch), - read or centralize the private-context fields listed in the privacy boundary above.
Out of scope
- Hosted backend, accounts, or cloud telemetry — none will be introduced.
- App Store / TestFlight distribution — none exists and none is planned.
- Automated signing —
scripts/release-pace.shremains human-triggered. - Automated physical-device smoke —
docs/operations/release-smoke-checklist.mdremains a manual hardware gate. - Companion Robot as a separately automated product — frozen input.