From 37dceb9ac5ce60c8a158890fec8061d1ba500006 Mon Sep 17 00:00:00 2001 From: Kaysser Kayyali Date: Mon, 22 Jun 2026 16:27:51 +0000 Subject: [PATCH] feat: refine workflow, take first steps towards thiking about other peoples preferences --- .claude/skills/fobsidian-sync-setup/SKILL.md | 157 +++++++ .env.example | 66 +++ .gitignore | 14 +- .obsidian/snippets/vault-colors.css | 38 ++ .raw/.gitkeep | 0 CLAUDE.md | 55 +++ README.md | 26 ++ _meta/hot.md | 24 ++ _meta/index.md | 47 +++ _meta/log.md | 20 + _meta/overview.md | 44 ++ _templates/component.md | 23 + _templates/decision.md | 22 + _templates/dependency.md | 25 ++ _templates/flow.md | 26 ++ _templates/module.md | 30 ++ docker-compose.yml | 59 +++ .../.decision-log.md | 79 ++++ .../prd.md | 304 ++++++++++++++ .../review-engineering.md | 297 +++++++++++++ .../review-launchable.md | 396 ++++++++++++++++++ .../review-rubric.md | 192 +++++++++ scripts/start-relay-session.js | 141 +++++++ src/dashboard.html | 40 +- src/server.ts | 211 +++++++++- wiki/components/_index.md | 13 + wiki/components/content-hash.md | 35 ++ wiki/components/foundry-block.md | 36 ++ wiki/components/name-uuid-resolver.md | 28 ++ wiki/decisions/001-external-foundry.md | 32 ++ wiki/decisions/002-tool-host-run.md | 30 ++ .../003-relay-no-journal-db-no-docker-stop.md | 35 ++ .../decisions/004-dropped-browser-obsidian.md | 37 ++ wiki/decisions/005-autosync-o-to-f-only.md | 40 ++ wiki/decisions/_index.md | 15 + wiki/dependencies/_index.md | 14 + wiki/dependencies/classic-level.md | 34 ++ .../foundryvtt-rest-api-module.md | 36 ++ wiki/dependencies/linkedom.md | 32 ++ wiki/dependencies/threehats-relay.md | 38 ++ wiki/flows/_index.md | 14 + wiki/flows/autosync-watch-loop.md | 41 ++ wiki/flows/index-recommend.md | 41 ++ wiki/flows/push-flow.md | 39 ++ wiki/flows/refresh-flow.md | 37 ++ wiki/modules/_index.md | 17 + wiki/modules/autosync.md | 50 +++ wiki/modules/batch.md | 56 +++ wiki/modules/cli.md | 32 ++ wiki/modules/config.md | 29 ++ wiki/modules/push.md | 48 +++ wiki/modules/relay-client.md | 43 ++ wiki/modules/server.md | 59 +++ 53 files changed, 3292 insertions(+), 5 deletions(-) create mode 100644 .claude/skills/fobsidian-sync-setup/SKILL.md create mode 100644 .env.example create mode 100644 .obsidian/snippets/vault-colors.css create mode 100644 .raw/.gitkeep create mode 100644 CLAUDE.md create mode 100644 _meta/hot.md create mode 100644 _meta/index.md create mode 100644 _meta/log.md create mode 100644 _meta/overview.md create mode 100644 _templates/component.md create mode 100644 _templates/decision.md create mode 100644 _templates/dependency.md create mode 100644 _templates/flow.md create mode 100644 _templates/module.md create mode 100644 docker-compose.yml create mode 100644 docs/prds/prd-foundry-obsidian-sync-2026-06-22/.decision-log.md create mode 100644 docs/prds/prd-foundry-obsidian-sync-2026-06-22/prd.md create mode 100644 docs/prds/prd-foundry-obsidian-sync-2026-06-22/review-engineering.md create mode 100644 docs/prds/prd-foundry-obsidian-sync-2026-06-22/review-launchable.md create mode 100644 docs/prds/prd-foundry-obsidian-sync-2026-06-22/review-rubric.md create mode 100644 scripts/start-relay-session.js create mode 100644 wiki/components/_index.md create mode 100644 wiki/components/content-hash.md create mode 100644 wiki/components/foundry-block.md create mode 100644 wiki/components/name-uuid-resolver.md create mode 100644 wiki/decisions/001-external-foundry.md create mode 100644 wiki/decisions/002-tool-host-run.md create mode 100644 wiki/decisions/003-relay-no-journal-db-no-docker-stop.md create mode 100644 wiki/decisions/004-dropped-browser-obsidian.md create mode 100644 wiki/decisions/005-autosync-o-to-f-only.md create mode 100644 wiki/decisions/_index.md create mode 100644 wiki/dependencies/_index.md create mode 100644 wiki/dependencies/classic-level.md create mode 100644 wiki/dependencies/foundryvtt-rest-api-module.md create mode 100644 wiki/dependencies/linkedom.md create mode 100644 wiki/dependencies/threehats-relay.md create mode 100644 wiki/flows/_index.md create mode 100644 wiki/flows/autosync-watch-loop.md create mode 100644 wiki/flows/index-recommend.md create mode 100644 wiki/flows/push-flow.md create mode 100644 wiki/flows/refresh-flow.md create mode 100644 wiki/modules/_index.md create mode 100644 wiki/modules/autosync.md create mode 100644 wiki/modules/batch.md create mode 100644 wiki/modules/cli.md create mode 100644 wiki/modules/config.md create mode 100644 wiki/modules/push.md create mode 100644 wiki/modules/relay-client.md create mode 100644 wiki/modules/server.md diff --git a/.claude/skills/fobsidian-sync-setup/SKILL.md b/.claude/skills/fobsidian-sync-setup/SKILL.md new file mode 100644 index 0000000..f989a17 --- /dev/null +++ b/.claude/skills/fobsidian-sync-setup/SKILL.md @@ -0,0 +1,157 @@ +--- +name: fobsidian-sync-setup +description: Onboard a fresh clone — bring up the relay, wire it to the user's external Foundry, launch the headless session, and start the sync dashboard. Run when the user says "setup", "/setup", "get me running", or "onboarding". +allowed-tools: Bash, Read, Write, Edit, AskUserQuestion, WebFetch +--- + +# Setup — get a fresh clone running + +You bring up the pieces this project needs and wire them to the user's +**external** Foundry. You run every command yourself — do NOT hand the user a shell +command to type. The only things the user types are values you can't know (their +Foundry URL, the API key the relay shows them) and those are collected through +`AskUserQuestion`. + +The stack: +- **relay** (container) — ThreeHats rest-api relay. Foundry's rest-api module + connects OUT to it over WebSocket. +- **dashboard** (host-run, `./sync.sh ui`) — the sync tool on :7788. The user edits + notes in their own Obsidian desktop app pointed at `$VAULT`; the dashboard + reads/writes the same files on disk. + +## Before you start + +Read `.env.example` so you know every var. The sync tool reads env directly +(`src/config.ts`); `sync.sh` loads `.env` and injects `--journal`. Pushing edits back +to Foundry happens through the dashboard's HTTP API (`POST /api/push`, +`/api/push-all`) or its **Auto-sync mode** (watch the vault + poll Foundry). + +If `.env` already exists and `RELAY_API_KEY` + `FOUNDRY_URL` are set, skip to the +"Bring up the relay" step (the user may be re-running you). + +## 1. Create `.env` from the template + +```bash +cp .env.example .env +``` + +## 2. Collect the user's Foundry details + +Use `AskUserQuestion` to gather: + +- `FOUNDRY_URL` — their Foundry server URL (e.g. `https://foundry.example.com`). +- `FOUNDRY_WORLD` — the world id/title to launch. +- `RELAY_USER` / `RELAY_PASSWORD` — a Foundry player account the headless session + logs in as. +- `VAULT` — absolute host path to their Obsidian vault root (the folder with + `.obsidian`); they edit this in their own Obsidian desktop app. `REFINED` defaults + to `${VAULT}/Refined`. + +Write each into `.env` with `Edit`. Leave `CC`, `JOURNAL`, and the +`FOUNDRY_CONTAINER`/`FOUNDRY_DATA_DIR` block commented/blank unless the user has them. + +## 3. Bring up the relay + +```bash +docker compose up -d relay +``` + +Poll until it answers (it does a first-run migration on `./relay-data`): + +```bash +# Retry for ~60s: +curl -sf http://localhost:${RELAY_PORT:-3010}/session +``` + +Print the relay UI URL to the user: `http://localhost:${RELAY_PORT:-3010}`. + +## 4. Create the relay API key (human-in-browser) + +This genuinely can't be scripted — the relay creates keys through its web signup: + +1. Tell the user: "Open `http://localhost:${RELAY_PORT}` in a browser, click Sign Up, + create an account, then copy the API key shown on your dashboard." +2. Use `AskUserQuestion` to collect the pasted key. +3. `Edit` it into `.env` as `RELAY_API_KEY=`. + +## 5. Connect Foundry's rest-api module (human-in-Foundry) + +Foundry's module connects OUT to the relay, so the relay must be reachable FROM the +Foundry host. Tell the user clearly: + +> In Foundry: install the "rest-api" module, enable it, and in its settings set the +> relay URL to `ws(s)://:`. If Foundry is on a +> different machine than this compose, expose `RELAY_PORT` (port-forward / tailnet / +> public domain) first — the module can't reach `localhost` from another host. + +Then verify a client connected: + +```bash +curl -s -H "x-api-key: ${RELAY_API_KEY}" http://localhost:${RELAY_PORT:-3010}/session +``` + +The response includes `activeClients` (or similar). If zero, stop and tell the user +the module isn't connected yet — do not proceed to the headless session. + +## 6. Launch the headless Foundry session + +```bash +node scripts/start-relay-session.js +``` + +This posts `/session-handshake` then `/start-session` (RSA-encrypts the password) to +spin up a headless browser logged into their Foundry. It can take up to 2 min. Poll +until `activeSessions` is non-empty: + +```bash +curl -s -H "x-api-key: ${RELAY_API_KEY}" http://localhost:${RELAY_PORT:-3010}/session +``` + +If it fails, the script prints which var is wrong (FOUNDRY_URL / RELAY_USER / +RELAY_PASSWORD / FOUNDRY_WORLD). Re-collect the bad value via `AskUserQuestion`, +`Edit` `.env`, and retry. + +## 7. Build the name↔uuid map + start the dashboard + +```bash +./sync.sh refresh --out "${OUT:-./out}" +``` + +This lists journal entries via the relay `/search` (zero Foundry downtime) and caches +`name-uuid.json`. + +Then launch the dashboard (omit `--cc` if `CC` is blank — the CLI auto-builds cc stubs +from the journal, or runs without if `JOURNAL` is also blank): + +```bash +./sync.sh ui --vault "${REFINED}" --out "${OUT:-./out}" --host 0.0.0.0 +``` + +Run it in the background (`run_in_background: true`) so it stays up. Print to the +user: + +- Dashboard: `http://:7788` +- It opens in dev mode with dry-run ON by default — actions preview without writing. + Uncheck dry-run to write into `--out`; restart with `--apply` to write back to the + real refined/cc dirs (with `.bak-` backups). +- Open `$VAULT` in your Obsidian desktop app to edit notes; the dashboard reads/writes + the same files. + +## 8. Tell the user how syncing works + +The project's hard rule: if it's not in the dashboard UI, it doesn't exist. From the +dashboard the user can: + +- **Push** one note (`POST /api/push {name}`) or **Push all** edited notes + (`POST /api/push-all`). +- Flip on **Auto-sync mode** — the dashboard watches the refined vault and pushes + Obsidian→Foundry automatically the moment a linked, seeded note is saved (guarded by + the note's `foundry.contentHash` baseline, so no-op saves don't re-push). Needs the + relay + a live headless session. Foundry→Obsidian is NOT automatic (the relay has no + change-push) — use the Sync / Re-pull buttons for that direction. + +## If something is already running + +- Relay already up + key already set → skip to step 5/6. +- Headless session already active (`activeSessions` non-empty) → skip to step 7. +- Dashboard already on :7788 → don't start a second one; just print its URL. \ No newline at end of file diff --git a/.env.example b/.env.example new file mode 100644 index 0000000..87d81c2 --- /dev/null +++ b/.env.example @@ -0,0 +1,66 @@ +# ───────────────────────────────────────────────────────────────────────────── +# foundry-obsidian-sync — environment template. +# Copy to .env and fill in: cp .env.example .env +# (Or just run the /setup skill, which fills most of this in for you.) +# .env is gitignored — never commit real keys or passwords. +# ───────────────────────────────────────────────────────────────────────────── + +# === Relay (ThreeHats foundryvtt-rest-api-relay) === +# Host port the relay publishes (its web UI + REST API + WS endpoint live here). +RELAY_PORT=3010 +RELAY_CONTAINER=foundry-rest-api-relay +# URL the relay advertises to itself (email links, etc.). Usually http://localhost:. +RELAY_FRONTEND_URL=http://localhost:3010 +# URL the sync TOOL calls the relay at. For a local compose this is +# http://localhost:. If Foundry is remote and you front the relay with +# a public/tailnet domain, set this to that URL instead. +RELAY_URL=http://localhost:3010 +# x-api-key for /get, /update, /search. Created via the relay's web UI on first run +# (sign up at http://localhost:, copy the key from the dashboard). +RELAY_API_KEY= +# Optional: pin a specific connected Foundry client (leave blank to auto-resolve +# when exactly one client is connected to the key). +RELAY_CLIENT_ID= + +# === Headless Foundry session (the relay drives a headless browser at Foundry) === +# Your Foundry server URL. The relay logs a headless user into THIS Foundry. +FOUNDRY_URL=https://your-foundry.example.com +# Foundry player credentials the headless session uses to log in. +RELAY_USER= +RELAY_PASSWORD= +# Foundry world id/title to launch in the headless session. +FOUNDRY_WORLD= +# Seconds before an idle headless session is reaped (relay env). +HEADLESS_SESSION_TIMEOUT=3600 + +# IMPORTANT — networking: Foundry's rest-api module connects OUT to the relay over +# WebSocket. So the relay must be REACHABLE FROM your Foundry host. If Foundry runs +# elsewhere, expose RELAY_PORT (port-forward / tailnet / public domain) and point the +# module's relay URL at ws(s)://:. + +# === Vault (the sync tool / dashboard reads & writes this directly on the host) === +# Absolute path on the HOST to your Obsidian vault root (the folder containing +# .obsidian). You edit these notes in your own Obsidian desktop app, pointed at this +# vault; the host-run dashboard reads/writes the same files. +# QUOTE any path with spaces — sync.sh sources this file in bash, so an unquoted +# "VAULT=/home/me/My Vault" tries to run "Vault" as a command and VAULT stays unset. +VAULT="/home/me/My Vault" +# The refined-notes subdirectory the dashboard's --vault points at. Defaults to +# ${VAULT}/Refined if you leave this blank. +REFINED="${VAULT}/Refined" +# Optional Campaign Codex export dir (omit to let the dashboard auto-build stubs). +CC= + +# === Sync tool === +# Optional: a Foundry journal LevelDB snapshot, for offline dashboard indexing +# (to-foundry / ui). Leave blank if you only ever push live via the relay. +JOURNAL= +# Sandbox output dir for the tool (name-uuid.json, index.json, converted notes). +OUT=./out + +# === Foundry host control (OPTIONAL — only for `refresh --full-index`) === +# Only needed if you run `./sync.sh refresh --full-index` to stop/start a LOCAL +# Foundry Docker container and read its live journal LevelDB. Live `push` and plain +# `refresh` go through the relay and never touch these. +FOUNDRY_CONTAINER= +FOUNDRY_DATA_DIR= \ No newline at end of file diff --git a/.gitignore b/.gitignore index 83774dd..01c2d69 100644 --- a/.gitignore +++ b/.gitignore @@ -2,8 +2,20 @@ node_modules/ # Local live-stack secrets (relay API key, Foundry container/world paths). .env .env.local +# Compose runtime state (relay sqlite DB + headless-chrome profile). +relay-data/ +# Tool sandbox output (name-uuid.json, index.json, converted notes). +out/ # Copied world-data fixtures (your private Foundry snapshot) — never commit. tests/fixtures/ # Apply-mode backups written by --apply. *.bak-* -*.foundry.json \ No newline at end of file +*.foundry.json + +# Wiki (Mode B): .raw/ holds local source dumps (private/large) — keep the folder via .gitkeep. +.raw/* +!.raw/.gitkeep +# Obsidian app config is machine-specific — but share the CSS snippet. +.obsidian/* +!.obsidian/snippets/ +!.obsidian/snippets/* \ No newline at end of file diff --git a/.obsidian/snippets/vault-colors.css b/.obsidian/snippets/vault-colors.css new file mode 100644 index 0000000..e6e45d2 --- /dev/null +++ b/.obsidian/snippets/vault-colors.css @@ -0,0 +1,38 @@ +/* vault-colors.css — Mode B (GitHub/Repository) folder colors + custom callouts. + Enable: Settings > Appearance > CSS Snippets > open folder > (file is here) > + refresh icon > toggle on. */ +:root { + --wiki-1: #4fc1ff; /* modules */ + --wiki-2: #c586c0; /* components */ + --wiki-3: #dcdcaa; /* decisions */ + --wiki-4: #ce9178; /* dependencies */ + --wiki-5: #6a9955; /* flows */ + --wiki-7: #569cd6; /* _meta */ +} + +/* Folder colors in the file explorer */ +.nav-folder-title[data-path^="wiki/modules"] { color: var(--wiki-1); } +.nav-folder-title[data-path^="wiki/components"] { color: var(--wiki-2); } +.nav-folder-title[data-path^="wiki/decisions"] { color: var(--wiki-3); } +.nav-folder-title[data-path^="wiki/dependencies"] { color: var(--wiki-4); } +.nav-folder-title[data-path^="wiki/flows"] { color: var(--wiki-5); } +.nav-folder-title[data-path^="_meta"] { color: var(--wiki-7); } +.nav-folder-title[data-path=".raw"] { color: #808080; opacity: 0.6; } + +/* Custom callouts */ +.callout[data-callout='contradiction'] { + --callout-color: 209, 105, 105; + --callout-icon: lucide-alert-triangle; +} +.callout[data-callout='gap'] { + --callout-color: 220, 220, 170; + --callout-icon: lucide-help-circle; +} +.callout[data-callout='key-insight'] { + --callout-color: 79, 193, 255; + --callout-icon: lucide-lightbulb; +} +.callout[data-callout='stale'] { + --callout-color: 128, 128, 128; + --callout-icon: lucide-clock; +} \ No newline at end of file diff --git a/.raw/.gitkeep b/.raw/.gitkeep new file mode 100644 index 0000000..e69de29 diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..dff89c9 --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,55 @@ +# obsidian-foundry-sync: LLM Wiki + +Mode: B (GitHub / Repository) +Purpose: Map the architecture of the obsidian-foundry-sync tool — modules, data flows, and the design decisions behind them. +Owner: Kaysser Kayyali +Created: 2026-06-22 + +## Structure + +``` +vault root (= this repo) +├── .raw/ # immutable source documents (README, code dumps, issue exports) +├── wiki/ +│ ├── modules/ # one note per major module / package (server, push, batch, autosync, …) +│ ├── components/ # reusable pieces (content-hash guard, foundry block, name↔uuid resolver) +│ ├── decisions/ # Architecture Decision Records (ADRs) +│ ├── dependencies/ # external deps + services (relay, classic-level, linkedom, rest-api module) +│ └── flows/ # data flows / request paths (push, refresh, auto-sync watch loop, index) +├── _meta/ +│ ├── index.md # master catalog of all wiki pages +│ ├── log.md # append-only operation log (newest at top) +│ ├── overview.md # executive summary of the whole wiki +│ └── hot.md # hot cache: ~500-word recent-context summary +├── _templates/ # one template per note type +└── CLAUDE.md # this file +``` + +This repo is ALSO the obsidian-foundry-sync code project. The wiki layer (above) documents +it; the code itself, build/test/usage, and the live-stack wiring live in `README.md`, +`src/`, `scripts/`, `docker-compose.yml`, and `.env.example`. When doing code work, prefer +those; when doing knowledge work, prefer the wiki. + +## Conventions + +- All notes use YAML frontmatter: `type`, `status`, `created`, `updated`, `tags` (minimum). + Module/flow/decision/dependency notes add the fields in their `_templates/` file. +- Wikilinks use `[[Note Name]]` format: filenames are unique, no paths needed. +- `.raw/` contains source documents: never modify them. +- `_meta/index.md` is the master catalog: update on every ingest/page creation. +- `_meta/log.md` is append-only: never edit past entries; new entries go at the TOP. +- `_meta/hot.md` is a cache, not a journal: overwrite it completely each time (keep <500 words). + +## Operations + +- **Ingest**: drop a source in `.raw/`, say "ingest [filename]" (→ `wiki-ingest`). +- **Query**: ask any question; Claude reads `_meta/index.md` first, then drills in (→ `wiki-query`). +- **Lint**: say "lint the wiki" to run a health check (→ `wiki-lint`). +- **Save**: say "save this" / "file this" (→ `save`). +- **Archive**: move cold sources to `.archive/` to keep `.raw/` clean. + +## Cross-project referencing + +Other Claude Code projects can point at this vault without duplicating context. In that +project's CLAUDE.md, add a `## Wiki Knowledge Base` block with `Path: `, and the +reading order: `_meta/hot.md` → `_meta/index.md` → `wiki//_index.md` → individual pages. \ No newline at end of file diff --git a/README.md b/README.md index 9d75f93..127b706 100644 --- a/README.md +++ b/README.md @@ -18,6 +18,32 @@ Foundry is the source of truth. Each Obsidian note carries a `foundry:` identity block (`cc_uuid`, `cc_type`, `folder_path`, `contentHash`, `syncedAt`) so the two sides stay linked across syncs and renames. +## Quick start (Docker Compose) + +Clone, then bring up the relay wired to your own **external** Foundry. You need: a +running Foundry VTT (license + the +[rest-api](https://github.com/ThreeHats/foundryvtt-rest-api-relay) module) and an +Obsidian vault on disk that you edit in your own Obsidian desktop app. + +1. `cp .env.example .env` and fill in `FOUNDRY_URL`, `FOUNDRY_WORLD`, `RELAY_USER`/ + `RELAY_PASSWORD`, and `VAULT` (or let the **`/setup`** skill do all of this for + you — it runs the commands itself; you only paste values it can't know). +2. `docker compose up -d relay` — relay on `:3010`. +3. Open `http://localhost:3010`, sign up, and copy the API key into `.env` as + `RELAY_API_KEY`. In Foundry, point the rest-api module at + `ws(s)://:3010`. +4. `node scripts/start-relay-session.js` — launches a headless Foundry session the + relay drives. +5. `./sync.sh refresh --out ./out` then `./sync.sh ui --vault "$VAULT/Refined" \ + --out ./out --host 0.0.0.0` — opens the dashboard on `:7788`. + +The sync tool stays host-run (it reads the journal LevelDB and shells out to +`docker`); the compose provides only the relay. You edit notes in your own Obsidian +desktop app pointed at `$VAULT`; the dashboard reads/writes the same files on disk. +Push edits back to Foundry from the **dashboard** (`/api/push`, `/api/push-all`), or +flip on **Auto-sync mode** to push Obsidian→Foundry automatically the moment you save +a note (Foundry→Obsidian stays manual — use Sync / Re-pull for that direction). + ## How it works 1. `db.ts` opens the journal LevelDB **read-only** with `classic-level`, indexes every diff --git a/_meta/hot.md b/_meta/hot.md new file mode 100644 index 0000000..956db86 --- /dev/null +++ b/_meta/hot.md @@ -0,0 +1,24 @@ +--- +type: meta +title: "Hot Cache" +updated: 2026-06-22T03:30:00 +--- + +# Recent Context + +## Last Updated +2026-06-22. Scaffolded the Mode B architecture wiki inside the obsidian-foundry-sync repo and seeded it from the codebase. + +## Key Recent Facts +- The compose now bundles ONLY the relay (browser Obsidian + plugin were removed 2026-06-21). See [[004-dropped-browser-obsidian]]. +- Auto-sync mode ([[autosync]]) is Obsidian→Foundry instant only, via `fs.watch` + a `foundry.contentHash` baseline guard. Foundry→Obsidian stays manual — the relay has no change-push and `/search` is minified with no content hash ([[005-autosync-o-to-f-only]]). +- `pushNote` always PUTs (no internal idempotency); the guard + `baselineNote` are what prevent loops/redundant pushes ([[push-flow]], [[autosync-watch-loop]]). +- The dashboard runs dev/dry-run by default; auto-sync baselines land in the `--out` mirror in dev, the real vault (with `.bak`) in apply. + +## Recent Changes +- Created: [[server]], [[autosync]], [[push]], [[relay-client]], [[batch]], [[content-hash]], [[foundry-block]], [[name-uuid-resolver]], [[push-flow]], [[refresh-flow]], [[autosync-watch-loop]], [[index-recommend]], ADRs [[001-external-foundry]]–[[005-autosync-o-to-f-only]], dependency pages. +- Wiki decisions recorded for: external Foundry, host-run tool, relay-over-LevelDB, dropped browser Obsidian, O→F-only auto-sync. + +## Active Threads +- Verify a real auto-sync push end-to-end once the dev headless session is back up (`activeClients > 0`); the scratch-note test proved the path but the session was down. +- MCP server for the wiki is NOT configured (deferred). \ No newline at end of file diff --git a/_meta/index.md b/_meta/index.md new file mode 100644 index 0000000..19c720b --- /dev/null +++ b/_meta/index.md @@ -0,0 +1,47 @@ +--- +type: meta +title: "Wiki Index" +updated: 2026-06-22 +--- + +# Wiki Index + +Master catalog of every page in the wiki. Update on every ingest / page creation. + +## modules +- [[server]] — HTTP dashboard + JSON API (dev/apply modes, mirror-aware reads, push/refresh/autosync) +- [[autosync]] — AutoSyncController: vault watcher → guard → live push → baseline +- [[push]] — pushNote: one refined note → live Foundry via relay (no Docker stop) +- [[relay-client]] — RelayClient over the ThreeHats rest-api relay (/get /update /create /search) +- [[batch]] — indexAll + recommend(); seed/sync/rePull/import row builders +- [[cli]] — cmdUi / refresh / push commands (env-driven) +- [[config]] — env-driven relay + Foundry host config + +## components +- [[content-hash]] — body contentHash idempotency guard +- [[foundry-block]] — the `foundry:` frontmatter block + read/baseline helpers +- [[name-uuid-resolver]] — name↔uuid map for link resolution + +## flows +- [[push-flow]] — Obsidian → live Foundry (relay /get → diff → /update) +- [[refresh-flow]] — name↔uuid map via relay /search (zero downtime) +- [[autosync-watch-loop]] — fs.watch → debounce → guard → pushNote → baselineNote +- [[index-recommend]] — indexAll + recommend() drift detection (local baselines + snapshot) + +## decisions (ADRs) +- [[001-external-foundry]] — cloner supplies FOUNDRY_URL; no Foundry in the compose +- [[002-tool-host-run]] — sync tool runs on the host, not in a container +- [[003-relay-no-journal-db-no-docker-stop]] — live push/refresh via relay; LevelDB only for full index +- [[004-dropped-browser-obsidian]] — removed containerized Obsidian + plugin; dashboard is the UI +- [[005-autosync-o-to-f-only]] — auto-sync is Obsidian→Foundry instant; Foundry→Obsidian stays manual + +## dependencies +- [[threehats-relay]] — foundryvtt-rest-api-relay (Go + headless Chrome) +- [[foundryvtt-rest-api-module]] — Foundry module that connects OUT to the relay +- [[classic-level]] — journal LevelDB reader +- [[linkedom]] — HTML parsing for the Obsidian→Foundry conversion + +## meta +- [[overview]] — executive summary +- [[hot]] — hot cache (~500 words) +- [[log]] — append-only operation log \ No newline at end of file diff --git a/_meta/log.md b/_meta/log.md new file mode 100644 index 0000000..1a2344e --- /dev/null +++ b/_meta/log.md @@ -0,0 +1,20 @@ +--- +type: meta +title: "Operation Log" +updated: 2026-06-22 +--- + +# Operation Log + +Append-only. Newest entry at the TOP. Never edit past entries. + +--- + +## 2026-06-22 — SCAFFOLD (Mode B) + +- Created vault structure inside the obsidian-foundry-sync repo: `wiki/{modules,components,decisions,dependencies,flows}`, `_meta/`, `_templates/`, `.raw/`. +- Created `_meta/index.md`, `_meta/log.md`, `_meta/overview.md`, `_meta/hot.md`; vault `CLAUDE.md`. +- Created `_templates/` for module, component, decision, dependency, flow. +- Added `.obsidian/snippets/vault-colors.css` (Mode B folder colors + custom callouts). +- Seeded real pages from the codebase: modules [[server]] [[autosync]] [[push]] [[relay-client]] [[batch]] [[cli]] [[config]]; components [[content-hash]] [[foundry-block]] [[name-uuid-resolver]]; flows [[push-flow]] [[refresh-flow]] [[autosync-watch-loop]] [[index-recommend]]; decisions [[001-external-foundry]]–[[005-autosync-o-to-f-only]]; dependencies [[threehats-relay]] [[foundryvtt-rest-api-module]] [[classic-level]] [[linkedom]]. +- Choices: vault inside the repo; no MCP server (yet); Obsidian app not installed (headless host). \ No newline at end of file diff --git a/_meta/overview.md b/_meta/overview.md new file mode 100644 index 0000000..5317a85 --- /dev/null +++ b/_meta/overview.md @@ -0,0 +1,44 @@ +--- +type: meta +title: "Wiki Overview" +updated: 2026-06-22 +--- + +# obsidian-foundry-sync — Architecture Wiki + +`obsidian-foundry-sync` is a host-run Node/TypeScript tool that bridges **Foundry VTT** +Campaign Codex journal data and an **Obsidian vault** bidirectionally. It reads the +Foundry journal LevelDB, converts between Obsidian markdown and Foundry's +`flags.campaign-codex` JSON, and pushes/pulls live through a ThreeHats rest-api relay +(Foundry keeps running — no Docker stop, no LevelDB write lock). + +## The three pieces + +1. **Relay** (container, `docker compose up -d relay`) — ThreeHats `foundryvtt-rest-api-relay`. + Foundry's rest-api module connects OUT to it over WebSocket. API keys come from its web + signup; a headless Foundry session drives it. +2. **Dashboard** (host-run, `./sync.sh ui`, :7788) — the sync tool. JSON API + UI for + seed/sync/import/link/refresh/push, and an **Auto-sync** toggle. +3. **Vault** (host path `$VAULT`, edited in your own Obsidian desktop app) — the + `Refined` subdir is the tool's `--vault`. + +## Key flows + +- **Push** ([[push-flow]]): refined note → relay `/get` live entry → build minimal diff + (`name` + dot-path `flags.campaign-codex`) → `/update`. Reversible (live-entry backup). +- **Refresh** ([[refresh-flow]]): relay `/search` (minified) → `name-uuid.json`. Zero downtime. +- **Auto-sync** ([[autosync-watch-loop]]): `fs.watch` the vault → guard on the note's + `foundry.contentHash` baseline → `pushNote` → `baselineNote`. Obsidian→Foundry instant. + Foundry→Obsidian is manual (Sync / Re-pull) — see [[005-autosync-o-to-f-only]]. + +## Key design decisions + +See [[decisions]]. Headlines: external Foundry ([[001-external-foundry]]); tool stays +host-run ([[002-tool-host-run]]); relay for live ops, LevelDB only for full index +([[003-relay-no-journal-db-no-docker-stop]]); browser Obsidian + plugin were dropped — the +dashboard is the UI ([[004-dropped-browser-obsidian]]). + +## Hard rule + +UI-only: if it's not in the dashboard UI, it doesn't exist. The agent runs commands; the +human only does UI steps (plus the unavoidable relay-signup + Foundry-module-connect). \ No newline at end of file diff --git a/_templates/component.md b/_templates/component.md new file mode 100644 index 0000000..ac28802 --- /dev/null +++ b/_templates/component.md @@ -0,0 +1,23 @@ +--- +type: component +path: "src/.ts" +status: active # active | deprecated | experimental +purpose: "" +depends_on: [] +used_by: [] +tags: [component] +created: YYYY-MM-DD +updated: YYYY-MM-DD +--- + +# + +## Purpose + +## API + +## How it works + +## Used by + +## Notes / gotchas \ No newline at end of file diff --git a/_templates/decision.md b/_templates/decision.md new file mode 100644 index 0000000..6dee375 --- /dev/null +++ b/_templates/decision.md @@ -0,0 +1,22 @@ +--- +type: decision +status: active # active | superseded | deprecated +date: YYYY-MM-DD +context: "" +decision: "" +consequences: "" +tags: [decision, adr] +created: YYYY-MM-DD +updated: YYYY-MM-DD +--- + +# ADR — + +## Context + +## Decision + +## Consequences + +## Related +- [[<other ADR>]] \ No newline at end of file diff --git a/_templates/dependency.md b/_templates/dependency.md new file mode 100644 index 0000000..9558d9c --- /dev/null +++ b/_templates/dependency.md @@ -0,0 +1,25 @@ +--- +type: dependency +name: "" +version: "" +kind: service # service | library | module | image +status: active # active | evaluating | deprecated +purpose: "" +risk: low # low | medium | high +tags: [dependency] +created: YYYY-MM-DD +updated: YYYY-MM-DD +--- + +# <Dependency Name> + +## What it is + +## Why we depend on it + +## How we use it + +## Risk / lock-in + +## Related +- [[<module that uses it>]] \ No newline at end of file diff --git a/_templates/flow.md b/_templates/flow.md new file mode 100644 index 0000000..da02a1b --- /dev/null +++ b/_templates/flow.md @@ -0,0 +1,26 @@ +--- +type: flow +status: active # active | draft | deprecated +purpose: "" +actors: [] # modules / components / services involved +steps: [] +tags: [flow] +created: YYYY-MM-DD +updated: YYYY-MM-DD +--- + +# <Flow Name> + +## Purpose + +## Steps +1. +2. + +## Actors +- [[<module>]] + +## Edge cases / failure modes + +## Related +- [[<other flow>]] \ No newline at end of file diff --git a/_templates/module.md b/_templates/module.md new file mode 100644 index 0000000..5ff2a5b --- /dev/null +++ b/_templates/module.md @@ -0,0 +1,30 @@ +--- +type: module +path: "src/<file>.ts" +status: active # active | deprecated | experimental | planned +language: typescript +purpose: "" +maintainer: "" +last_updated: YYYY-MM-DD +linked_issues: [] +depends_on: [] # [[other module]] / [[component]] / [[dependency]] +used_by: [] +tags: [module] +created: YYYY-MM-DD +updated: YYYY-MM-DD +--- + +# <Module Name> + +## Purpose + +## Public surface +- `export fn …` — + +## How it works + +## Depends on + +## Used by + +## Notes / gotchas \ No newline at end of file diff --git a/docker-compose.yml b/docker-compose.yml new file mode 100644 index 0000000..34f9e2f --- /dev/null +++ b/docker-compose.yml @@ -0,0 +1,59 @@ +# Shareable stack: ThreeHats rest-api relay + browser-served Obsidian. +# +# Brings up the two containers the sync tool needs alongside an EXTERNAL Foundry +# instance (you supply FOUNDRY_URL — Foundry itself is not bundled; it needs a +# license + the rest-api module installed). The sync tool itself stays host-run +# (./sync.sh ui) so it can read the journal LevelDB and shell out to `docker`. +# +# Copy .env.example -> .env and fill it in, or run the /setup skill which does it +# for you. Then: docker compose up -d +# +# NOTE on networking: Foundry's rest-api module connects OUT to the relay over +# WebSocket. If your Foundry is on a different host than this compose, the relay's +# port (RELAY_PORT) must be reachable from Foundry (port-forward / tailnet / public +# domain). RELAY_URL below is what the *tool* calls (usually localhost). + +services: + relay: + # Pin to a release tag (e.g. :2.1.0) for production stability. + image: threehats/foundryvtt-rest-api-relay:latest + container_name: ${RELAY_CONTAINER:-foundry-rest-api-relay} + environment: + - DB_TYPE=sqlite + - APP_ENV=production + - ALLOW_HEADLESS=true + - PORT=3010 + - HEADLESS_SESSION_TIMEOUT=${HEADLESS_SESSION_TIMEOUT:-3600} + - FRONTEND_URL=${RELAY_FRONTEND_URL:-http://localhost:3010} + volumes: + - ./relay-data:/app/data + ports: + - "${RELAY_PORT:-3010}:3010" + # Headless Chrome (Puppeteer) needs a sizable /dev/shm to avoid crashes. + shm_size: "1g" + restart: unless-stopped + + # Optional GPU-accelerated relay for hosts with a /dev/dri device. Software + # rendering (the default `relay` service above) works without it, just slower. + # Enable with: docker compose --profile gpu up -d relay-gpu + relay-gpu: + image: threehats/foundryvtt-rest-api-relay:latest + container_name: ${RELAY_CONTAINER:-foundry-rest-api-relay}-gpu + profiles: ["gpu"] + environment: + - DB_TYPE=sqlite + - APP_ENV=production + - ALLOW_HEADLESS=true + - PORT=3010 + - HEADLESS_SESSION_TIMEOUT=${HEADLESS_SESSION_TIMEOUT:-3600} + - FRONTEND_URL=${RELAY_FRONTEND_URL:-http://localhost:3010} + - CHROME_GPU_MODE=gpu + - CAPTURE_BROWSER_CONSOLE=error + volumes: + - ./relay-data:/app/data + ports: + - "${RELAY_PORT:-3010}:3010" + devices: + - /dev/dri:/dev/dri + shm_size: "1g" + restart: unless-stopped \ No newline at end of file diff --git a/docs/prds/prd-foundry-obsidian-sync-2026-06-22/.decision-log.md b/docs/prds/prd-foundry-obsidian-sync-2026-06-22/.decision-log.md new file mode 100644 index 0000000..7a6533e --- /dev/null +++ b/docs/prds/prd-foundry-obsidian-sync-2026-06-22/.decision-log.md @@ -0,0 +1,79 @@ +# Decision Log — Live Relay Sync PRD + +Canonical memory + audit trail for this PRD run. Newest at top. Every decision, +change, and override lands here as the conversation unfolds. + +--- + +## 2026-06-22 — Divergence / conflict posture (the no-clobber decisions) + +- **Foundry-side baseline hash scope = content + name + folder_path.** Catches + Foundry-side renames and folder moves as divergence (surfaces them instead of + silently breaking name↔uuid links). Stored alongside the existing Obsidian + `foundry.contentHash` (new field, e.g. `foundry.ccHash`). +- **Both-sides-diverged default = surface conflict, DM resolves.** Never + auto-overwrite. Conflict shows as a dashboard row; DM picks a side or merges + via manual buttons. Strictest "never clobber work" posture. +- **Fail-safe when relay /get can't read Foundry = skip push + surface error.** + Do not fall back to Obsidian-side-only check (that reintroduces clobber risk). + DM sees a "couldn't verify Foundry side — not pushed" row. +- **Implies a hard NFR:** no auto-sync operation may overwrite a side that has + changed since the last sync. The current O→F auto-sync code violates this + (checks Obsidian side only) → must be fixed before/within delivery A. + +## 2026-06-22 — Vision follow-ups (source of truth, divergence, status) + +- **Source of truth = newest version of a doc** — not a fixed side. But **never + clobber work**: both-sides-diverged → reconcile, not overwrite. Manual buttons + stay as override/repair for unsynced-on-either-side or both-deviate cases. +- **Foundry = equal-origin editing surface during prep.** "Foundry is source of + truth" demoted to both-diverged tie-breaker rule (newest-wins vs Foundry-wins + vs manual merge — undecided). +- **"Not syncing" indicator location → DEFERRED.** No easy answer; Foundry UI not + ours; dashboard is the only surface we control now. Custom Foundry module is + future, not this PRD. +- **Sync-status-when-ON → dashboard, for now.** User proposed a maintained + **status note inside the vault** (`Sync Status.md`) as a lightweight parity + indicator — candidate feature, to confirm. +- **Divergence-detection question raised by user** → answered in conversation: + yes, guaranteeable, but only with a Foundry-side baseline hash + relay /get + before any auto push/pull. Current code lacks this (only checks Obsidian side) + → can silently clobber Foundry edits today. Gap for delivery A to close. + +## 2026-06-22 — Vision input (prep / run-the-match / status) + +- **Prep is the heart.** DM edits many notes at once, draws links, summarizes/ + closes prior journal notes, adds images/lore, creates objects. Data originates + from whichever tool is easier — Obsidian **or** Foundry. Files must stay synced + the whole time, bidirectionally, no button-babysitting. +- **Run-the-match = sync OFF, Foundry-centric.** DM generates many notes, mostly + in Foundry. System must make "not syncing" legible (Foundry should show it). +- **Status always visible** — on or off, the DM should see sync status at a + glance: when to expect parity and when not. +- **Open (probed next):** + - Confirm Foundry is an equal-origin editing surface during prep (challenges + ADR-005 / README "Foundry is source of truth, O→F hot path"). Reconcile. + - Where does the "not syncing" indicator live? Foundry UI is rendered by the + rest-api module (not ours). Candidate: a journal entry/flag the sync tool + writes that the module surfaces; or dashboard "SYNC PAUSED" state; or both. + - Sync-status-when-ON: is the existing autosync activity panel enough, or want + a persistent parity indicator ("in parity / N pending / last sync 12s ago")? + - Manual buttons (Sync/Re-pull/Push-all) — repair/override tools (stay) vs retire. + +## 2026-06-22 — Run opened + +- **Scope chosen:** A + B + C — full live-sync surface. A = ship & verify O→F + auto-sync (uncommitted controller/UI + empty `RELAY_CLIENT_ID` gap + end-to-end + live verification). B = Foundry→Obsidian auto direction (no relay push channel; + needs polling/snapshot diff). C = operational hardening (watch fallback, + concurrency/debounce, error surfacing, retry). +- **Stakes chosen:** public/launchable → full NFRs, error contracts, onboarding + rigor. +- **Working mode chosen:** coaching path, Vision + Features entry (capability-first; + single operator role = GM/world-builder). +- **Workspace bound:** `docs/prds/prd-foundry-obsidian-sync-2026-06-22/`. +- **No `_bmad/` project config** — running on neutral defaults. `planning_artifacts` + resolved to `docs/`. +- **External session blocker noted:** relay has no connected Foundry client right + now and `RELAY_CLIENT_ID` is empty; user will bring the headless session up + themselves. Live end-to-end verification (A) gated on that. \ No newline at end of file diff --git a/docs/prds/prd-foundry-obsidian-sync-2026-06-22/prd.md b/docs/prds/prd-foundry-obsidian-sync-2026-06-22/prd.md new file mode 100644 index 0000000..8981676 --- /dev/null +++ b/docs/prds/prd-foundry-obsidian-sync-2026-06-22/prd.md @@ -0,0 +1,304 @@ +--- +title: "Live Relay Sync — Auto-Sync & Bidirectional Hardening" +status: draft +created: 2026-06-22 +updated: 2026-06-22 +--- + +# Live Relay Sync — Auto-Sync & Bidirectional Hardening + +> Status: **draft** — being authored via bmad-prd coaching path. +> Scope: full live-sync surface over the ThreeHats relay — (A) ship & verify +> Obsidian→Foundry instant auto-sync, (B) add Foundry→Obsidian auto direction, +> (C) operational hardening. Stakes: public/launchable. + +## 1. Vision + +A DM's work has two phases, and live sync serves them differently. + +**Prep** is the heart of the tool. The DM is thinking through situations, lore, +and NPCs — editing many notes at once, drawing new links between items, +summarizing or closing out previous journal notes, dropping in images and lore, +creating new objects to build out the world. Data is coming from **whichever +tool is easier in the moment** — Obsidian or Foundry — and files must stay in +sync the whole time, across both directions, without the DM babysitting a sync +button. The feeling to deliver: *always in lockstep while I work*, bidirectionally, +across a flurry of simultaneous edits. + +**Running the match** is the live session. The DM is generating a lot of notes, +almost entirely inside Foundry, and the sync service is almost certainly **off** +in this phase. The system should make that state legible — Foundry should show +"not syncing" somewhere, so the DM never wonders whether prep edits are +propagating when they aren't. Sync is deliberately paused, not silently absent. + +**Status, always.** Whether sync is on or off, the DM should be able to see the +sync status at a glance — when to expect parity and when not to. No guessing +whether the vault and Foundry agree. + +_[CONFIRMED] Source of truth = the newest version of a document — not a fixed +side. But the system must **never clobber work**: when both sides have diverged +since the last sync, it must route to reconciliation instead of overwriting +either side. The manual buttons (Sync / Re-pull / Push-all) stay precisely for +this — quick overrides when a file is unsynced on any side or both deviate._ + +_[CONFIRMED] Foundry is an equal-origin editing surface during prep — data +originates in whichever tool is easier. "Foundry is source of truth" is demoted +to a tie-breaker rule for the both-diverged conflict case (to be decided: +newest-wins, Foundry-wins, or manual merge)._ + +_[DEFERRED] Where the "not syncing" indicator lives during run-the-match — no +easy answer yet; decision deferred. Foundry UI is rendered by the rest-api +module (not ours); the dashboard is the only realistic surface we control now. +A custom Foundry module (giving us what the relay does, plus indicators) is a +future exploration, not this PRD._ + +_[CONFIRMED] Sync-status-when-ON lives in the dashboard for now. Candidate +addition: a single **status note** the sync tool maintains inside the vault +(e.g. `Sync Status.md`) showing last-sync time/state — a lightweight, +in-our-control way for the user to see parity at a glance without a custom +module. (Proposed by user; to be confirmed as a feature.)_ + +## 2. Problem & Context + +**Where the tool is today.** obsidian-foundry-sync is an offline bidirectional +converter between a Foundry VTT Campaign Codex LevelDB snapshot and an Obsidian +vault, plus a dashboard that does **manual** live relay operations — push one, +push-all, refresh the name↔uuid index, sync, re-pull. An Obsidian→Foundry +instant auto-sync (`AutoSyncController` + dashboard toggle) is **code-complete +but uncommitted**, and it only checks the Obsidian side before pushing (it +compares the note's body hash to its `foundry.contentHash` baseline and pushes +on any difference). There is **no Foundry→Obsidian auto direction** — the relay +has no change-push channel and `/search` is minified with no content hash, so +F→O stays manual (Sync / Re-pull buttons). + +**The prep workflow doesn't fit manual sync.** A DM prepping a session edits +many notes at once, draws new links, summarizes/closes prior journal notes, +drops in images and lore, and creates new objects — with data originating in +**whichever tool is easier in the moment**, Obsidian or Foundry. Pressing +Sync/Re-pull per file doesn't scale to that churn; the value proposition of +auto-sync is exactly this phase. But auto-sync that only watches one direction +leaves half the prep edits unpropagated, and auto-sync that pushes without +checking the other side **can silently clobber Foundry edits** — the current +O→F code has this risk. + +**The run-the-match phase is Foundry-centric with sync off.** During the live +session the DM generates many notes, mostly inside Foundry, and the sync +service is expected to be off. Today "off" is silent — the DM can't tell +whether prep edits are still propagating. Status legibility (on/off, parity, +last sync) is missing on both sides. + +**Config/onboarding gap.** Live relay sync needs a connected Foundry client and +a valid `clientId`. `RELAY_CLIENT_ID` is currently empty in `.env`, and the +headless Foundry session that the relay drives is not always up. A launchable +tool needs this onboarding path to be discoverable and self-correcting. + +**Audience.** Today a single operator (the DM/world-builder). Intended to be +launchable — other DMs running it against their own Foundry worlds — so the +no-clobber, error-surfacing, and onboarding rigor must hold for non-author +users, not just for the one who wrote it. + +_[ASSUMPTION] The relay remains the live transport for this PRD; the "custom +Foundry module that gives us what the relay does" is explicitly future work, +not a dependency here._ + +## 3. Goals & Non-Goals + +### Goals + +- **G1 — No-clobber bidirectional auto-sync for prep.** Both directions sync + automatically; neither side is ever overwritten if it changed since last sync; + both-diverged conflicts surface for the DM to resolve. +- **G2 — Ship & verify the O→F auto-sync safely.** Commit the existing + controller/UI **with the divergence guard added** (Foundry-side baseline hash + + relay `/get` before acting), and verify one end-to-end live push. +- **G3 — Foundry→Obsidian auto direction.** A working F→O auto path despite the + relay having no push channel (polling / snapshot-diff design). +- **G4 — Legible sync status at all times.** The DM can see, at a glance, + whether sync is on or off, whether the vault and Foundry are in parity, and + when the last sync landed — in the dashboard, plus a maintained status note + inside the vault. +- **G5 — Operational hardening.** Recursive-watch fallback, tuned + concurrency/debounce, retry on transient relay 404/timeout, and visible + error rows in the dashboard. +- **G6 — Closed onboarding/config.** `RELAY_CLIENT_ID` and the connected-Foundry- + client requirement are discoverable and self-correcting from the dashboard. + +### Non-Goals + +- **Custom Foundry module** (indicators inside Foundry UI, relay replacement) — + future exploration, out of scope. +- **"Not syncing" indicator rendered inside Foundry** — deferred; Foundry UI is + not a surface we control. Surfaced via dashboard + vault status note instead. +- **Syncing during run-the-match.** Sync stays off by design in that phase; the + goal is legibility of the off state, not automation during the session. +- **Automatic semantic/3-way content merge.** Both-diverged = pick a side or + merge manually via the buttons; no auto content-merge engine. +- **Auto-sync of unlinked or unseeded notes.** Seed/link first remains a manual + prerequisite (unchanged from current behavior). +- **Full LevelDB / docker-stop index in the dashboard.** Remains CLI-only + (unchanged). + +## 4. Features & Functional Requirements + +FR IDs are stable. Grouped F1–F6; IDs are `FR-<group>.<n>`. + +### F1 — Obsidian→Foundry auto-sync (safe) + +- **FR-1.1** Watch the refined vault dir for `.md` saves using recursive + `fs.watch`, with a per-subdir fallback (re-scanning on subdir create/rename) + for platforms/Node versions without recursive watch. Skip `.obsidian` and + dotfiles. +- **FR-1.2** On a save, read the note and skip it if it has no + `foundry.cc_uuid` (unlinked) or no `foundry.contentHash` baseline (unseeded) + — seed/link remain manual prerequisites. +- **FR-1.3** Compute the current Obsidian body hash; if it equals the + `foundry.contentHash` baseline, skip (covers no-op saves and the watcher's + own post-push baseline write — no feedback loop). +- **FR-1.4** **Before pushing**, `relay /get` the live Foundry entry and + compute its Foundry-side hash over **content + name + folder_path**; compare + to the stored Foundry-side baseline (`foundry.ccHash`, new field). +- **FR-1.5** Route: Obsidian-changed **and** Foundry-unchanged (F-hash equals + `ccHash` baseline) → push O→F via the same `pushNote` path the manual push + button uses; re-baseline both sides on success. +- **FR-1.6** If the Foundry side is unreadable (`/get` 404 / timeout / session + down), **skip the push and surface an error row** — do not fall back to an + Obsidian-side-only check (that reintroduces clobber risk). +- **FR-1.7** After a successful push, re-baseline **both** `foundry.contentHash` + (Obsidian body) **and** `foundry.ccHash` (Foundry-side) to the new values. + Dev mode baselines land in the `--out` mirror; apply mode in the real vault + with a `.bak`. +- **FR-1.8** Auto-sync always applies live (dry-run not honored) — unchanged + from current behavior; the whole point is hands-off live push. + +### F2 — Foundry→Obsidian auto-sync + +- **FR-2.1** While auto-sync is ON, poll `relay /search` + (`documentType:JournalEntry`, minified) on a configurable cadence; build a + current `{uuid → name/img/folder}` snapshot. +- **FR-2.2** Diff the current snapshot against the last snapshot and the + vault's known linked notes (via `foundry.cc_uuid`) to detect Foundry-side + changes: renamed (name change, same uuid), moved (folder change), + content-changed (detected via `/get` hash compare), missing, or new. +- **FR-2.3** For each Foundry-changed **linked** note where the Obsidian side + is unchanged: `/get` the live entry, convert to refined markdown, write into + the vault, and re-baseline both sides. +- **FR-2.4** Never clobber an Obsidian-side change: vault-newer or + both-diverged notes route to F3 conflict handling, not auto-pull. +- **FR-2.5** New (cc-only) Foundry entries surface as **import candidates** + (existing import row) — do not auto-import. +- **FR-2.6** Poll cadence is configurable with a prep-tuned default (seconds); + a manual "catch up now" trigger is available alongside the background poll. + +### F3 — Divergence detection & conflict routing + +- **FR-3.1** Every sync tick (O→F or F→O) computes both-side hashes and routes + per the 2×2: parity / O-changed / F-changed / both-changed. +- **FR-3.2** both-changed → **do not auto-overwrite**; create a conflict row + in the dashboard summarizing both versions and highlighting the diff. +- **FR-3.3** The conflict row offers manual resolution: "push vault → + Foundry", "pull Foundry → vault", "mark resolved (no change)". +- **FR-3.4** Conflict state persists until the DM resolves it; a re-save on + either side does not auto-clear a known conflict. +- **FR-3.5** Foundry-side renames and folder moves (caught via name + folder in + the hash) surface as changes/conflicts, not silently absorbed. + +### F4 — Sync status & parity + +- **FR-4.1** Dashboard shows a persistent sync-status header: ON/OFF, mode + (dev/apply), watched dir. +- **FR-4.2** Dashboard shows a parity indicator: counts of in-parity / + O-pending / F-pending / conflict / unsynced-linked notes, plus a last-sync + timestamp. +- **FR-4.3** The sync tool maintains a `Sync Status.md` note in the vault (on a + path excluded from the watcher, never synced to Foundry) showing on/off, last + sync time, parity counts, and recent events — updated each tick. +- **FR-4.4** When sync is OFF, the dashboard shows a loud "SYNC PAUSED" state, + not a silent absence. +- **FR-4.5** Dashboard parity and the vault status note reflect one underlying + state (single source of truth for status). + +### F5 — Operational hardening + +- **FR-5.1** Recursive-watch fallback (FR-1.1) verified on the host kernel, + including re-scan on subdir create/rename so new folders get watched. +- **FR-5.2** Debounce window and max concurrency configurable; defaults tuned + for prep so a burst of simultaneous saves doesn't thrash or drop events. +- **FR-5.3** Transient relay errors (404 invalid client, 408/504 timeout, 5xx) + retried with bounded backoff; persistent failures surface as error rows. +- **FR-5.4** Every auto-sync op (push/skip/error/conflict) logged to the + activity panel with time, note, status, message; panel capped and scrollable. +- **FR-5.5** Inflight dedup + queue drain verified under burst — no dropped + events, no duplicate pushes. + +### F6 — Onboarding & config + +- **FR-6.1** If `RELAY_CLIENT_ID` is unset/empty, the dashboard surfaces a + clear "no clientId configured" state with guidance — not a silent 404 at push + time. +- **FR-6.2** If no Foundry client is connected to the relay (`activeClients = + 0`), the dashboard surfaces "Foundry not connected" and disables + enable-auto-sync until resolved; re-checked on a cadence. +- **FR-6.3** The dashboard can list connected relay clients (relay `/search` + with no clientId returns the client list on >1, or "No connected clients" on + 0) so the DM can pick/copy a valid `clientId` from the UI — no shell command. +- **FR-6.4** The existing `/setup` skill covers env wiring; the dashboard + reflects setup state and links into it. + +## 5. Non-Functional Requirements + +- **NFR-1 — No-clobber safety.** No auto-sync operation may overwrite a side + that has changed since the last sync. Both-diverged → conflict, never + auto-overwrite. (The hard requirement; the current O→F code violates it.) +- **NFR-2 — Fail-safe.** If the relay cannot read the Foundry side, the + operation is skipped and surfaced — never a blind push on Obsidian-side-only + evidence. +- **NFR-3 — Performance.** Debounce + bounded concurrency handle a ~50-note + prep burst without dropped events or relay thrash; F→O poll cadence does not + overload the relay or the host. +- **NFR-4 — Reliability.** Transient relay errors retried with backoff; + persistent errors surfaced within one tick. +- **NFR-5 — Observability.** Every operation is visible in the dashboard + activity panel and the vault status note; no silent skips or silent + overwrites. +- **NFR-6 — Onboardability.** A non-author user can get from clone → live sync + using the dashboard + `/setup`, without editing shell commands (per the + UI-only convention). +- **NFR-7 — Configurability.** Poll cadence, debounce, concurrency, and + status-note path are env/config-driven with safe defaults. +- **NFR-8 — Backward compatibility.** Existing manual buttons, seed/sync/ + rePull/import rows, dev/apply modes, and the CLI-only full LevelDB index all + keep working unchanged. + +## 6. Open Questions + +- **OQ-1** Conflict row quick-actions: beyond "push vault / pull Foundry / mark + resolved," do we also offer a one-click "newest mtime wins" convenience, or + keep every conflict fully manual? (Default posture is manual; this is a + convenience question.) +- **OQ-2** Where the "not syncing" indicator lives during run-the-match — + **deferred**. Foundry UI is not a surface we control; dashboard + vault status + note cover the legible case for now. Reopens when a custom Foundry module is + explored. +- **OQ-3** F→O poll cadence default (seconds) — pick during delivery B with + relay-load testing. +- **OQ-4** `Sync Status.md` path/name and its exclusion from both the O→F + watcher and the F→O pull — confirm during delivery. +- **OQ-5** New (cc-only) Foundry entries created during prep: stay as import + candidates (FR-2.5) or auto-import? Current spec: candidates only. +- **OQ-6** Conflict-row diff format — full unified diff vs. a condensed + summary — design during delivery. + +## 7. Out of Scope / Future + +Mirrors §3 Non-Goals, plus: + +- **Custom Foundry module** — a module that gives us what the relay does plus + in-Foundry indicators (sync status, parity). Future exploration; would reopen + OQ-2. +- **Syncing during run-the-match** — sync stays off by design in the live + session; legibility of the off state is in scope, automation during the + session is not. +- **Automatic semantic / 3-way content merge** — both-diverged is manual + resolution via buttons; no auto content-merge engine. +- **Auto-sync of unlinked or unseeded notes** — seed/link first stays manual. +- **Full LevelDB / docker-stop index in the dashboard** — stays CLI-only. \ No newline at end of file diff --git a/docs/prds/prd-foundry-obsidian-sync-2026-06-22/review-engineering.md b/docs/prds/prd-foundry-obsidian-sync-2026-06-22/review-engineering.md new file mode 100644 index 0000000..9167c2d --- /dev/null +++ b/docs/prds/prd-foundry-obsidian-sync-2026-06-22/review-engineering.md @@ -0,0 +1,297 @@ +--- +title: "Engineering-feasibility review — Live Relay Sync PRD" +reviewer: Claude (engineering-feasibility pass) +prd: docs/prds/prd-foundry-obsidian-sync-2026-06-22/prd.md +date: 2026-06-22 +scope: "Stress-test PRD technical claims against the actual codebase + relay constraints." +--- + +# Engineering-feasibility review + +Verdict at the end. Findings tiered **BLOCKER / HIGH / MEDIUM / LOW** with file:line +citations and a recommended fix for each. + +## What the code actually does today (baseline facts) + +- `contentHash(text)` hashes **only the canonicalized body** — wikilink-canonicalize + + whitespace-canonicalize, then SHA-256. It does **not** include the note name, the + folder path, or any frontmatter. (`src/normalize.ts:24`, `canonicalize` at + `src/normalize.ts:19`.) +- The `foundry:` block has no `ccHash` field. `readFoundryBlock` returns whatever keys + are in the nested map (`src/frontmatter.ts:9`); `baselineFoundryBlock` + (`src/server.ts:289`) rewrites **only** `contentHash:` and `syncedAt:` lines. So the + Foundry-side baseline the PRD depends on does not exist yet — it is genuinely new. +- `AutoSyncController.process()` (`src/server.ts:582`) reads the note, compares + `contentHash(body)` to `fb.contentHash`, and on a difference calls `pushNote` **with + no relay `/get` of its own**. `pushNote` does call `relay.getEntry(id)` + (`src/push.ts:142`) but **only to build the diff** (preserve ownership/pages/ + existing image) — it never compares a Foundry-side hash to a baseline. So if the + Foundry side changed since the last sync, `pushNote` still PUTs `{ name, + "flags.campaign-codex": cc }` and **overwrites the Foundry-side edit silently**. + Clobber risk is **real, not theoretical**. +- Relay `/get` returns the full doc: `{ data: { name, type, _id, uuid, folder, pages, + ownership, flags, ... } }` (`docs/relay-api.md:31-33`). `folder` is present on + `/get`. +- Relay `/search` minified returns **only** `{ uuid, id, name, img, documentType }` + (`docs/relay-api.md:55-59`, mirrored in `src/relay/client.ts:19-25`, + `searchJournalEntries` at `src/relay/client.ts:97`). **No `folder`. No content. No + content hash. No mtime.** This is the load-bearing constraint for F2. +- ADR 005 (`wiki/decisions/005-autosync-o-to-f-only.md`) already analyzed F→O polling + and rejected it on cost grounds: ~800 `/get` calls/min for a ~200-entry world at a + 15s cadence. The PRD does not cite or reconcile with this ADR. + +--- + +## BLOCKER + +### B1 — FR-2.1 / FR-2.2: `/search` minified has NO `folder`. Folder-move detection from the poll snapshot is impossible. + +**Claim (PRD FR-2.1):** "poll `relay /search` … build a current `{uuid → +name/img/folder}` snapshot." + +**Claim (PRD FR-2.2):** "moved (folder change)" is detected by diffing the current +snapshot against the last snapshot. + +**Reality:** `relay /search?minified=true` returns `{ uuid, id, name, img, +documentType }` only (`docs/relay-api.md:55-59`, `src/relay/client.ts:19-25`, +`src/relay/client.ts:97-105`). There is **no `folder` field** in the minified result. +The PRD's own §2 admits "/search is minified with no content hash" but then FR-2.1 +quietly assumes `folder` is present — it is not. You cannot build a +`{uuid → folder}` snapshot from `/search`, and you cannot detect folder moves by +snapshot-diffing alone. + +**Impact:** FR-2.2's "moved (folder change)" detection and FR-3.5's "folder moves +surface as changes/conflicts" are **not implementable as specified**. Folder moves +would be invisible to the F→O poll entirely. + +**Fix:** Pick one and write it into the PRD: +1. **Per-note `/get` on every poll** to read `folder` (and content). This is the only + way to detect moves from the relay at all, and it collapses F2 into "every poll = + `/get` per linked note" (see B2). Accept the cost explicitly and override ADR 005. +2. **Drop folder-move detection from F2.** Foundry-side renames are detectable + (name is in `/search`); folder moves are not. State this as a known gap and route + moves to the next manual Sync / Re-pull. This is the cheaper path and is honest + about the relay's surface. +3. Propose a relay change (non-goal per PRD §3) — not available for this PRD. + +Recommended: **option 2** for this PRD, with option 1 as an explicit opt-in +"deep poll" mode. Either way FR-2.1 and FR-2.2 must be rewritten to drop `folder` +from the `/search`-derived snapshot. + +--- + +### B2 — FR-2.2: content-change detection requires a `/get` per linked note per poll. The PRD does not account for the load, and this directly contradicts ADR 005. + +**Claim (PRD FR-2.2):** "content-changed (detected via `/get` hash compare)." + +**Reality:** `/search` minified has no content and no content hash +(`docs/relay-api.md:55-59`). The only way to detect a Foundry-side content edit is to +`/get` the full entry and hash it. So "detect content changes" = " `/get` every +linked note every poll cycle." For the ~200-entry world in ADR 005 at the PRD's +"seconds" default cadence (FR-2.6, OQ-3), this is **thousands of `/get` calls/min**, +all funneled through one headless Foundry WebSocket. ADR 005 rejected exactly this +design at 15s/800-calls-min as "heavy and fragile." + +**Impact:** NFR-3 ("does not overload the relay or the host") is **unsatisfiable** at +the cadence the PRD hints at, with the design the PRD specifies. The PRD cannot +both (a) detect Foundry content changes automatically and (b) avoid per-note `/get` +load. One has to give. + +**Fix:** Make the F→O design explicit and costed in the PRD: +- **Default: shallow poll.** `/search` only → detect **renames + new + missing** + (all detectable from minified `/search`). Content changes and folder moves are + **not** auto-detected; they surface on the next manual Sync / Re-pull or on a + user-triggered "catch up now" deep sweep (FR-2.6 already has the trigger — promote + it to the only content-change path). +- **Opt-in deep poll:** a `/get`-per-linked-note sweep, cadence in **minutes**, not + seconds, with a concurrency cap (the existing `mapPool` at `src/server.ts:317` is + the right primitive). State the call-rate budget in NFR-3. +- Add a line to §6 acknowledging this supersedes ADR 005's "F→O stays manual" + conclusion **only for rename/new/missing**, not for content. + +Without this, F2 is not shippable as written. + +--- + +## HIGH + +### H1 — FR-1.4 / NFR-1: the "Foundry-side hash over content + name + folder" is a brand-new function, not derivable from `contentHash`. The PRD under-specifies it. + +**Claim (PRD FR-1.4):** compute a Foundry-side hash over "content + name + +folder_path" from the `/get` response and compare to a new `foundry.ccHash` baseline. + +**Reality:** +- `contentHash` (`src/normalize.ts:24`) hashes **canonicalized Obsidian markdown + body only**. It cannot be reused for the Foundry side — the Foundry `/get` response + stores content inside `flags["campaign-codex"].data` (HTML per + `obsidianToFoundryJsonLive`), not as refined markdown. Hashing the raw HTML is + fragile (Foundry may round-trip HTML attributes/whitespace); hashing requires + either (a) a stable HTML canonicalizer or (b) converting `cc.data` HTML back to + refined markdown and reusing `contentHash` — which is a non-trivial round-trip + through `entryToObsidian`-style logic that the PRD does not mention. +- `name` and `folder` are present on `/get` (`docs/relay-api.md:31-33`) and are + stable scalars — hashing those is trivial. +- The hard part is **content**. The PRD says "content + name + folder_path" as if + it's obvious; it is not. The PRD needs to specify the exact Foundry-side hash + input: is it `cc.data` raw, canonicalized HTML, or re-converted markdown? Each + has different drift properties. + +**Impact:** If the Foundry-side hash is unstable (e.g. Foundry re-serializes HTML +between pushes), the divergence guard false-positives every tick → every save becomes +a conflict → the tool is unusable. This is the single most likely delivery failure. + +**Fix:** Add a subsection to FR-1.4 specifying the hash: +- Input: `canonicalize(htmlToMarkdown(flags["campaign-codex"].data)) + "\n" + name + + "\n" + folder` — i.e. reuse the existing `contentHash` pipeline by converting the + Foundry HTML body back to refined markdown first (the inverse of + `obsidianToFoundryJsonLive`). This makes the Foundry-side hash and the Obsidian + body hash **directly comparable** (same canonical form), which also makes the 2×2 + routing in F3 well-defined. +- Specify the HTML→markdown conversion (linkedom is already a dep per the wiki; + confirm it round-trips stably). Add a unit test for hash stability across a + push→/get round-trip before shipping FR-1.4. + +Also: `baselineFoundryBlock` (`src/server.ts:289`) and `baselineNote` +(`src/server.ts:307`) currently touch only `contentHash`/`syncedAt`. FR-1.7 requires +baseline of **both** `contentHash` and `ccHash` — both functions need a `ccHash:` +line added to the rewrite, and `frontmatter.ts`'s `readFoundryBlock` consumers need +to read it. Call this out as a concrete code change in the PRD. + +--- + +### H2 — FR-1.6 fail-safe: the PRD correctly identifies the gap, but the fix has a subtle hole — the `/get`-then-`/update` race window. + +**Confirmed:** current `process()` (`src/server.ts:582-617`) does **not** `/get` to +compare a Foundry-side hash before pushing. It pushes on Obsidian-body-hash-diff +only. `pushNote`'s internal `/get` (`src/push.ts:142`) builds the diff but does not +gate the push. So NFR-1 is **violated today** — the PRD is right to call this out. + +**Subtle hole in the proposed fix:** even with FR-1.4's "`/get` then compare ccHash +then push" sequence, there is a TOCTOU window between the `/get` (read F-side +baseline) and the `/update` (write). If a Foundry edit lands in that window, the +push still clobbers it. The window is small (one relay round-trip) but nonzero. + +**Impact:** Low probability, but it's the same class of bug the PRD is trying to +eliminate. Worth a defensive measure. + +**Fix:** After `pushNote`'s `relay.updateEntry` succeeds, re-`/get` and verify the +entry's content hash matches what we just wrote; if it diverges (concurrent edit +landed mid-flight), surface a conflict row instead of baselining. Cheaper +alternative: document the window as accepted residual risk in NFR-1. Either is +fine; the PRD should pick one explicitly. + +--- + +### H3 — F3 routing: the O→F `inflight` set and the F→O poll do not share a lock. Concurrent O-save + F-poll on the same note can double-write. + +**Reality:** `AutoSyncController.inflight` (`src/server.ts:463`, used at +`src/server.ts:583`) is keyed by **relPath** and only guards the O→F watcher path. +The proposed F→O poll is a separate loop with no shared lock. If an O-save and an +F-poll fire on the same uuid in the same tick: +- O→F sees O-changed, F-unchanged → pushes O→F. +- F→O sees F-changed (from the just-pushed O content? or a real F edit?), O-unchanged + (pre-save snapshot) → pulls F→O. +Result depends on timing and can oscillate or clobber. + +**Impact:** NFR-1 ("never clobber") can be violated under concurrent cross-direction +activity on the same note. Prep bursts are exactly this pattern. + +**Fix:** Add a **per-uuid lock** shared between the watcher path and the poll path +(not per-relPath). While a sync op is in flight for a uuid, the other direction +queues/skips. Add to FR-3.1 / FR-5.5. Also: the debounce window +(`src/server.ts:567`, 700ms) is O-side only; an F edit arriving during the debounce +is invisible to the watcher. The `/get` at push time (FR-1.4) is what catches it — +confirm FR-1.4 is evaluated **after** debounce drains, not on the raw save event. + +--- + +## MEDIUM + +### M1 — FR-1.4: one extra `/get` per changed note is acceptable, but the PRD should state the cost and cap it. + +One additional `/get` per push (on top of the `/get` `pushNote` already does at +`src/push.ts:142`) doubles relay reads per O→F op. At the existing concurrency=3 +(`src/server.ts:466`) and debounce=700ms (`src/server.ts:467`) this is fine for a +prep burst. But **reuse the `/get` `pushNote` already makes** — don't do two `/get`s. +The cleanest implementation is to have `pushNote` (or a new wrapper) compute the F +hash from the `liveEntry` it already fetched, compare to `ccHash` baseline, and +gate the `/update` on that. The PRD reads as if FR-1.4 adds a **separate** `/get` +before the push — that would be wasteful. Clarify: "the `/get` that `pushNote` +already performs is reused for the F-side hash compare; no extra round-trip." + +### M2 — FR-2.5 new-entry detection: feasible, but `img` is the only extra signal beyond name+uuid. + +`/search` minified gives `{uuid, id, name, img, documentType}`. For new Foundry +JournalEntries, `name + uuid` is enough to surface as import candidates (the +existing import row at `src/server.ts:182-187` and `importRow` already works off +`row.entry`). **Feasible.** One caveat: the dashboard's `ccOnly` pool +(`src/server.ts:154`) is built from the **LevelDB journal snapshot** via `indexAll`, +not from live `/search`. So "new entries surface as import candidates" requires +either (a) merging live `/search` results into the index's `ccOnly` pool, or (b) a +separate "live new entries" list in the dashboard. The PRD doesn't specify which. +Recommend (b) to avoid conflating the static LevelDB index with live relay state. +Add to FR-2.5. + +### M3 — FR-1.7 dev-mode baseline landing: `baselineNote` writes via `targetPath` (mirror in dev, real vault in apply) — correct, but the F→O pull writes the **vault** file and must use the same path resolution. + +`resolveRefined` (`src/server.ts:85`) prefers the dev mirror if it's newer, else the +real vault. The F→O pull (FR-2.3) writes into the vault; in dev mode it must write +to the mirror (`targetPath(state, "refined", rel)`), not the real vault, or dev +mode stops being a safe preview. The PRD says "Dev mode baselines land in the +`--out` mirror" (FR-1.7) — good — but FR-2.3 doesn't repeat the constraint for the +**pulled content**. Add it. + +### M4 — FR-4.3 `Sync Status.md` watcher exclusion: the watcher's exclusion list today is `.obsidian` + dotfiles only (`src/server.ts:561`, `src/server.ts:533`). + +`onChange` skips `.obsidian` and dotfiles (`src/server.ts:561`). A `Sync Status.md` +at the vault root is **not** a dotfile and **not** in `.obsidian`, so the watcher +will fire on every status-note write → auto-sync will try to push it (it has no +`foundry.cc_uuid` → skipped at `src/server.ts:591`, but still generates noise and a +`/get`-less skip). The PRD (OQ-4) flags the path as TBD but doesn't note that the +exclusion must be **added to `onChange`** explicitly. Add a concrete code-change +note to FR-4.3: extend the skip predicate at `src/server.ts:561` to include the +status-note path. + +--- + +## LOW + +### L1 — FR-6.3 client-list discovery: the relay returns the client list on `>1` connected clients with no clientId, and "No connected clients" on 0 (`docs/relay-api.md:13-14`). With exactly 1, the relay auto-resolves and returns the doc — **no client list is surfaced**. So FR-6.3's "list connected relay clients" works only when there are 0 or >1 clients. The single-client case (the most common during prep) returns no list. The PRD should note this: when exactly one client is connected, the dashboard should treat that as "clientId auto-resolved, no pick needed" rather than showing an empty list. + +### L2 — FR-5.3 retry on 404: a 404 from the relay is **not always** transient. `404 {"error":"No connected Foundry clients found"}` (`docs/relay-api.md:13`) is a **persistent** condition (Foundry down) — retrying with backoff just delays the error. The PRD's FR-5.3 lists "404 invalid client" as transient-retryable; distinguish "invalid clientId" (retryable after re-fetching the client list) from "no connected clients" (persistent, surface immediately). Update FR-5.3. + +### L3 — FR-1.2 / FR-1.3 skip semantics: `process()` skips on no `cc_uuid` or no `contentHash` (`src/server.ts:591-592`) and on body-hash-equal (`src/server.ts:594`). All three are correct and already implemented. The PRD's FR-1.2/1.3 matches the code. No change needed; noted for traceability. + +### L4 — `foundry.ccHash` naming: the existing `foundry:` block uses snake_case-ish `cc_uuid`, `cc_type`, `folder_path`, `contentHash` (mixed). `ccHash` is fine but consider `cc_content_hash` or `foundryHash` for grep-ability and consistency with `cc_uuid`/`cc_type`. Cosmetic; decide during delivery. + +--- + +## Overall verdict + +**Conditional-go.** The PRD's O→F hardening (F1, FR-1.4–1.7) is feasible and the +clobber-risk diagnosis is accurate, but the Foundry-side hash (H1) is under-specified +and is the most likely thing to derail delivery. The F→O direction (F2) has **two +blockers**: the PRD assumes `/search` returns `folder` (it does not — B1) and assumes +content-change detection is cheap (it requires a `/get` per linked note per poll, +which ADR 005 already rejected on cost grounds — B2). F2 as written is not +shippable; it needs to be scoped down to "shallow poll: renames + new + missing" with +content/folder changes routed to the manual trigger, or it needs an explicit, +costed deep-poll mode in minutes not seconds. + +The two-baseline 2×2 routing (F3) is sound in principle but has a real cross-direction +lock gap (H3) and a TOCTOU window (H2) that the PRD should address. + +Recommended pre-delivery edits to the PRD: +1. Rewrite FR-2.1/2.2 to drop `folder` from the `/search` snapshot (B1) and split + shallow vs deep poll (B2). +2. Specify the exact Foundry-side hash input and the HTML→markdown reuse (H1). +3. Add a shared per-uuid lock to F3/FR-5.5 (H3) and decide on the TOCTOU fix (H2). +4. Clarify that FR-1.4 **reuses** the `/get` `pushNote` already makes (M1). +5. Note the watcher-exclusion code change for `Sync Status.md` (M4) and the + single-client blind spot in FR-6.3 (L1). + +Top findings: **B1** (folder not in `/search`), **B2** (content detection = per-note +`/get` every poll, contradicts ADR 005), **H1** (Foundry-side hash under-specified), +**H2** (TOCTOU after `/get`), **H3** (no shared cross-direction lock). + +File: `/home/kaykayyali/docker/obsidian-foundry-sync/docs/prds/prd-foundry-obsidian-sync-2026-06-22/review-engineering.md` \ No newline at end of file diff --git a/docs/prds/prd-foundry-obsidian-sync-2026-06-22/review-launchable.md b/docs/prds/prd-foundry-obsidian-sync-2026-06-22/review-launchable.md new file mode 100644 index 0000000..ccc8c94 --- /dev/null +++ b/docs/prds/prd-foundry-obsidian-sync-2026-06-22/review-launchable.md @@ -0,0 +1,396 @@ +--- +title: "Launchability & Onboarding Adversarial Review" +review_of: prd-foundry-obsidian-sync-2026-06-22/prd.md +reviewer_role: adversarial reviewer (launchability / onboarding) +stakes: public/launchable +created: 2026-06-22 +--- + +# Launchability & Onboarding Adversarial Review + +**Verdict: NOT launchable as specified.** The PRD sets stakes = "public/launchable" and +NFR-6 promises "clone → live sync using the dashboard + `/setup`, without editing shell +commands," but the F6 feature set only closes two of roughly seven onboarding gates, and +several launchable-grade concerns (dashboard auth, Foundry-side data integrity, error +contracts, status-note feedback-loop safety, conflict UX comprehension) are unspecified, +hand-waved to "design during delivery," or silently dropped. The PRD reads as a strong +*author-hardening* PRD wearing a *launchable* label. Two blockers and four high-severity +findings stand between this draft and a non-author DM actually succeeding. + +Cross-references in this review: PRD = `prd.md` in this dir; README = +`/home/kaykayyali/docker/obsidian-foundry-sync/README.md`; env = +`/home/kaykayyali/docker/obsidian-foundry-sync/.env.example`. + +--- + +## BLOCKER + +### B1 — `/setup` is not a product surface; NFR-6 is unmet as written + +**Finding.** FR-6.4 ("The existing `/setup` skill covers env wiring; the dashboard +reflects setup state and links into it") and NFR-6 ("A non-author user can get from +clone → live sync using the dashboard + `/setup`, without editing shell commands") +treat `/setup` as a product feature. It is not. `/setup` is a Claude Code skill — an +authoring-time convenience available only to someone running Claude Code against this +repo. A non-author DM who clones the repo has no `/setup`. README §"Quick start" +confirms the real onboarding path is five shell steps: `cp .env.example .env`, +`docker compose up -d relay`, browser signup to copy `RELAY_API_KEY`, +`node scripts/start-relay-session.js`, and `./sync.sh refresh` + `./sync.sh ui`. + +The F6 FRs only cover *detection* of missing config (FR-6.1 clientId empty, FR-6.2 no +connected client, FR-6.3 list clients). They do not cover *acquisition* of: + +1. The `RELAY_API_KEY` (requires bringing up the relay container + browser signup — + shell + browser, not dashboard). +2. Starting the headless Foundry session (`scripts/start-relay-session.js` — shell). +3. Pointing Foundry's rest-api module at the relay WebSocket URL (Foundry-side admin + config — cannot be driven from the dashboard at all). +4. Bringing up the relay container itself (`docker compose up -d relay` — shell). +5. Installing deps / starting the dashboard (`npm install`, `./sync.sh ui` — shell). + +FR-6.2 detects "Foundry not connected" but the dashboard cannot *start* the headless +session that would connect it. So the DM is stuck at a red banner with no in-UI remedy. + +**Why it matters for launchable.** This is the single promise that defines "launchable" +for this product, and it is the one the PRD most overstates. A non-author DM following +the PRD literally cannot reach live sync without shell, and the PRD does not acknowledge +the gap — it papers over it by referencing a tool the end user does not have. + +**Recommended fix.** Pick one of two honest paths and state it in §3 and F6: + +- **(a) Downscope NFR-6** to "clone → live sync using the dashboard, *given a running + relay + headless session + REST-API module wired by the operator*," and add a §2 + "Operator prerequisites" block that enumerates the shell/Foundry steps the DM (or + their operator) must complete once. This matches reality and is still a real + improvement over today. +- **(b) Actually build the in-UI onboarding** for the five gates above: a dashboard + "first run" wizard that brings up the relay container via a host-side helper, guides + API-key creation (deep-link to the relay's signup page, paste key into a form), + launches the headless session via a server endpoint (not a shell command the user + types), and shows step-by-step instructions for the Foundry-side rest-api module + wiring with a connectivity check. Then NFR-6 stands. This is substantially more + delivery work than F6 currently implies and must be sized as such. + +Either way, drop `/setup` from the FRs and NFR-6 — it is an author tool, not a product +surface. If (b) is chosen, add FR-6.5..6.8 for the four remaining gates. + +--- + +### B2 — Dashboard binds `0.0.0.0:7788` with no auth; security is absent from the PRD + +**Finding.** README §"Batch dashboard" confirms the dashboard "binds 0.0.0.0 by default +so it's reachable on your tailnet at this VM's IP, port 7788; pass `--host 127.0.0.1` to +restrict to localhost." The PRD has **no security section**. There is no FR or NFR for +authentication, authorization, transport security, or access control on the dashboard. +The dashboard can push to a live Foundry world (`/api/push`, `/api/push-all`), read and +write the vault on disk, and presumably surfaces the relay API key in some form. With +auto-sync (F1) the dashboard also drives automatic live writes to Foundry. An +unauthenticated `0.0.0.0` listener that can overwrite a live Foundry world is a +launchable-grade security hole. + +The PRD also does not discuss: storage of `RELAY_API_KEY` and `RELAY_PASSWORD` (plaintext +in `.env` per env.example), whether the dashboard ever reflects secrets to the browser, +CSRF on POST endpoints, or what happens when the dashboard is exposed on a tailnet/shared +network. + +**Why it matters for launchable.** A launchable product that can clobber a live Foundry +world over an open port with no auth is irresponsible to ship. "Bind localhost" is a +mitigation the README mentions but the PRD never requires; the default is `0.0.0.0`. + +**Recommended fix.** Add a §5 NFR-9 — Security, and a F7 — Security & access control +feature group, minimum: + +- **FR-7.1** Dashboard authenticates by default (token or password set via env or + first-run prompt); unauthenticated requests get 401, not the UI. +- **FR-7.2** Default bind changes to `127.0.0.1`; `0.0.0.0` requires explicit opt-in + *and* an auth token set (refuse to start otherwise). +- **FR-7.3** Secrets (`RELAY_API_KEY`, `RELAY_PASSWORD`) are never rendered to the + browser; the dashboard shows only masked presence/absence. +- **FR-7.4** POST mutation endpoints require a CSRF token or same-origin check. +- **NFR-9** No unauthenticated mutation path; no secret egress to the client; TLS + recommended when bound beyond localhost. + +--- + +## HIGH + +### H1 — Error contracts are unspecified; "surface an error row" is not actionable for a non-author DM + +**Finding.** FR-1.6, FR-5.3, NFR-2, NFR-4 all reference "error rows" / "surfaced" but +none specify the error *vocabulary*, *messages*, or *remediation guidance*. The relay +returns at least four distinct failure modes that a non-author DM cannot distinguish +from a raw status code: relay down (network), session down (relay up but no Foundry +client), clientId invalid (relay up, client up, wrong/empty id), hash-mismatch (conflict, +not an error), and transient 404/408/504 (retryable). FR-5.3 collapses these to +"transient relay errors (404 invalid client, 408/504 timeout, 5xx)" — but "404 invalid +client" is *not* transient (a wrong clientId will 404 forever), and "session down" is +not in the retry list at all. The mapping from relay response → user-facing message → +recommended action is absent. + +**Why it matters for launchable.** NFR-5 promises "no silent skips," but a red row that +says `Relay 404` is silent in the sense that matters: the DM does not know whether to +fix `.env`, restart the headless session, or pick a different clientId. The author can +debug from a status code; a non-author DM cannot. + +**Recommended fix.** Add a §5a "Error contracts" table mapping each failure mode to: +(a) detection signal, (b) retry-or-not, (c) the exact user-facing message, (d) the +remediation hint shown. Minimum rows: relay unreachable, relay 401 (bad API key), +clientId invalid/empty, no connected Foundry client, session idle-reaped, hash mismatch +(route to F3, not error),Foundry-side /get 404 on a known uuid (deleted on Foundry), +and persistent 5xx after backoff. Specify that error rows include a one-line "what to +do" string, not just a status. Split FR-5.3's retry list into *transient* (timeout, 5xx, +session-temporarily-unavailable) vs *persistent* (404 invalid client, 401 bad key) with +no retry on persistent. + +### H2 — `Sync Status.md` exclusion is not airtight; feedback-loop risk is unaddressed + +**Finding.** FR-4.3 maintains `Sync Status.md` in the vault, "on a path excluded from +the watcher, never synced to Foundry, updated each tick." OQ-4 defers the path/exclusion +mechanism to delivery. The vault is what the watcher watches (FR-1.1). If the exclusion +is path/filename-based and the DM renames or moves the note, or the tool's path config +drifts, the watcher fires on every tick's status-note write → auto-sync attempts to push +a `foundry:`-less note (skipped by FR-1.2, but it still churns the activity panel and +wakes the pipeline) or, worse, a future version seeds it and a feedback loop starts. +The PRD does not specify: the exclusion *mechanism* (glob? exact path? frontmatter flag? +a dedicated subdir?), what guarantees it across renames, whether the F→O poll also skips +it (FR-2.x doesn't mention it), and what happens on a name collision with a real note. + +**Why it matters for launchable.** A status note that triggers sync is a self-inflicted +loop that a non-author DM will not diagnose. "Excluded" stated as a requirement without +a mechanism is a bug waiting to ship. + +**Recommended fix.** Promote OQ-4 to a specification: define the exclusion as a +*dedicated, reserved path* (e.g. `${VAULT}/.sync-status.md` or a dot-prefixed file that +FR-1.1's "skip dotfiles" rule already covers), AND a frontmatter sentinel (e.g. +`foundry.sync_status: true`) that both the O→F watcher and the F→O poll check and skip +on, so the exclusion holds even if the path changes. Add FR-4.6: "The status note is +excluded from both O→F watch and F→O poll by *both* a path rule and a content sentinel; +a status note that loses its sentinel is treated as user error and surfaced, not +synced." Add an explicit test to NFR-5: status-note writes never produce a sync op. + +### H3 — Conflict UX is unspecified; "mark resolved (no change)" is a data-loss footgun for a non-author DM + +**Finding.** F3 routes both-diverged to a conflict row with "push vault → Foundry", +"pull Foundry → vault", "mark resolved (no change)" (FR-3.3). OQ-6 defers diff format +to delivery. Two problems: + +1. **Diff format is unspecified.** "Summarizing both versions and highlighting the diff" + (FR-3.2) could be a unified diff, a side-by-side, or a condensed summary — three very + different UX outcomes for a non-author DM staring at a conflict. +2. **"Mark resolved (no change)" is genuinely ambiguous.** Its actual semantic (per + FR-3.1 + the no-clobber rule) is presumably "accept the current diverged state as the + new parity baseline without transferring content in either direction" — i.e. *both* + sides keep their diverged content and both hashes re-baseline to current values. A + DM reading "mark resolved (no change)" will reasonably think "do nothing, come back + later" or "keep both" — and may click it expecting safety, then discover later that + the two sides now permanently disagree and neither has the other's edits. There is no + FR specifying what "mark resolved" actually does to the baselines, whether the + divergence is preserved or collapsed, or what the DM is told just happened. + +**Why it matters for launchable.** This is the one screen where the DM makes an +irreversible content decision. For the author it is obvious; for a non-author DM it is +the screen most likely to lose work. "Design during delivery" is not a launchable +specification for the highest-stakes UX in the product. + +**Recommended fix.** Resolve OQ-6 in the PRD, not in delivery. Specify: + +- **FR-3.6** Conflict diff format: side-by-side with a one-line plain-language summary + ("Vault adds 3 paragraphs about X; Foundry renamed to Y and changed folder to Z"), + not a raw unified diff. +- **FR-3.7** "Mark resolved" is renamed to an explicit label — e.g. "Accept both as-is + (keep divergence)" — with a confirmation dialog that states in plain text: "The vault + and Foundry will keep their current versions. They will be treated as in-sync from + now on. Neither side's changes will be copied to the other." +- **FR-3.8** Each conflict action states, before commit, what it will do to each side + and to the baselines (one-line preview). No irreversible action without a confirm. +- **FR-3.9** A resolved conflict produces a visible activity-panel entry stating which + side won and that the other side's edits were *not* transferred. + +### H4 — Foundry-side data integrity has no backup/recovery story; auto-sync writes live with no Foundry-side undo + +**Finding.** FR-1.8 states "Auto-sync always applies live (dry-run not honored)." Manual +`--apply` mode writes a timestamped `.bak-<iso>` of vault files it overwrites (README +§"Modes"). But push to Foundry goes through the relay `/update`, which overwrites the +live Foundry entry — there is no Foundry-side `.bak`, no undo, no rollback specified. +For a non-author DM who auto-syncs a bad edit (or hits a bug, or a hash race), the +Foundry world is the side with no recovery path. The PRD has no §"Data integrity" or +backups section. The vault side has `.bak`; the Foundry side has nothing. + +**Why it matters for launchable.** A launchable sync tool that can overwrite a live +Foundry world automatically, with no specified recovery path on the overwritten side, +is a data-loss hazard. The author can restore from a world backup; a non-author DM may +not have one scheduled. + +**Recommended fix.** Add a §5 NFR-10 — Data integrity, minimum: + +- **FR-5.6** Before any auto or manual push to Foundry, the prior Foundry-side content + is fetched and cached locally (a `foundry-backups/<uuid>/<iso>.json` dir) so a bad + push can be reversed from the dashboard. Specify retention (e.g. last N per uuid). +- **FR-5.7** A "Revert last push" action exists for the most recent push per note. +- **NFR-10** Foundry-side overwrites are always preceded by a local backup of the + pre-push Foundry state; the dashboard exposes a restore path. +- Document in §2 that Foundry-side world backups (Foundry's own backup feature) remain + the DM's responsibility and are recommended before enabling auto-sync. + +--- + +## MEDIUM + +### M1 — Performance NFR is sized to the author's vault, not real-world vaults + +**Finding.** NFR-3 bounds the burst target at "~50-note prep burst" with no stated basis. +FR-2.1 polls `/search` for all `JournalEntry` minified on every tick — on a large world +(hundreds or thousands of JournalEntries) this is a full scan per poll. No vault-size +assumption is stated, no relay-load ceiling, no concurrency default numbers (FR-5.2 +"defaults tuned for prep" — what numbers?). The `RELAY_URL=http://localhost:3010` +topology in `.env.example` reflects the author's all-on-one-host setup; the PRD does +not acknowledge that a DM with remote Foundry has higher latency and a different +retry/backoff profile. + +**Recommended fix.** State the operating envelope in NFR-3: tested vault size range, +JournalEntry count range the poll is validated against, default poll cadence (resolve +OQ-3 with a number + rationale), default debounce and max-concurrency numbers, and the +relay-load ceiling (concurrent `/update` calls). Add NFR-3a: "F→O poll and O→F push +paths are validated against a vault of at least N notes and M JournalEntries," with N +and M chosen above the author's own size. + +### M2 — Auto-sync in dev vs apply mode is contradictory + +**Finding.** FR-1.7 says "Dev mode baselines land in the `--out` mirror; apply mode in +the real vault with a `.bak`." FR-1.8 says "Auto-sync always applies live (dry-run not +honored)." These interact unclearly: can auto-sync be enabled in dev mode at all? If +yes, "applies live" to what — the `--out` mirror or the real vault? If auto-sync only +makes sense in apply mode, say so and gate the toggle on `--apply`. If it can run in dev +mode, define what "live" means there. A non-author DM who starts the dashboard in the +default dev mode and flips on auto-sync will not know what is being written where. + +**Recommended fix.** Add FR-1.9: "Auto-sync requires apply mode; enabling it in dev mode +is either blocked with an explanatory banner or explicitly writes to `--out` only (pick +one and state it)." Reconcile FR-1.7/1.8 accordingly. + +### M3 — Status state single-source-of-truth has no persistence story across restart + +**Finding.** FR-4.5 says dashboard parity and the vault status note reflect "one +underlying state (single source of truth for status)." The PRD doesn't say where that +state lives. If it is in-memory in the server, a restart loses parity counts, conflict +state (FR-3.4 "persists until resolved"), and last-sync time — but the vault status note +persists. On restart, does the dashboard re-derive parity from a full re-scan, or does +it show stale/empty until the next tick? Conflict persistence (FR-3.4) is especially +load-bearing: a restart that drops conflict state could let a re-save auto-clear a known +conflict, violating FR-3.4. + +**Recommended fix.** Specify the status store: either (a) a small persisted state file +on disk (`sync-state.json`) that survives restart and is the single source the dashboard +and status note both read from, or (b) re-derive on restart with an explicit "recomputing +parity…" state. Add FR-3.4a: "Conflict state persists across server restarts, not just +across ticks." + +### M4 — Versioning, updates, and frontmatter schema migration are unaddressed + +**Finding.** The PRD introduces a new frontmatter field `foundry.ccHash` (FR-1.4) and +relies on existing `foundry.contentHash`/`cc_uuid`/`folder_path`. There is no schema +version field on the `foundry:` block and no migration story. A non-author DM who +updates the tool to a later version that changes hash algorithm or adds fields has no +specified upgrade path — old baselines will mismatch and every note may route to +conflict on first sync after upgrade. No §"Versioning & updates" section. + +**Recommended fix.** Add `foundry.schema_version` (or similar) to the identity block and +an FR in F1/F5: on startup, notes with a missing/older schema version are migrated +(re-hash, re-baseline) in a single pass before auto-sync is allowed to engage, with a +dashboard migration banner. Add a §5 NFR-11 — Upgrades: "A version bump that changes +hashing or identity fields ships with an idempotent migration; auto-sync stays disabled +until migration completes." + +### M5 — Supportability: no persistent log / diagnostics export + +**Finding.** NFR-5 mandates observability via the activity panel and status note, but +FR-5.4 says the panel is "capped and scrollable" — i.e. not persistent. A non-author DM +who hits a problem and asks for help has no log file to share. There is no FR for a +persistent log, a diagnostics export, or a "copy debug info" action. For a launchable +product this is the difference between "I can help you" and "I can't see what happened." + +**Recommended fix.** Add FR-5.7 (renumber from H4's FR-5.7): "All auto-sync ops +additionally append to a persistent, rotated log file on disk +(`logs/sync-<date>.log`)." Add FR-5.8: "A 'Copy diagnostics' dashboard action bundles +recent log tail, current config (secrets redacted), parity counts, and relay/clientId +status into a single redacted blob for support." NFR-5a: "Operation history is +persistent across restarts, not just visible in-session." + +--- + +## LOW + +### L1 — Polling load on shared relays is a courtesy concern + +**Finding.** FR-2.1 polls `/search` on a cadence for every DM running the tool. If the +relay is a shared instance (multiple DMs), uncoordinated polling multiplies. The PRD +doesn't note this as a consideration or suggest a jittered/backoff poll. + +**Recommended fix.** Add jitter to the poll cadence and a note in §2 that on shared +relays the cadence should be raised. Non-blocking. + +### L2 — New (cc-only) Foundry entries during prep: candidate vs auto-import is an open question with launchability impact + +**Finding.** OQ-5 leaves "auto-import new Foundry entries" open with current spec = +candidates only. For a non-author DM, "import candidates" with no further guidance may +mean a growing pile of unimported rows they don't know what to do with. + +**Recommended fix.** Resolve OQ-5 in the PRD: keep candidates-only (safe) but specify +that the candidate row has a one-click "Import as new refined note" action with a clear +explanation of what import does. The current import row exists per README; confirm it is +comprehensible to a non-author DM. + +### L3 — "Foundry-wins" tie-breaker for both-diverged is deferred but conflict UX depends on it + +**Finding.** §1 confirms "Foundry is source of truth" is demoted to a tie-breaker for +both-diverged, "to be decided: newest-wins, Foundry-wins, or manual merge." F3 then +specifies manual resolution via buttons, which makes the tie-breaker moot *if* the DM +always resolves manually. But the conflict row needs a *default ordering* (which side is +shown on the left, which action is highlighted) and that ordering embeds a tie-breaker +opinion. Leaving it undecided leaks into UX. + +**Recommended fix.** Pick a neutral default (e.g. vault on left, Foundry on right, no +pre-highlighted action) so the undecided tie-breaker doesn't bias the DM. State it in F3. + +### L4 — README still describes Foundry as "source of truth"; PRD demotes it + +**Finding.** README L17 says "Foundry is the source of truth." The PRD §1 demotes that +to a tie-breaker. For a launchable product the README is the onboarding doc and the PRD +is the spec; a non-author DM reading both will be confused about which side wins. + +**Recommended fix.** Note in §6 / §"Out of Scope" that README wording will be updated to +match the demoted-source-of-truth model before launch. Track as a doc task. + +--- + +## Summary table + +| ID | Tier | Theme | One-line | +|-----|---------|------------------------|--------------------------------------------------------------------------| +| B1 | BLOCKER | Onboarding | `/setup` is not a product surface; NFR-6 unmet as written | +| B2 | BLOCKER | Security | `0.0.0.0:7788` dashboard, no auth, no security section | +| H1 | HIGH | Error contracts | Error vocabulary/messages/remediation unspecified | +| H2 | HIGH | Status-note loop | `Sync Status.md` exclusion mechanism not airtight | +| H3 | HIGH | Conflict UX | Diff format + "mark resolved" semantics unspecified; data-loss footgun | +| H4 | HIGH | Data integrity | Auto-sync writes Foundry live with no Foundry-side backup/undo | +| M1 | MEDIUM | Perf sizing | "~50-note burst" not sized to real vaults; no relay-load ceiling | +| M2 | MEDIUM | Dev/apply vs auto-sync | FR-1.7/1.8 interaction contradictory | +| M3 | MEDIUM | Status persistence | Single-source-of-truth state has no restart story; conflict state at risk| +| M4 | MEDIUM | Versioning | No schema version / migration story for `foundry:` block | +| M5 | MEDIUM | Supportability | No persistent log or diagnostics export | +| L1 | LOW | Poll courtesy | Jittered poll on shared relays | +| L2 | LOW | Import candidates UX | OQ-5 candidate UX needs comprehensibility check | +| L3 | LOW | Tie-breaker default | Undecided tie-breaker leaks into conflict-row ordering | +| L4 | LOW | Doc drift | README "source of truth" contradicts PRD demotion | + +--- + +## Overall recommendation + +**Do not mark this PRD ready for launchable delivery** until B1 and B2 are resolved +(either by honest downscoping of NFR-6 or by committing to build the in-UI onboarding +and auth) and H1–H4 are specified concretely enough to implement against. As a +*hardening* PRD for the single author it is solid; as a *launchable* PRD it currently +treats onboarding, security, error contracts, status-loop safety, conflict UX, and +Foundry-side data integrity as "design during delivery" — exactly the surfaces where a +non-author DM falls off. \ No newline at end of file diff --git a/docs/prds/prd-foundry-obsidian-sync-2026-06-22/review-rubric.md b/docs/prds/prd-foundry-obsidian-sync-2026-06-22/review-rubric.md new file mode 100644 index 0000000..1a53764 --- /dev/null +++ b/docs/prds/prd-foundry-obsidian-sync-2026-06-22/review-rubric.md @@ -0,0 +1,192 @@ +# PRD Quality Review — Live Relay Sync — Auto-Sync & Bidirectional Hardening + +## Overall verdict + +This is a tightly-argued, capability-spec PRD that earns most of its length: the +no-clobber thesis is crisp, the trade-offs are named with what was given up (not +just what was chosen), and Non-Goals do real work. What's at risk is downstream +extractability and thesis-validation: there is no Glossary and no Success Metrics +section, so a story author will re-derive domain terms and the PRD has no +machine-readable way to confirm the thesis landed. A stale "to be decided" in the +Vision also drifts from a decision the FRs already made. None of this blocks a +green-light; all of it is one editing pass away from closed. + +## Decision-readiness — strong + +Decisions are stated as decisions, not buried as considerations. The `[CONFIRMED]` / +`[DEFERRED]` tags in §1 are honest — e.g. _"Foundry is source of truth" is demoted +to a tie-breaker rule for the both-diverged conflict case"_ names both the choice +and the demotion. Trade-offs name what was given up: FR-1.6 explicitly says _"do +not fall back to an Obsidian-side-only check (that reintroduces clobber risk)"_ — +the rejected path is on the page, not hidden. The Open Questions are actually open: +OQ-1 carries a default posture (_"Default posture is manual"_), OQ-3/OQ-4/OQ-6 are +genuinely delivery-time (_"pick during delivery B with relay-load testing"_), and +OQ-2 is a real deferral with the reopening condition named (_"Reopens when a +custom Foundry module is explored"_). The decision-log shows real tensions +surfaced (the clobber gap in the current O→F code) rather than smoothed to neutral. + +### Findings +- **medium** Stale "to be decided" in Vision contradicts the FRs (§1, _"to be + decided: newest-wins, Foundry-wins, or manual merge"_). FR-3.2 and FR-3.3 + already specify manual resolution with three explicit actions (_"push vault → + Foundry", "pull Foundry → vault", "mark resolved"_); OQ-1 only asks whether to + add a newest-mtime convenience. The §1 phrasing reads as unresolved when the + PRD has in fact decided. *Fix:* rewrite the §1 parenthetical to + _"default = manual resolution (FR-3.2); a newest-mtime convenience is open + (OQ-1)"_. + +## Substance over theater — strong + +No personas, no User Journeys, no differentiation section — and that is correct +for a single-operator brownfield capability spec (see Shape fit). The Vision is +product-specific, not swappable: the prep / run-the-match duality, the +_"always in lockstep while I work"_ feeling, and the _"sync deliberately paused, +not silently absent"_ framing would not transfer to another tool. NFRs carry +product-specific thresholds, not boilerplate: NFR-3 names a _"~50-note prep +burst"_, NFR-4 _"within one tick"_, NFR-6 names the _"UI-only convention"_. No +innovation theater, no persona theater. Nothing here reads like furniture. + +### Findings +- (none) + +## Strategic coherence — adequate + +The thesis is stated and the features serve it. G1 (no-clobber bidirectional +auto-sync for prep) is the thesis; F1+F2+F3 implement it directly. G4 (legible +status) is the second arc; F4 serves it. G5/G6 (hardening/onboarding) are the +launchable-stakes rigor that follows from the chosen stakes, not a bolt-on. +Prioritization follows the thesis: delivery A (ship O→F *with the divergence +guard*) is gated on the safety fix, not on "what's easy first." + +The gap is that there is **no Success Metrics section**. The rubric asks whether +SMs validate the thesis and whether counter-metrics are named. G2 embeds one +criterion (_"verify one end-to-end live push"_), but there is no consolidated set +of operational metrics that would let a reviewer confirm the thesis — e.g. +"zero silent clobbers across a 50-note prep burst", "F→O poll detects a Foundry +rename within N ticks", "non-author user reaches live sync via dashboard + /setup +without shell". For a public/launchable PRD, the absence is a coherence tell: the +PRD argues the thesis but doesn't state how it would know it landed. + +### Findings +- **medium** No Success Metrics section (whole PRD). The thesis is clear but + unmeasured; activity metrics are absent too. *Fix:* add an SM block with 3–5 + operational metrics tied to G1/G2/G4/G6 (e.g. _SM-1: zero auto-overwrite + events on a side that changed since last sync, across a 50-note burst_; + _SM-2: one verified end-to-end live O→F push_; _SM-3: F→O detects a Foundry + rename/move within ≤2 poll ticks_; _SM-6: a non-author user clones, runs + /setup, and reaches a connected live sync with no shell command_). Name a + counter-metric for SM-1 (e.g. _conflict-rows-surfaced > 0 when both sides + diverged_). + +## Done-ness clarity — adequate + +Most FRs carry testable consequences. FR-1.4's hash-over-_content + name + +folder_path_ is concrete and verifiable. FR-1.5's 2×2 routing is a truth table. +FR-3.1 makes the routing explicit. FR-2.2 enumerates the change kinds +(_renamed / moved / content-changed / missing / new_). G2's _"verify one +end-to-end live push"_ is an acceptance gate for delivery A. NFR-1 is the hard +contract (_"No auto-sync operation may overwrite a side that has changed since +the last sync"_). + +The unforgiving part: a few FRs use adjectives where bounds belong. FR-5.2 says +_"defaults tuned for prep so a burst of simultaneous saves doesn't thrash or drop +events"_ — "thrash" is undefined and "tuned for prep" is an adjective, not a +bound; NFR-3 supplies the ~50-note number but FR-5.2 doesn't reference it. +FR-5.4's _"panel capped and scrollable"_ leaves the cap unspecified. These are +the spots where an engineer writing stories will have to go back and ask "how +many?" + +### Findings +- **low** FR-5.2 "thrash" / "tuned for prep" — bound-less adjectives (§F5). + *Fix:* bind FR-5.2 to NFR-3's ~50-note burst and name concrete defaults + ranges (debounce window, max concurrency) or defer the numbers to a config + table with placeholders. +- **low** FR-5.4 "panel capped" — cap value missing (§F5). *Fix:* state the cap + (e.g. _"last 200 events, scrollable for older"_). + +## Scope honesty — strong + +Non-Goals do real work and name what could be silently assumed: _"Not syncing" +indicator rendered inside Foundry_ is explicitly deferred with the reason +(_"Foundry UI is not a surface we control"_); _"Syncing during run-the-match"_ +is called out as a design decision, not an omission; _"Automatic semantic / 3-way +content merge"_ is excluded so the conflict UI doesn't get over-built. The +`[ASSUMPTION]` tag in §2 (_"The relay remains the live transport for this PRD; +the custom Foundry module … is explicitly future work, not a dependency here"_) +flags the key inference. Open-items density is moderate (6 OQs + 1 ASSUMPTION) +against launchable stakes; every OQ carries a default or a named +delivery-trigger, so the density is honest rather than a blocker. + +### Findings +- **low** Implicit relay-load assumption not flagged (§F2 / NFR-3). FR-2.2 + detects content-changed entries _"via /get hash compare"_, which implies one + /get per changed linked note per poll tick — a load assumption that + interacts with OQ-3's cadence. This is an inference the user didn't directly + confirm and should be tagged `[ASSUMPTION]`. *Fix:* add + _`[ASSUMPTION]` one relay /get per changed linked note per poll tick is + acceptable relay load at the chosen cadence (OQ-3)_ to §F2, and index it. + +## Downstream usability — adequate + +FR IDs are contiguous (FR-1.1 → FR-6.4), unique, and cross-references resolve +(FR-5.1 → FR-1.1 ✓; OQ-5 → FR-2.5 ✓; NFR-1 → the divergence guard ✓). Brownfield +references are accurate against the decision log and code memory +(_AutoSyncController_, _pushNote_, _foundry.cc_uuid_, _foundry.contentHash_, +_rest-api module_, _relay /search_ minified, _relay /get_). No floating UJs (no +UJs at all — appropriate). Each section reads sensibly pulled out alone. + +The gap is the **missing Glossary**. The PRD uses a thick domain vocabulary — +_cc_uuid_, _contentHash_, _ccHash_ (new field, introduced in FR-1.4), _linked +note_, _seeded_, _refined vault dir_, _refined markdown_, _import candidate_, +_dev/apply mode_, _activity panel_, _parity_, _cc-only entry_ — and these are +defined only inline and inconsistently (FR-1.2 glosses _foundry.cc_uuid_ as +"unlinked"; FR-1.4 defines _ccHash_ in passing). A story author source-extracting +from this PRD will have to reassemble the term set, and drift risk is real +(_ccHash_ vs _contentHash_ vs _cc_uuid_ are easy to confuse). For a PRD that +feeds architecture and stories, this is the dimension that needs the most work. + +### Findings +- **medium** No Glossary section (whole PRD). Domain nouns are inline-defined + and not consolidated; _ccHash_ is introduced mid-FR without a term-of-art + definition. *Fix:* add a Glossary section defining at minimum: _cc_uuid_, + _contentHash_, _ccHash_, _linked note_, _seeded_, _refined vault dir_, + _refined markdown_, _import candidate_, _cc-only entry_, _dev/apply mode_, + _activity panel_, _parity_, _tick_. Reference Glossary terms from the FRs + instead of re-defining inline. + +## Shape fit — strong + +The shape matches the product. This is a brownfield, single-operator (DM/world- +builder) capability spec intended to go launchable — exactly the case the rubric +flags as _"Internal tool, single-operator role → capability spec shape; UJs may +be overhead; SMs may be operational."_ No personas, no UJs: correct, not a gap. +Operational SMs would fit the shape but their absence (see Strategic coherence) +is a coherence gap, not a shape mismatch. The PRD is neither over-formalized (no +UJ theater for a one-operator tool) nor under-formalized (NFRs and error +contracts are present at launchable rigor). Existing-code references are +distinguished from new behavior (_"code-complete but uncommitted"_, _"new field, +e.g. foundry.ccHash"_), which is what brownfield shape demands. + +### Findings +- **low** Operational SMs would fit this shape and are absent (§5 / §SM). *Fix:* + see the Strategic-coherence finding — operational metrics (zero silent + clobbers, ticks-to-detect, onboard-without-shell) are the right SM shape for + this product and would close both dimensions at once. + +## Mechanical notes + +- **Glossary drift:** No glossary; _ccHash_ vs _contentHash_ vs _cc_uuid_ are + used close together and differ only in case/short suffix — high confusion + risk. _"cc-only entry"_ (FR-2.5) and _"import candidate"_ (FR-2.5) are used + as synonyms without being linked. _"refined vault dir"_ / _"refined markdown"_ + appear without definition (FR-1.1, FR-2.3). +- **ID continuity:** FR IDs contiguous and unique; cross-refs (FR-5.1→FR-1.1, + OQ-5→FR-2.5, NFR-1→the divergence guard) all resolve. No duplicates, no gaps. +- **Assumptions Index roundtrip:** broken. One inline `[ASSUMPTION]` exists + (§2, relay-as-transport); no Assumptions Index at the end of the PRD. The + implicit relay-load assumption (Scope honesty finding) is not tagged at all. +- **UJ persona linkage:** N/A — no UJs, no personas (appropriate for shape). +- **Required sections for stakes:** Vision ✓, Problem & Context ✓, Goals & + Non-Goals ✓, FRs ✓, NFRs ✓, Open Questions ✓, Out of Scope ✓. **Missing: + Success Metrics, Glossary.** For public/launchable stakes, SMs are expected; + Glossary is expected for a PRD that feeds architecture + stories. \ No newline at end of file diff --git a/scripts/start-relay-session.js b/scripts/start-relay-session.js new file mode 100644 index 0000000..73bd0b8 --- /dev/null +++ b/scripts/start-relay-session.js @@ -0,0 +1,141 @@ +// Starts a headless Foundry session on a ThreeHats foundryvtt-rest-api-relay. +// +// Generalized version: every value comes from the environment (no hardcoded +// host). Reads this repo's .env first, then process env. Required: RELAY_API_KEY, +// RELAY_URL, FOUNDRY_URL, RELAY_USER, RELAY_PASSWORD, FOUNDRY_WORLD. +// +// Flow (per docs/relay-api.md): +// 1. POST /session-handshake (x-api-key, x-foundry-url, x-username) -> {nonce, publicKey, token} +// 2. RSA-OAEP encrypt {password, nonce} with the relay's public key. +// 3. POST /start-session {handshakeToken, encryptedPassword, world} -> headless launch. +// +// Run from the repo root: node scripts/start-relay-session.js +// (The /setup skill does this for you after the Foundry rest-api module is connected.) + +const crypto = require('crypto'); +const http = require('http'); +const https = require('https'); +const fs = require('fs'); +const path = require('path'); +const { URL } = require('url'); + +// --- Load .env (simple KEY=value; real process.env wins) --- +(function loadDotEnv() { + const p = path.join(__dirname, '..', '.env'); + let txt; + try { txt = fs.readFileSync(p, 'utf8'); } catch { return; } // no .env is fine + for (const line of txt.split(/\r?\n/)) { + const m = line.match(/^\s*([A-Za-z_][A-Za-z0-9_]*)\s*=\s*(.*)\s*$/); + if (!m) continue; + const [, k, vRaw] = m; + if (process.env[k] !== undefined) continue; // real env wins over .env + const v = vRaw.replace(/^['"]|['"]$/g, ''); + if (v !== '') process.env[k] = v; + } +})(); + +const env = (k) => (process.env[k] !== undefined && process.env[k] !== '' ? process.env[k] : ''); + +const API_KEY = env('RELAY_API_KEY'); +const RELAY_URL = env('RELAY_URL'); +const FOUNDRY_URL = env('FOUNDRY_URL'); +const USERNAME = env('RELAY_USER'); +const PASSWORD = env('RELAY_PASSWORD'); +const WORLD = env('FOUNDRY_WORLD'); + +const missing = [ + ['RELAY_API_KEY', API_KEY], + ['RELAY_URL', RELAY_URL], + ['FOUNDRY_URL', FOUNDRY_URL], + ['RELAY_USER', USERNAME], + ['RELAY_PASSWORD', PASSWORD], + ['FOUNDRY_WORLD', WORLD], +].filter(([, v]) => !v).map(([k]) => k); +if (missing.length) { + console.error(`Missing required env var(s): ${missing.join(', ')}.\n` + + `Copy .env.example to .env and fill them in (or run the /setup skill).`); + process.exit(1); +} + +let relay; +try { + relay = new URL(RELAY_URL); +} catch { + console.error(`RELAY_URL is not a valid URL: ${RELAY_URL}`); + process.exit(1); +} +const transport = relay.protocol === 'https:' ? https : http; + +function post(pathname, headers, body) { + return new Promise((resolve, reject) => { + const data = JSON.stringify(body || {}); + const req = transport.request( + { + protocol: relay.protocol, + hostname: relay.hostname, + port: relay.port || (relay.protocol === 'https:' ? 443 : 80), + path: pathname, + method: 'POST', + headers: { + 'Content-Type': 'application/json', + 'Content-Length': Buffer.byteLength(data), + 'x-api-key': API_KEY, + ...headers, + }, + }, + (res) => { + let d = ''; + res.on('data', (c) => (d += c)); + res.on('end', () => { + if (res.statusCode && res.statusCode >= 400) { + reject(new Error(`relay ${res.statusCode} POST ${pathname}: ${d.slice(0, 300)}`)); + return; + } + try { resolve(JSON.parse(d)); } catch { resolve(d); } + }); + } + ); + req.on('error', reject); + req.write(data); + req.end(); + }); +} + +(async () => { + console.log(`Starting handshake on ${RELAY_URL.href} (world: ${WORLD}, foundry: ${FOUNDRY_URL}, user: ${USERNAME})...`); + let hs; + try { + hs = await post('/session-handshake', { + 'x-foundry-url': FOUNDRY_URL, + 'x-username': USERNAME, + }, {}); + } catch (e) { + console.error(`Handshake failed: ${e.message}\n` + + `Is the relay up (docker compose up -d relay) and is the Foundry rest-api module connected to it?`); + process.exit(1); + } + if (!hs || !hs.nonce || !hs.publicKey || !hs.token) { + console.error('Handshake returned an unexpected response:', JSON.stringify(hs)); + process.exit(1); + } + + const payload = JSON.stringify({ password: PASSWORD, nonce: hs.nonce }); + const encryptedPassword = crypto.publicEncrypt( + { key: hs.publicKey, padding: crypto.constants.RSA_PKCS1_OAEP_PADDING, oaepHash: 'sha256' }, + Buffer.from(payload) + ).toString('base64'); + + console.log('Launching headless session (may take up to 2 min)...'); + try { + const result = await post('/start-session', {}, { + handshakeToken: hs.token, + encryptedPassword, + world: WORLD, + }); + console.log('Result:', JSON.stringify(result, null, 2)); + } catch (e) { + console.error(`Start-session failed: ${e.message}\n` + + `Check FOUNDRY_URL/RELAY_USER/RELAY_PASSWORD/FOUNDRY_WORLD are correct and the world exists.`); + process.exit(1); + } +})(); \ No newline at end of file diff --git a/src/dashboard.html b/src/dashboard.html index a4792d3..f96f808 100644 --- a/src/dashboard.html +++ b/src/dashboard.html @@ -89,7 +89,12 @@ <button onclick="act('importAll')" title="Pull every cc-only Foundry entry (not yet in your vault) into a new refined note under refined/imported/<folder>. Un-curated staging for review.">Import all</button> <button onclick="pushAll()" title="Push every vault-newer (sync→cc) note into the LIVE Foundry world via the relay in one run. dry-run (default) lists what would be pushed; uncheck dry-run to apply — each live entry is backed up to <out>/bak first and the note's foundry.contentHash is baselined so a re-run only catches new edits. Replaces scripts/resync.ts.">Push all changed</button> <button onclick="refreshLive()" title="Rebuild the cached name↔uuid map (for link resolution in pushes) via the relay /search — zero Foundry downtime. The heavy docker-stop full index is CLI-only: npx tsx src/cli.ts refresh --full-index.">Refresh live index</button> + <button id="autoSyncBtn" onclick="toggleAutosync()" title="Toggle Obsidian→Foundry auto-sync. ON = saving a linked, seeded note in your vault pushes it into LIVE Foundry instantly (guarded by the note's foundry.contentHash baseline, so no-op saves and the post-push baseline write don't re-push; unlinked/unseeded notes are skipped). Foundry→Obsidian is NOT auto — use Sync / Re-pull for that. Needs the relay (RELAY_API_KEY). Dev mode baselines land in the --out mirror; apply mode in the real vault.">Auto-sync: off</button> </header> +<details id="autoSyncPanel" class="autosync-panel" open style="display:none;margin:0 12px;border-top:1px solid #ddd;padding:6px 12px;background:#fafafa"> + <summary style="cursor:pointer">Auto-sync activity <span id="autoSyncCounts" class="meta"></span><span id="autoSyncNote" class="meta" style="margin-left:8px"></span></summary> + <pre id="autoSyncLog" class="autosync-log" style="max-height:180px;overflow:auto;margin:6px 0 8px;font-size:12px;background:#fff;border:1px solid #eee;padding:6px">(no activity yet)</pre> +</details> <main> <section class="list"> <div class="toolbar"> @@ -118,7 +123,7 @@ <section class="detail" id="detail">Select a row to inspect.</section> </main> <script> -let INDEX = null, STATUS = null, SEL = null, REC_FILTER = null; +let INDEX = null, STATUS = null, SEL = null, REC_FILTER = null, AUTO = null, autoPoll = null; const dryEl = () => document.getElementById('dryRun'); // Recommendation -> display label, badge class, bulk op, and one-line guidance. @@ -147,6 +152,7 @@ async function init() { `matched <b>${c.matched}</b> · cc-only <b>${c.ccOnly}</b> · refined-only <b>${c.refinedOnly}</b> · unlinked <b>${c.unlinked}</b>`; renderRecPanel(); render(); + refreshAutosync(); } function esc(s){ return (s==null?'':String(s)).replace(/[&<>]/g,c=>({'&':'&','<':'<','>':'>'}[c])); } function attr(s){ return (s==null?'':String(s)).replace(/[&"]/g,c=>({'&':'&','"':'"'}[c])); } @@ -318,6 +324,38 @@ async function refreshLive(){ if (r.error){ toast('error: ' + r.error); return; } toast(`live index refreshed: ${r.pairs} name↔uuid pairs cached`); } +// Auto-sync (Obsidian→Foundry, instant): the server watches the vault and pushes +// saved notes into live Foundry. Toggle here; poll for the activity log while on. +async function refreshAutosync(){ + const r = await fetch('/api/autosync').then(r=>r.json()).catch(()=>null); + if (!r) return; + AUTO = r; + const btn = document.getElementById('autoSyncBtn'); + btn.textContent = `Auto-sync: ${r.enabled ? 'on' : 'off'}`; + btn.classList.toggle('primary', r.enabled); + const panel = document.getElementById('autoSyncPanel'); + panel.style.display = r.enabled ? '' : 'none'; + document.getElementById('autoSyncCounts').textContent = + r.enabled ? `pushed ${r.pushed} · skipped ${r.skipped} · errors ${r.errors}` : ''; + document.getElementById('autoSyncNote').textContent = + r.enabled && STATUS && STATUS.mode !== 'apply' + ? '(dev mode — baselines land in the --out mirror, not the real vault; run --apply to baseline the real vault)' + : ''; + const log = document.getElementById('autoSyncLog'); + log.textContent = r.events && r.events.length + ? r.events.map(e => `${e.time.replace('T',' ').slice(5,19)} ${e.status.padEnd(7)} ${e.name} — ${e.message}`).join('\n') + : '(no activity yet — save a linked, seeded note in your vault to trigger a push)'; + if (r.enabled && !autoPoll) autoPoll = setInterval(refreshAutosync, 2000); + if (!r.enabled && autoPoll) { clearInterval(autoPoll); autoPoll = null; } +} +async function toggleAutosync(){ + const want = !(AUTO && AUTO.enabled); + toast(`turning auto-sync ${want ? 'on' : 'off'}…`); + const r = await fetch('/api/autosync', {method:'POST', headers:{'content-type':'application/json'}, body: JSON.stringify({enabled: want})}).then(r=>r.json()); + if (r.error){ toast('error: ' + r.error); return; } + toast(`auto-sync ${r.enabled ? 'ON — saving a note pushes it to live Foundry' : 'off'}`); + refreshAutosync(); +} // Push every vault-newer (sync-cc) note into live Foundry in one run. dry-run (default) // lists what would be pushed; apply pushes each, backs up the live entry, and baselines // the note so a re-run only catches new edits. Replaces scripts/resync.ts. diff --git a/src/server.ts b/src/server.ts index 92b8715..2553e60 100644 --- a/src/server.ts +++ b/src/server.ts @@ -10,14 +10,15 @@ import { createServer, type IncomingMessage, type ServerResponse, type Server } from "node:http"; import { readFile, writeFile, mkdir, copyFile, access, stat } from "node:fs/promises"; -import { join, dirname } from "node:path"; +import { watch, readdirSync, statSync, type FSWatcher } from "node:fs"; +import { join, dirname, relative, basename, extname } from "node:path"; import { fileURLToPath } from "node:url"; import { JournalDb } from "./db.js"; import { indexAll, seedRow, syncRow, rePullRow, importRow, buildBlock, seedBlockContent, type FileRow, type IndexResult } from "./batch.js"; import { RelayClient } from "./relay/client.js"; import { pushNote } from "./push.js"; import { nameUuidIndexFromEntries, saveNameUuidIndex, loadNameUuidIndex, MapNameResolver, type NameResolver } from "./resolver.js"; -import { splitFrontmatter } from "./frontmatter.js"; +import { splitFrontmatter, readFoundryBlock } from "./frontmatter.js"; import { contentHash } from "./normalize.js"; import { backupStamp } from "./write.js"; import type { RelayConfig, FoundryHostConfig } from "./config.js"; @@ -53,6 +54,7 @@ interface State { db: JournalDb; cfg: ServerConfig; index: IndexResult | null; + autosync: AutoSyncController; } function send(res: ServerResponse, code: number, body: unknown): void { @@ -427,9 +429,198 @@ async function handleLink(state: State, req: IncomingMessage, res: ServerRespons } } +export interface AutoSyncEvent { + time: string; + name: string; + status: "pushed" | "skipped" | "error"; + message: string; +} + +/** + * Auto-sync (Obsidian→Foundry, instant). Watches the refined vault dir for .md saves + * and pushes each changed, linked, seeded note into live Foundry via the relay — the + * same `pushNote` the manual "push" button uses. Foundry keeps running; no LevelDB, + * no Docker stop. + * + * Loop/idempotency guard: a save only pushes if the note's body contentHash differs + * from its `foundry.contentHash` baseline. After a push, `baselineNote` rewrites that + * baseline to the new body hash, so the watcher's own baseline write (and any no-op + * save that didn't change the body) is skipped — no feedback loop, no redundant push. + * Unlinked (no `foundry.cc_uuid`) or unseeded (no `foundry.contentHash`) notes are + * skipped; seed/link them manually first. Foundry→Obsidian is NOT auto (the relay has + * no change-push; use the dashboard's Sync / Re-pull buttons for that direction). + * + * In dev mode the baseline lands in the `--out` mirror (consistent with dev semantics); + * in apply mode it lands in the real vault with a `.bak`. The push itself goes live via + * the relay regardless of mode. + */ +class AutoSyncController { + enabled = false; + private recursive = false; + private root: FSWatcher | null = null; + private subs: { w: FSWatcher; prefix: string }[] = []; + private timers = new Map<string, NodeJS.Timeout>(); + private inflight = new Set<string>(); + private queue: string[] = []; + private running = 0; + private readonly concurrency = 3; + private readonly debounceMs = 700; + events: AutoSyncEvent[] = []; + private readonly maxEvents = 100; + pushed = 0; + skipped = 0; + errors = 0; + + constructor(private state: State) {} + + status() { + return { + enabled: this.enabled, + mode: this.state.cfg.mode, + watching: this.state.cfg.refinedDir, + pushed: this.pushed, + skipped: this.skipped, + errors: this.errors, + events: this.events, + }; + } + + private log(name: string, status: AutoSyncEvent["status"], message: string): void { + this.events.unshift({ time: new Date().toISOString(), name, status, message }); + if (this.events.length > this.maxEvents) this.events.length = this.maxEvents; + if (status === "pushed") this.pushed++; + else if (status === "error") this.errors++; + else this.skipped++; + } + + async setEnabled(on: boolean): Promise<void> { + if (on === this.enabled) return; + if (on) await this.start(); + else this.stop(); + } + + private async start(): Promise<void> { + if (!this.state.cfg.relayCfg) throw new Error("relay not configured — auto-sync needs RELAY_API_KEY / --relay-api-key"); + const dir = this.state.cfg.refinedDir; + try { + this.root = watch(dir, { recursive: true }, (evt, fn) => this.onChange(evt, fn, "")); + this.recursive = true; + } catch { + // Older Node / platform without recursive fs.watch: watch the root + every + // subdir individually, and re-scan on renames so new folders get watched too. + this.root = watch(dir, (evt, fn) => this.onChange(evt, fn, "")); + this.recursive = false; + } + if (!this.recursive) this.rescanSubs(dir); + this.enabled = true; + this.log("(watcher)", "skipped", `auto-sync ON — watching ${dir}${this.recursive ? " (recursive)" : " (per-dir fallback)"}`); + } + + /** Attach non-recursive watchers to every subdir (skipping .obsidian). Fallback path. */ + private rescanSubs(dir: string): void { + const root = this.state.cfg.refinedDir; + const seen = new Set(this.subs.map((s) => s.prefix)); + const stack = [dir]; + while (stack.length) { + const d = stack.pop()!; + let entries: string[]; + try { entries = readdirSync(d); } catch { continue; } + for (const e of entries) { + const full = join(d, e); + let isDir = false; + try { isDir = statSync(full).isDirectory(); } catch { continue; } + if (!isDir) continue; + if (e === ".obsidian" || e.startsWith(".")) continue; + const prefix = relative(root, full).replace(/\\/g, "/"); + if (seen.has(prefix)) { stack.push(full); continue; } + seen.add(prefix); + try { + const w = watch(full, (evt, fn) => this.onChange(evt, fn, prefix)); + this.subs.push({ w, prefix }); + } catch { /* unreadable dir — skip */ } + stack.push(full); + } + } + } + + stop(): void { + this.root?.close(); this.root = null; + for (const s of this.subs) s.w.close(); + this.subs = []; + for (const t of this.timers.values()) clearTimeout(t); + this.timers.clear(); + this.queue.length = 0; + this.enabled = false; + this.log("(watcher)", "skipped", "auto-sync OFF"); + } + + private onChange(_evt: string, filename: string | null, prefix: string): void { + if (!filename) return; + const rel = (prefix ? `${prefix}/${filename}` : filename).replace(/\\/g, "/"); + if (!rel.endsWith(".md")) return; + if (rel.split("/").includes(".obsidian")) return; + // Fallback path: a rename may have created a new subdir — re-scan to watch it. + if (!this.recursive) this.rescanSubs(this.state.cfg.refinedDir); + const prev = this.timers.get(rel); + if (prev) clearTimeout(prev); + this.timers.set(rel, setTimeout(() => { + this.timers.delete(rel); + if (this.inflight.has(rel)) return; + this.queue.push(rel); + this.drain(); + }, this.debounceMs)); + } + + private drain(): void { + while (this.running < this.concurrency && this.queue.length) { + const rel = this.queue.shift()!; + this.running++; + this.process(rel).catch(() => {}).finally(() => { this.running--; this.drain(); }); + } + } + + private async process(relPath: string): Promise<void> { + this.inflight.add(relPath); + const name = basename(relPath, extname(relPath)); + try { + const abs = await resolveRefined(this.state, relPath); + let md: string; + try { md = await readFile(abs, "utf8"); } catch { this.log(name, "skipped", "file removed"); return; } + const { fm, body } = splitFrontmatter(md); + const fb = readFoundryBlock(fm); + if (!fb?.cc_uuid) { this.log(name, "skipped", "not linked — no foundry.cc_uuid (seed/link first)"); return; } + if (!fb.contentHash) { this.log(name, "skipped", "not seeded — no foundry.contentHash baseline"); return; } + const bodyHash = contentHash(body); + if (bodyHash === fb.contentHash) { this.log(name, "skipped", "unchanged since last push"); return; } + + const relay = relayClient(this.state); + const outcome = await pushNote({ + notePath: abs, + noteName: name, + outDir: this.state.cfg.outDir, + relay, + foundryDataDir: this.state.cfg.foundryCfg?.dataDir ?? "", + world: this.state.cfg.foundryCfg?.world ?? "", + dryRun: false, // auto-sync always applies — the whole point is hands-off live push + log: () => {}, + }); + // Baseline foundry.contentHash to the new body hash so a re-save with no further + // edit (and the watcher's own baseline write) is a no-op. Idempotency lives here, + // not in pushNote (which always PUTs). + const baselined = await baselineNote(this.state, relPath, abs); + this.log(name, "pushed", `→ ${outcome.ccUuid}${outcome.updatedName ? ` ("${outcome.updatedName}")` : ""}${baselined ? " · baselined" : ""}`); + } catch (e) { + this.log(name, "error", (e as Error).message); + } finally { + this.inflight.delete(relPath); + } + } +} + export async function startServer(cfg: ServerConfig): Promise<{ server: Server; state: State }> { const db = await JournalDb.open(cfg.journal); - const state: State = { db, cfg, index: null }; + const state = { db, cfg, index: null } as State; + state.autosync = new AutoSyncController(state); // Build the index once at startup. In dev mode overlay the <out> mirror so dev // writes are reflected (the corpora are otherwise static while the server runs). const ov = overlayDirs(state); @@ -489,6 +680,20 @@ export async function startServer(cfg: ServerConfig): Promise<{ server: Server; if (req.method === "POST" && url.pathname === "/api/refresh") { return handleRefresh(state, req, res); } + if (req.method === "GET" && url.pathname === "/api/autosync") { + return send(res, 200, state.autosync.status()); + } + if (req.method === "POST" && url.pathname === "/api/autosync") { + const body = await readJsonBody(req); + if (body === null) return send(res, 400, { error: "bad json" }); + try { + await state.autosync.setEnabled(body.enabled === true); + send(res, 200, state.autosync.status()); + } catch (e) { + send(res, 500, { error: (e as Error).message }); + } + return; + } send(res, 404, { error: "not found" }); } catch (e) { send(res, 500, { error: (e as Error).message }); diff --git a/wiki/components/_index.md b/wiki/components/_index.md new file mode 100644 index 0000000..3cb0f3d --- /dev/null +++ b/wiki/components/_index.md @@ -0,0 +1,13 @@ +--- +type: meta +title: "Components Index" +updated: 2026-06-22 +--- + +# Components + +Reusable pieces shared across modules. See `_templates/component.md`. + +- [[content-hash]] — body contentHash idempotency guard +- [[foundry-block]] — the `foundry:` frontmatter block + read/baseline helpers +- [[name-uuid-resolver]] — name↔uuid map for link resolution \ No newline at end of file diff --git a/wiki/components/content-hash.md b/wiki/components/content-hash.md new file mode 100644 index 0000000..6096e05 --- /dev/null +++ b/wiki/components/content-hash.md @@ -0,0 +1,35 @@ +--- +type: component +path: "src/normalize.ts" +status: active +purpose: "Stable body hash used as the sync baseline / idempotency guard." +depends_on: [] +used_by: [[batch]], [[server]], [[autosync]], [[push]] +tags: [component] +created: 2026-06-22 +updated: 2026-06-22 +--- + +# content-hash + +`contentHash(body)` in `src/normalize.ts`. Hashes a note's BODY (frontmatter excluded) to a +stable string, stored in the note's `foundry.contentHash` and the cc file's `cc_sync_hash`. + +## How it's used +- **Drift detection** ([[index-recommend]]): `refinedChanged` = `foundry.contentHash` != + `contentHash(body)` (vault newer than last baseline). `ccChanged` = `cc_sync_hash` != + cc body hash. +- **Idempotency guard** ([[autosync]]): a save only pushes if `contentHash(body) != + foundry.contentHash`. After a push, `baselineNote` sets `foundry.contentHash = + contentHash(body)` → subsequent no-op saves skip. +- **push-all**: only rows with `recommendation === "sync-cc"` (i.e. body hash drifted from + baseline) are pushed; each is baselined after. + +## Why it matters +`pushNote` always PUTs. The hash is the ONLY thing preventing redundant pushes and the +auto-sync feedback loop. It's the project's core sync primitive. + +## Notes / gotchas +- It hashes the BODY only, so a frontmatter-only edit (e.g. the post-push baseline rewrite) + does NOT change the body hash → that's exactly why the baseline write is a no-op for the + watcher. \ No newline at end of file diff --git a/wiki/components/foundry-block.md b/wiki/components/foundry-block.md new file mode 100644 index 0000000..6d01e1f --- /dev/null +++ b/wiki/components/foundry-block.md @@ -0,0 +1,36 @@ +--- +type: component +path: "src/frontmatter.ts" +status: active +purpose: "Parse / read / rewrite the `foundry:` identity block in a refined note's frontmatter." +depends_on: [] +used_by: [[batch]], [[push]], [[server]], [[autosync]] +tags: [component] +created: 2026-06-22 +updated: 2026-06-22 +--- + +# foundry-block + +The `foundry:` nested mapping in a refined note's YAML frontmatter ties the note to its +Foundry journal entry. Fields: `cc_uuid` (e.g. `JournalEntry.<id>`), `cc_type`, `folder_path`, +`contentHash`, `syncedAt`. + +## Helpers (src/frontmatter.ts) +- `splitFrontmatter(md)` → `{fm, body}`. +- `readFoundryBlock(fm)` → the `foundry:` mapping as `Record<string,string> | undefined` + (so `fb.cc_uuid`, `fb.contentHash` are available). +- `buildBlock(entry, body, stamp)` → the block content for seeding (used by `seedRow`/`handleLink`). +- `seedBlockContent(md, block)` → inject/refresh the block in a note. + +## In server.ts +- `baselineFoundryBlock(md, newHash, newSyncedAt)` — text surgery that rewrites ONLY the + `contentHash` + `syncedAt` lines inside the `foundry:` block, leaving `cc_uuid`/`cc_type`/ + `folder_path` and all curation untouched. Mirrors `scripts/resync.ts`. +- `baselineNote(state, relPath, absPath)` — reads the note, computes `contentHash(body)`, + calls `baselineFoundryBlock`, writes via the mirror-aware `targetPath`. Returns true if it + wrote. Used by push-all and [[autosync]]. + +## Notes / gotchas +- Baseline text-surgery is line-based: it scans for `foundry:` then the indented ` ` lines + beneath it. A note without a `foundry:` block is returned unchanged (no baseline). \ No newline at end of file diff --git a/wiki/components/name-uuid-resolver.md b/wiki/components/name-uuid-resolver.md new file mode 100644 index 0000000..23d9f78 --- /dev/null +++ b/wiki/components/name-uuid-resolver.md @@ -0,0 +1,28 @@ +--- +type: component +path: "src/resolver.ts" +status: active +purpose: "name↔uuid map for resolving wikilinks/mentions in notes to Foundry journal UUIDs during push." +depends_on: [[relay-client]] +used_by: [[push]], [[server]], [[cli]] +tags: [component] +created: 2026-06-22 +updated: 2026-06-22 +--- + +# name-uuid-resolver + +`src/resolver.ts` — the `NameResolver` interface + `MapNameResolver` impl, plus +`nameUuidIndexFromEntries`, `saveNameUuidIndex`, `loadNameUuidIndex`. + +## How it's built / cached +- `refresh` ([[refresh-flow]]) calls `relay.searchJournalEntries()` (minified) → + `nameUuidIndexFromEntries([{name,uuid}…])` → saves `<out>/name-uuid.json`. +- `pushNote` resolves links via: a preloaded `resolver` arg → else `loadNameUuidIndex(<out>/ + name-uuid.json)` → else build via relay `/search` (and cache it). + +## Notes / gotchas +- The map is keyed by entry NAME. Curated notes whose filename differs from the Foundry entry + name (e.g. "Angro Harn" vs "Angro Harn - Journal") still resolve because the cc payload + carries explicit UUID references, and the index's uuid-fallback matching handles filenames. +- `name-uuid.json` is gitignored output (lives under `--out`). \ No newline at end of file diff --git a/wiki/decisions/001-external-foundry.md b/wiki/decisions/001-external-foundry.md new file mode 100644 index 0000000..80539b0 --- /dev/null +++ b/wiki/decisions/001-external-foundry.md @@ -0,0 +1,32 @@ +--- +type: decision +status: active +date: 2026-06-20 +context: "The shareable compose bundle must let a cloner get running against their own Foundry." +decision: "Foundry is external — the cloner supplies FOUNDRY_URL. No Foundry service in the compose." +consequences: "Compose bundles only the relay (+ the host-run dashboard). The relay must be reachable FROM the Foundry host (the rest-api module connects OUT to it)." +tags: [decision, adr] +created: 2026-06-22 +updated: 2026-06-22 +--- + +# ADR 001 — External Foundry + +## Context +The live stack on the original host runs Foundry in its own compose (`~/docker/foundryvtt`, +Traefik-fronted). For a shareable bundle, baking Foundry in is impractical (license, world +data, secrets). The cloner already has Foundry somewhere. + +## Decision +Foundry stays external. The cloner supplies `FOUNDRY_URL` (+ world, user, password). The +compose provides only the [[threehats-relay | relay]] and the (host-run) dashboard. + +## Consequences +- One fewer service to bundle/license. +- The relay must be reachable from the Foundry host (the rest-api module connects OUT over + WebSocket). If Foundry is remote, expose the relay port (port-forward / tailnet / public domain). +- The `/setup` skill collects `FOUNDRY_URL`/`FOUNDRY_WORLD`/`RELAY_USER`/`RELAY_PASSWORD` from + the user and points their rest-api module at the relay. + +## Related +- [[002-tool-host-run]], [[003-relay-no-journal-db-no-docker-stop]] \ No newline at end of file diff --git a/wiki/decisions/002-tool-host-run.md b/wiki/decisions/002-tool-host-run.md new file mode 100644 index 0000000..29a1e1f --- /dev/null +++ b/wiki/decisions/002-tool-host-run.md @@ -0,0 +1,30 @@ +--- +type: decision +status: active +date: 2026-06-20 +context: "The sync tool reads the Foundry journal LevelDB and shells out to `docker` for refresh --full-index." +decision: "The sync tool runs on the HOST, not in a container. The compose never includes it." +consequences: "The tool needs host FS access (LevelDB) + the docker socket. The dashboard (:7788) is host-run via ./sync.sh ui." +tags: [decision, adr] +created: 2026-06-22 +updated: 2026-06-22 +--- + +# ADR 002 — Tool stays host-run + +## Context +The tool opens the Foundry journal LevelDB (read-only) for the dashboard's full index, and +`refresh --full-index` stops/starts a local Foundry container via `docker`. Both need host +access. Containerizing it would require bind-mounting the LevelDB + docker socket — fragile. + +## Decision +The sync tool (dashboard, CLI) runs on the host. The compose bundles only the relay. The +dashboard binds `0.0.0.0:7788` (reachable on the tailnet) via `./sync.sh ui`. + +## Consequences +- Cloner needs Node on the host (the tool runs via `npx tsx`). +- `.env` carries host-specific paths (`VAULT`, `JOURNAL`, `FOUNDRY_DATA_DIR`) — gitignored. +- The `/setup` skill starts the dashboard on the host, not via compose. + +## Related +- [[001-external-foundry]], [[003-relay-no-journal-db-no-docker-stop]] \ No newline at end of file diff --git a/wiki/decisions/003-relay-no-journal-db-no-docker-stop.md b/wiki/decisions/003-relay-no-journal-db-no-docker-stop.md new file mode 100644 index 0000000..c4548c7 --- /dev/null +++ b/wiki/decisions/003-relay-no-journal-db-no-docker-stop.md @@ -0,0 +1,35 @@ +--- +type: decision +status: active +date: 2026-06-20 +context: "Live push/refresh must not stop Foundry or fight the LevelDB write lock." +decision: "Live ops (push, push-all, refresh) go through the relay. The LevelDB snapshot is only for the full dashboard index (CLI, docker-stop)." +consequences: "Zero-downtime live sync. The index's pull ops reflect the SNAPSHOT, not live Foundry — a known limitation." +tags: [decision, adr] +created: 2026-06-22 +updated: 2026-06-22 +--- + +# ADR 003 — Relay for live ops; LevelDB only for full index + +## Context +Foundry holds the journal LevelDB open with a write lock. Reading it live requires stopping +Foundry (the docker-stop path). That's downtime — unacceptable for routine push/refresh. + +## Decision +- **Push / push-all / refresh** use the [[relay-client | relay]] (`/get`, `/update`, `/search`). + Foundry keeps running; no LevelDB open, no Docker stop. `cp -r` of the live journal dir is + used for the dev dashboard's snapshot (works while locked — only opening it is blocked). +- The **full LevelDB read** (full entry docs, not minified) is CLI-only: `refresh --full-index` + stops/starts a local Foundry container. The dashboard's `indexAll` uses the snapshot copy. + +## Consequences +- Live sync is zero-downtime. +- The dashboard index's `entry` is the SNAPSHOT (`db.byId`), so pull ops (`rePullRow`) reflect + the snapshot, not live Foundry edits. Live edits need a fresh snapshot or a future live-pull + path. See [[index-recommend]]. +- `/search` is minified (no content hash) → can't detect live content changes cheaply + ([[005-autosync-o-to-f-only]]). + +## Related +- [[002-tool-host-run]], [[push-flow]], [[refresh-flow]] \ No newline at end of file diff --git a/wiki/decisions/004-dropped-browser-obsidian.md b/wiki/decisions/004-dropped-browser-obsidian.md new file mode 100644 index 0000000..22dd224 --- /dev/null +++ b/wiki/decisions/004-dropped-browser-obsidian.md @@ -0,0 +1,37 @@ +--- +type: decision +status: active +date: 2026-06-21 +context: "The containerized browser-Obsidian (noVNC) + auto-installed plugin were clunky and had restricted-mode friction." +decision: "Removed the obsidian + obsidian-init compose services and the plugin/. Edit in your own Obsidian desktop app; the dashboard is the only UI." +consequences: "Compose bundles only the relay. The foundry-sync Obsidian plugin no longer exists. Dashboard push/pull/sync + auto-sync replace it." +tags: [decision, adr] +created: 2026-06-22 +updated: 2026-06-22 +--- + +# ADR 004 — Dropped browser Obsidian + plugin + +## Context +Phase 1 of the shareable bundle added `ghcr.io/sytone/obsidian-remote` (browser Obsidian via +noVNC) bind-mounted to the vault, plus a `foundry-sync` Obsidian plugin auto-installed by an +`obsidian-init` one-shot. In practice: the noVNC desktop was heavy/laggy, plugin loading hit +Obsidian's restricted-mode trust prompt (uncertain whether a preseed would be honored), and the +user already edits a real desktop-Obsidian vault on this host (`Land of Mardonar`). + +## Decision +Removed entirely (2026-06-21): +- `obsidian` + `obsidian-init` services deleted from `docker-compose.yml`. +- `plugin/` directory and `scripts/obsidian-init.js` deleted. +- `.env` / `.env.example` obsidian vars (`OBSIDIAN_*`, `DASHBOARD_URL`, `PUID/PGID/TZ`) removed. +- README + `/setup` skill rewritten: relay + dashboard only; edit notes in your own Obsidian + desktop app pointed at `$VAULT`. + +## Consequences +- Compose now bundles ONLY the relay (`docker compose config` → `relay` + `relay-gpu` profile). +- The dashboard (:7788) is the sole UI — consistent with the user's hard rule: "if it's not in + the dashboard UI, it doesn't exist." Push/pull/sync + auto-sync cover what the plugin did. +- The user edits the vault in their own Obsidian app; the dashboard reads/writes the same files. + +## Related +- [[005-autosync-o-to-f-only]] — the dashboard feature that replaced the plugin's push button. \ No newline at end of file diff --git a/wiki/decisions/005-autosync-o-to-f-only.md b/wiki/decisions/005-autosync-o-to-f-only.md new file mode 100644 index 0000000..76f84cc --- /dev/null +++ b/wiki/decisions/005-autosync-o-to-f-only.md @@ -0,0 +1,40 @@ +--- +type: decision +status: active +date: 2026-06-21 +context: "User asked for an auto-sync button so 'any file syncs from either direction instantly.'" +decision: "Auto-sync is Obsidian→Foundry instant only (inotify + contentHash guard). Foundry→Obsidian stays manual (Sync / Re-pull)." +consequences: "The common pain (manual push after editing a note) is solved. Foundry→Obsidian remains a dashboard button click — the relay can't push change events." +tags: [decision, adr] +created: 2026-06-22 +updated: 2026-06-22 +--- + +# ADR 005 — Auto-sync is Obsidian→Foundry only + +## Context +The user wanted hands-off sync both ways. Two facts shaped feasibility: +1. The relay has **no change-push / events stream**. `/search` is minified (`{uuid,id,name,img}`) + with **no content hash** — so a live Foundry content edit can't be detected from `/search`. +2. Detecting a live Foundry edit needs a `/get` per matched entry per cycle and a diff. For a + ~200-entry world, a fast poll (~15s) is ~800 `/get` calls/min through the headless Foundry — + heavy and fragile. The index's drift detection compares LOCAL baselines + a stale snapshot, + not live Foundry ([[index-recommend]]). + +So "instantly from either direction" is impossible: Obsidian→Foundry can be instant (inotify on +the host vault); Foundry→Obsidian cannot. + +## Decision +Auto-sync ([[autosync]]) is **Obsidian→Foundry instant only**. Foundry→Obsidian stays manual +via the dashboard's Sync / Re-pull buttons. The user was offered three scopes (both + 60s poll, +O→F only, both + 15s poll) and chose O→F only. + +## Consequences +- Solves the actual pain: saving a linked, seeded note pushes it to live Foundry within ~1s. +- Foundry→Obsidian requires a manual button click. A future live-pull path (relay `/get` per + matched entry + `entryToObsidian` on a poll) is feasible but not built — cost scales with + matched-entry count. +- The guard + `baselineNote` are what keep it loop-free ([[autosync-watch-loop]]). + +## Related +- [[autosync]], [[autosync-watch-loop]], [[003-relay-no-journal-db-no-docker-stop]] \ No newline at end of file diff --git a/wiki/decisions/_index.md b/wiki/decisions/_index.md new file mode 100644 index 0000000..11ead81 --- /dev/null +++ b/wiki/decisions/_index.md @@ -0,0 +1,15 @@ +--- +type: meta +title: "Decisions Index (ADRs)" +updated: 2026-06-22 +--- + +# Decisions (ADRs) + +Architecture Decision Records. Newest context at the bottom of the file; highest number = most recent. See `_templates/decision.md`. + +- [[001-external-foundry]] — external Foundry (cloner supplies FOUNDRY_URL) +- [[002-tool-host-run]] — sync tool runs on the host, not in a container +- [[003-relay-no-journal-db-no-docker-stop]] — live ops via relay; LevelDB only for full index +- [[004-dropped-browser-obsidian]] — removed containerized Obsidian + plugin +- [[005-autosync-o-to-f-only]] — auto-sync is Obsidian→Foundry instant only \ No newline at end of file diff --git a/wiki/dependencies/_index.md b/wiki/dependencies/_index.md new file mode 100644 index 0000000..fcabbd3 --- /dev/null +++ b/wiki/dependencies/_index.md @@ -0,0 +1,14 @@ +--- +type: meta +title: "Dependencies Index" +updated: 2026-06-22 +--- + +# Dependencies + +External services, libraries, and modules. See `_templates/dependency.md`. + +- [[threehats-relay]] — foundryvtt-rest-api-relay (Go + headless Chrome) +- [[foundryvtt-rest-api-module]] — Foundry module that connects OUT to the relay +- [[classic-level]] — journal LevelDB reader (classic-level npm) +- [[linkedom]] — HTML parsing for Obsidian→Foundry conversion \ No newline at end of file diff --git a/wiki/dependencies/classic-level.md b/wiki/dependencies/classic-level.md new file mode 100644 index 0000000..82aff7f --- /dev/null +++ b/wiki/dependencies/classic-level.md @@ -0,0 +1,34 @@ +--- +type: dependency +name: "classic-level" +version: "^3.0.0" +kind: library +status: active +purpose: "Read the Foundry journal LevelDB (read-only) for the dashboard's full index." +risk: low +tags: [dependency] +created: 2026-06-22 +updated: 2026-06-22 +--- + +# classic-level + +## What it is +The `classic-level` npm package — a Node binding for LevelDB. Used by `src/db.ts` +(`JournalDb.open`, `all()`, `byId()`). + +## Why we depend on it +The dashboard's full index (`indexAll`) needs full journal entry docs (not the minified +`/search` list). Those live in Foundry's journal LevelDB. + +## How we use it +- `JournalDb` opens the LevelDB read-only (a `cp -r` snapshot in dev, since Foundry holds the + live DB with a write lock — `cp` works while locked, opening doesn't). +- `db.byId(ccId)` returns the snapshot entry used by [[batch]] row builders + `handleEntries`/`handleLink`. + +## Risk / lock-in +- Low: stable, narrow use (read-only). Only the full-index path needs the live DB (docker-stop, + CLI-only — [[003-relay-no-journal-db-no-docker-stop]]). + +## Related +- [[batch]], [[index-recommend]], [[003-relay-no-journal-db-no-docker-stop]] \ No newline at end of file diff --git a/wiki/dependencies/foundryvtt-rest-api-module.md b/wiki/dependencies/foundryvtt-rest-api-module.md new file mode 100644 index 0000000..9fac843 --- /dev/null +++ b/wiki/dependencies/foundryvtt-rest-api-module.md @@ -0,0 +1,36 @@ +--- +type: dependency +name: "Foundry rest-api module" +version: "" +kind: module +status: active +purpose: "The Foundry-side module that exposes the relay's endpoints by connecting OUT to it." +risk: medium +tags: [dependency] +created: 2026-06-22 +updated: 2026-06-22 +--- + +# Foundry rest-api module + +## What it is +A Foundry VTT module (`github.com/ThreeHats/foundryvtt-rest-api-relay` ecosystem) installed in +the user's Foundry world. It connects OUT to the relay over WebSocket and services the +relay's forwarded requests (CRUD on journal entries, search). + +## Why we depend on it +Without it, the relay has nothing to forward to — `activeClients` is 0 and every +`/get`/`/update`/`/search` 404s ("No connected Foundry clients found" / "Invalid client ID"). + +## How we use it +- The `/setup` skill tells the user: install the module in Foundry, enable it, and set its + relay URL to `ws(s)://<host-Foundry-can-reach>:3010`. The relay must be reachable FROM the + Foundry host ([[001-external-foundry]]). +- With >1 connected client, requests need a `clientId` query param ([[relay-client]]). + +## Risk / lock-in +- Medium: Foundry-version compatibility; module defines the response envelopes. +- Must be reconnected after Foundry/server restarts. + +## Related +- [[threehats-relay]], [[relay-client]], [[001-external-foundry]] \ No newline at end of file diff --git a/wiki/dependencies/linkedom.md b/wiki/dependencies/linkedom.md new file mode 100644 index 0000000..ff69cfd --- /dev/null +++ b/wiki/dependencies/linkedom.md @@ -0,0 +1,32 @@ +--- +type: dependency +name: "linkedom" +version: "^0.18.12" +kind: library +status: active +purpose: "Parse HTML in the Obsidian→Foundry conversion (markdown→HTML→Campaign Codex JSON)." +risk: low +tags: [dependency] +created: 2026-06-22 +updated: 2026-06-22 +--- + +# linkedom + +## What it is +A DOM-like HTML parser (`linkedom` npm). Used in the Obsidian→Foundry conversion path +(`src/toFoundry.ts` / `obsidianToFoundryJsonLive`). + +## Why we depend on it +The note body is markdown → HTML → the Campaign Codex `flags.campaign-codex.data` structure. +linkedom parses that HTML without a browser. + +## How we use it +- `obsidianToFoundryJsonLive` (called by `buildPushPayload` in [[push]]) builds the full + JournalEntry + the minimal `/update` diff. + +## Risk / lock-in +- Low: pure parsing, no network. Only the conversion path depends on it. + +## Related +- [[push]], [[push-flow]] \ No newline at end of file diff --git a/wiki/dependencies/threehats-relay.md b/wiki/dependencies/threehats-relay.md new file mode 100644 index 0000000..5b8b9d9 --- /dev/null +++ b/wiki/dependencies/threehats-relay.md @@ -0,0 +1,38 @@ +--- +type: dependency +name: "ThreeHats foundryvtt-rest-api-relay" +version: latest +kind: service +status: active +purpose: "REST/WebSocket bridge to a running Foundry world for live push/refresh without downtime." +risk: medium +tags: [dependency] +created: 2026-06-22 +updated: 2026-06-22 +--- + +# ThreeHats foundryvtt-rest-api-relay + +## What it is +A Go relay (`github.com/ThreeHats/foundryvtt-rest-api-relay`) that runs a headless Chrome +session logged into Foundry. Foundry's rest-api module connects OUT to it over WebSocket; API +clients call the relay's HTTP endpoints (`/get`, `/update`, `/create`, `/search`, `/session`, +`/session-handshake`, `/start-session`), which it forwards to the module. + +## Why we depend on it +Live push/refresh without stopping Foundry or touching the LevelDB write lock ([[003-relay-no-journal-db-no-docker-stop]]). + +## How we use it +- Bundled in `docker-compose.yml` as the `relay` service (port 3010, sqlite, `ALLOW_HEADLESS`, + `shm_size 1g`; `relay-gpu` profile for `/dev/dri` passthrough). +- API keys created via the relay's web signup (no env/CLI) → `RELAY_API_KEY`. +- Headless session launched by `scripts/start-relay-session.js` (handshake → RSA-OAEP encrypt + password → start-session). On the original host, `relay-keepalive.sh` keeps it alive. +- [[relay-client]] wraps the endpoints we use. + +## Risk / lock-in +- Medium: external project; envelope shapes are module-defined and per-endpoint inconsistent. +- No change-push/events — `/search` is minified with no content hash ([[005-autosync-o-to-f-only]]). + +## Related +- [[relay-client]], [[foundryvtt-rest-api-module]], [[refresh-flow]], [[push-flow]] \ No newline at end of file diff --git a/wiki/flows/_index.md b/wiki/flows/_index.md new file mode 100644 index 0000000..6041022 --- /dev/null +++ b/wiki/flows/_index.md @@ -0,0 +1,14 @@ +--- +type: meta +title: "Flows Index" +updated: 2026-06-22 +--- + +# Flows + +Data flows / request paths. See `_templates/flow.md`. + +- [[push-flow]] — Obsidian → live Foundry (relay /get → diff → /update) +- [[refresh-flow]] — name↔uuid map via relay /search (zero downtime) +- [[autosync-watch-loop]] — fs.watch → debounce → guard → push → baseline +- [[index-recommend]] — indexAll + recommend() drift detection \ No newline at end of file diff --git a/wiki/flows/autosync-watch-loop.md b/wiki/flows/autosync-watch-loop.md new file mode 100644 index 0000000..1d46408 --- /dev/null +++ b/wiki/flows/autosync-watch-loop.md @@ -0,0 +1,41 @@ +--- +type: flow +status: active +purpose: "On a vault save, push the changed note to live Foundry instantly (Obsidian→Foundry only)." +actors: [[autosync]], [[push]], [[foundry-block]], [[content-hash]], [[relay-client]] +steps: [fs.watch event, filter .md, debounce, guard, pushNote, baselineNote, log event] +tags: [flow] +created: 2026-06-22 +updated: 2026-06-22 +--- + +# autosync-watch-loop + +Driven by [[autosync]] `AutoSyncController`, toggled from the dashboard's "Auto-sync" button. + +## Steps +1. `fs.watch(refinedDir, {recursive:true})` (per-dir fallback) fires on a `.md` change. +2. Filter: must end `.md`, not be a `.bak`, not under `.obsidian/`. +3. Debounce 700ms per file (coalesce rapid saves). +4. `process(relPath)`: + - read via `resolveRefined` (mirror-aware); + - **guard**: skip unless `foundry.cc_uuid` (linked) AND `foundry.contentHash` (seeded) AND + `contentHash(body) !== foundry.contentHash` (actually changed); + - else `pushNote({dryRun:false})` ([[push-flow]]) → `baselineNote`. +5. Log to the 100-event ring buffer (`pushed` / `skipped` / `error`); UI polls `GET /api/autosync`. + +## Why no loop +After a push, `baselineNote` sets `foundry.contentHash = contentHash(body)`. The watcher then +fires on that baseline write too, but the guard reads `body hash == contentHash` → `skipped: +unchanged`. So the baseline write is a no-op for the watcher. No feedback loop, no double-push. + +## Edge cases / failure modes +- Unlinked note (no `cc_uuid`) → `skipped: not linked` (seed/link first). +- Unseeded note (no `contentHash` baseline) → `skipped: not seeded`. +- Relay down / no headless session → `pushNote` → `relay.getEntry` 404 → `error` logged (no + crash, no Foundry write); the note is NOT baselined, so it retries on the next real save. +- New note created from scratch → no `foundry:` block → skipped (seed/link it manually first). + +## Related +- [[005-autosync-o-to-f-only]] — why Foundry→Obsidian isn't in this loop. +- [[push-flow]] — the per-note push this loop invokes. \ No newline at end of file diff --git a/wiki/flows/index-recommend.md b/wiki/flows/index-recommend.md new file mode 100644 index 0000000..cbac3bb --- /dev/null +++ b/wiki/flows/index-recommend.md @@ -0,0 +1,41 @@ +--- +type: flow +status: active +purpose: "Build the file↔entry index and decide a per-row recommendation from local baselines." +actors: [[batch]], [[content-hash]], [[foundry-block]], db.ts +steps: [walk refined+cc, read meta, basename match + uuid fallback, look up snapshot entry, recommend] +tags: [flow] +created: 2026-06-22 +updated: 2026-06-22 +--- + +# index-recommend + +`indexAll` + `recommend` in [[batch]]. Rebuilt on every `GET /api/index` (hash-only, cheap) and +after mutating ops. Powers the dashboard's recommendation badges + bulk actions. + +## Steps +1. Walk refined + cc dirs (with dev-mode `<out>` overlays). +2. Read each file's meta: refined → `foundry.contentHash` (storedHash), body hash, mtime; + cc → `cc_id`, `cc_sync_hash`, body hash, mtime. +3. Match by **basename** (canonical); **uuid fallback** via `foundry.cc_uuid` for curated + notes whose filename differs. +4. `entry = db.byId(ccId)` — from the LevelDB **snapshot**. +5. `recommend(...)`: `seed` | `sync-cc` (vault newer) | `repull` (cc newer) | `conflict` | + `in-sync` | `import` | `review`. + +## Drift is LOCAL-baseline-based, not live +- `refinedChanged` = `foundry.contentHash` != `contentHash(body)` — vault newer than the LAST + BASELINE (set by a prior push/sync). Not "newer than live Foundry." +- `ccChanged` = `cc_sync_hash` != cc body hash — cc file newer than its baseline. +- `entry` is the snapshot, so pull ops (`rePullRow`) reflect the snapshot, not live Foundry. + +## Implications +- The index is a fast, local, offline drift signal — great for "what have I edited since I + last pushed?" It does NOT know about live Foundry edits someone else made. +- Live Foundry edits are only surfaced via the relay push/refresh path. Detecting a live + content change requires `/get` per entry (no hash in `/search`) — see [[005-autosync-o-to-f-only]]. + +## Related +- [[content-hash]], [[foundry-block]] — the baselines this compares. +- [[push-flow]] / [[refresh-flow]] — the live paths that complement this offline index. \ No newline at end of file diff --git a/wiki/flows/push-flow.md b/wiki/flows/push-flow.md new file mode 100644 index 0000000..fc22715 --- /dev/null +++ b/wiki/flows/push-flow.md @@ -0,0 +1,39 @@ +--- +type: flow +status: active +purpose: "Push one refined note's content into LIVE Foundry without stopping Foundry." +actors: [[push]], [[relay-client]], [[foundry-block]], [[name-uuid-resolver]], foundry/assets.ts +steps: [read note, process body images, read foundry.cc_uuid, resolve name↔uuid, relay /get live entry, build minimal diff, backup live entry, relay /update, baseline note] +tags: [flow] +created: 2026-06-22 +updated: 2026-06-22 +--- + +# push-flow + +Obsidian → live Foundry. Driven by [[push]] `pushNote`, invoked from the dashboard's push / +push-all buttons and from [[autosync]]. + +## Steps +1. Read the refined note; pre-process body `![[image]]` embeds → upload to Foundry's uploads + dir, rewrite to `![](servedPath)` (drop non-local). Frontmatter untouched. +2. `readNoteFoundryMeta` → `foundry.cc_uuid` (required — else "run seed first") + portrait. +3. Resolve the name↔uuid map (preloaded → `<out>/name-uuid.json` → build via `/search`). +4. `relay.getEntry(cc_uuid)` — the LIVE entry (keeps ownership/folder/pages/existing image). +5. Upload portrait if present (needs `FOUNDRY_DATA_DIR`/`WORLD`). +6. `buildPushPayload` → minimal diff `{ name, "flags.campaign-codex": cc }` (dot-path merge + preserves sibling flags; no `_id`/`pages`/`ownership`). +7. dryRun → return diff. Apply → back up live entry to `<out>/bak/<name>.<stamp>.json`, then + `relay.updateEntry(uuid, diff)`. +8. (caller) `baselineNote` → set `foundry.contentHash = contentHash(body)` so a re-run skips. + +## Edge cases / failure modes +- No `foundry.cc_uuid` → throws (seed/link first). +- No connected Foundry client → relay `404`/`400` ("Invalid client ID" / lists clients). Start + the headless session. +- Relay timeout `408`/`504` (~10s WS round-trip). +- Image upload skipped when `FOUNDRY_DATA_DIR`/`WORLD` unset (existing image kept). + +## Related +- [[autosync-watch-loop]] — same flow, triggered by a vault save. +- [[refresh-flow]] — builds the name↔uuid map this depends on. \ No newline at end of file diff --git a/wiki/flows/refresh-flow.md b/wiki/flows/refresh-flow.md new file mode 100644 index 0000000..8058750 --- /dev/null +++ b/wiki/flows/refresh-flow.md @@ -0,0 +1,37 @@ +--- +type: flow +status: active +purpose: "Rebuild the cached name↔uuid map live via relay /search — zero Foundry downtime." +actors: [[relay-client]], [[name-uuid-resolver]], [[server]], [[cli]] +steps: [relay /search minified, nameUuidIndexFromEntries, save name-uuid.json] +tags: [flow] +created: 2026-06-22 +updated: 2026-06-22 +--- + +# refresh-flow + +`handleRefresh` (dashboard `POST /api/refresh`) and `cmd refresh` (CLI). Builds `name-uuid.json` +so pushes can resolve note mentions to Foundry UUIDs. + +## Steps +1. `relay.searchJournalEntries()` → `GET /search?filter=documentType:JournalEntry&minified=true` + → `results: [{uuid,id,name,img}]`. +2. `nameUuidIndexFromEntries(results.map(r => ({name, uuid})))` → `{nameToUuid, uuidToName}`. +3. `saveNameUuidIndex(idx, <out>/name-uuid.json)`. +4. Returns `{pairs, path}`. + +## Why /search (not the LevelDB) +`/search` is minified and live — it lists every journal entry with zero Foundry downtime +(no Docker stop, no LevelDB lock). The heavy docker-stop full LevelDB read is CLI-only +(`refresh --full-index`) and only needed for the dashboard's full `indexAll` (which needs full +entry docs, not minified). See [[003-relay-no-journal-db-no-docker-stop]]. + +## Edge cases / failure modes +- 0 connected clients → `404`; >1 → `400` (pass `clientId`). +- Minified results have NO content hash — so refresh detects new/renamed entries but NOT + content changes. That asymmetry is why Foundry→Obsidian isn't auto ([[005-autosync-o-to-f-only]]). + +## Related +- [[push-flow]] — consumes the map. +- [[index-recommend]] — the index uses the LevelDB snapshot, not this map. \ No newline at end of file diff --git a/wiki/modules/_index.md b/wiki/modules/_index.md new file mode 100644 index 0000000..aa7b688 --- /dev/null +++ b/wiki/modules/_index.md @@ -0,0 +1,17 @@ +--- +type: meta +title: "Modules Index" +updated: 2026-06-22 +--- + +# Modules + +One note per major module / package. See `_templates/module.md`. + +- [[server]] — HTTP dashboard + JSON API +- [[autosync]] — vault watcher → live push +- [[push]] — single-note live push +- [[relay-client]] — ThreeHats relay client +- [[batch]] — index + row builders +- [[cli]] — commands +- [[config]] — env-driven config \ No newline at end of file diff --git a/wiki/modules/autosync.md b/wiki/modules/autosync.md new file mode 100644 index 0000000..4af28c6 --- /dev/null +++ b/wiki/modules/autosync.md @@ -0,0 +1,50 @@ +--- +type: module +path: "src/server.ts (AutoSyncController)" +status: active +language: typescript +purpose: "Watch the refined vault and push each saved linked+seeded note into live Foundry instantly (Obsidian→Foundry only)." +maintainer: Kaysser Kayyali +last_updated: 2026-06-22 +linked_issues: [] +depends_on: [[push]], [[relay-client]], [[content-hash]], [[foundry-block]] +used_by: [[server]] +tags: [module] +created: 2026-06-22 +updated: 2026-06-22 +--- + +# autosync + +`AutoSyncController` in `src/server.ts`. Toggled via the dashboard's "Auto-sync" button +(`POST /api/autosync {enabled}`); status + activity log at `GET /api/autosync` (the UI polls +every 2s while on). See [[005-autosync-o-to-f-only]] for why it's one-directional. + +## How it works +1. `fs.watch(refinedDir, {recursive:true})` (per-dir fallback for older Node) → `onChange`. +2. Filter to `*.md`, skip `.obsidian/` and `.bak`. Debounce 700ms per file. +3. `process(relPath)`: + - read the note via `resolveRefined` (mirror-aware); + - **guard**: skip unless `foundry.cc_uuid` exists (linked), `foundry.contentHash` exists + (seeded), and `contentHash(body) !== foundry.contentHash` (actually changed); + - `pushNote` (live, `dryRun:false`) → `baselineNote` (rewrite `foundry.contentHash` to the + new body hash). +4. Bounded concurrency (3), per-file in-flight mutex, 100-event ring buffer (`events`). + +## The guard is the loop/idempotency key +`pushNote` always PUTs. The guard skips no-op saves (body hash == baseline). After a push, +`baselineNote` sets the baseline to the new body hash, so the watcher's own baseline write +(also a `.md` change) reads as unchanged → skipped. No feedback loop, no redundant push. + +## Mode behavior +- **dev**: baseline lands in the `--out` mirror (consistent with dev semantics); the mirror + then wins `resolveRefined` until the real vault is edited again. +- **apply**: baseline lands in the real vault (with `.bak`). +- The push itself goes live via the relay regardless of mode. + +## Notes / gotchas +- Needs the relay + a live headless session (`activeClients > 0`), or `pushNote` → `relay.getEntry` + 404s ("Invalid client ID" / no connected clients) and logs an `error` (no crash, no Foundry write). +- Verified 2026-06-22: no-op save → `skipped: unchanged`; scratch note with fake uuid + body + change → guard passed → push attempted → relay 404 → `error` logged. Real-push end-to-end + pending the dev headless session coming back up. \ No newline at end of file diff --git a/wiki/modules/batch.md b/wiki/modules/batch.md new file mode 100644 index 0000000..bd19cc6 --- /dev/null +++ b/wiki/modules/batch.md @@ -0,0 +1,56 @@ +--- +type: module +path: "src/batch.ts" +status: active +language: typescript +purpose: "Index both corpora against the journal DB; produce match rows + recommendations; build per-row outputs." +maintainer: Kaysser Kayyali +last_updated: 2026-06-22 +linked_issues: [] +depends_on: [[content-hash]], [[foundry-block]], db.ts, toObsidian.ts +used_by: [[server]], [[cli]] +tags: [module] +created: 2026-06-22 +updated: 2026-06-22 +--- + +# batch + +`src/batch.ts` — `indexAll`, `recommend`, and the row builders (`seedRow`, `syncRow`, +`rePullRow`, `importRow`). See [[index-recommend]]. + +## indexAll(db, ccDir, refinedDir, refinedOverlay?, ccOverlay?) +Walks refined + cc dirs (with dev-mode overlays), reads each file's meta, matches by +**basename** (canonical — the exporter names cc files by entry name), with **uuid fallback** +for curated notes whose filename differs but are already linked by `foundry.cc_uuid`. Produces: +`matched`, `ccOnly`, `refinedOnly`, `counts`, `byRecommendation`. In dev mode the server +passes the `<out>/refined` + `<out>/cc` mirrors as overlays so dev writes are reflected. + +## recommend(params) → Recommendation +``` +refined-only → review +cc-only (entry?) → import : review +matched-unlinked → review +matched+linked, !seeded → seed +!hasCc | !ccSynced → sync-cc +refinedChanged && ccChanged → conflict +refinedChanged → sync-cc (vault newer → push) +ccChanged → repull (cc/foundry newer → pull) +else → in-sync +``` +`refinedChanged` = note's `foundry.contentHash` != note body hash (vault newer than last +baseline). `ccChanged` = cc file's `cc_sync_hash` != cc body hash. **Both compare against +LOCAL baselines, not live Foundry.** + +## Row builders +- `seedRow` — inject/refresh the `foundry:` block from the journal entry (curation untouched). +- `syncRow` — regenerate cc.md from the refined note AND refresh the refined `foundry:` block; + baselines both sides. +- `rePullRow` — regenerate the refined body from Foundry (`entryToObsidian(row.entry, …)`), + preserving curated type/aliases/status tags. Uses the snapshot `row.entry`. +- `importRow` — cc-only entry → new un-curated refined note under `refined/imported/<folder>/`. + +## Notes / gotchas +- `row.entry` comes from `db.byId(ccId)` — the LevelDB **snapshot**, not live Foundry. So + pull ops reflect the snapshot; live Foundry edits need a fresh snapshot (docker-stop full + index) or a future live-pull path. See [[003-relay-no-journal-db-no-docker-stop]]. \ No newline at end of file diff --git a/wiki/modules/cli.md b/wiki/modules/cli.md new file mode 100644 index 0000000..5e039d4 --- /dev/null +++ b/wiki/modules/cli.md @@ -0,0 +1,32 @@ +--- +type: module +path: "src/cli.ts" +status: active +language: typescript +purpose: "Entry point (tsx). Commands: ui (dashboard), refresh, push. Env-driven via [[config]]." +maintainer: Kaysser Kayyali +last_updated: 2026-06-22 +linked_issues: [] +depends_on: [[server]], [[batch]], [[relay-client]], [[push]], [[config]] +used_by: [] +tags: [module] +created: 2026-06-22 +updated: 2026-06-22 +--- + +# cli + +`src/cli.ts` — run via `npx tsx src/cli.ts <cmd>` or `./sync.sh <cmd>` (which sources `.env` +and injects `--journal`). + +## Commands (to document fully) +- `ui` → `startServer` ([[server]]). `--vault`, `--cc` (optional — auto-builds stubs), + `--out`, `--host` (default `0.0.0.0`), `--apply` (write real dirs; default dev/dry-run). + Relay enabled when `RELAY_API_KEY` is set. Binds `0.0.0.0:7788`. +- `refresh` → `name-uuid.json` via relay `/search` ([[refresh-flow]]). `--full-index` does + the heavy docker-stop LevelDB read (CLI-only by design). +- `push` → [[push]] one note live. + +## Notes +- `sync.sh` loads `.env` (`set -a; . ./.env`) and injects `--journal` from `$JOURNAL`. +- This page is a stub — expand with exact flags + the `cmdUi`/`cmdRefresh`/`cmdPush` bodies. \ No newline at end of file diff --git a/wiki/modules/config.md b/wiki/modules/config.md new file mode 100644 index 0000000..dbec9de --- /dev/null +++ b/wiki/modules/config.md @@ -0,0 +1,29 @@ +--- +type: module +path: "src/config.ts" +status: active +language: typescript +purpose: "Env-driven config loaders for the relay + Foundry host control." +maintainer: Kaysser Kayyali +last_updated: 2026-06-22 +linked_issues: [] +depends_on: [] +used_by: [[server]], [[cli]], [[relay-client]] +tags: [module] +created: 2026-06-22 +updated: 2026-06-22 +--- + +# config + +`src/config.ts` — `loadRelayConfig` / `loadFoundryConfig` (and the `RelayConfig` / +`FoundryHostConfig` types). Reads `RELAY_URL`, `RELAY_API_KEY`, `RELAY_CLIENT_ID` for the +relay; `FOUNDRY_CONTAINER`, `FOUNDRY_DATA_DIR`, world for host control. See `.env.example` +for the full var list. + +## Notes +- The relay is enabled in the dashboard iff `RELAY_API_KEY` is set. +- `FOUNDRY_CONTAINER` / `FOUNDRY_DATA_DIR` are only used by `refresh --full-index` (stop/start + a local Foundry container + read its live LevelDB). Live push and plain refresh go through + the relay and never touch them. See [[003-relay-no-journal-db-no-docker-stop]]. +- This page is a stub — expand with the exact env→field mapping. \ No newline at end of file diff --git a/wiki/modules/push.md b/wiki/modules/push.md new file mode 100644 index 0000000..f6a8136 --- /dev/null +++ b/wiki/modules/push.md @@ -0,0 +1,48 @@ +--- +type: module +path: "src/push.ts" +status: active +language: typescript +purpose: "Push one refined note into LIVE Foundry via the relay (Foundry keeps running)." +maintainer: Kaysser Kayyali +last_updated: 2026-06-22 +linked_issues: [] +depends_on: [[relay-client]], [[name-uuid-resolver]], [[foundry-block]], foundry/assets.ts, toFoundry.ts +used_by: [[server]], [[autosync]], [[cli]] +tags: [module] +created: 2026-06-22 +updated: 2026-06-22 +--- + +# push + +`src/push.ts` — `pushNote(deps)` and the pure `buildPushPayload(md, noteName, liveEntry, resolver, imageOverride?)`. + +## pushNote flow (see [[push-flow]]) +1. Read the note; pre-process body `![[image]]` embeds (upload co-located files to Foundry's + uploads dir, rewrite to `![](servedPath)`, drop non-local). Frontmatter untouched. +2. `readNoteFoundryMeta` → `foundry.cc_uuid` (throws if absent — "run seed first") + portrait. +3. Resolve the name↔uuid map: preloaded `resolver`, else `<out>/name-uuid.json`, else build via + relay `/search`. +4. `relay.getEntry(id)` — fetch the LIVE entry (preserves ownership/folder/pages/existing image). +5. Portrait upload if the note has one (and `foundryDataDir`/`world` set). +6. `buildPushPayload` → `obsidianToFoundryJsonLive` (full entry with `name` + `flags.campaign-codex` + overridden, links resolved) → minimal diff `{ name, "flags.campaign-codex": cc }` (dot-path merge + preserves sibling flags; never echoes `_id`/`pages`/`ownership`). +7. `dryRun` → return diff. Apply → back up the live entry to `<out>/bak/<name>.<stamp>.json` + (reversible), then `relay.updateEntry(id, diff)`. + +## Critical: no internal idempotency +`pushNote` ALWAYS PUTs (unless `dryRun`). It does NOT compare the note body to anything. The +caller is responsible for gating: the dashboard's push-all and [[autosync]] call `baselineNote` +afterward to set `foundry.contentHash = contentHash(body)` so a re-run skips unchanged notes. +Without that, repeated pushes re-send the same diff. + +## Depends on +- [[relay-client]], [[name-uuid-resolver]], [[foundry-block]] (`splitFrontmatter`, `readFoundryBlock`), + `src/foundry/assets.ts` (portrait + body-image upload), `src/toFoundry.ts` (`obsidianToFoundryJsonLive`). + +## Notes / gotchas +- Image upload needs `FOUNDRY_DATA_DIR` + `FOUNDRY_WORLD`; without them, images are skipped and + existing ones kept. See [[foundry-uploads-convention]] (in the user's memory, not this wiki). +- `skipImageUpload` is used for fast batch dry-run previews that shouldn't touch the Foundry data dir. \ No newline at end of file diff --git a/wiki/modules/relay-client.md b/wiki/modules/relay-client.md new file mode 100644 index 0000000..a39fd3d --- /dev/null +++ b/wiki/modules/relay-client.md @@ -0,0 +1,43 @@ +--- +type: module +path: "src/relay/client.ts" +status: active +language: typescript +purpose: "HTTP client over the ThreeHats foundryvtt-rest-api-relay (x-api-key auth, clientId query param)." +maintainer: Kaysser Kayyali +last_updated: 2026-06-22 +linked_issues: [] +depends_on: [[threehats-relay]], [[config]] +used_by: [[push]], [[server]], [[cli]] +tags: [module] +created: 2026-06-22 +updated: 2026-06-22 +--- + +# relay-client + +`src/relay/client.ts` — `RelayClient(cfg)` wraps the relay's pass-through REST API. See +`docs/relay-api.md` for envelope shapes. + +## Methods +| Method | Relay call | Returns | +|---|---|---| +| `getEntry(uuid)` | `GET /get?uuid=` | `data: <JournalEntry>` | +| `updateEntry(uuid, diff)` | `PUT /update?uuid=` body `{data:<diff>}` | `entity: [<doc>]` | +| `createEntry(entityType, data)` | `POST /create` | `{uuid, data}` | +| `searchJournalEntries()` | `GET /search?filter=documentType:JournalEntry&minified=true` | `results: [{uuid,id,name,img}]` | + +## Mechanics +- Base URL: no `/api` prefix. Auth: `x-api-key` header on every request. +- **World selection**: `clientId` query param. Omit only when exactly one Foundry client is + connected to the key (relay auto-resolves); 0 → `404`, >1 → `400` listing clients. +- The relay forwards to Foundry's rest-api module over WebSocket and returns the module's + response verbatim. Envelopes are per-endpoint (`data` / `entity` / `results`). +- WS round-trip timeout ~10s → `408`/`504`. + +## Notes / gotchas +- For `update`, send a minimal diff (dot-path `flags.campaign-codex`), never the full doc — + echoing `_id`/`pages`/`ownership` would clobber the live entry. +- `/search` is **minified** (`{uuid,id,name,img}`) — no content hash. So you cannot detect a + Foundry content change from `/search` alone; you'd need `/get` per entry. This is why + Foundry→Obsidian auto-sync wasn't built ([[005-autosync-o-to-f-only]]). \ No newline at end of file diff --git a/wiki/modules/server.md b/wiki/modules/server.md new file mode 100644 index 0000000..e8381ad --- /dev/null +++ b/wiki/modules/server.md @@ -0,0 +1,59 @@ +--- +type: module +path: "src/server.ts" +status: active +language: typescript +purpose: "Local review dashboard + JSON API over the batch engine; binds 0.0.0.0:7788." +maintainer: Kaysser Kayyali +last_updated: 2026-06-22 +linked_issues: [] +depends_on: [[batch]], [[push]], [[relay-client]], [[content-hash]], [[foundry-block]], [[name-uuid-resolver]] +used_by: [[cli]] +tags: [module] +created: 2026-06-22 +updated: 2026-06-22 +--- + +# server + +`src/server.ts` — `startServer(cfg)` builds the index once, serves `dashboard.html` at `/` +and a JSON API under `/api/*`. The mode (`dev` | `apply`) is fixed at startup; the UI cannot +escalate to apply unless the server was started with `--apply`. + +## Mode + mirror-aware reads +- **dev**: writes land under `--out/<bucket>/<relPath>` (a mirror). `resolveRefined` / + `resolveCc` read the mirror only if it exists AND is at least as new (mtime) as the real + file — otherwise the real vault (edited in Obsidian) wins. This makes dev a true preview + that never masks live vault edits. +- **apply**: `targetPath` writes the real refined/cc dir; `writeWithBackup` copies a + `.bak-<stamp>` first (reversible). +- dry-run never writes; it collects `preview` entries. + +## Routes +| Method | Path | Handler | +|---|---|---| +| GET | `/` | `dashboard.html` | +| GET | `/api/index` | rebuild index, return `IndexResult` (rebuilt every request — hash-only, cheap) | +| GET | `/api/status` | `{mode, refinedDir, ccDir, outDir}` | +| GET | `/api/file?name=` | per-row detail + seed/sync/re-pull previews | +| GET | `/api/entries` | Foundry journal entries (name+uuid+type) from the snapshot, for the Link picker | +| POST | `/api/action` | `runAction` — seed/sync/repull/import + All variants | +| POST | `/api/push` | [[push]] one note (live) | +| POST | `/api/push-all` | push every vault-newer matched note (live), baseline each | +| POST | `/api/link` | inject the `foundry:` identity block (link a refined-only note by uuid) | +| POST | `/api/refresh` | rebuild `name-uuid.json` via relay `/search` | +| GET/POST | `/api/autosync` | [[autosync]] status / toggle | + +## State +`{ db, cfg, index, autosync }`. `index` is rebuilt on every `/api/index` and after mutating +ops (push-all, link) so recommendations reflect current files. + +## Depends on +- [[batch]] (`indexAll`, row builders), [[push]] (`pushNote`), [[relay-client]], [[content-hash]], + [[foundry-block]] (`readFoundryBlock`, `splitFrontmatter`), [[name-uuid-resolver]]. + +## Notes / gotchas +- The index's drift detection compares against LOCAL baselines + the LevelDB snapshot — not + live Foundry. Live edits are only seen via the relay push/refresh path. See [[index-recommend]]. +- `pushNote` has no internal idempotency; the dashboard's push-all (and [[autosync]]) baseline + the note's `foundry.contentHash` afterward so re-runs skip unchanged notes. \ No newline at end of file