Compare commits
30 Commits
feat/live-
...
main
| Author | SHA1 | Date | |
|---|---|---|---|
| ac6288088a | |||
| 2b8b0302a8 | |||
| 42a9ae4378 | |||
| d006311f3e | |||
| 8f48f356ce | |||
| 2457596f26 | |||
| 19a76a0c01 | |||
| 4f868da9b8 | |||
| cba0b60798 | |||
| 55c03b671f | |||
| da59bb8c40 | |||
| 721a348fbb | |||
| d7b06d7071 | |||
| fa4d36dbe4 | |||
| 4ae4876695 | |||
| 32ed68eb4f | |||
| fe925d4aec | |||
| 7207cc887d | |||
| d8517b0ea4 | |||
| 9c56b22cc8 | |||
| 51bbb350e7 | |||
| 3202115735 | |||
| f189fd739e | |||
| 8406a0a52a | |||
| 70c6d982fa | |||
| 348ab30f03 | |||
| 5d96bf1267 | |||
| d404929a84 | |||
| 8fd56a22d9 | |||
| 37dceb9ac5 |
157
.claude/skills/fobsidian-sync-setup/SKILL.md
Normal file
157
.claude/skills/fobsidian-sync-setup/SKILL.md
Normal file
@@ -0,0 +1,157 @@
|
||||
---
|
||||
name: fobsidian-sync-setup
|
||||
description: Onboard a fresh clone — bring up the relay, wire it to the user's external Foundry, launch the headless session, and start the sync dashboard. Run when the user says "setup", "/setup", "get me running", or "onboarding".
|
||||
allowed-tools: Bash, Read, Write, Edit, AskUserQuestion, WebFetch
|
||||
---
|
||||
|
||||
# Setup — get a fresh clone running
|
||||
|
||||
You bring up the pieces this project needs and wire them to the user's
|
||||
**external** Foundry. You run every command yourself — do NOT hand the user a shell
|
||||
command to type. The only things the user types are values you can't know (their
|
||||
Foundry URL, the API key the relay shows them) and those are collected through
|
||||
`AskUserQuestion`.
|
||||
|
||||
The stack:
|
||||
- **relay** (container) — ThreeHats rest-api relay. Foundry's rest-api module
|
||||
connects OUT to it over WebSocket.
|
||||
- **dashboard** (host-run, `./sync.sh ui`) — the sync tool on :7788. The user edits
|
||||
notes in their own Obsidian desktop app pointed at `$VAULT`; the dashboard
|
||||
reads/writes the same files on disk.
|
||||
|
||||
## Before you start
|
||||
|
||||
Read `.env.example` so you know every var. The sync tool reads env directly
|
||||
(`src/config.ts`); `sync.sh` loads `.env` and injects `--journal`. Pushing edits back
|
||||
to Foundry happens through the dashboard's HTTP API (`POST /api/push`,
|
||||
`/api/push-all`) or its **Auto-sync mode** (watch the vault + poll Foundry).
|
||||
|
||||
If `.env` already exists and `RELAY_API_KEY` + `FOUNDRY_URL` are set, skip to the
|
||||
"Bring up the relay" step (the user may be re-running you).
|
||||
|
||||
## 1. Create `.env` from the template
|
||||
|
||||
```bash
|
||||
cp .env.example .env
|
||||
```
|
||||
|
||||
## 2. Collect the user's Foundry details
|
||||
|
||||
Use `AskUserQuestion` to gather:
|
||||
|
||||
- `FOUNDRY_URL` — their Foundry server URL (e.g. `https://foundry.example.com`).
|
||||
- `FOUNDRY_WORLD` — the world id/title to launch.
|
||||
- `RELAY_USER` / `RELAY_PASSWORD` — a Foundry player account the headless session
|
||||
logs in as.
|
||||
- `VAULT` — absolute host path to their Obsidian vault root (the folder with
|
||||
`.obsidian`); they edit this in their own Obsidian desktop app. `REFINED` defaults
|
||||
to `${VAULT}/Refined`.
|
||||
|
||||
Write each into `.env` with `Edit`. Leave `CC`, `JOURNAL`, and the
|
||||
`FOUNDRY_CONTAINER`/`FOUNDRY_DATA_DIR` block commented/blank unless the user has them.
|
||||
|
||||
## 3. Bring up the relay
|
||||
|
||||
```bash
|
||||
docker compose up -d relay
|
||||
```
|
||||
|
||||
Poll until it answers (it does a first-run migration on `./relay-data`):
|
||||
|
||||
```bash
|
||||
# Retry for ~60s:
|
||||
curl -sf http://localhost:${RELAY_PORT:-3010}/session
|
||||
```
|
||||
|
||||
Print the relay UI URL to the user: `http://localhost:${RELAY_PORT:-3010}`.
|
||||
|
||||
## 4. Create the relay API key (human-in-browser)
|
||||
|
||||
This genuinely can't be scripted — the relay creates keys through its web signup:
|
||||
|
||||
1. Tell the user: "Open `http://localhost:${RELAY_PORT}` in a browser, click Sign Up,
|
||||
create an account, then copy the API key shown on your dashboard."
|
||||
2. Use `AskUserQuestion` to collect the pasted key.
|
||||
3. `Edit` it into `.env` as `RELAY_API_KEY=`.
|
||||
|
||||
## 5. Connect Foundry's rest-api module (human-in-Foundry)
|
||||
|
||||
Foundry's module connects OUT to the relay, so the relay must be reachable FROM the
|
||||
Foundry host. Tell the user clearly:
|
||||
|
||||
> In Foundry: install the "rest-api" module, enable it, and in its settings set the
|
||||
> relay URL to `ws(s)://<host-Foundry-can-reach>:<RELAY_PORT>`. If Foundry is on a
|
||||
> different machine than this compose, expose `RELAY_PORT` (port-forward / tailnet /
|
||||
> public domain) first — the module can't reach `localhost` from another host.
|
||||
|
||||
Then verify a client connected:
|
||||
|
||||
```bash
|
||||
curl -s -H "x-api-key: ${RELAY_API_KEY}" http://localhost:${RELAY_PORT:-3010}/session
|
||||
```
|
||||
|
||||
The response includes `activeClients` (or similar). If zero, stop and tell the user
|
||||
the module isn't connected yet — do not proceed to the headless session.
|
||||
|
||||
## 6. Launch the headless Foundry session
|
||||
|
||||
```bash
|
||||
node scripts/start-relay-session.js
|
||||
```
|
||||
|
||||
This posts `/session-handshake` then `/start-session` (RSA-encrypts the password) to
|
||||
spin up a headless browser logged into their Foundry. It can take up to 2 min. Poll
|
||||
until `activeSessions` is non-empty:
|
||||
|
||||
```bash
|
||||
curl -s -H "x-api-key: ${RELAY_API_KEY}" http://localhost:${RELAY_PORT:-3010}/session
|
||||
```
|
||||
|
||||
If it fails, the script prints which var is wrong (FOUNDRY_URL / RELAY_USER /
|
||||
RELAY_PASSWORD / FOUNDRY_WORLD). Re-collect the bad value via `AskUserQuestion`,
|
||||
`Edit` `.env`, and retry.
|
||||
|
||||
## 7. Build the name↔uuid map + start the dashboard
|
||||
|
||||
```bash
|
||||
./sync.sh refresh --out "${OUT:-./out}"
|
||||
```
|
||||
|
||||
This lists journal entries via the relay `/search` (zero Foundry downtime) and caches
|
||||
`name-uuid.json`.
|
||||
|
||||
Then launch the dashboard (omit `--cc` if `CC` is blank — the CLI auto-builds cc stubs
|
||||
from the journal, or runs without if `JOURNAL` is also blank):
|
||||
|
||||
```bash
|
||||
./sync.sh ui --vault "${REFINED}" --out "${OUT:-./out}" --host 0.0.0.0
|
||||
```
|
||||
|
||||
Run it in the background (`run_in_background: true`) so it stays up. Print to the
|
||||
user:
|
||||
|
||||
- Dashboard: `http://<this-host-ip>:7788`
|
||||
- It opens in dev mode with dry-run ON by default — actions preview without writing.
|
||||
Uncheck dry-run to write into `--out`; restart with `--apply` to write back to the
|
||||
real refined/cc dirs (with `.bak-<iso>` backups).
|
||||
- Open `$VAULT` in your Obsidian desktop app to edit notes; the dashboard reads/writes
|
||||
the same files.
|
||||
|
||||
## 8. Tell the user how syncing works
|
||||
|
||||
The project's hard rule: if it's not in the dashboard UI, it doesn't exist. From the
|
||||
dashboard the user can:
|
||||
|
||||
- **Push** one note (`POST /api/push {name}`) or **Push all** edited notes
|
||||
(`POST /api/push-all`).
|
||||
- Flip on **Auto-sync mode** — the dashboard watches the refined vault and pushes
|
||||
Obsidian→Foundry automatically the moment a linked, seeded note is saved (guarded by
|
||||
the note's `foundry.contentHash` baseline, so no-op saves don't re-push). Needs the
|
||||
relay + a live headless session. Foundry→Obsidian is NOT automatic (the relay has no
|
||||
change-push) — use the Sync / Re-pull buttons for that direction.
|
||||
|
||||
## If something is already running
|
||||
|
||||
- Relay already up + key already set → skip to step 5/6.
|
||||
- Headless session already active (`activeSessions` non-empty) → skip to step 7.
|
||||
- Dashboard already on :7788 → don't start a second one; just print its URL.
|
||||
76
.env.example
Normal file
76
.env.example
Normal file
@@ -0,0 +1,76 @@
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# foundry-obsidian-sync — environment template.
|
||||
# Copy to .env and fill in: cp .env.example .env
|
||||
# (Or just run the /setup skill, which fills most of this in for you.)
|
||||
# .env is gitignored — never commit real keys or passwords.
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
|
||||
# === Relay (ThreeHats foundryvtt-rest-api-relay) ===
|
||||
# Host port the relay publishes (its web UI + REST API + WS endpoint live here).
|
||||
RELAY_PORT=3010
|
||||
RELAY_CONTAINER=foundry-rest-api-relay
|
||||
# URL the relay advertises to itself (email links, etc.). Usually http://localhost:<RELAY_PORT>.
|
||||
RELAY_FRONTEND_URL=http://localhost:3010
|
||||
# URL the sync TOOL calls the relay at. For a local compose this is
|
||||
# http://localhost:<RELAY_PORT>. If Foundry is remote and you front the relay with
|
||||
# a public/tailnet domain, set this to that URL instead.
|
||||
RELAY_URL=http://localhost:3010
|
||||
# x-api-key for /get, /update, /search. Created via the relay's web UI on first run
|
||||
# (sign up at http://localhost:<RELAY_PORT>, copy the key from the dashboard).
|
||||
RELAY_API_KEY=
|
||||
# Optional: pin a specific connected Foundry client (leave blank to auto-resolve
|
||||
# when exactly one client is connected to the key).
|
||||
RELAY_CLIENT_ID=
|
||||
|
||||
# === Headless Foundry session (the relay drives a headless browser at Foundry) ===
|
||||
# Your Foundry server URL. The relay logs a headless user into THIS Foundry.
|
||||
FOUNDRY_URL=https://your-foundry.example.com
|
||||
# Foundry player credentials the headless session uses to log in.
|
||||
RELAY_USER=
|
||||
RELAY_PASSWORD=
|
||||
# Foundry world id/title to launch in the headless session.
|
||||
FOUNDRY_WORLD=
|
||||
# Seconds before an idle headless session is reaped (relay env).
|
||||
HEADLESS_SESSION_TIMEOUT=3600
|
||||
|
||||
# === Dashboard auth (E7) ===
|
||||
# The dashboard binds 127.0.0.1 by default (localhost-only, no auth needed).
|
||||
# To expose it on your tailnet, pass --host 0.0.0.0 AND set a token below, then
|
||||
# flip ENABLE_AUTH_MIDDLEWARE=on. With the flag on, a 0.0.0.0 bind WITHOUT a
|
||||
# token is refused at boot (safe-by-default); the flag on without a token is
|
||||
# also refused (self-lockout guard — you'd otherwise brick the dashboard).
|
||||
# (Optional; off-by-default — the dashboard is localhost-only without these.)
|
||||
DASHBOARD_AUTH_TOKEN=
|
||||
ENABLE_AUTH_MIDDLEWARE=false
|
||||
|
||||
# IMPORTANT — networking: Foundry's rest-api module connects OUT to the relay over
|
||||
# WebSocket. So the relay must be REACHABLE FROM your Foundry host. If Foundry runs
|
||||
# elsewhere, expose RELAY_PORT (port-forward / tailnet / public domain) and point the
|
||||
# module's relay URL at ws(s)://<that-reachable-host>:<RELAY_PORT>.
|
||||
|
||||
# === Vault (the sync tool / dashboard reads & writes this directly on the host) ===
|
||||
# Absolute path on the HOST to your Obsidian vault root (the folder containing
|
||||
# .obsidian). You edit these notes in your own Obsidian desktop app, pointed at this
|
||||
# vault; the host-run dashboard reads/writes the same files.
|
||||
# QUOTE any path with spaces — sync.sh sources this file in bash, so an unquoted
|
||||
# "VAULT=/home/me/My Vault" tries to run "Vault" as a command and VAULT stays unset.
|
||||
VAULT="/home/me/My Vault"
|
||||
# The refined-notes subdirectory the dashboard's --vault points at. Defaults to
|
||||
# ${VAULT}/Refined if you leave this blank.
|
||||
REFINED="${VAULT}/Refined"
|
||||
# Optional Campaign Codex export dir (omit to let the dashboard auto-build stubs).
|
||||
CC=
|
||||
|
||||
# === Sync tool ===
|
||||
# Optional: a Foundry journal LevelDB snapshot, for offline dashboard indexing
|
||||
# (to-foundry / ui). Leave blank if you only ever push live via the relay.
|
||||
JOURNAL=
|
||||
# Sandbox output dir for the tool (name-uuid.json, index.json, converted notes).
|
||||
OUT=./out
|
||||
|
||||
# === Foundry host control (OPTIONAL — only for `refresh --full-index`) ===
|
||||
# Only needed if you run `./sync.sh refresh --full-index` to stop/start a LOCAL
|
||||
# Foundry Docker container and read its live journal LevelDB. Live `push` and plain
|
||||
# `refresh` go through the relay and never touch these.
|
||||
FOUNDRY_CONTAINER=
|
||||
FOUNDRY_DATA_DIR=
|
||||
14
.gitignore
vendored
14
.gitignore
vendored
@@ -2,8 +2,20 @@ node_modules/
|
||||
# Local live-stack secrets (relay API key, Foundry container/world paths).
|
||||
.env
|
||||
.env.local
|
||||
# Compose runtime state (relay sqlite DB + headless-chrome profile).
|
||||
relay-data/
|
||||
# Tool sandbox output (name-uuid.json, index.json, converted notes).
|
||||
out/
|
||||
# Copied world-data fixtures (your private Foundry snapshot) — never commit.
|
||||
tests/fixtures/
|
||||
# Apply-mode backups written by --apply.
|
||||
*.bak-*
|
||||
*.foundry.json
|
||||
*.foundry.json
|
||||
|
||||
# Wiki (Mode B): .raw/ holds local source dumps (private/large) — keep the folder via .gitkeep.
|
||||
.raw/*
|
||||
!.raw/.gitkeep
|
||||
# Obsidian app config is machine-specific — but share the CSS snippet.
|
||||
.obsidian/*
|
||||
!.obsidian/snippets/
|
||||
!.obsidian/snippets/*
|
||||
38
.obsidian/snippets/vault-colors.css
vendored
Normal file
38
.obsidian/snippets/vault-colors.css
vendored
Normal file
@@ -0,0 +1,38 @@
|
||||
/* vault-colors.css — Mode B (GitHub/Repository) folder colors + custom callouts.
|
||||
Enable: Settings > Appearance > CSS Snippets > open folder > (file is here) >
|
||||
refresh icon > toggle on. */
|
||||
:root {
|
||||
--wiki-1: #4fc1ff; /* modules */
|
||||
--wiki-2: #c586c0; /* components */
|
||||
--wiki-3: #dcdcaa; /* decisions */
|
||||
--wiki-4: #ce9178; /* dependencies */
|
||||
--wiki-5: #6a9955; /* flows */
|
||||
--wiki-7: #569cd6; /* _meta */
|
||||
}
|
||||
|
||||
/* Folder colors in the file explorer */
|
||||
.nav-folder-title[data-path^="wiki/modules"] { color: var(--wiki-1); }
|
||||
.nav-folder-title[data-path^="wiki/components"] { color: var(--wiki-2); }
|
||||
.nav-folder-title[data-path^="wiki/decisions"] { color: var(--wiki-3); }
|
||||
.nav-folder-title[data-path^="wiki/dependencies"] { color: var(--wiki-4); }
|
||||
.nav-folder-title[data-path^="wiki/flows"] { color: var(--wiki-5); }
|
||||
.nav-folder-title[data-path^="_meta"] { color: var(--wiki-7); }
|
||||
.nav-folder-title[data-path=".raw"] { color: #808080; opacity: 0.6; }
|
||||
|
||||
/* Custom callouts */
|
||||
.callout[data-callout='contradiction'] {
|
||||
--callout-color: 209, 105, 105;
|
||||
--callout-icon: lucide-alert-triangle;
|
||||
}
|
||||
.callout[data-callout='gap'] {
|
||||
--callout-color: 220, 220, 170;
|
||||
--callout-icon: lucide-help-circle;
|
||||
}
|
||||
.callout[data-callout='key-insight'] {
|
||||
--callout-color: 79, 193, 255;
|
||||
--callout-icon: lucide-lightbulb;
|
||||
}
|
||||
.callout[data-callout='stale'] {
|
||||
--callout-color: 128, 128, 128;
|
||||
--callout-icon: lucide-clock;
|
||||
}
|
||||
0
.raw/.gitkeep
Normal file
0
.raw/.gitkeep
Normal file
55
CLAUDE.md
Normal file
55
CLAUDE.md
Normal file
@@ -0,0 +1,55 @@
|
||||
# obsidian-foundry-sync: LLM Wiki
|
||||
|
||||
Mode: B (GitHub / Repository)
|
||||
Purpose: Map the architecture of the obsidian-foundry-sync tool — modules, data flows, and the design decisions behind them.
|
||||
Owner: Kaysser Kayyali
|
||||
Created: 2026-06-22
|
||||
|
||||
## Structure
|
||||
|
||||
```
|
||||
vault root (= this repo)
|
||||
├── .raw/ # immutable source documents (README, code dumps, issue exports)
|
||||
├── wiki/
|
||||
│ ├── modules/ # one note per major module / package (server, push, batch, autosync, …)
|
||||
│ ├── components/ # reusable pieces (content-hash guard, foundry block, name↔uuid resolver)
|
||||
│ ├── decisions/ # Architecture Decision Records (ADRs)
|
||||
│ ├── dependencies/ # external deps + services (relay, classic-level, linkedom, rest-api module)
|
||||
│ └── flows/ # data flows / request paths (push, refresh, auto-sync watch loop, index)
|
||||
├── _meta/
|
||||
│ ├── index.md # master catalog of all wiki pages
|
||||
│ ├── log.md # append-only operation log (newest at top)
|
||||
│ ├── overview.md # executive summary of the whole wiki
|
||||
│ └── hot.md # hot cache: ~500-word recent-context summary
|
||||
├── _templates/ # one template per note type
|
||||
└── CLAUDE.md # this file
|
||||
```
|
||||
|
||||
This repo is ALSO the obsidian-foundry-sync code project. The wiki layer (above) documents
|
||||
it; the code itself, build/test/usage, and the live-stack wiring live in `README.md`,
|
||||
`src/`, `scripts/`, `docker-compose.yml`, and `.env.example`. When doing code work, prefer
|
||||
those; when doing knowledge work, prefer the wiki.
|
||||
|
||||
## Conventions
|
||||
|
||||
- All notes use YAML frontmatter: `type`, `status`, `created`, `updated`, `tags` (minimum).
|
||||
Module/flow/decision/dependency notes add the fields in their `_templates/` file.
|
||||
- Wikilinks use `[[Note Name]]` format: filenames are unique, no paths needed.
|
||||
- `.raw/` contains source documents: never modify them.
|
||||
- `_meta/index.md` is the master catalog: update on every ingest/page creation.
|
||||
- `_meta/log.md` is append-only: never edit past entries; new entries go at the TOP.
|
||||
- `_meta/hot.md` is a cache, not a journal: overwrite it completely each time (keep <500 words).
|
||||
|
||||
## Operations
|
||||
|
||||
- **Ingest**: drop a source in `.raw/`, say "ingest [filename]" (→ `wiki-ingest`).
|
||||
- **Query**: ask any question; Claude reads `_meta/index.md` first, then drills in (→ `wiki-query`).
|
||||
- **Lint**: say "lint the wiki" to run a health check (→ `wiki-lint`).
|
||||
- **Save**: say "save this" / "file this" (→ `save`).
|
||||
- **Archive**: move cold sources to `.archive/` to keep `.raw/` clean.
|
||||
|
||||
## Cross-project referencing
|
||||
|
||||
Other Claude Code projects can point at this vault without duplicating context. In that
|
||||
project's CLAUDE.md, add a `## Wiki Knowledge Base` block with `Path: <this repo>`, and the
|
||||
reading order: `_meta/hot.md` → `_meta/index.md` → `wiki/<domain>/_index.md` → individual pages.
|
||||
26
README.md
26
README.md
@@ -18,6 +18,32 @@ Foundry is the source of truth. Each Obsidian note carries a `foundry:` identity
|
||||
block (`cc_uuid`, `cc_type`, `folder_path`, `contentHash`, `syncedAt`) so the two
|
||||
sides stay linked across syncs and renames.
|
||||
|
||||
## Quick start (Docker Compose)
|
||||
|
||||
Clone, then bring up the relay wired to your own **external** Foundry. You need: a
|
||||
running Foundry VTT (license + the
|
||||
[rest-api](https://github.com/ThreeHats/foundryvtt-rest-api-relay) module) and an
|
||||
Obsidian vault on disk that you edit in your own Obsidian desktop app.
|
||||
|
||||
1. `cp .env.example .env` and fill in `FOUNDRY_URL`, `FOUNDRY_WORLD`, `RELAY_USER`/
|
||||
`RELAY_PASSWORD`, and `VAULT` (or let the **`/setup`** skill do all of this for
|
||||
you — it runs the commands itself; you only paste values it can't know).
|
||||
2. `docker compose up -d relay` — relay on `:3010`.
|
||||
3. Open `http://localhost:3010`, sign up, and copy the API key into `.env` as
|
||||
`RELAY_API_KEY`. In Foundry, point the rest-api module at
|
||||
`ws(s)://<host-Foundry-can-reach>:3010`.
|
||||
4. `node scripts/start-relay-session.js` — launches a headless Foundry session the
|
||||
relay drives.
|
||||
5. `./sync.sh refresh --out ./out` then `./sync.sh ui --vault "$VAULT/Refined" \
|
||||
--out ./out --host 0.0.0.0` — opens the dashboard on `:7788`.
|
||||
|
||||
The sync tool stays host-run (it reads the journal LevelDB and shells out to
|
||||
`docker`); the compose provides only the relay. You edit notes in your own Obsidian
|
||||
desktop app pointed at `$VAULT`; the dashboard reads/writes the same files on disk.
|
||||
Push edits back to Foundry from the **dashboard** (`/api/push`, `/api/push-all`), or
|
||||
flip on **Auto-sync mode** to push Obsidian→Foundry automatically the moment you save
|
||||
a note (Foundry→Obsidian stays manual — use Sync / Re-pull for that direction).
|
||||
|
||||
## How it works
|
||||
|
||||
1. `db.ts` opens the journal LevelDB **read-only** with `classic-level`, indexes every
|
||||
|
||||
24
_meta/hot.md
Normal file
24
_meta/hot.md
Normal file
@@ -0,0 +1,24 @@
|
||||
---
|
||||
type: meta
|
||||
title: "Hot Cache"
|
||||
updated: 2026-06-22T03:30:00
|
||||
---
|
||||
|
||||
# Recent Context
|
||||
|
||||
## Last Updated
|
||||
2026-06-22. Scaffolded the Mode B architecture wiki inside the obsidian-foundry-sync repo and seeded it from the codebase.
|
||||
|
||||
## Key Recent Facts
|
||||
- The compose now bundles ONLY the relay (browser Obsidian + plugin were removed 2026-06-21). See [[004-dropped-browser-obsidian]].
|
||||
- Auto-sync mode ([[autosync]]) is Obsidian→Foundry instant only, via `fs.watch` + a `foundry.contentHash` baseline guard. Foundry→Obsidian stays manual — the relay has no change-push and `/search` is minified with no content hash ([[005-autosync-o-to-f-only]]).
|
||||
- `pushNote` always PUTs (no internal idempotency); the guard + `baselineNote` are what prevent loops/redundant pushes ([[push-flow]], [[autosync-watch-loop]]).
|
||||
- The dashboard runs dev/dry-run by default; auto-sync baselines land in the `--out` mirror in dev, the real vault (with `.bak`) in apply.
|
||||
|
||||
## Recent Changes
|
||||
- Created: [[server]], [[autosync]], [[push]], [[relay-client]], [[batch]], [[content-hash]], [[foundry-block]], [[name-uuid-resolver]], [[push-flow]], [[refresh-flow]], [[autosync-watch-loop]], [[index-recommend]], ADRs [[001-external-foundry]]–[[005-autosync-o-to-f-only]], dependency pages.
|
||||
- Wiki decisions recorded for: external Foundry, host-run tool, relay-over-LevelDB, dropped browser Obsidian, O→F-only auto-sync.
|
||||
|
||||
## Active Threads
|
||||
- Verify a real auto-sync push end-to-end once the dev headless session is back up (`activeClients > 0`); the scratch-note test proved the path but the session was down.
|
||||
- MCP server for the wiki is NOT configured (deferred).
|
||||
47
_meta/index.md
Normal file
47
_meta/index.md
Normal file
@@ -0,0 +1,47 @@
|
||||
---
|
||||
type: meta
|
||||
title: "Wiki Index"
|
||||
updated: 2026-06-22
|
||||
---
|
||||
|
||||
# Wiki Index
|
||||
|
||||
Master catalog of every page in the wiki. Update on every ingest / page creation.
|
||||
|
||||
## modules
|
||||
- [[server]] — HTTP dashboard + JSON API (dev/apply modes, mirror-aware reads, push/refresh/autosync)
|
||||
- [[autosync]] — AutoSyncController: vault watcher → guard → live push → baseline
|
||||
- [[push]] — pushNote: one refined note → live Foundry via relay (no Docker stop)
|
||||
- [[relay-client]] — RelayClient over the ThreeHats rest-api relay (/get /update /create /search)
|
||||
- [[batch]] — indexAll + recommend(); seed/sync/rePull/import row builders
|
||||
- [[cli]] — cmdUi / refresh / push commands (env-driven)
|
||||
- [[config]] — env-driven relay + Foundry host config
|
||||
|
||||
## components
|
||||
- [[content-hash]] — body contentHash idempotency guard
|
||||
- [[foundry-block]] — the `foundry:` frontmatter block + read/baseline helpers
|
||||
- [[name-uuid-resolver]] — name↔uuid map for link resolution
|
||||
|
||||
## flows
|
||||
- [[push-flow]] — Obsidian → live Foundry (relay /get → diff → /update)
|
||||
- [[refresh-flow]] — name↔uuid map via relay /search (zero downtime)
|
||||
- [[autosync-watch-loop]] — fs.watch → debounce → guard → pushNote → baselineNote
|
||||
- [[index-recommend]] — indexAll + recommend() drift detection (local baselines + snapshot)
|
||||
|
||||
## decisions (ADRs)
|
||||
- [[001-external-foundry]] — cloner supplies FOUNDRY_URL; no Foundry in the compose
|
||||
- [[002-tool-host-run]] — sync tool runs on the host, not in a container
|
||||
- [[003-relay-no-journal-db-no-docker-stop]] — live push/refresh via relay; LevelDB only for full index
|
||||
- [[004-dropped-browser-obsidian]] — removed containerized Obsidian + plugin; dashboard is the UI
|
||||
- [[005-autosync-o-to-f-only]] — auto-sync is Obsidian→Foundry instant; Foundry→Obsidian stays manual
|
||||
|
||||
## dependencies
|
||||
- [[threehats-relay]] — foundryvtt-rest-api-relay (Go + headless Chrome)
|
||||
- [[foundryvtt-rest-api-module]] — Foundry module that connects OUT to the relay
|
||||
- [[classic-level]] — journal LevelDB reader
|
||||
- [[linkedom]] — HTML parsing for the Obsidian→Foundry conversion
|
||||
|
||||
## meta
|
||||
- [[overview]] — executive summary
|
||||
- [[hot]] — hot cache (~500 words)
|
||||
- [[log]] — append-only operation log
|
||||
20
_meta/log.md
Normal file
20
_meta/log.md
Normal file
@@ -0,0 +1,20 @@
|
||||
---
|
||||
type: meta
|
||||
title: "Operation Log"
|
||||
updated: 2026-06-22
|
||||
---
|
||||
|
||||
# Operation Log
|
||||
|
||||
Append-only. Newest entry at the TOP. Never edit past entries.
|
||||
|
||||
---
|
||||
|
||||
## 2026-06-22 — SCAFFOLD (Mode B)
|
||||
|
||||
- Created vault structure inside the obsidian-foundry-sync repo: `wiki/{modules,components,decisions,dependencies,flows}`, `_meta/`, `_templates/`, `.raw/`.
|
||||
- Created `_meta/index.md`, `_meta/log.md`, `_meta/overview.md`, `_meta/hot.md`; vault `CLAUDE.md`.
|
||||
- Created `_templates/` for module, component, decision, dependency, flow.
|
||||
- Added `.obsidian/snippets/vault-colors.css` (Mode B folder colors + custom callouts).
|
||||
- Seeded real pages from the codebase: modules [[server]] [[autosync]] [[push]] [[relay-client]] [[batch]] [[cli]] [[config]]; components [[content-hash]] [[foundry-block]] [[name-uuid-resolver]]; flows [[push-flow]] [[refresh-flow]] [[autosync-watch-loop]] [[index-recommend]]; decisions [[001-external-foundry]]–[[005-autosync-o-to-f-only]]; dependencies [[threehats-relay]] [[foundryvtt-rest-api-module]] [[classic-level]] [[linkedom]].
|
||||
- Choices: vault inside the repo; no MCP server (yet); Obsidian app not installed (headless host).
|
||||
44
_meta/overview.md
Normal file
44
_meta/overview.md
Normal file
@@ -0,0 +1,44 @@
|
||||
---
|
||||
type: meta
|
||||
title: "Wiki Overview"
|
||||
updated: 2026-06-22
|
||||
---
|
||||
|
||||
# obsidian-foundry-sync — Architecture Wiki
|
||||
|
||||
`obsidian-foundry-sync` is a host-run Node/TypeScript tool that bridges **Foundry VTT**
|
||||
Campaign Codex journal data and an **Obsidian vault** bidirectionally. It reads the
|
||||
Foundry journal LevelDB, converts between Obsidian markdown and Foundry's
|
||||
`flags.campaign-codex` JSON, and pushes/pulls live through a ThreeHats rest-api relay
|
||||
(Foundry keeps running — no Docker stop, no LevelDB write lock).
|
||||
|
||||
## The three pieces
|
||||
|
||||
1. **Relay** (container, `docker compose up -d relay`) — ThreeHats `foundryvtt-rest-api-relay`.
|
||||
Foundry's rest-api module connects OUT to it over WebSocket. API keys come from its web
|
||||
signup; a headless Foundry session drives it.
|
||||
2. **Dashboard** (host-run, `./sync.sh ui`, :7788) — the sync tool. JSON API + UI for
|
||||
seed/sync/import/link/refresh/push, and an **Auto-sync** toggle.
|
||||
3. **Vault** (host path `$VAULT`, edited in your own Obsidian desktop app) — the
|
||||
`Refined` subdir is the tool's `--vault`.
|
||||
|
||||
## Key flows
|
||||
|
||||
- **Push** ([[push-flow]]): refined note → relay `/get` live entry → build minimal diff
|
||||
(`name` + dot-path `flags.campaign-codex`) → `/update`. Reversible (live-entry backup).
|
||||
- **Refresh** ([[refresh-flow]]): relay `/search` (minified) → `name-uuid.json`. Zero downtime.
|
||||
- **Auto-sync** ([[autosync-watch-loop]]): `fs.watch` the vault → guard on the note's
|
||||
`foundry.contentHash` baseline → `pushNote` → `baselineNote`. Obsidian→Foundry instant.
|
||||
Foundry→Obsidian is manual (Sync / Re-pull) — see [[005-autosync-o-to-f-only]].
|
||||
|
||||
## Key design decisions
|
||||
|
||||
See [[decisions]]. Headlines: external Foundry ([[001-external-foundry]]); tool stays
|
||||
host-run ([[002-tool-host-run]]); relay for live ops, LevelDB only for full index
|
||||
([[003-relay-no-journal-db-no-docker-stop]]); browser Obsidian + plugin were dropped — the
|
||||
dashboard is the UI ([[004-dropped-browser-obsidian]]).
|
||||
|
||||
## Hard rule
|
||||
|
||||
UI-only: if it's not in the dashboard UI, it doesn't exist. The agent runs commands; the
|
||||
human only does UI steps (plus the unavoidable relay-signup + Foundry-module-connect).
|
||||
23
_templates/component.md
Normal file
23
_templates/component.md
Normal file
@@ -0,0 +1,23 @@
|
||||
---
|
||||
type: component
|
||||
path: "src/<file>.ts"
|
||||
status: active # active | deprecated | experimental
|
||||
purpose: ""
|
||||
depends_on: []
|
||||
used_by: []
|
||||
tags: [component]
|
||||
created: YYYY-MM-DD
|
||||
updated: YYYY-MM-DD
|
||||
---
|
||||
|
||||
# <Component Name>
|
||||
|
||||
## Purpose
|
||||
|
||||
## API
|
||||
|
||||
## How it works
|
||||
|
||||
## Used by
|
||||
|
||||
## Notes / gotchas
|
||||
22
_templates/decision.md
Normal file
22
_templates/decision.md
Normal file
@@ -0,0 +1,22 @@
|
||||
---
|
||||
type: decision
|
||||
status: active # active | superseded | deprecated
|
||||
date: YYYY-MM-DD
|
||||
context: ""
|
||||
decision: ""
|
||||
consequences: ""
|
||||
tags: [decision, adr]
|
||||
created: YYYY-MM-DD
|
||||
updated: YYYY-MM-DD
|
||||
---
|
||||
|
||||
# ADR <NNN> — <Title>
|
||||
|
||||
## Context
|
||||
|
||||
## Decision
|
||||
|
||||
## Consequences
|
||||
|
||||
## Related
|
||||
- [[<other ADR>]]
|
||||
25
_templates/dependency.md
Normal file
25
_templates/dependency.md
Normal file
@@ -0,0 +1,25 @@
|
||||
---
|
||||
type: dependency
|
||||
name: ""
|
||||
version: ""
|
||||
kind: service # service | library | module | image
|
||||
status: active # active | evaluating | deprecated
|
||||
purpose: ""
|
||||
risk: low # low | medium | high
|
||||
tags: [dependency]
|
||||
created: YYYY-MM-DD
|
||||
updated: YYYY-MM-DD
|
||||
---
|
||||
|
||||
# <Dependency Name>
|
||||
|
||||
## What it is
|
||||
|
||||
## Why we depend on it
|
||||
|
||||
## How we use it
|
||||
|
||||
## Risk / lock-in
|
||||
|
||||
## Related
|
||||
- [[<module that uses it>]]
|
||||
26
_templates/flow.md
Normal file
26
_templates/flow.md
Normal file
@@ -0,0 +1,26 @@
|
||||
---
|
||||
type: flow
|
||||
status: active # active | draft | deprecated
|
||||
purpose: ""
|
||||
actors: [] # modules / components / services involved
|
||||
steps: []
|
||||
tags: [flow]
|
||||
created: YYYY-MM-DD
|
||||
updated: YYYY-MM-DD
|
||||
---
|
||||
|
||||
# <Flow Name>
|
||||
|
||||
## Purpose
|
||||
|
||||
## Steps
|
||||
1.
|
||||
2.
|
||||
|
||||
## Actors
|
||||
- [[<module>]]
|
||||
|
||||
## Edge cases / failure modes
|
||||
|
||||
## Related
|
||||
- [[<other flow>]]
|
||||
30
_templates/module.md
Normal file
30
_templates/module.md
Normal file
@@ -0,0 +1,30 @@
|
||||
---
|
||||
type: module
|
||||
path: "src/<file>.ts"
|
||||
status: active # active | deprecated | experimental | planned
|
||||
language: typescript
|
||||
purpose: ""
|
||||
maintainer: ""
|
||||
last_updated: YYYY-MM-DD
|
||||
linked_issues: []
|
||||
depends_on: [] # [[other module]] / [[component]] / [[dependency]]
|
||||
used_by: []
|
||||
tags: [module]
|
||||
created: YYYY-MM-DD
|
||||
updated: YYYY-MM-DD
|
||||
---
|
||||
|
||||
# <Module Name>
|
||||
|
||||
## Purpose
|
||||
|
||||
## Public surface
|
||||
- `export fn …` —
|
||||
|
||||
## How it works
|
||||
|
||||
## Depends on
|
||||
|
||||
## Used by
|
||||
|
||||
## Notes / gotchas
|
||||
59
docker-compose.yml
Normal file
59
docker-compose.yml
Normal file
@@ -0,0 +1,59 @@
|
||||
# Shareable stack: ThreeHats rest-api relay + browser-served Obsidian.
|
||||
#
|
||||
# Brings up the two containers the sync tool needs alongside an EXTERNAL Foundry
|
||||
# instance (you supply FOUNDRY_URL — Foundry itself is not bundled; it needs a
|
||||
# license + the rest-api module installed). The sync tool itself stays host-run
|
||||
# (./sync.sh ui) so it can read the journal LevelDB and shell out to `docker`.
|
||||
#
|
||||
# Copy .env.example -> .env and fill it in, or run the /setup skill which does it
|
||||
# for you. Then: docker compose up -d
|
||||
#
|
||||
# NOTE on networking: Foundry's rest-api module connects OUT to the relay over
|
||||
# WebSocket. If your Foundry is on a different host than this compose, the relay's
|
||||
# port (RELAY_PORT) must be reachable from Foundry (port-forward / tailnet / public
|
||||
# domain). RELAY_URL below is what the *tool* calls (usually localhost).
|
||||
|
||||
services:
|
||||
relay:
|
||||
# Pin to a release tag (e.g. :2.1.0) for production stability.
|
||||
image: threehats/foundryvtt-rest-api-relay:latest
|
||||
container_name: ${RELAY_CONTAINER:-foundry-rest-api-relay}
|
||||
environment:
|
||||
- DB_TYPE=sqlite
|
||||
- APP_ENV=production
|
||||
- ALLOW_HEADLESS=true
|
||||
- PORT=3010
|
||||
- HEADLESS_SESSION_TIMEOUT=${HEADLESS_SESSION_TIMEOUT:-3600}
|
||||
- FRONTEND_URL=${RELAY_FRONTEND_URL:-http://localhost:3010}
|
||||
volumes:
|
||||
- ./relay-data:/app/data
|
||||
ports:
|
||||
- "${RELAY_PORT:-3010}:3010"
|
||||
# Headless Chrome (Puppeteer) needs a sizable /dev/shm to avoid crashes.
|
||||
shm_size: "1g"
|
||||
restart: unless-stopped
|
||||
|
||||
# Optional GPU-accelerated relay for hosts with a /dev/dri device. Software
|
||||
# rendering (the default `relay` service above) works without it, just slower.
|
||||
# Enable with: docker compose --profile gpu up -d relay-gpu
|
||||
relay-gpu:
|
||||
image: threehats/foundryvtt-rest-api-relay:latest
|
||||
container_name: ${RELAY_CONTAINER:-foundry-rest-api-relay}-gpu
|
||||
profiles: ["gpu"]
|
||||
environment:
|
||||
- DB_TYPE=sqlite
|
||||
- APP_ENV=production
|
||||
- ALLOW_HEADLESS=true
|
||||
- PORT=3010
|
||||
- HEADLESS_SESSION_TIMEOUT=${HEADLESS_SESSION_TIMEOUT:-3600}
|
||||
- FRONTEND_URL=${RELAY_FRONTEND_URL:-http://localhost:3010}
|
||||
- CHROME_GPU_MODE=gpu
|
||||
- CAPTURE_BROWSER_CONSOLE=error
|
||||
volumes:
|
||||
- ./relay-data:/app/data
|
||||
ports:
|
||||
- "${RELAY_PORT:-3010}:3010"
|
||||
devices:
|
||||
- /dev/dri:/dev/dri
|
||||
shm_size: "1g"
|
||||
restart: unless-stopped
|
||||
1222
docs/epics.md
Normal file
1222
docs/epics.md
Normal file
File diff suppressed because it is too large
Load Diff
131
docs/prds/prd-foundry-obsidian-sync-2026-06-22/.decision-log.md
Normal file
131
docs/prds/prd-foundry-obsidian-sync-2026-06-22/.decision-log.md
Normal file
@@ -0,0 +1,131 @@
|
||||
# Decision Log — Live Relay Sync PRD
|
||||
|
||||
Canonical memory + audit trail for this PRD run. Newest at top. Every decision,
|
||||
change, and override lands here as the conversation unfolds.
|
||||
|
||||
---
|
||||
|
||||
## 2026-06-22 — Reviewer gate + finalize (PRD → final)
|
||||
|
||||
Three parallel reviewers dispatched; full reviews on disk:
|
||||
`review-rubric.md`, `review-engineering.md`, `review-launchable.md`.
|
||||
|
||||
**Reviewer outcomes:**
|
||||
- Rubric: 0 critical/high; 3 medium (stale TBD in §1, no Success Metrics, no
|
||||
Glossary), 4 low. → all applied (§8 SMs, §9 Glossary, §1 tie-breaker fix,
|
||||
FR-5.2/5.4 bounds).
|
||||
- Engineering: **Conditional-go** — 2 BLOCKER (F2: `/search` minified has no
|
||||
`folder` and no content hash; content detection = per-note `/get` per poll,
|
||||
contradicts ADR-005), 3 HIGH (Foundry-side hash under-specified, `/get`→
|
||||
`/update` TOCTOU, no shared cross-direction lock), 4 MEDIUM, 4 LOW. → all
|
||||
applied (F2 two-layer rescope, FR-1.4 hash definition, FR-1.10 TOCTOU
|
||||
re-verify, FR-3.1 per-uuid lock, M1–M4, L1–L2).
|
||||
- Launchable: **NOT launchable as specified** — 2 BLOCKER (`/setup` is an
|
||||
author tool not a product surface; `0.0.0.0:7788` no auth), 4 HIGH (error
|
||||
contracts, status-note loop, conflict UX footgun, no Foundry-side undo),
|
||||
5 MEDIUM, 4 LOW. → all applied.
|
||||
|
||||
**Four user decisions (the real crux):**
|
||||
1. **B1-launchable → Downscope NFR-6.** Honest "given operator-wired relay +
|
||||
headless session + rest-api module"; §2 Operator prerequisites block added;
|
||||
`/setup` dropped from FRs. In-UI wizard → future PRD (§7).
|
||||
2. **B2-launchable → Auth-by-default + 127.0.0.1.** F7 + NFR-9 added: auth by
|
||||
default, default bind localhost, `0.0.0.0` requires token, no secret egress,
|
||||
CSRF/same-origin on mutations. (Behavior change from today's `0.0.0.0`
|
||||
no-auth default — acknowledged.)
|
||||
3. **H4-launchable → Cache + revert last push.** NFR-10 + FR-5.6/5.7: pre-push
|
||||
Foundry entry cached to `foundry-backups/<uuid>/<iso>.json` (last N) +
|
||||
dashboard "Revert last push."
|
||||
4. **F2 deep poll → On by default (minutes cadence).** F2 = shallow poll
|
||||
(renames/new/missing, faster) + deep poll (content/moves via per-note
|
||||
`/get`, minutes cadence, `mapPool`-capped), both on by default + manual
|
||||
catch-up trigger.
|
||||
|
||||
**Auto-applied (no user decision):** conflict-UX rename + confirmation
|
||||
(FR-3.6/3.7/3.8/3.9); status-note dot-path + sentinel exclusion (FR-4.3/4.6);
|
||||
error-contracts table (§5a); schema_version + migration (NFR-11); persistent
|
||||
log + diagnostics (FR-5.8/5.9); auto-sync gated to apply mode (FR-1.9);
|
||||
TOCTOU post-push re-verify (FR-1.10); per-uuid shared lock (FR-3.1); transient/
|
||||
persistent retry split (FR-5.3); neutral conflict ordering (FR-3.10);
|
||||
single-client auto-resolve (FR-6.3); OQ-4 and OQ-6 resolved in-PRD; README
|
||||
doc-drift task (§7).
|
||||
|
||||
**Finalize:** decision-log audit ✓ (all entries reflected). Input
|
||||
reconciliation — conversational input only, already folded; no external docs
|
||||
to extract. Reviewer pass ✓. Triage — OQs are non-blockers with defaults
|
||||
(OQ-1/3/7 open with defaults; OQ-2 deferred with reopening condition; OQ-4/5/6
|
||||
resolved). Editorial polish deferred to dev (PRD reads clean; full subagent
|
||||
editorial pass available on request, not blocking dev). **status: final.**
|
||||
|
||||
## 2026-06-22 — Divergence / conflict posture (the no-clobber decisions)
|
||||
|
||||
- **Foundry-side baseline hash scope = content + name + folder_path.** Catches
|
||||
Foundry-side renames and folder moves as divergence (surfaces them instead of
|
||||
silently breaking name↔uuid links). Stored alongside the existing Obsidian
|
||||
`foundry.contentHash` (new field, e.g. `foundry.ccHash`).
|
||||
- **Both-sides-diverged default = surface conflict, DM resolves.** Never
|
||||
auto-overwrite. Conflict shows as a dashboard row; DM picks a side or merges
|
||||
via manual buttons. Strictest "never clobber work" posture.
|
||||
- **Fail-safe when relay /get can't read Foundry = skip push + surface error.**
|
||||
Do not fall back to Obsidian-side-only check (that reintroduces clobber risk).
|
||||
DM sees a "couldn't verify Foundry side — not pushed" row.
|
||||
- **Implies a hard NFR:** no auto-sync operation may overwrite a side that has
|
||||
changed since the last sync. The current O→F auto-sync code violates this
|
||||
(checks Obsidian side only) → must be fixed before/within delivery A.
|
||||
|
||||
## 2026-06-22 — Vision follow-ups (source of truth, divergence, status)
|
||||
|
||||
- **Source of truth = newest version of a doc** — not a fixed side. But **never
|
||||
clobber work**: both-sides-diverged → reconcile, not overwrite. Manual buttons
|
||||
stay as override/repair for unsynced-on-either-side or both-deviate cases.
|
||||
- **Foundry = equal-origin editing surface during prep.** "Foundry is source of
|
||||
truth" demoted to both-diverged tie-breaker rule (newest-wins vs Foundry-wins
|
||||
vs manual merge — undecided).
|
||||
- **"Not syncing" indicator location → DEFERRED.** No easy answer; Foundry UI not
|
||||
ours; dashboard is the only surface we control now. Custom Foundry module is
|
||||
future, not this PRD.
|
||||
- **Sync-status-when-ON → dashboard, for now.** User proposed a maintained
|
||||
**status note inside the vault** (`Sync Status.md`) as a lightweight parity
|
||||
indicator — candidate feature, to confirm.
|
||||
- **Divergence-detection question raised by user** → answered in conversation:
|
||||
yes, guaranteeable, but only with a Foundry-side baseline hash + relay /get
|
||||
before any auto push/pull. Current code lacks this (only checks Obsidian side)
|
||||
→ can silently clobber Foundry edits today. Gap for delivery A to close.
|
||||
|
||||
## 2026-06-22 — Vision input (prep / run-the-match / status)
|
||||
|
||||
- **Prep is the heart.** DM edits many notes at once, draws links, summarizes/
|
||||
closes prior journal notes, adds images/lore, creates objects. Data originates
|
||||
from whichever tool is easier — Obsidian **or** Foundry. Files must stay synced
|
||||
the whole time, bidirectionally, no button-babysitting.
|
||||
- **Run-the-match = sync OFF, Foundry-centric.** DM generates many notes, mostly
|
||||
in Foundry. System must make "not syncing" legible (Foundry should show it).
|
||||
- **Status always visible** — on or off, the DM should see sync status at a
|
||||
glance: when to expect parity and when not.
|
||||
- **Open (probed next):**
|
||||
- Confirm Foundry is an equal-origin editing surface during prep (challenges
|
||||
ADR-005 / README "Foundry is source of truth, O→F hot path"). Reconcile.
|
||||
- Where does the "not syncing" indicator live? Foundry UI is rendered by the
|
||||
rest-api module (not ours). Candidate: a journal entry/flag the sync tool
|
||||
writes that the module surfaces; or dashboard "SYNC PAUSED" state; or both.
|
||||
- Sync-status-when-ON: is the existing autosync activity panel enough, or want
|
||||
a persistent parity indicator ("in parity / N pending / last sync 12s ago")?
|
||||
- Manual buttons (Sync/Re-pull/Push-all) — repair/override tools (stay) vs retire.
|
||||
|
||||
## 2026-06-22 — Run opened
|
||||
|
||||
- **Scope chosen:** A + B + C — full live-sync surface. A = ship & verify O→F
|
||||
auto-sync (uncommitted controller/UI + empty `RELAY_CLIENT_ID` gap + end-to-end
|
||||
live verification). B = Foundry→Obsidian auto direction (no relay push channel;
|
||||
needs polling/snapshot diff). C = operational hardening (watch fallback,
|
||||
concurrency/debounce, error surfacing, retry).
|
||||
- **Stakes chosen:** public/launchable → full NFRs, error contracts, onboarding
|
||||
rigor.
|
||||
- **Working mode chosen:** coaching path, Vision + Features entry (capability-first;
|
||||
single operator role = GM/world-builder).
|
||||
- **Workspace bound:** `docs/prds/prd-foundry-obsidian-sync-2026-06-22/`.
|
||||
- **No `_bmad/` project config** — running on neutral defaults. `planning_artifacts`
|
||||
resolved to `docs/`.
|
||||
- **External session blocker noted:** relay has no connected Foundry client right
|
||||
now and `RELAY_CLIENT_ID` is empty; user will bring the headless session up
|
||||
themselves. Live end-to-end verification (A) gated on that.
|
||||
@@ -0,0 +1,72 @@
|
||||
# E1a Hash Spike — Findings & Verdict
|
||||
|
||||
**Date:** 2026-06-22
|
||||
**Story:** E1a.1 (Build `htmlToMarkdown` inverse + round-trip hash-stability test suite)
|
||||
**Verdict:** **NO-GO** — the markdown-hash divergence guard is not viable; E1b must adopt the **E1b-alt** fork.
|
||||
**Gate artifact:** `tests/e1a-hash-spike.test.ts` (13 tests, green — the 5 unstable fixtures assert `toBe(false)` to pin the NO-GO evidence; if any becomes stable, the assertion flips red and forces re-evaluation).
|
||||
**Inverse attempted:** `src/fromFoundry.ts` `htmlToMarkdown(html: string): string` (linkedom, no resolver, tuned for round-trip: `<img>`→``, two-column → left column only, table best-effort).
|
||||
|
||||
## What the spike did
|
||||
|
||||
For each fixture: Obsidian body → real forward transform (`obsidianToFoundryJsonLive` / `buildFoundryJson` in `src/toFoundry.ts`) → extract `flags["campaign-codex"].data.description` + `.notes` → inverse (`htmlToMarkdown`) → reassemble (left body + `## Secrets\n\n` + notes) → `canonicalize` → compare to `canonicalize(originalBody)`. GO requires every fixture to round-trip; any failure → NO-GO.
|
||||
|
||||
Fixtures use **realistic Obsidian formatting** (blank line after each heading/block — the project's normal style).
|
||||
|
||||
## Result: 7 stable / 5 unstable
|
||||
|
||||
### Stable (round-trip OK)
|
||||
plain text · headings (blank-line formatted) · unordered list · ordered list · image (`` ↔ `<img src>`) · entity/whitespace drift (ampersand, blank-line runs, trailing spaces — `canonicalize` normalizes these) · secrets-last (the contract's order assumption, blank-line formatted)
|
||||
|
||||
### Unstable (NO-GO evidence) — 5 distinct reasons
|
||||
|
||||
1. **Wikilinks / @UUID (fundamental, contract).** The forward (`markdownToHtml` + `wikiToUuid`) converts `[[Susan Delgado]]` → `@UUID[JournalEntry.susan]{Susan Delgado}` via a resolver. The E0.2 seam is `(html: string) => string` with **no resolver**, so the inverse leaves `@UUID` in the markdown; the original body has `[[Susan Delgado]]` → mismatch. **Any linked note with cross-references fails.** Fixing requires changing the seam to `(html, resolver) => string` (a contract change) AND a resolver available at guard time (E1b's push path has it; E2's deep-poll builds one) — but even with that, reasons #3–#5 remain.
|
||||
|
||||
2. **Tables (fundamental, forward gap).** `markdownToHtml` (`src/mdToHtml.ts`) has no table branch — table rows are parsed as paragraphs. A table body cannot round-trip through the forward transform regardless of the inverse. Fix requires adding table support to the forward push transform (a feature, not a seam tweak).
|
||||
|
||||
3. **## Secrets section order (fundamental, convention).** The forward MOVES `## Secrets` to `data.notes` (always last); reconstruction puts it last, reordering vs. an original where another section follows `## Secrets`. Fix requires a project convention that `## Secrets` is always last — not enforceable by the hash, and the forward itself doesn't enforce it.
|
||||
|
||||
4. **## Secrets heading case (fundamental, convention).** The contract re-inserts `## Secrets` (title case); `canonicalize` does NOT normalize case. An uppercase `## SECRETS` heading won't match. Fix requires the project to standardize on exactly `## Secrets` (case-sensitive) — not enforceable by the hash.
|
||||
|
||||
5. **Body starts with `**bold**` (forward bug).** `parseBody` (`src/toFoundry.ts:18`) extracts a tagline via `/\*([^*]+)\*/`, which matches the inner `*his revolver*` of `**his revolver**`, mangling any body that starts with bold. The inverse faithfully converts the mangled HTML back, so the round-trip can never match. This is a forward-transform bug distinct from the inverse, but it makes the markdown round-trip unstable for a whole class of bodies regardless of inverse quality.
|
||||
|
||||
## Additional note (not a fixture): tight formatting
|
||||
|
||||
Bodies with **tight formatting** (no blank line after a heading, e.g. `## H\nText`) also fail: the forward correctly splits into `<h2>` + `<p>`, the inverse rejoins as `## H\n\nText` (block form), and `canonicalize` does not bridge single-vs-double newline after a heading. This is input-dependent (well-formatted notes round-trip; tight notes don't) and could be papered over with a `canonicalize` contract change (normalize heading-newline) — but the spike forbids silently changing `canonicalize`, and reasons #1–#5 already decide NO-GO regardless.
|
||||
|
||||
## Remediation attempted
|
||||
|
||||
- Tuned the inverse for images (`<img>`→`` to match the forward's ``, vs the pull inverse's `![[basename]]`) — **fixed** (images now round-trip).
|
||||
- Added best-effort table handling to the inverse — **does not help** (the forward has no table branch; the gap is upstream).
|
||||
- Considered a resolver seam `(html, resolver) => string` — would fix #1 only; #2–#5 remain. Not worth the contract change for a partial fix.
|
||||
- Considered a `canonicalize` change for blank-line normalization — forbidden silently, and insufficient alone.
|
||||
|
||||
## Recommendation: E1b-alt
|
||||
|
||||
Adopt the **E1b-alt** fork as the E1b design (the epics document already catalogs it as the explicit NO-GO fork, not a void):
|
||||
|
||||
> `ccHash = contentHash(canonicalizeHtml(flags["campaign-codex"].data.description + "\n" + (data.notes ?? "")) + "\n" + name + "\n" + folder)`
|
||||
> — canonicalize the Foundry HTML **directly**, hash the HTML, never hash markdown.
|
||||
|
||||
This sidesteps **all 5** failure reasons:
|
||||
- No inverse → no `@UUID`-without-resolver problem (#1).
|
||||
- No markdown round-trip → no table forward gap (#2), no secrets order (#3), no heading case (#4), no `parseBody` tagline coupling (#5).
|
||||
- Both the baseline (`foundry.ccHash`, stored at push time) and the live ccHash hash the **same HTML** from the live entry → comparable by construction; Foundry-side changes detected by HTML-hash change.
|
||||
|
||||
### E1b-alt contract changes (re-baseline downstream consumers)
|
||||
|
||||
- **E0.2 `ccHash` contract** is superseded: the `HtmlToMarkdown` seam is dropped; a `CanonicalizeHtml` seam is added. `ccHash(entry)` becomes `contentHash(canonicalizeHtml(data.description + data.notes) + name + folder)`. `CC_HASH_CONTRACT` rewrites. `src/cchash.ts` and its tests update. The `(html: string) => string` `htmlToMarkdown` (`src/fromFoundry.ts`) is no longer consumed by ccHash (it can remain as the F→O pull helper, or be retired in favor of `src/htmlMd.ts`).
|
||||
- **`canonicalizeHtml`** is the new spike question: it must normalize incidental HTML whitespace/entity drift (the relay may serialize HTML differently across `/get` calls) so the HTML hash is stable across calls for unchanged content. That is E1b-alt's own mini-gate (a `canonicalizeHtml` stability test: same entry → same canonicalized HTML across plausible serialization variations).
|
||||
- **E2 (F→O deep-pull)** compares `canonicalizeHtml`-based ccHash to `foundry.ccHash` — same comparator, different hash input.
|
||||
- **E3 (conflict) / E4 (parity)** consume the ccHash as an opaque string; the contract change is transparent to them as long as `foundry.ccHash` is stored/restored consistently.
|
||||
- A Foundry-side rename (name-field change) still changes ccHash (name is in the input) — correct, routed through `pushNote`'s `updatedName` path, not a content divergence (E3.5 still holds).
|
||||
|
||||
### What ships from E1a as-is
|
||||
|
||||
- `src/fromFoundry.ts` (`htmlToMarkdown`) — the inverse attempted; **ships unwired** (NO-GO means it is not wired into the push path or ccHash). It remains as the gate artifact's inverse and a reference for F→O pull if useful.
|
||||
- `tests/e1a-hash-spike.test.ts` — the reproducible gate artifact (green, NO-GO pinned).
|
||||
- No change to the forward push path (`pushNote`, `markdownToHtml`) — unchanged, per the AC.
|
||||
- No change to `canonicalize` / `contentHash` signatures — per the AC.
|
||||
|
||||
## Next step
|
||||
|
||||
Stand up **E1b-alt** as the E1b design: re-baseline the E0.2 ccHash contract to the `CanonicalizeHtml` form, build `canonicalizeHtml` + its own stability test (the HTML-serialization-drift mini-gate), then proceed with E1b's controller hardening on the HTML-hash divergence guard.
|
||||
522
docs/prds/prd-foundry-obsidian-sync-2026-06-22/prd.md
Normal file
522
docs/prds/prd-foundry-obsidian-sync-2026-06-22/prd.md
Normal file
@@ -0,0 +1,522 @@
|
||||
---
|
||||
title: "Live Relay Sync — Auto-Sync & Bidirectional Hardening"
|
||||
status: final
|
||||
created: 2026-06-22
|
||||
updated: 2026-06-22
|
||||
reviewers: [rubric, engineering-feasibility, launchable]
|
||||
---
|
||||
|
||||
# Live Relay Sync — Auto-Sync & Bidirectional Hardening
|
||||
|
||||
> Scope: full live-sync surface over the ThreeHats relay — (A) ship & verify
|
||||
> Obsidian→Foundry instant auto-sync **with a no-clobber divergence guard**,
|
||||
> (B) add Foundry→Obsidian auto direction, (C) operational hardening, plus
|
||||
> launchable-grade security, error contracts, and data integrity.
|
||||
> Stakes: **public/launchable** (operator-wired prerequisites — see §2).
|
||||
> Reviewed by three parallel reviewers; findings applied (see
|
||||
> `.decision-log.md` and `review-{rubric,engineering,launchable}.md`).
|
||||
|
||||
## 1. Vision
|
||||
|
||||
A DM's work has two phases, and live sync serves them differently.
|
||||
|
||||
**Prep** is the heart of the tool. The DM is thinking through situations, lore,
|
||||
and NPCs — editing many notes at once, drawing new links between items,
|
||||
summarizing or closing out previous journal notes, dropping in images and lore,
|
||||
creating new objects to build out the world. Data is coming from **whichever
|
||||
tool is easier in the moment** — Obsidian or Foundry — and files must stay in
|
||||
sync the whole time, across both directions, without the DM babysitting a sync
|
||||
button. The feeling to deliver: *always in lockstep while I work*,
|
||||
bidirectionally, across a flurry of simultaneous edits.
|
||||
|
||||
**Running the match** is the live session. The DM is generating a lot of notes,
|
||||
almost entirely inside Foundry, and the sync service is almost certainly **off**
|
||||
in this phase. The system should make that state legible — Foundry should show
|
||||
"not syncing" somewhere, so the DM never wonders whether prep edits are
|
||||
propagating when they aren't. Sync is deliberately paused, not silently absent.
|
||||
|
||||
**Status, always.** Whether sync is on or off, the DM should be able to see the
|
||||
sync status at a glance — when to expect parity and when not to. No guessing
|
||||
whether the vault and Foundry agree.
|
||||
|
||||
_[CONFIRMED] Source of truth = the newest version of a document — not a fixed
|
||||
side. But the system must **never clobber work**: when both sides have diverged
|
||||
since the last sync, it routes to reconciliation instead of overwriting either
|
||||
side. Default resolution = **manual** (FR-3.2): the conflict row offers
|
||||
explicit actions; a newest-mtime convenience is open (OQ-1) but not assumed.
|
||||
The manual buttons (Sync / Re-pull / Push-all) stay precisely for this — quick
|
||||
overrides when a file is unsynced on any side or both deviate._
|
||||
|
||||
_[CONFIRMED] Foundry is an equal-origin editing surface during prep — data
|
||||
originates in whichever tool is easier. "Foundry is source of truth" is demoted
|
||||
to a tie-breaker rule for the both-diverged conflict case, and the conflict UI
|
||||
defaults to a neutral ordering (vault left, Foundry right, no pre-highlighted
|
||||
action) so the undecided tie-breaker does not bias the DM (FR-3.10)._
|
||||
|
||||
_[DEFERRED] Where the "not syncing" indicator lives during run-the-match — no
|
||||
easy answer yet; decision deferred. Foundry UI is rendered by the rest-api
|
||||
module (not ours); the dashboard is the only realistic surface we control now.
|
||||
A custom Foundry module (giving us what the relay does, plus indicators) is a
|
||||
future exploration, not this PRD._
|
||||
|
||||
_[CONFIRMED] Sync-status-when-ON lives in the dashboard for now, plus a single
|
||||
maintained **status note** inside the vault (`Sync Status.md`) showing
|
||||
last-sync time/state — a lightweight, in-our-control parity indicator (FR-4.3)._
|
||||
|
||||
## 2. Problem & Context
|
||||
|
||||
**Where the tool is today.** obsidian-foundry-sync is an offline bidirectional
|
||||
converter between a Foundry VTT Campaign Codex LevelDB snapshot and an Obsidian
|
||||
vault, plus a dashboard that does **manual** live relay operations — push one,
|
||||
push-all, refresh the name↔uuid index, sync, re-pull. An Obsidian→Foundry
|
||||
instant auto-sync (`AutoSyncController` + dashboard toggle) is **code-complete
|
||||
but uncommitted**, and it only checks the Obsidian side before pushing (it
|
||||
compares the note's body hash to its `foundry.contentHash` baseline and pushes
|
||||
on any difference). There is **no Foundry→Obsidian auto direction** — the relay
|
||||
has no change-push channel and `/search` is minified with no content hash, so
|
||||
F→O stays manual (Sync / Re-pull buttons).
|
||||
|
||||
**The prep workflow doesn't fit manual sync.** A DM prepping a session edits
|
||||
many notes at once, draws new links, summarizes/closes prior journal notes,
|
||||
drops in images and lore, and creates new objects — with data originating in
|
||||
**whichever tool is easier in the moment**, Obsidian or Foundry. Pressing
|
||||
Sync/Re-pull per file doesn't scale to that churn; the value proposition of
|
||||
auto-sync is exactly this phase. But auto-sync that only watches one direction
|
||||
leaves half the prep edits unpropagated, and auto-sync that pushes without
|
||||
checking the other side **can silently clobber Foundry edits** — the current
|
||||
O→F code has this risk (confirmed against `src/server.ts:582-617`).
|
||||
|
||||
**The run-the-match phase is Foundry-centric with sync off.** During the live
|
||||
session the DM generates many notes, mostly inside Foundry, and the sync
|
||||
service is expected to be off. Today "off" is silent — the DM can't tell
|
||||
whether prep edits are still propagating. Status legibility (on/off, parity,
|
||||
last sync) is missing on both sides.
|
||||
|
||||
### Operator prerequisites (the honest onboarding boundary)
|
||||
|
||||
Live relay sync sits on top of infrastructure the **operator** (the DM or
|
||||
whoever runs their host) wires **once**, outside the dashboard. The dashboard
|
||||
handles config **detection** and all live operations, but it does not bring up
|
||||
the infrastructure. A non-author DM must complete, or have completed, these
|
||||
gates before the dashboard can reach live sync:
|
||||
|
||||
1. Bring up the ThreeHats relay container (`docker compose up -d relay`).
|
||||
2. Create a relay account and copy `RELAY_API_KEY` into `.env` (browser signup).
|
||||
3. Start the headless Foundry session the relay drives
|
||||
(`scripts/start-relay-session.js`) — shell.
|
||||
4. Point Foundry's rest-api module at the relay WebSocket URL (Foundry-side
|
||||
admin config — cannot be driven from the dashboard at all).
|
||||
5. Install deps and start the dashboard (`npm install`, `./sync.sh ui`).
|
||||
|
||||
The dashboard **surfaces** each unmet gate (FR-6.1/6.2/6.3) and guides
|
||||
remediation, but building an in-UI first-run wizard that performs these steps
|
||||
is **out of scope for this PRD** (future work — see §7). NFR-6 is scoped
|
||||
accordingly. Foundry-side world backups (Foundry's own backup feature) remain
|
||||
the DM's responsibility and are **recommended before enabling auto-sync** —
|
||||
this PRD adds a local pre-push cache + revert (FR-5.6/5.7) as a recovery path,
|
||||
but it is not a substitute for world backups.
|
||||
|
||||
_[ASSUMPTION] The relay remains the live transport for this PRD; the "custom
|
||||
Foundry module that gives us what the relay does" is explicitly future work,
|
||||
not a dependency here._
|
||||
|
||||
_[ASSUMPTION] One relay `/get` per changed linked note per deep-poll tick is
|
||||
acceptable relay load at a minutes cadence with a concurrency cap (OQ-3)._
|
||||
|
||||
**Audience.** Today a single operator (the DM/world-builder). Intended to be
|
||||
launchable — other DMs running it against their own Foundry worlds **given the
|
||||
operator prerequisites above** — so the no-clobber, error-surfacing, security,
|
||||
and data-integrity rigor must hold for non-author users, not just for the one
|
||||
who wrote it.
|
||||
|
||||
## 3. Goals & Non-Goals
|
||||
|
||||
### Goals
|
||||
|
||||
- **G1 — No-clobber bidirectional auto-sync for prep.** Both directions sync
|
||||
automatically; neither side is ever overwritten if it changed since last
|
||||
sync; both-diverged conflicts surface for the DM to resolve.
|
||||
- **G2 — Ship & verify the O→F auto-sync safely.** Commit the existing
|
||||
controller/UI **with the divergence guard added** (Foundry-side baseline hash
|
||||
+ relay `/get` before acting + post-push re-verify), and verify one
|
||||
end-to-end live push.
|
||||
- **G3 — Foundry→Obsidian auto direction.** A working F→O auto path via
|
||||
shallow poll (renames/new/missing) + deep poll (content/moves), within the
|
||||
relay's actual constraints.
|
||||
- **G4 — Legible sync status at all times.** The DM can see, at a glance,
|
||||
whether sync is on or off, whether the vault and Foundry are in parity, and
|
||||
when the last sync landed — in the dashboard plus a maintained vault status
|
||||
note; state persists across restarts.
|
||||
- **G5 — Operational hardening.** Recursive-watch fallback, tuned
|
||||
concurrency/debounce, retry with transient/persistent split, visible error
|
||||
rows, persistent log, shared cross-direction locking.
|
||||
- **G6 — Honest onboarding & config.** Given the operator prerequisites, the
|
||||
dashboard detects and guides every unmet config/relay gate with no shell
|
||||
command from the DM.
|
||||
- **G7 — Security.** The dashboard authenticates by default, binds localhost
|
||||
by default, never exposes secrets to the browser, and guards mutations.
|
||||
- **G8 — Foundry-side data integrity.** Every push to Foundry is preceded by a
|
||||
local backup of the pre-push Foundry state, with a dashboard revert path.
|
||||
|
||||
### Non-Goals
|
||||
|
||||
- **In-UI first-run onboarding wizard** that performs the operator
|
||||
prerequisites (bring up relay, acquire API key, launch headless session, wire
|
||||
rest-api module) — future PRD.
|
||||
- **Custom Foundry module** (indicators inside Foundry UI, relay replacement) —
|
||||
future exploration; would reopen OQ-2.
|
||||
- **"Not syncing" indicator rendered inside Foundry** — deferred; Foundry UI is
|
||||
not a surface we control. Surfaced via dashboard + vault status note instead.
|
||||
- **Syncing during run-the-match.** Sync stays off by design in that phase; the
|
||||
goal is legibility of the off state, not automation during the session.
|
||||
- **Automatic semantic/3-way content merge.** Both-diverged = pick a side or
|
||||
accept divergence via the buttons; no auto content-merge engine.
|
||||
- **Auto-sync of unlinked or unseeded notes.** Seed/link first remains a manual
|
||||
prerequisite (unchanged from current behavior).
|
||||
- **Full LevelDB / docker-stop index in the dashboard.** Remains CLI-only
|
||||
(unchanged).
|
||||
|
||||
## 4. Features & Functional Requirements
|
||||
|
||||
FR IDs are stable. Grouped F1–F7; IDs are `FR-<group>.<n>`.
|
||||
|
||||
### F1 — Obsidian→Foundry auto-sync (safe)
|
||||
|
||||
- **FR-1.1** Watch the refined vault dir for `.md` saves using recursive
|
||||
`fs.watch`, with a per-subdir fallback (re-scanning on subdir create/rename)
|
||||
for platforms/Node versions without recursive watch. Skip `.obsidian`,
|
||||
dotfiles, and the reserved status-note path (FR-4.3).
|
||||
- **FR-1.2** On a save, read the note and skip it if it has no
|
||||
`foundry.cc_uuid` (unlinked) or no `foundry.contentHash` baseline (unseeded)
|
||||
— seed/link remain manual prerequisites.
|
||||
- **FR-1.3** Compute the current Obsidian body hash; if it equals the
|
||||
`foundry.contentHash` baseline, skip (covers no-op saves and the watcher's
|
||||
own post-push baseline write — no feedback loop).
|
||||
- **FR-1.4** **Before pushing**, compute the Foundry-side hash and compare to
|
||||
the stored Foundry-side baseline (`foundry.ccHash`, **new field**). The
|
||||
`/get` that `pushNote` already performs (`src/push.ts:142`) is **reused** for
|
||||
this — no extra round-trip. The Foundry-side hash input is
|
||||
`canonicalize(htmlToMarkdown(flags["campaign-codex"].data)) + "\n" + name +
|
||||
"\n" + folder_path`, i.e. the Foundry HTML body is converted back to refined
|
||||
markdown (the inverse of `obsidianToFoundryJsonLive`, via linkedom) and run
|
||||
through the **same `contentHash` pipeline** so the two sides are directly
|
||||
comparable and the F3 2×2 routing is well-defined. `baselineFoundryBlock`
|
||||
(`src/server.ts:289`) and `baselineNote` (`src/server.ts:307`) must be
|
||||
extended to also rewrite a `ccHash:` line; `readFoundryBlock` consumers read
|
||||
it. A hash-stability unit test across a push→`/get` round-trip is required
|
||||
before FR-1.4 ships.
|
||||
- **FR-1.5** Route: Obsidian-changed **and** Foundry-unchanged (F-hash equals
|
||||
`ccHash` baseline) → push O→F via the same `pushNote` path the manual push
|
||||
button uses; re-baseline both sides on success.
|
||||
- **FR-1.6** If the Foundry side is unreadable (`/get` 404 / timeout / session
|
||||
down), **skip the push and surface an error row** — do not fall back to an
|
||||
Obsidian-side-only check (that reintroduces clobber risk).
|
||||
- **FR-1.7** After a successful push, re-baseline **both** `foundry.contentHash`
|
||||
(Obsidian body) **and** `foundry.ccHash` (Foundry-side) to the new values.
|
||||
Baselines land in the real vault with a `.bak` (apply mode only — see FR-1.9).
|
||||
- **FR-1.8** Auto-sync always applies live to Foundry (dry-run not honored) —
|
||||
the whole point is hands-off live push.
|
||||
- **FR-1.9** **Auto-sync requires apply mode.** Enabling it in dev mode is
|
||||
blocked with an explanatory banner (auto-sync writes live Foundry; dev mode
|
||||
is a preview). This reconciles FR-1.7/1.8 — baselines and live writes both
|
||||
target the real vault/Foundry, never the `--out` mirror.
|
||||
- **FR-1.10** **TOCTOU guard.** After `pushNote`'s `relay.updateEntry`
|
||||
succeeds, re-`/get` and verify the entry's Foundry-side hash matches what was
|
||||
just written; if it diverges (a concurrent Foundry edit landed mid-flight),
|
||||
surface a conflict row instead of baselining. The pre-push Foundry `/get`
|
||||
(FR-5.6) is the prior-state backup used by the revert path.
|
||||
|
||||
### F2 — Foundry→Obsidian auto-sync
|
||||
|
||||
The relay has no push channel and `/search` is minified
|
||||
(`{uuid,id,name,img,documentType}` — **no `folder`, no content, no content
|
||||
hash**). F2 therefore runs **two layers**:
|
||||
|
||||
- **FR-2.1 (shallow poll, default ON)** — poll `relay /search`
|
||||
(`documentType:JournalEntry`, minified) on a configurable cadence; build a
|
||||
`{uuid → name/img}` snapshot. Diff against the last snapshot to detect
|
||||
**renames** (name change on a known uuid), **new** entries, and **missing**
|
||||
entries. Folder moves and content changes are **not** detectable here (no
|
||||
`folder`, no content in minified `/search`).
|
||||
- **FR-2.2 (deep poll, default ON at a minutes cadence)** — for each linked
|
||||
note, `relay /get` the live entry and compute its Foundry-side hash
|
||||
(FR-1.4's input); compare to `foundry.ccHash` to detect **content changes**
|
||||
and **folder moves**. Concurrency-capped (reuse `mapPool`,
|
||||
`src/server.ts:317`); cadence in **minutes**, not seconds. This supersedes
|
||||
ADR-005's "F→O stays manual" conclusion **for rename/new/missing (shallow)
|
||||
and content/move (deep)** — ADR-005's cost rejection is respected by the
|
||||
minutes cadence + concurrency cap.
|
||||
- **FR-2.3** For each Foundry-changed **linked** note where the Obsidian side
|
||||
is unchanged: `/get` the live entry, convert to refined markdown, write into
|
||||
the vault, re-baseline both sides. (Apply mode only — FR-1.9.)
|
||||
- **FR-2.4** Never clobber an Obsidian-side change: vault-newer or
|
||||
both-diverged notes route to F3 conflict handling, not auto-pull.
|
||||
- **FR-2.5** New (cc-only) Foundry entries surface in a **separate "live new
|
||||
entries" list** in the dashboard (not conflated with the LevelDB `ccOnly`
|
||||
pool, which is built from the static journal snapshot). Each row has a
|
||||
one-click "Import as new refined note" action with a plain-language
|
||||
explanation of what import does. No auto-import.
|
||||
- **FR-2.6** A manual "catch up now" trigger forces an immediate deep sweep
|
||||
(FR-2.2) alongside the background polls. Poll cadences are configurable
|
||||
(OQ-3) with jitter to be courteous on shared relays.
|
||||
|
||||
### F3 — Divergence detection & conflict routing
|
||||
|
||||
- **FR-3.1** Every sync tick (O→F or F→O) computes both-side hashes and routes
|
||||
per the 2×2: parity / O-changed / F-changed / both-changed. A **per-uuid
|
||||
lock shared by the watcher path and the poll path** (not per-relPath) ensures
|
||||
only one direction acts on a uuid at a time; the other queues/skips. FR-1.4's
|
||||
`/get` is evaluated **after** the debounce drains, not on the raw save event.
|
||||
- **FR-3.2** both-changed → **do not auto-overwrite**; create a conflict row
|
||||
showing a **side-by-side diff with a one-line plain-language summary**
|
||||
("Vault adds 3 paragraphs about X; Foundry renamed to Y and changed folder to
|
||||
Z"), not a raw unified diff.
|
||||
- **FR-3.3** The conflict row offers three explicit actions: "Push vault →
|
||||
Foundry", "Pull Foundry → vault", and **"Accept both as-is (keep
|
||||
divergence)"** (renamed from "mark resolved (no change)" — see FR-3.7).
|
||||
- **FR-3.4** Conflict state persists until the DM resolves it, **across ticks
|
||||
and across server restarts** (persisted in `sync-state.json`, FR-4.7); a
|
||||
re-save on either side does not auto-clear a known conflict.
|
||||
- **FR-3.5** Foundry-side renames and folder moves (caught via name + folder in
|
||||
the Foundry-side hash) surface as changes/conflicts, not silently absorbed.
|
||||
- **FR-3.6** Conflict diff format = side-by-side with the plain-language
|
||||
summary line (FR-3.2).
|
||||
- **FR-3.7** "Accept both as-is (keep divergence)" re-baselines **both** hashes
|
||||
to the current values **without transferring content in either direction**;
|
||||
the two sides keep their diverged content and are treated as in-sync from
|
||||
then on. A confirmation dialog states in plain text: "The vault and Foundry
|
||||
will keep their current versions. They will be treated as in-sync from now
|
||||
on. Neither side's changes will be copied to the other."
|
||||
- **FR-3.8** Each conflict action states, before commit, what it will do to
|
||||
each side and to the baselines (one-line preview). No irreversible action
|
||||
without a confirm.
|
||||
- **FR-3.9** A resolved conflict produces a visible activity-panel entry
|
||||
stating which side won and that the other side's edits were **not**
|
||||
transferred.
|
||||
- **FR-3.10** Conflict-row ordering is neutral: vault on the left, Foundry on
|
||||
the right, no pre-highlighted action — so the undecided tie-breaker does not
|
||||
bias the DM.
|
||||
|
||||
### F4 — Sync status & parity
|
||||
|
||||
- **FR-4.1** Dashboard shows a persistent sync-status header: ON/OFF, mode
|
||||
(apply only for auto-sync — FR-1.9), watched dir.
|
||||
- **FR-4.2** Dashboard shows a parity indicator: counts of in-parity /
|
||||
O-pending / F-pending / conflict / unsynced-linked notes, plus a last-sync
|
||||
timestamp.
|
||||
- **FR-4.3** The sync tool maintains a status note at a **reserved dot-path**
|
||||
(`${VAULT}/.sync-status.md`, covered by FR-1.1's dotfile skip) **and** carries
|
||||
a `foundry.sync_status: true` content sentinel. Both the O→F watcher and the
|
||||
F→O poll check **both** the path rule and the sentinel and skip on either.
|
||||
If a status note loses its sentinel (user edit), it is surfaced as user error
|
||||
and **not** synced. Status-note writes must never produce a sync op (NFR-5).
|
||||
- **FR-4.4** When sync is OFF, the dashboard shows a loud "SYNC PAUSED" state,
|
||||
not a silent absence.
|
||||
- **FR-4.5** Dashboard parity and the vault status note reflect one underlying
|
||||
state — the persisted `sync-state.json` (FR-4.7) — so they never disagree.
|
||||
- **FR-4.6** The status note's exclusion is airtight by **both** path and
|
||||
sentinel (FR-4.3); a rename/move of the note does not start a feedback loop
|
||||
because the sentinel is checked on content, not path alone.
|
||||
- **FR-4.7** Status state (parity counts, conflict state, last-sync time) lives
|
||||
in a persisted `sync-state.json` that survives server restart; on restart the
|
||||
dashboard reads it rather than showing stale/empty until the next tick.
|
||||
|
||||
### F5 — Operational hardening
|
||||
|
||||
- **FR-5.1** Recursive-watch fallback (FR-1.1) verified on the host kernel,
|
||||
including re-scan on subdir create/rename so new folders get watched.
|
||||
- **FR-5.2** Debounce window and max concurrency configurable; defaults
|
||||
**debounce 700ms, max concurrency 3** (current values), validated against the
|
||||
NFR-3 ~50-note prep burst. Tuned so a burst doesn't thrash or drop events.
|
||||
- **FR-5.3** Retry split into **transient** (timeout 408/504, 5xx,
|
||||
session-temporarily-unavailable) — retried with bounded backoff — vs
|
||||
**persistent** (404 invalid clientId, 401 bad API key, 404 no connected
|
||||
Foundry clients) — **no retry**, surfaced immediately with remediation. (A
|
||||
wrong clientId 404s forever; retrying only delays the error.)
|
||||
- **FR-5.4** Every auto-sync op (push/skip/error/conflict) logged to the
|
||||
activity panel with time, note, status, message; panel capped at the **last
|
||||
200 events**, scrollable for older.
|
||||
- **FR-5.5** Inflight dedup + the shared per-uuid lock (FR-3.1) verified under
|
||||
burst — no dropped events, no duplicate pushes, no cross-direction
|
||||
oscillation.
|
||||
- **FR-5.6** **Before any auto or manual push to Foundry**, the prior
|
||||
Foundry-side entry is `/get`-fetched and cached locally to
|
||||
`foundry-backups/<uuid>/<iso>.json`; the last N per uuid are retained
|
||||
(configurable). (This `/get` is the same one FR-1.4 reuses — no extra
|
||||
round-trip.)
|
||||
- **FR-5.7** A **"Revert last push"** dashboard action restores the most
|
||||
recent cached Foundry state for a note (writes it back via `/update`).
|
||||
- **FR-5.8** All auto-sync ops additionally append to a persistent, rotated
|
||||
log file on disk (`logs/sync-<date>.log`) — survives restart, for support.
|
||||
- **FR-5.9** A **"Copy diagnostics"** dashboard action bundles recent log tail,
|
||||
current config (secrets redacted), parity counts, and relay/clientId status
|
||||
into a single redacted blob for support.
|
||||
|
||||
### F6 — Onboarding & config (given operator prerequisites)
|
||||
|
||||
- **FR-6.1** If `RELAY_CLIENT_ID` is unset/empty, the dashboard surfaces a
|
||||
clear "no clientId configured" state with guidance — not a silent 404 at push
|
||||
time.
|
||||
- **FR-6.2** If no Foundry client is connected to the relay (`activeClients =
|
||||
0`), the dashboard surfaces "Foundry not connected" and disables
|
||||
enable-auto-sync until resolved; re-checked on a cadence.
|
||||
- **FR-6.3** The dashboard can list connected relay clients (relay `/search`
|
||||
with no clientId returns the client list on >1, or "No connected clients" on
|
||||
0) so the DM can pick/copy a valid `clientId` from the UI. When **exactly
|
||||
one** client is connected, the relay auto-resolves — the dashboard treats
|
||||
that as "clientId auto-resolved, no pick needed" rather than showing an empty
|
||||
list.
|
||||
|
||||
### F7 — Security & access control
|
||||
|
||||
- **FR-7.1** Dashboard authenticates by default (token or password set via env
|
||||
or a first-run prompt); unauthenticated requests get 401, not the UI.
|
||||
- **FR-7.2** Default bind changes to `127.0.0.1`; `0.0.0.0` requires explicit
|
||||
opt-in **and** an auth token set — the server refuses to start on `0.0.0.0`
|
||||
without auth.
|
||||
- **FR-7.3** Secrets (`RELAY_API_KEY`, `RELAY_PASSWORD`) are never rendered to
|
||||
the browser; the dashboard shows only masked presence/absence.
|
||||
- **FR-7.4** POST mutation endpoints require a CSRF token or same-origin
|
||||
check.
|
||||
|
||||
### §5a — Error contracts
|
||||
|
||||
Every error row includes a one-line "what to do" string, not just a status.
|
||||
|
||||
| Failure mode | Detection | Retry? | User message | Remediation |
|
||||
|---|---|---|---|---|
|
||||
| Relay unreachable | network error / connect refused | transient (backoff) | "Can't reach the relay at <url>." | Check relay container is up; check network. |
|
||||
| Relay 401 | HTTP 401 | **no** | "Relay rejected the API key." | Re-create `RELAY_API_KEY` (operator gate 2). |
|
||||
| clientId invalid/empty | 404 "Invalid client ID" | **no** | "The relay clientId is wrong or empty." | Pick a valid clientId from the client list (FR-6.3). |
|
||||
| No connected Foundry client | 404 "No connected Foundry clients found" | **no** | "Foundry isn't connected to the relay." | Start the headless session (operator gate 3); re-check runs automatically. |
|
||||
| Session idle-reaped | was connected, now 404 | transient (re-check) | "The Foundry session dropped." | Restart the headless session; dashboard re-checks. |
|
||||
| Hash mismatch (both diverged) | F-hash ≠ ccHash AND O-hash ≠ contentHash | n/a (route to F3) | "Both sides changed since last sync — conflict." | Open the conflict row (FR-3.2). |
|
||||
| `/get` 404 on a known uuid | 404 for a previously-linked uuid | **no** | "Foundry entry <name> was deleted on the Foundry side." | Re-link or remove the orphaned note. |
|
||||
| Persistent 5xx after backoff | 5xx after retries exhausted | **no** (already retried) | "The relay keeps erroring on this note." | See diagnostics (FR-5.9); contact support. |
|
||||
|
||||
## 5. Non-Functional Requirements
|
||||
|
||||
- **NFR-1 — No-clobber safety.** No auto-sync operation may overwrite a side
|
||||
that has changed since the last sync. Both-diverged → conflict, never
|
||||
auto-overwrite. TOCTOU window closed by FR-1.10 (post-push re-verify). The
|
||||
current O→F code violates this (pushes on Obsidian-body-diff only) — must be
|
||||
fixed before/within delivery A.
|
||||
- **NFR-2 — Fail-safe.** If the relay cannot read the Foundry side, the
|
||||
operation is skipped and surfaced (error-contracts table) — never a blind
|
||||
push on Obsidian-side-only evidence.
|
||||
- **NFR-3 — Performance.** Debounce (700ms) + bounded concurrency (3) handle a
|
||||
~50-note prep burst without dropped events or relay thrash. **Operating
|
||||
envelope:** validated against a vault of ≥N notes and ≥M JournalEntries
|
||||
(N/M chosen above the author's own size — OQ-3). Default cadences: shallow
|
||||
poll seconds-tens-of-seconds, deep poll minutes. Relay-load ceiling = a
|
||||
documented max concurrent `/update` + `/get` budget; deep-poll concurrency
|
||||
capped via `mapPool`.
|
||||
- **NFR-4 — Reliability.** Transient relay errors retried with backoff;
|
||||
persistent errors surfaced within one tick (no retry on persistent — FR-5.3).
|
||||
- **NFR-5 — Observability.** Every operation is visible in the dashboard
|
||||
activity panel and the vault status note; no silent skips or silent
|
||||
overwrites. Status-note writes never produce a sync op. Operation history is
|
||||
**persistent across restarts** (FR-5.8).
|
||||
- **NFR-6 — Onboardability (honest).** Given the operator prerequisites (§2),
|
||||
a non-author DM can reach a connected live sync using the dashboard with no
|
||||
shell command beyond those prerequisites — the dashboard detects and guides
|
||||
every unmet gate.
|
||||
- **NFR-7 — Configurability.** Poll cadences, debounce, concurrency, status-note
|
||||
path, backup retention, and auth token are env/config-driven with safe
|
||||
defaults.
|
||||
- **NFR-8 — Backward compatibility.** Existing manual buttons, seed/sync/
|
||||
rePull/import rows, dev/apply modes, and the CLI-only full LevelDB index all
|
||||
keep working unchanged. Auto-sync is newly gated to apply mode (FR-1.9).
|
||||
- **NFR-9 — Security.** No unauthenticated mutation path; no secret egress to
|
||||
the client; default bind localhost; TLS recommended when bound beyond
|
||||
localhost.
|
||||
- **NFR-10 — Data integrity.** Foundry-side overwrites are always preceded by a
|
||||
local backup of the pre-push Foundry state (FR-5.6); the dashboard exposes a
|
||||
restore path (FR-5.7). This complements, not replaces, Foundry world backups.
|
||||
- **NFR-11 — Upgrades.** The `foundry:` block carries a `schema_version`; a
|
||||
version bump that changes hashing or identity fields ships with an idempotent
|
||||
migration (re-hash, re-baseline) run in a single pass at startup before
|
||||
auto-sync is allowed to engage, with a dashboard migration banner. (Old notes
|
||||
lacking `ccHash` would otherwise route to conflict on first sync after
|
||||
upgrade.)
|
||||
|
||||
## 6. Open Questions
|
||||
|
||||
- **OQ-1** Beyond the three explicit conflict actions, do we also offer a
|
||||
one-click "newest mtime wins" convenience? Default posture = manual only.
|
||||
- **OQ-2** Where the "not syncing" indicator lives during run-the-match —
|
||||
**deferred**. Reopens when a custom Foundry module is explored.
|
||||
- **OQ-3** Concrete cadence numbers (shallow poll, deep poll) and the NFR-3
|
||||
operating-envelope N/M — pick during delivery B with relay-load testing.
|
||||
- **OQ-4** **Resolved** — status note lives at `${VAULT}/.sync-status.md` with
|
||||
a `foundry.sync_status: true` sentinel; exclusion by both path and sentinel
|
||||
(FR-4.3/4.6).
|
||||
- **OQ-5** New Foundry entries: candidates only with a one-click import + plain
|
||||
explanation (FR-2.5) — confirmed; no auto-import.
|
||||
- **OQ-6** **Resolved** — conflict diff is side-by-side + plain-language
|
||||
summary; "mark resolved" renamed to "Accept both as-is (keep divergence)"
|
||||
with a confirmation (FR-3.6/3.7/3.8).
|
||||
- **OQ-7** `foundry.ccHash` field naming (`ccHash` vs `cc_content_hash` vs
|
||||
`foundryHash`) — cosmetic, decide during delivery for grep-ability/
|
||||
consistency with `cc_uuid`/`cc_type`.
|
||||
|
||||
## 7. Out of Scope / Future
|
||||
|
||||
- **In-UI first-run onboarding wizard** performing the operator prerequisites
|
||||
(relay bring-up, API-key acquisition, headless-session launch, rest-api
|
||||
wiring) — future PRD; this PRD honestly scopes NFR-6 to operator-wired prereqs.
|
||||
- **Custom Foundry module** — indicators inside Foundry UI + relay replacement;
|
||||
would reopen OQ-2.
|
||||
- **Syncing during run-the-match** — sync stays off by design in the live
|
||||
session.
|
||||
- **Automatic semantic / 3-way content merge** — both-diverged is manual
|
||||
resolution via buttons; no auto content-merge engine.
|
||||
- **Auto-sync of unlinked or unseeded notes** — seed/link first stays manual.
|
||||
- **Full LevelDB / docker-stop index in the dashboard** — stays CLI-only.
|
||||
- **Doc task:** update README's "Foundry is the source of truth" wording to
|
||||
match the demoted source-of-truth model (§1) before launch — tracked here to
|
||||
avoid spec/onboarding drift.
|
||||
|
||||
## 8. Success Metrics
|
||||
|
||||
- **SM-1** Zero auto-overwrite events on a side that changed since last sync,
|
||||
across a 50-note prep burst. **Counter-metric:** conflict-rows-surfaced > 0
|
||||
whenever both sides diverged (the guard is working).
|
||||
- **SM-2** One verified end-to-end live O→F push (delivery A acceptance gate).
|
||||
- **SM-3** F→O detects a Foundry rename within ≤2 shallow poll ticks; a content
|
||||
change within ≤1 deep poll tick.
|
||||
- **SM-4** Status-note writes never produce a sync op (FR-4.3/NFR-5 test).
|
||||
- **SM-5** A non-author DM, given the operator prerequisites, clones the repo
|
||||
and reaches a connected live sync via the dashboard with no shell command
|
||||
beyond those prerequisites.
|
||||
- **SM-6** Zero unauthenticated mutation paths (security gate; NFR-9).
|
||||
|
||||
## 9. Glossary
|
||||
|
||||
- **cc_uuid** — Foundry Campaign Codex UUID stored in a note's `foundry:` block;
|
||||
its presence means the note is **linked**.
|
||||
- **contentHash** — SHA-256 of the canonicalized Obsidian note **body** (the
|
||||
O-side baseline; `src/normalize.ts`).
|
||||
- **ccHash** — *(new)* SHA-256 of the canonicalized Foundry-side representation
|
||||
(HTML→markdown + name + folder; the F-side baseline). See FR-1.4.
|
||||
- **linked note** — a refined note with a `foundry.cc_uuid`.
|
||||
- **seeded** — a linked note that also carries a `foundry.contentHash`
|
||||
baseline (auto-sync prerequisites: linked + seeded).
|
||||
- **refined vault dir** — the Obsidian vault subdirectory holding curated notes
|
||||
the tool syncs.
|
||||
- **refined markdown** — the curated Obsidian markdown format the tool
|
||||
converts to/from Foundry's campaign-codex HTML.
|
||||
- **import candidate** — a Foundry JournalEntry not yet present in the vault;
|
||||
surfaced for one-click import (FR-2.5).
|
||||
- **cc-only entry** — a Foundry entry with no matching refined note (≡ import
|
||||
candidate from the Foundry side).
|
||||
- **dev / apply mode** — dev = preview into `--out` mirror; apply = writes the
|
||||
real vault + live Foundry. Auto-sync is apply-only (FR-1.9).
|
||||
- **activity panel** — the dashboard feed of auto-sync events (FR-5.4).
|
||||
- **parity** — vault and Foundry agree for a note (both-side hashes match
|
||||
baselines).
|
||||
- **tick** — one evaluation of a sync op (O→F on save, or F→O on poll).
|
||||
- **shallow poll / deep poll** — F→O's two layers (FR-2.1 / FR-2.2).
|
||||
- **conflict row** — the dashboard UI for a both-diverged note (F3).
|
||||
- **operator prerequisites** — the five infrastructure gates the operator
|
||||
wires once before the dashboard can reach live sync (§2).
|
||||
@@ -0,0 +1,297 @@
|
||||
---
|
||||
title: "Engineering-feasibility review — Live Relay Sync PRD"
|
||||
reviewer: Claude (engineering-feasibility pass)
|
||||
prd: docs/prds/prd-foundry-obsidian-sync-2026-06-22/prd.md
|
||||
date: 2026-06-22
|
||||
scope: "Stress-test PRD technical claims against the actual codebase + relay constraints."
|
||||
---
|
||||
|
||||
# Engineering-feasibility review
|
||||
|
||||
Verdict at the end. Findings tiered **BLOCKER / HIGH / MEDIUM / LOW** with file:line
|
||||
citations and a recommended fix for each.
|
||||
|
||||
## What the code actually does today (baseline facts)
|
||||
|
||||
- `contentHash(text)` hashes **only the canonicalized body** — wikilink-canonicalize +
|
||||
whitespace-canonicalize, then SHA-256. It does **not** include the note name, the
|
||||
folder path, or any frontmatter. (`src/normalize.ts:24`, `canonicalize` at
|
||||
`src/normalize.ts:19`.)
|
||||
- The `foundry:` block has no `ccHash` field. `readFoundryBlock` returns whatever keys
|
||||
are in the nested map (`src/frontmatter.ts:9`); `baselineFoundryBlock`
|
||||
(`src/server.ts:289`) rewrites **only** `contentHash:` and `syncedAt:` lines. So the
|
||||
Foundry-side baseline the PRD depends on does not exist yet — it is genuinely new.
|
||||
- `AutoSyncController.process()` (`src/server.ts:582`) reads the note, compares
|
||||
`contentHash(body)` to `fb.contentHash`, and on a difference calls `pushNote` **with
|
||||
no relay `/get` of its own**. `pushNote` does call `relay.getEntry(id)`
|
||||
(`src/push.ts:142`) but **only to build the diff** (preserve ownership/pages/
|
||||
existing image) — it never compares a Foundry-side hash to a baseline. So if the
|
||||
Foundry side changed since the last sync, `pushNote` still PUTs `{ name,
|
||||
"flags.campaign-codex": cc }` and **overwrites the Foundry-side edit silently**.
|
||||
Clobber risk is **real, not theoretical**.
|
||||
- Relay `/get` returns the full doc: `{ data: { name, type, _id, uuid, folder, pages,
|
||||
ownership, flags, ... } }` (`docs/relay-api.md:31-33`). `folder` is present on
|
||||
`/get`.
|
||||
- Relay `/search` minified returns **only** `{ uuid, id, name, img, documentType }`
|
||||
(`docs/relay-api.md:55-59`, mirrored in `src/relay/client.ts:19-25`,
|
||||
`searchJournalEntries` at `src/relay/client.ts:97`). **No `folder`. No content. No
|
||||
content hash. No mtime.** This is the load-bearing constraint for F2.
|
||||
- ADR 005 (`wiki/decisions/005-autosync-o-to-f-only.md`) already analyzed F→O polling
|
||||
and rejected it on cost grounds: ~800 `/get` calls/min for a ~200-entry world at a
|
||||
15s cadence. The PRD does not cite or reconcile with this ADR.
|
||||
|
||||
---
|
||||
|
||||
## BLOCKER
|
||||
|
||||
### B1 — FR-2.1 / FR-2.2: `/search` minified has NO `folder`. Folder-move detection from the poll snapshot is impossible.
|
||||
|
||||
**Claim (PRD FR-2.1):** "poll `relay /search` … build a current `{uuid →
|
||||
name/img/folder}` snapshot."
|
||||
|
||||
**Claim (PRD FR-2.2):** "moved (folder change)" is detected by diffing the current
|
||||
snapshot against the last snapshot.
|
||||
|
||||
**Reality:** `relay /search?minified=true` returns `{ uuid, id, name, img,
|
||||
documentType }` only (`docs/relay-api.md:55-59`, `src/relay/client.ts:19-25`,
|
||||
`src/relay/client.ts:97-105`). There is **no `folder` field** in the minified result.
|
||||
The PRD's own §2 admits "/search is minified with no content hash" but then FR-2.1
|
||||
quietly assumes `folder` is present — it is not. You cannot build a
|
||||
`{uuid → folder}` snapshot from `/search`, and you cannot detect folder moves by
|
||||
snapshot-diffing alone.
|
||||
|
||||
**Impact:** FR-2.2's "moved (folder change)" detection and FR-3.5's "folder moves
|
||||
surface as changes/conflicts" are **not implementable as specified**. Folder moves
|
||||
would be invisible to the F→O poll entirely.
|
||||
|
||||
**Fix:** Pick one and write it into the PRD:
|
||||
1. **Per-note `/get` on every poll** to read `folder` (and content). This is the only
|
||||
way to detect moves from the relay at all, and it collapses F2 into "every poll =
|
||||
`/get` per linked note" (see B2). Accept the cost explicitly and override ADR 005.
|
||||
2. **Drop folder-move detection from F2.** Foundry-side renames are detectable
|
||||
(name is in `/search`); folder moves are not. State this as a known gap and route
|
||||
moves to the next manual Sync / Re-pull. This is the cheaper path and is honest
|
||||
about the relay's surface.
|
||||
3. Propose a relay change (non-goal per PRD §3) — not available for this PRD.
|
||||
|
||||
Recommended: **option 2** for this PRD, with option 1 as an explicit opt-in
|
||||
"deep poll" mode. Either way FR-2.1 and FR-2.2 must be rewritten to drop `folder`
|
||||
from the `/search`-derived snapshot.
|
||||
|
||||
---
|
||||
|
||||
### B2 — FR-2.2: content-change detection requires a `/get` per linked note per poll. The PRD does not account for the load, and this directly contradicts ADR 005.
|
||||
|
||||
**Claim (PRD FR-2.2):** "content-changed (detected via `/get` hash compare)."
|
||||
|
||||
**Reality:** `/search` minified has no content and no content hash
|
||||
(`docs/relay-api.md:55-59`). The only way to detect a Foundry-side content edit is to
|
||||
`/get` the full entry and hash it. So "detect content changes" = " `/get` every
|
||||
linked note every poll cycle." For the ~200-entry world in ADR 005 at the PRD's
|
||||
"seconds" default cadence (FR-2.6, OQ-3), this is **thousands of `/get` calls/min**,
|
||||
all funneled through one headless Foundry WebSocket. ADR 005 rejected exactly this
|
||||
design at 15s/800-calls-min as "heavy and fragile."
|
||||
|
||||
**Impact:** NFR-3 ("does not overload the relay or the host") is **unsatisfiable** at
|
||||
the cadence the PRD hints at, with the design the PRD specifies. The PRD cannot
|
||||
both (a) detect Foundry content changes automatically and (b) avoid per-note `/get`
|
||||
load. One has to give.
|
||||
|
||||
**Fix:** Make the F→O design explicit and costed in the PRD:
|
||||
- **Default: shallow poll.** `/search` only → detect **renames + new + missing**
|
||||
(all detectable from minified `/search`). Content changes and folder moves are
|
||||
**not** auto-detected; they surface on the next manual Sync / Re-pull or on a
|
||||
user-triggered "catch up now" deep sweep (FR-2.6 already has the trigger — promote
|
||||
it to the only content-change path).
|
||||
- **Opt-in deep poll:** a `/get`-per-linked-note sweep, cadence in **minutes**, not
|
||||
seconds, with a concurrency cap (the existing `mapPool` at `src/server.ts:317` is
|
||||
the right primitive). State the call-rate budget in NFR-3.
|
||||
- Add a line to §6 acknowledging this supersedes ADR 005's "F→O stays manual"
|
||||
conclusion **only for rename/new/missing**, not for content.
|
||||
|
||||
Without this, F2 is not shippable as written.
|
||||
|
||||
---
|
||||
|
||||
## HIGH
|
||||
|
||||
### H1 — FR-1.4 / NFR-1: the "Foundry-side hash over content + name + folder" is a brand-new function, not derivable from `contentHash`. The PRD under-specifies it.
|
||||
|
||||
**Claim (PRD FR-1.4):** compute a Foundry-side hash over "content + name +
|
||||
folder_path" from the `/get` response and compare to a new `foundry.ccHash` baseline.
|
||||
|
||||
**Reality:**
|
||||
- `contentHash` (`src/normalize.ts:24`) hashes **canonicalized Obsidian markdown
|
||||
body only**. It cannot be reused for the Foundry side — the Foundry `/get` response
|
||||
stores content inside `flags["campaign-codex"].data` (HTML per
|
||||
`obsidianToFoundryJsonLive`), not as refined markdown. Hashing the raw HTML is
|
||||
fragile (Foundry may round-trip HTML attributes/whitespace); hashing requires
|
||||
either (a) a stable HTML canonicalizer or (b) converting `cc.data` HTML back to
|
||||
refined markdown and reusing `contentHash` — which is a non-trivial round-trip
|
||||
through `entryToObsidian`-style logic that the PRD does not mention.
|
||||
- `name` and `folder` are present on `/get` (`docs/relay-api.md:31-33`) and are
|
||||
stable scalars — hashing those is trivial.
|
||||
- The hard part is **content**. The PRD says "content + name + folder_path" as if
|
||||
it's obvious; it is not. The PRD needs to specify the exact Foundry-side hash
|
||||
input: is it `cc.data` raw, canonicalized HTML, or re-converted markdown? Each
|
||||
has different drift properties.
|
||||
|
||||
**Impact:** If the Foundry-side hash is unstable (e.g. Foundry re-serializes HTML
|
||||
between pushes), the divergence guard false-positives every tick → every save becomes
|
||||
a conflict → the tool is unusable. This is the single most likely delivery failure.
|
||||
|
||||
**Fix:** Add a subsection to FR-1.4 specifying the hash:
|
||||
- Input: `canonicalize(htmlToMarkdown(flags["campaign-codex"].data)) + "\n" + name +
|
||||
"\n" + folder` — i.e. reuse the existing `contentHash` pipeline by converting the
|
||||
Foundry HTML body back to refined markdown first (the inverse of
|
||||
`obsidianToFoundryJsonLive`). This makes the Foundry-side hash and the Obsidian
|
||||
body hash **directly comparable** (same canonical form), which also makes the 2×2
|
||||
routing in F3 well-defined.
|
||||
- Specify the HTML→markdown conversion (linkedom is already a dep per the wiki;
|
||||
confirm it round-trips stably). Add a unit test for hash stability across a
|
||||
push→/get round-trip before shipping FR-1.4.
|
||||
|
||||
Also: `baselineFoundryBlock` (`src/server.ts:289`) and `baselineNote`
|
||||
(`src/server.ts:307`) currently touch only `contentHash`/`syncedAt`. FR-1.7 requires
|
||||
baseline of **both** `contentHash` and `ccHash` — both functions need a `ccHash:`
|
||||
line added to the rewrite, and `frontmatter.ts`'s `readFoundryBlock` consumers need
|
||||
to read it. Call this out as a concrete code change in the PRD.
|
||||
|
||||
---
|
||||
|
||||
### H2 — FR-1.6 fail-safe: the PRD correctly identifies the gap, but the fix has a subtle hole — the `/get`-then-`/update` race window.
|
||||
|
||||
**Confirmed:** current `process()` (`src/server.ts:582-617`) does **not** `/get` to
|
||||
compare a Foundry-side hash before pushing. It pushes on Obsidian-body-hash-diff
|
||||
only. `pushNote`'s internal `/get` (`src/push.ts:142`) builds the diff but does not
|
||||
gate the push. So NFR-1 is **violated today** — the PRD is right to call this out.
|
||||
|
||||
**Subtle hole in the proposed fix:** even with FR-1.4's "`/get` then compare ccHash
|
||||
then push" sequence, there is a TOCTOU window between the `/get` (read F-side
|
||||
baseline) and the `/update` (write). If a Foundry edit lands in that window, the
|
||||
push still clobbers it. The window is small (one relay round-trip) but nonzero.
|
||||
|
||||
**Impact:** Low probability, but it's the same class of bug the PRD is trying to
|
||||
eliminate. Worth a defensive measure.
|
||||
|
||||
**Fix:** After `pushNote`'s `relay.updateEntry` succeeds, re-`/get` and verify the
|
||||
entry's content hash matches what we just wrote; if it diverges (concurrent edit
|
||||
landed mid-flight), surface a conflict row instead of baselining. Cheaper
|
||||
alternative: document the window as accepted residual risk in NFR-1. Either is
|
||||
fine; the PRD should pick one explicitly.
|
||||
|
||||
---
|
||||
|
||||
### H3 — F3 routing: the O→F `inflight` set and the F→O poll do not share a lock. Concurrent O-save + F-poll on the same note can double-write.
|
||||
|
||||
**Reality:** `AutoSyncController.inflight` (`src/server.ts:463`, used at
|
||||
`src/server.ts:583`) is keyed by **relPath** and only guards the O→F watcher path.
|
||||
The proposed F→O poll is a separate loop with no shared lock. If an O-save and an
|
||||
F-poll fire on the same uuid in the same tick:
|
||||
- O→F sees O-changed, F-unchanged → pushes O→F.
|
||||
- F→O sees F-changed (from the just-pushed O content? or a real F edit?), O-unchanged
|
||||
(pre-save snapshot) → pulls F→O.
|
||||
Result depends on timing and can oscillate or clobber.
|
||||
|
||||
**Impact:** NFR-1 ("never clobber") can be violated under concurrent cross-direction
|
||||
activity on the same note. Prep bursts are exactly this pattern.
|
||||
|
||||
**Fix:** Add a **per-uuid lock** shared between the watcher path and the poll path
|
||||
(not per-relPath). While a sync op is in flight for a uuid, the other direction
|
||||
queues/skips. Add to FR-3.1 / FR-5.5. Also: the debounce window
|
||||
(`src/server.ts:567`, 700ms) is O-side only; an F edit arriving during the debounce
|
||||
is invisible to the watcher. The `/get` at push time (FR-1.4) is what catches it —
|
||||
confirm FR-1.4 is evaluated **after** debounce drains, not on the raw save event.
|
||||
|
||||
---
|
||||
|
||||
## MEDIUM
|
||||
|
||||
### M1 — FR-1.4: one extra `/get` per changed note is acceptable, but the PRD should state the cost and cap it.
|
||||
|
||||
One additional `/get` per push (on top of the `/get` `pushNote` already does at
|
||||
`src/push.ts:142`) doubles relay reads per O→F op. At the existing concurrency=3
|
||||
(`src/server.ts:466`) and debounce=700ms (`src/server.ts:467`) this is fine for a
|
||||
prep burst. But **reuse the `/get` `pushNote` already makes** — don't do two `/get`s.
|
||||
The cleanest implementation is to have `pushNote` (or a new wrapper) compute the F
|
||||
hash from the `liveEntry` it already fetched, compare to `ccHash` baseline, and
|
||||
gate the `/update` on that. The PRD reads as if FR-1.4 adds a **separate** `/get`
|
||||
before the push — that would be wasteful. Clarify: "the `/get` that `pushNote`
|
||||
already performs is reused for the F-side hash compare; no extra round-trip."
|
||||
|
||||
### M2 — FR-2.5 new-entry detection: feasible, but `img` is the only extra signal beyond name+uuid.
|
||||
|
||||
`/search` minified gives `{uuid, id, name, img, documentType}`. For new Foundry
|
||||
JournalEntries, `name + uuid` is enough to surface as import candidates (the
|
||||
existing import row at `src/server.ts:182-187` and `importRow` already works off
|
||||
`row.entry`). **Feasible.** One caveat: the dashboard's `ccOnly` pool
|
||||
(`src/server.ts:154`) is built from the **LevelDB journal snapshot** via `indexAll`,
|
||||
not from live `/search`. So "new entries surface as import candidates" requires
|
||||
either (a) merging live `/search` results into the index's `ccOnly` pool, or (b) a
|
||||
separate "live new entries" list in the dashboard. The PRD doesn't specify which.
|
||||
Recommend (b) to avoid conflating the static LevelDB index with live relay state.
|
||||
Add to FR-2.5.
|
||||
|
||||
### M3 — FR-1.7 dev-mode baseline landing: `baselineNote` writes via `targetPath` (mirror in dev, real vault in apply) — correct, but the F→O pull writes the **vault** file and must use the same path resolution.
|
||||
|
||||
`resolveRefined` (`src/server.ts:85`) prefers the dev mirror if it's newer, else the
|
||||
real vault. The F→O pull (FR-2.3) writes into the vault; in dev mode it must write
|
||||
to the mirror (`targetPath(state, "refined", rel)`), not the real vault, or dev
|
||||
mode stops being a safe preview. The PRD says "Dev mode baselines land in the
|
||||
`--out` mirror" (FR-1.7) — good — but FR-2.3 doesn't repeat the constraint for the
|
||||
**pulled content**. Add it.
|
||||
|
||||
### M4 — FR-4.3 `Sync Status.md` watcher exclusion: the watcher's exclusion list today is `.obsidian` + dotfiles only (`src/server.ts:561`, `src/server.ts:533`).
|
||||
|
||||
`onChange` skips `.obsidian` and dotfiles (`src/server.ts:561`). A `Sync Status.md`
|
||||
at the vault root is **not** a dotfile and **not** in `.obsidian`, so the watcher
|
||||
will fire on every status-note write → auto-sync will try to push it (it has no
|
||||
`foundry.cc_uuid` → skipped at `src/server.ts:591`, but still generates noise and a
|
||||
`/get`-less skip). The PRD (OQ-4) flags the path as TBD but doesn't note that the
|
||||
exclusion must be **added to `onChange`** explicitly. Add a concrete code-change
|
||||
note to FR-4.3: extend the skip predicate at `src/server.ts:561` to include the
|
||||
status-note path.
|
||||
|
||||
---
|
||||
|
||||
## LOW
|
||||
|
||||
### L1 — FR-6.3 client-list discovery: the relay returns the client list on `>1` connected clients with no clientId, and "No connected clients" on 0 (`docs/relay-api.md:13-14`). With exactly 1, the relay auto-resolves and returns the doc — **no client list is surfaced**. So FR-6.3's "list connected relay clients" works only when there are 0 or >1 clients. The single-client case (the most common during prep) returns no list. The PRD should note this: when exactly one client is connected, the dashboard should treat that as "clientId auto-resolved, no pick needed" rather than showing an empty list.
|
||||
|
||||
### L2 — FR-5.3 retry on 404: a 404 from the relay is **not always** transient. `404 {"error":"No connected Foundry clients found"}` (`docs/relay-api.md:13`) is a **persistent** condition (Foundry down) — retrying with backoff just delays the error. The PRD's FR-5.3 lists "404 invalid client" as transient-retryable; distinguish "invalid clientId" (retryable after re-fetching the client list) from "no connected clients" (persistent, surface immediately). Update FR-5.3.
|
||||
|
||||
### L3 — FR-1.2 / FR-1.3 skip semantics: `process()` skips on no `cc_uuid` or no `contentHash` (`src/server.ts:591-592`) and on body-hash-equal (`src/server.ts:594`). All three are correct and already implemented. The PRD's FR-1.2/1.3 matches the code. No change needed; noted for traceability.
|
||||
|
||||
### L4 — `foundry.ccHash` naming: the existing `foundry:` block uses snake_case-ish `cc_uuid`, `cc_type`, `folder_path`, `contentHash` (mixed). `ccHash` is fine but consider `cc_content_hash` or `foundryHash` for grep-ability and consistency with `cc_uuid`/`cc_type`. Cosmetic; decide during delivery.
|
||||
|
||||
---
|
||||
|
||||
## Overall verdict
|
||||
|
||||
**Conditional-go.** The PRD's O→F hardening (F1, FR-1.4–1.7) is feasible and the
|
||||
clobber-risk diagnosis is accurate, but the Foundry-side hash (H1) is under-specified
|
||||
and is the most likely thing to derail delivery. The F→O direction (F2) has **two
|
||||
blockers**: the PRD assumes `/search` returns `folder` (it does not — B1) and assumes
|
||||
content-change detection is cheap (it requires a `/get` per linked note per poll,
|
||||
which ADR 005 already rejected on cost grounds — B2). F2 as written is not
|
||||
shippable; it needs to be scoped down to "shallow poll: renames + new + missing" with
|
||||
content/folder changes routed to the manual trigger, or it needs an explicit,
|
||||
costed deep-poll mode in minutes not seconds.
|
||||
|
||||
The two-baseline 2×2 routing (F3) is sound in principle but has a real cross-direction
|
||||
lock gap (H3) and a TOCTOU window (H2) that the PRD should address.
|
||||
|
||||
Recommended pre-delivery edits to the PRD:
|
||||
1. Rewrite FR-2.1/2.2 to drop `folder` from the `/search` snapshot (B1) and split
|
||||
shallow vs deep poll (B2).
|
||||
2. Specify the exact Foundry-side hash input and the HTML→markdown reuse (H1).
|
||||
3. Add a shared per-uuid lock to F3/FR-5.5 (H3) and decide on the TOCTOU fix (H2).
|
||||
4. Clarify that FR-1.4 **reuses** the `/get` `pushNote` already makes (M1).
|
||||
5. Note the watcher-exclusion code change for `Sync Status.md` (M4) and the
|
||||
single-client blind spot in FR-6.3 (L1).
|
||||
|
||||
Top findings: **B1** (folder not in `/search`), **B2** (content detection = per-note
|
||||
`/get` every poll, contradicts ADR 005), **H1** (Foundry-side hash under-specified),
|
||||
**H2** (TOCTOU after `/get`), **H3** (no shared cross-direction lock).
|
||||
|
||||
File: `/home/kaykayyali/docker/obsidian-foundry-sync/docs/prds/prd-foundry-obsidian-sync-2026-06-22/review-engineering.md`
|
||||
@@ -0,0 +1,396 @@
|
||||
---
|
||||
title: "Launchability & Onboarding Adversarial Review"
|
||||
review_of: prd-foundry-obsidian-sync-2026-06-22/prd.md
|
||||
reviewer_role: adversarial reviewer (launchability / onboarding)
|
||||
stakes: public/launchable
|
||||
created: 2026-06-22
|
||||
---
|
||||
|
||||
# Launchability & Onboarding Adversarial Review
|
||||
|
||||
**Verdict: NOT launchable as specified.** The PRD sets stakes = "public/launchable" and
|
||||
NFR-6 promises "clone → live sync using the dashboard + `/setup`, without editing shell
|
||||
commands," but the F6 feature set only closes two of roughly seven onboarding gates, and
|
||||
several launchable-grade concerns (dashboard auth, Foundry-side data integrity, error
|
||||
contracts, status-note feedback-loop safety, conflict UX comprehension) are unspecified,
|
||||
hand-waved to "design during delivery," or silently dropped. The PRD reads as a strong
|
||||
*author-hardening* PRD wearing a *launchable* label. Two blockers and four high-severity
|
||||
findings stand between this draft and a non-author DM actually succeeding.
|
||||
|
||||
Cross-references in this review: PRD = `prd.md` in this dir; README =
|
||||
`/home/kaykayyali/docker/obsidian-foundry-sync/README.md`; env =
|
||||
`/home/kaykayyali/docker/obsidian-foundry-sync/.env.example`.
|
||||
|
||||
---
|
||||
|
||||
## BLOCKER
|
||||
|
||||
### B1 — `/setup` is not a product surface; NFR-6 is unmet as written
|
||||
|
||||
**Finding.** FR-6.4 ("The existing `/setup` skill covers env wiring; the dashboard
|
||||
reflects setup state and links into it") and NFR-6 ("A non-author user can get from
|
||||
clone → live sync using the dashboard + `/setup`, without editing shell commands")
|
||||
treat `/setup` as a product feature. It is not. `/setup` is a Claude Code skill — an
|
||||
authoring-time convenience available only to someone running Claude Code against this
|
||||
repo. A non-author DM who clones the repo has no `/setup`. README §"Quick start"
|
||||
confirms the real onboarding path is five shell steps: `cp .env.example .env`,
|
||||
`docker compose up -d relay`, browser signup to copy `RELAY_API_KEY`,
|
||||
`node scripts/start-relay-session.js`, and `./sync.sh refresh` + `./sync.sh ui`.
|
||||
|
||||
The F6 FRs only cover *detection* of missing config (FR-6.1 clientId empty, FR-6.2 no
|
||||
connected client, FR-6.3 list clients). They do not cover *acquisition* of:
|
||||
|
||||
1. The `RELAY_API_KEY` (requires bringing up the relay container + browser signup —
|
||||
shell + browser, not dashboard).
|
||||
2. Starting the headless Foundry session (`scripts/start-relay-session.js` — shell).
|
||||
3. Pointing Foundry's rest-api module at the relay WebSocket URL (Foundry-side admin
|
||||
config — cannot be driven from the dashboard at all).
|
||||
4. Bringing up the relay container itself (`docker compose up -d relay` — shell).
|
||||
5. Installing deps / starting the dashboard (`npm install`, `./sync.sh ui` — shell).
|
||||
|
||||
FR-6.2 detects "Foundry not connected" but the dashboard cannot *start* the headless
|
||||
session that would connect it. So the DM is stuck at a red banner with no in-UI remedy.
|
||||
|
||||
**Why it matters for launchable.** This is the single promise that defines "launchable"
|
||||
for this product, and it is the one the PRD most overstates. A non-author DM following
|
||||
the PRD literally cannot reach live sync without shell, and the PRD does not acknowledge
|
||||
the gap — it papers over it by referencing a tool the end user does not have.
|
||||
|
||||
**Recommended fix.** Pick one of two honest paths and state it in §3 and F6:
|
||||
|
||||
- **(a) Downscope NFR-6** to "clone → live sync using the dashboard, *given a running
|
||||
relay + headless session + REST-API module wired by the operator*," and add a §2
|
||||
"Operator prerequisites" block that enumerates the shell/Foundry steps the DM (or
|
||||
their operator) must complete once. This matches reality and is still a real
|
||||
improvement over today.
|
||||
- **(b) Actually build the in-UI onboarding** for the five gates above: a dashboard
|
||||
"first run" wizard that brings up the relay container via a host-side helper, guides
|
||||
API-key creation (deep-link to the relay's signup page, paste key into a form),
|
||||
launches the headless session via a server endpoint (not a shell command the user
|
||||
types), and shows step-by-step instructions for the Foundry-side rest-api module
|
||||
wiring with a connectivity check. Then NFR-6 stands. This is substantially more
|
||||
delivery work than F6 currently implies and must be sized as such.
|
||||
|
||||
Either way, drop `/setup` from the FRs and NFR-6 — it is an author tool, not a product
|
||||
surface. If (b) is chosen, add FR-6.5..6.8 for the four remaining gates.
|
||||
|
||||
---
|
||||
|
||||
### B2 — Dashboard binds `0.0.0.0:7788` with no auth; security is absent from the PRD
|
||||
|
||||
**Finding.** README §"Batch dashboard" confirms the dashboard "binds 0.0.0.0 by default
|
||||
so it's reachable on your tailnet at this VM's IP, port 7788; pass `--host 127.0.0.1` to
|
||||
restrict to localhost." The PRD has **no security section**. There is no FR or NFR for
|
||||
authentication, authorization, transport security, or access control on the dashboard.
|
||||
The dashboard can push to a live Foundry world (`/api/push`, `/api/push-all`), read and
|
||||
write the vault on disk, and presumably surfaces the relay API key in some form. With
|
||||
auto-sync (F1) the dashboard also drives automatic live writes to Foundry. An
|
||||
unauthenticated `0.0.0.0` listener that can overwrite a live Foundry world is a
|
||||
launchable-grade security hole.
|
||||
|
||||
The PRD also does not discuss: storage of `RELAY_API_KEY` and `RELAY_PASSWORD` (plaintext
|
||||
in `.env` per env.example), whether the dashboard ever reflects secrets to the browser,
|
||||
CSRF on POST endpoints, or what happens when the dashboard is exposed on a tailnet/shared
|
||||
network.
|
||||
|
||||
**Why it matters for launchable.** A launchable product that can clobber a live Foundry
|
||||
world over an open port with no auth is irresponsible to ship. "Bind localhost" is a
|
||||
mitigation the README mentions but the PRD never requires; the default is `0.0.0.0`.
|
||||
|
||||
**Recommended fix.** Add a §5 NFR-9 — Security, and a F7 — Security & access control
|
||||
feature group, minimum:
|
||||
|
||||
- **FR-7.1** Dashboard authenticates by default (token or password set via env or
|
||||
first-run prompt); unauthenticated requests get 401, not the UI.
|
||||
- **FR-7.2** Default bind changes to `127.0.0.1`; `0.0.0.0` requires explicit opt-in
|
||||
*and* an auth token set (refuse to start otherwise).
|
||||
- **FR-7.3** Secrets (`RELAY_API_KEY`, `RELAY_PASSWORD`) are never rendered to the
|
||||
browser; the dashboard shows only masked presence/absence.
|
||||
- **FR-7.4** POST mutation endpoints require a CSRF token or same-origin check.
|
||||
- **NFR-9** No unauthenticated mutation path; no secret egress to the client; TLS
|
||||
recommended when bound beyond localhost.
|
||||
|
||||
---
|
||||
|
||||
## HIGH
|
||||
|
||||
### H1 — Error contracts are unspecified; "surface an error row" is not actionable for a non-author DM
|
||||
|
||||
**Finding.** FR-1.6, FR-5.3, NFR-2, NFR-4 all reference "error rows" / "surfaced" but
|
||||
none specify the error *vocabulary*, *messages*, or *remediation guidance*. The relay
|
||||
returns at least four distinct failure modes that a non-author DM cannot distinguish
|
||||
from a raw status code: relay down (network), session down (relay up but no Foundry
|
||||
client), clientId invalid (relay up, client up, wrong/empty id), hash-mismatch (conflict,
|
||||
not an error), and transient 404/408/504 (retryable). FR-5.3 collapses these to
|
||||
"transient relay errors (404 invalid client, 408/504 timeout, 5xx)" — but "404 invalid
|
||||
client" is *not* transient (a wrong clientId will 404 forever), and "session down" is
|
||||
not in the retry list at all. The mapping from relay response → user-facing message →
|
||||
recommended action is absent.
|
||||
|
||||
**Why it matters for launchable.** NFR-5 promises "no silent skips," but a red row that
|
||||
says `Relay 404` is silent in the sense that matters: the DM does not know whether to
|
||||
fix `.env`, restart the headless session, or pick a different clientId. The author can
|
||||
debug from a status code; a non-author DM cannot.
|
||||
|
||||
**Recommended fix.** Add a §5a "Error contracts" table mapping each failure mode to:
|
||||
(a) detection signal, (b) retry-or-not, (c) the exact user-facing message, (d) the
|
||||
remediation hint shown. Minimum rows: relay unreachable, relay 401 (bad API key),
|
||||
clientId invalid/empty, no connected Foundry client, session idle-reaped, hash mismatch
|
||||
(route to F3, not error),Foundry-side /get 404 on a known uuid (deleted on Foundry),
|
||||
and persistent 5xx after backoff. Specify that error rows include a one-line "what to
|
||||
do" string, not just a status. Split FR-5.3's retry list into *transient* (timeout, 5xx,
|
||||
session-temporarily-unavailable) vs *persistent* (404 invalid client, 401 bad key) with
|
||||
no retry on persistent.
|
||||
|
||||
### H2 — `Sync Status.md` exclusion is not airtight; feedback-loop risk is unaddressed
|
||||
|
||||
**Finding.** FR-4.3 maintains `Sync Status.md` in the vault, "on a path excluded from
|
||||
the watcher, never synced to Foundry, updated each tick." OQ-4 defers the path/exclusion
|
||||
mechanism to delivery. The vault is what the watcher watches (FR-1.1). If the exclusion
|
||||
is path/filename-based and the DM renames or moves the note, or the tool's path config
|
||||
drifts, the watcher fires on every tick's status-note write → auto-sync attempts to push
|
||||
a `foundry:`-less note (skipped by FR-1.2, but it still churns the activity panel and
|
||||
wakes the pipeline) or, worse, a future version seeds it and a feedback loop starts.
|
||||
The PRD does not specify: the exclusion *mechanism* (glob? exact path? frontmatter flag?
|
||||
a dedicated subdir?), what guarantees it across renames, whether the F→O poll also skips
|
||||
it (FR-2.x doesn't mention it), and what happens on a name collision with a real note.
|
||||
|
||||
**Why it matters for launchable.** A status note that triggers sync is a self-inflicted
|
||||
loop that a non-author DM will not diagnose. "Excluded" stated as a requirement without
|
||||
a mechanism is a bug waiting to ship.
|
||||
|
||||
**Recommended fix.** Promote OQ-4 to a specification: define the exclusion as a
|
||||
*dedicated, reserved path* (e.g. `${VAULT}/.sync-status.md` or a dot-prefixed file that
|
||||
FR-1.1's "skip dotfiles" rule already covers), AND a frontmatter sentinel (e.g.
|
||||
`foundry.sync_status: true`) that both the O→F watcher and the F→O poll check and skip
|
||||
on, so the exclusion holds even if the path changes. Add FR-4.6: "The status note is
|
||||
excluded from both O→F watch and F→O poll by *both* a path rule and a content sentinel;
|
||||
a status note that loses its sentinel is treated as user error and surfaced, not
|
||||
synced." Add an explicit test to NFR-5: status-note writes never produce a sync op.
|
||||
|
||||
### H3 — Conflict UX is unspecified; "mark resolved (no change)" is a data-loss footgun for a non-author DM
|
||||
|
||||
**Finding.** F3 routes both-diverged to a conflict row with "push vault → Foundry",
|
||||
"pull Foundry → vault", "mark resolved (no change)" (FR-3.3). OQ-6 defers diff format
|
||||
to delivery. Two problems:
|
||||
|
||||
1. **Diff format is unspecified.** "Summarizing both versions and highlighting the diff"
|
||||
(FR-3.2) could be a unified diff, a side-by-side, or a condensed summary — three very
|
||||
different UX outcomes for a non-author DM staring at a conflict.
|
||||
2. **"Mark resolved (no change)" is genuinely ambiguous.** Its actual semantic (per
|
||||
FR-3.1 + the no-clobber rule) is presumably "accept the current diverged state as the
|
||||
new parity baseline without transferring content in either direction" — i.e. *both*
|
||||
sides keep their diverged content and both hashes re-baseline to current values. A
|
||||
DM reading "mark resolved (no change)" will reasonably think "do nothing, come back
|
||||
later" or "keep both" — and may click it expecting safety, then discover later that
|
||||
the two sides now permanently disagree and neither has the other's edits. There is no
|
||||
FR specifying what "mark resolved" actually does to the baselines, whether the
|
||||
divergence is preserved or collapsed, or what the DM is told just happened.
|
||||
|
||||
**Why it matters for launchable.** This is the one screen where the DM makes an
|
||||
irreversible content decision. For the author it is obvious; for a non-author DM it is
|
||||
the screen most likely to lose work. "Design during delivery" is not a launchable
|
||||
specification for the highest-stakes UX in the product.
|
||||
|
||||
**Recommended fix.** Resolve OQ-6 in the PRD, not in delivery. Specify:
|
||||
|
||||
- **FR-3.6** Conflict diff format: side-by-side with a one-line plain-language summary
|
||||
("Vault adds 3 paragraphs about X; Foundry renamed to Y and changed folder to Z"),
|
||||
not a raw unified diff.
|
||||
- **FR-3.7** "Mark resolved" is renamed to an explicit label — e.g. "Accept both as-is
|
||||
(keep divergence)" — with a confirmation dialog that states in plain text: "The vault
|
||||
and Foundry will keep their current versions. They will be treated as in-sync from
|
||||
now on. Neither side's changes will be copied to the other."
|
||||
- **FR-3.8** Each conflict action states, before commit, what it will do to each side
|
||||
and to the baselines (one-line preview). No irreversible action without a confirm.
|
||||
- **FR-3.9** A resolved conflict produces a visible activity-panel entry stating which
|
||||
side won and that the other side's edits were *not* transferred.
|
||||
|
||||
### H4 — Foundry-side data integrity has no backup/recovery story; auto-sync writes live with no Foundry-side undo
|
||||
|
||||
**Finding.** FR-1.8 states "Auto-sync always applies live (dry-run not honored)." Manual
|
||||
`--apply` mode writes a timestamped `.bak-<iso>` of vault files it overwrites (README
|
||||
§"Modes"). But push to Foundry goes through the relay `/update`, which overwrites the
|
||||
live Foundry entry — there is no Foundry-side `.bak`, no undo, no rollback specified.
|
||||
For a non-author DM who auto-syncs a bad edit (or hits a bug, or a hash race), the
|
||||
Foundry world is the side with no recovery path. The PRD has no §"Data integrity" or
|
||||
backups section. The vault side has `.bak`; the Foundry side has nothing.
|
||||
|
||||
**Why it matters for launchable.** A launchable sync tool that can overwrite a live
|
||||
Foundry world automatically, with no specified recovery path on the overwritten side,
|
||||
is a data-loss hazard. The author can restore from a world backup; a non-author DM may
|
||||
not have one scheduled.
|
||||
|
||||
**Recommended fix.** Add a §5 NFR-10 — Data integrity, minimum:
|
||||
|
||||
- **FR-5.6** Before any auto or manual push to Foundry, the prior Foundry-side content
|
||||
is fetched and cached locally (a `foundry-backups/<uuid>/<iso>.json` dir) so a bad
|
||||
push can be reversed from the dashboard. Specify retention (e.g. last N per uuid).
|
||||
- **FR-5.7** A "Revert last push" action exists for the most recent push per note.
|
||||
- **NFR-10** Foundry-side overwrites are always preceded by a local backup of the
|
||||
pre-push Foundry state; the dashboard exposes a restore path.
|
||||
- Document in §2 that Foundry-side world backups (Foundry's own backup feature) remain
|
||||
the DM's responsibility and are recommended before enabling auto-sync.
|
||||
|
||||
---
|
||||
|
||||
## MEDIUM
|
||||
|
||||
### M1 — Performance NFR is sized to the author's vault, not real-world vaults
|
||||
|
||||
**Finding.** NFR-3 bounds the burst target at "~50-note prep burst" with no stated basis.
|
||||
FR-2.1 polls `/search` for all `JournalEntry` minified on every tick — on a large world
|
||||
(hundreds or thousands of JournalEntries) this is a full scan per poll. No vault-size
|
||||
assumption is stated, no relay-load ceiling, no concurrency default numbers (FR-5.2
|
||||
"defaults tuned for prep" — what numbers?). The `RELAY_URL=http://localhost:3010`
|
||||
topology in `.env.example` reflects the author's all-on-one-host setup; the PRD does
|
||||
not acknowledge that a DM with remote Foundry has higher latency and a different
|
||||
retry/backoff profile.
|
||||
|
||||
**Recommended fix.** State the operating envelope in NFR-3: tested vault size range,
|
||||
JournalEntry count range the poll is validated against, default poll cadence (resolve
|
||||
OQ-3 with a number + rationale), default debounce and max-concurrency numbers, and the
|
||||
relay-load ceiling (concurrent `/update` calls). Add NFR-3a: "F→O poll and O→F push
|
||||
paths are validated against a vault of at least N notes and M JournalEntries," with N
|
||||
and M chosen above the author's own size.
|
||||
|
||||
### M2 — Auto-sync in dev vs apply mode is contradictory
|
||||
|
||||
**Finding.** FR-1.7 says "Dev mode baselines land in the `--out` mirror; apply mode in
|
||||
the real vault with a `.bak`." FR-1.8 says "Auto-sync always applies live (dry-run not
|
||||
honored)." These interact unclearly: can auto-sync be enabled in dev mode at all? If
|
||||
yes, "applies live" to what — the `--out` mirror or the real vault? If auto-sync only
|
||||
makes sense in apply mode, say so and gate the toggle on `--apply`. If it can run in dev
|
||||
mode, define what "live" means there. A non-author DM who starts the dashboard in the
|
||||
default dev mode and flips on auto-sync will not know what is being written where.
|
||||
|
||||
**Recommended fix.** Add FR-1.9: "Auto-sync requires apply mode; enabling it in dev mode
|
||||
is either blocked with an explanatory banner or explicitly writes to `--out` only (pick
|
||||
one and state it)." Reconcile FR-1.7/1.8 accordingly.
|
||||
|
||||
### M3 — Status state single-source-of-truth has no persistence story across restart
|
||||
|
||||
**Finding.** FR-4.5 says dashboard parity and the vault status note reflect "one
|
||||
underlying state (single source of truth for status)." The PRD doesn't say where that
|
||||
state lives. If it is in-memory in the server, a restart loses parity counts, conflict
|
||||
state (FR-3.4 "persists until resolved"), and last-sync time — but the vault status note
|
||||
persists. On restart, does the dashboard re-derive parity from a full re-scan, or does
|
||||
it show stale/empty until the next tick? Conflict persistence (FR-3.4) is especially
|
||||
load-bearing: a restart that drops conflict state could let a re-save auto-clear a known
|
||||
conflict, violating FR-3.4.
|
||||
|
||||
**Recommended fix.** Specify the status store: either (a) a small persisted state file
|
||||
on disk (`sync-state.json`) that survives restart and is the single source the dashboard
|
||||
and status note both read from, or (b) re-derive on restart with an explicit "recomputing
|
||||
parity…" state. Add FR-3.4a: "Conflict state persists across server restarts, not just
|
||||
across ticks."
|
||||
|
||||
### M4 — Versioning, updates, and frontmatter schema migration are unaddressed
|
||||
|
||||
**Finding.** The PRD introduces a new frontmatter field `foundry.ccHash` (FR-1.4) and
|
||||
relies on existing `foundry.contentHash`/`cc_uuid`/`folder_path`. There is no schema
|
||||
version field on the `foundry:` block and no migration story. A non-author DM who
|
||||
updates the tool to a later version that changes hash algorithm or adds fields has no
|
||||
specified upgrade path — old baselines will mismatch and every note may route to
|
||||
conflict on first sync after upgrade. No §"Versioning & updates" section.
|
||||
|
||||
**Recommended fix.** Add `foundry.schema_version` (or similar) to the identity block and
|
||||
an FR in F1/F5: on startup, notes with a missing/older schema version are migrated
|
||||
(re-hash, re-baseline) in a single pass before auto-sync is allowed to engage, with a
|
||||
dashboard migration banner. Add a §5 NFR-11 — Upgrades: "A version bump that changes
|
||||
hashing or identity fields ships with an idempotent migration; auto-sync stays disabled
|
||||
until migration completes."
|
||||
|
||||
### M5 — Supportability: no persistent log / diagnostics export
|
||||
|
||||
**Finding.** NFR-5 mandates observability via the activity panel and status note, but
|
||||
FR-5.4 says the panel is "capped and scrollable" — i.e. not persistent. A non-author DM
|
||||
who hits a problem and asks for help has no log file to share. There is no FR for a
|
||||
persistent log, a diagnostics export, or a "copy debug info" action. For a launchable
|
||||
product this is the difference between "I can help you" and "I can't see what happened."
|
||||
|
||||
**Recommended fix.** Add FR-5.7 (renumber from H4's FR-5.7): "All auto-sync ops
|
||||
additionally append to a persistent, rotated log file on disk
|
||||
(`logs/sync-<date>.log`)." Add FR-5.8: "A 'Copy diagnostics' dashboard action bundles
|
||||
recent log tail, current config (secrets redacted), parity counts, and relay/clientId
|
||||
status into a single redacted blob for support." NFR-5a: "Operation history is
|
||||
persistent across restarts, not just visible in-session."
|
||||
|
||||
---
|
||||
|
||||
## LOW
|
||||
|
||||
### L1 — Polling load on shared relays is a courtesy concern
|
||||
|
||||
**Finding.** FR-2.1 polls `/search` on a cadence for every DM running the tool. If the
|
||||
relay is a shared instance (multiple DMs), uncoordinated polling multiplies. The PRD
|
||||
doesn't note this as a consideration or suggest a jittered/backoff poll.
|
||||
|
||||
**Recommended fix.** Add jitter to the poll cadence and a note in §2 that on shared
|
||||
relays the cadence should be raised. Non-blocking.
|
||||
|
||||
### L2 — New (cc-only) Foundry entries during prep: candidate vs auto-import is an open question with launchability impact
|
||||
|
||||
**Finding.** OQ-5 leaves "auto-import new Foundry entries" open with current spec =
|
||||
candidates only. For a non-author DM, "import candidates" with no further guidance may
|
||||
mean a growing pile of unimported rows they don't know what to do with.
|
||||
|
||||
**Recommended fix.** Resolve OQ-5 in the PRD: keep candidates-only (safe) but specify
|
||||
that the candidate row has a one-click "Import as new refined note" action with a clear
|
||||
explanation of what import does. The current import row exists per README; confirm it is
|
||||
comprehensible to a non-author DM.
|
||||
|
||||
### L3 — "Foundry-wins" tie-breaker for both-diverged is deferred but conflict UX depends on it
|
||||
|
||||
**Finding.** §1 confirms "Foundry is source of truth" is demoted to a tie-breaker for
|
||||
both-diverged, "to be decided: newest-wins, Foundry-wins, or manual merge." F3 then
|
||||
specifies manual resolution via buttons, which makes the tie-breaker moot *if* the DM
|
||||
always resolves manually. But the conflict row needs a *default ordering* (which side is
|
||||
shown on the left, which action is highlighted) and that ordering embeds a tie-breaker
|
||||
opinion. Leaving it undecided leaks into UX.
|
||||
|
||||
**Recommended fix.** Pick a neutral default (e.g. vault on left, Foundry on right, no
|
||||
pre-highlighted action) so the undecided tie-breaker doesn't bias the DM. State it in F3.
|
||||
|
||||
### L4 — README still describes Foundry as "source of truth"; PRD demotes it
|
||||
|
||||
**Finding.** README L17 says "Foundry is the source of truth." The PRD §1 demotes that
|
||||
to a tie-breaker. For a launchable product the README is the onboarding doc and the PRD
|
||||
is the spec; a non-author DM reading both will be confused about which side wins.
|
||||
|
||||
**Recommended fix.** Note in §6 / §"Out of Scope" that README wording will be updated to
|
||||
match the demoted-source-of-truth model before launch. Track as a doc task.
|
||||
|
||||
---
|
||||
|
||||
## Summary table
|
||||
|
||||
| ID | Tier | Theme | One-line |
|
||||
|-----|---------|------------------------|--------------------------------------------------------------------------|
|
||||
| B1 | BLOCKER | Onboarding | `/setup` is not a product surface; NFR-6 unmet as written |
|
||||
| B2 | BLOCKER | Security | `0.0.0.0:7788` dashboard, no auth, no security section |
|
||||
| H1 | HIGH | Error contracts | Error vocabulary/messages/remediation unspecified |
|
||||
| H2 | HIGH | Status-note loop | `Sync Status.md` exclusion mechanism not airtight |
|
||||
| H3 | HIGH | Conflict UX | Diff format + "mark resolved" semantics unspecified; data-loss footgun |
|
||||
| H4 | HIGH | Data integrity | Auto-sync writes Foundry live with no Foundry-side backup/undo |
|
||||
| M1 | MEDIUM | Perf sizing | "~50-note burst" not sized to real vaults; no relay-load ceiling |
|
||||
| M2 | MEDIUM | Dev/apply vs auto-sync | FR-1.7/1.8 interaction contradictory |
|
||||
| M3 | MEDIUM | Status persistence | Single-source-of-truth state has no restart story; conflict state at risk|
|
||||
| M4 | MEDIUM | Versioning | No schema version / migration story for `foundry:` block |
|
||||
| M5 | MEDIUM | Supportability | No persistent log or diagnostics export |
|
||||
| L1 | LOW | Poll courtesy | Jittered poll on shared relays |
|
||||
| L2 | LOW | Import candidates UX | OQ-5 candidate UX needs comprehensibility check |
|
||||
| L3 | LOW | Tie-breaker default | Undecided tie-breaker leaks into conflict-row ordering |
|
||||
| L4 | LOW | Doc drift | README "source of truth" contradicts PRD demotion |
|
||||
|
||||
---
|
||||
|
||||
## Overall recommendation
|
||||
|
||||
**Do not mark this PRD ready for launchable delivery** until B1 and B2 are resolved
|
||||
(either by honest downscoping of NFR-6 or by committing to build the in-UI onboarding
|
||||
and auth) and H1–H4 are specified concretely enough to implement against. As a
|
||||
*hardening* PRD for the single author it is solid; as a *launchable* PRD it currently
|
||||
treats onboarding, security, error contracts, status-loop safety, conflict UX, and
|
||||
Foundry-side data integrity as "design during delivery" — exactly the surfaces where a
|
||||
non-author DM falls off.
|
||||
192
docs/prds/prd-foundry-obsidian-sync-2026-06-22/review-rubric.md
Normal file
192
docs/prds/prd-foundry-obsidian-sync-2026-06-22/review-rubric.md
Normal file
@@ -0,0 +1,192 @@
|
||||
# PRD Quality Review — Live Relay Sync — Auto-Sync & Bidirectional Hardening
|
||||
|
||||
## Overall verdict
|
||||
|
||||
This is a tightly-argued, capability-spec PRD that earns most of its length: the
|
||||
no-clobber thesis is crisp, the trade-offs are named with what was given up (not
|
||||
just what was chosen), and Non-Goals do real work. What's at risk is downstream
|
||||
extractability and thesis-validation: there is no Glossary and no Success Metrics
|
||||
section, so a story author will re-derive domain terms and the PRD has no
|
||||
machine-readable way to confirm the thesis landed. A stale "to be decided" in the
|
||||
Vision also drifts from a decision the FRs already made. None of this blocks a
|
||||
green-light; all of it is one editing pass away from closed.
|
||||
|
||||
## Decision-readiness — strong
|
||||
|
||||
Decisions are stated as decisions, not buried as considerations. The `[CONFIRMED]` /
|
||||
`[DEFERRED]` tags in §1 are honest — e.g. _"Foundry is source of truth" is demoted
|
||||
to a tie-breaker rule for the both-diverged conflict case"_ names both the choice
|
||||
and the demotion. Trade-offs name what was given up: FR-1.6 explicitly says _"do
|
||||
not fall back to an Obsidian-side-only check (that reintroduces clobber risk)"_ —
|
||||
the rejected path is on the page, not hidden. The Open Questions are actually open:
|
||||
OQ-1 carries a default posture (_"Default posture is manual"_), OQ-3/OQ-4/OQ-6 are
|
||||
genuinely delivery-time (_"pick during delivery B with relay-load testing"_), and
|
||||
OQ-2 is a real deferral with the reopening condition named (_"Reopens when a
|
||||
custom Foundry module is explored"_). The decision-log shows real tensions
|
||||
surfaced (the clobber gap in the current O→F code) rather than smoothed to neutral.
|
||||
|
||||
### Findings
|
||||
- **medium** Stale "to be decided" in Vision contradicts the FRs (§1, _"to be
|
||||
decided: newest-wins, Foundry-wins, or manual merge"_). FR-3.2 and FR-3.3
|
||||
already specify manual resolution with three explicit actions (_"push vault →
|
||||
Foundry", "pull Foundry → vault", "mark resolved"_); OQ-1 only asks whether to
|
||||
add a newest-mtime convenience. The §1 phrasing reads as unresolved when the
|
||||
PRD has in fact decided. *Fix:* rewrite the §1 parenthetical to
|
||||
_"default = manual resolution (FR-3.2); a newest-mtime convenience is open
|
||||
(OQ-1)"_.
|
||||
|
||||
## Substance over theater — strong
|
||||
|
||||
No personas, no User Journeys, no differentiation section — and that is correct
|
||||
for a single-operator brownfield capability spec (see Shape fit). The Vision is
|
||||
product-specific, not swappable: the prep / run-the-match duality, the
|
||||
_"always in lockstep while I work"_ feeling, and the _"sync deliberately paused,
|
||||
not silently absent"_ framing would not transfer to another tool. NFRs carry
|
||||
product-specific thresholds, not boilerplate: NFR-3 names a _"~50-note prep
|
||||
burst"_, NFR-4 _"within one tick"_, NFR-6 names the _"UI-only convention"_. No
|
||||
innovation theater, no persona theater. Nothing here reads like furniture.
|
||||
|
||||
### Findings
|
||||
- (none)
|
||||
|
||||
## Strategic coherence — adequate
|
||||
|
||||
The thesis is stated and the features serve it. G1 (no-clobber bidirectional
|
||||
auto-sync for prep) is the thesis; F1+F2+F3 implement it directly. G4 (legible
|
||||
status) is the second arc; F4 serves it. G5/G6 (hardening/onboarding) are the
|
||||
launchable-stakes rigor that follows from the chosen stakes, not a bolt-on.
|
||||
Prioritization follows the thesis: delivery A (ship O→F *with the divergence
|
||||
guard*) is gated on the safety fix, not on "what's easy first."
|
||||
|
||||
The gap is that there is **no Success Metrics section**. The rubric asks whether
|
||||
SMs validate the thesis and whether counter-metrics are named. G2 embeds one
|
||||
criterion (_"verify one end-to-end live push"_), but there is no consolidated set
|
||||
of operational metrics that would let a reviewer confirm the thesis — e.g.
|
||||
"zero silent clobbers across a 50-note prep burst", "F→O poll detects a Foundry
|
||||
rename within N ticks", "non-author user reaches live sync via dashboard + /setup
|
||||
without shell". For a public/launchable PRD, the absence is a coherence tell: the
|
||||
PRD argues the thesis but doesn't state how it would know it landed.
|
||||
|
||||
### Findings
|
||||
- **medium** No Success Metrics section (whole PRD). The thesis is clear but
|
||||
unmeasured; activity metrics are absent too. *Fix:* add an SM block with 3–5
|
||||
operational metrics tied to G1/G2/G4/G6 (e.g. _SM-1: zero auto-overwrite
|
||||
events on a side that changed since last sync, across a 50-note burst_;
|
||||
_SM-2: one verified end-to-end live O→F push_; _SM-3: F→O detects a Foundry
|
||||
rename/move within ≤2 poll ticks_; _SM-6: a non-author user clones, runs
|
||||
/setup, and reaches a connected live sync with no shell command_). Name a
|
||||
counter-metric for SM-1 (e.g. _conflict-rows-surfaced > 0 when both sides
|
||||
diverged_).
|
||||
|
||||
## Done-ness clarity — adequate
|
||||
|
||||
Most FRs carry testable consequences. FR-1.4's hash-over-_content + name +
|
||||
folder_path_ is concrete and verifiable. FR-1.5's 2×2 routing is a truth table.
|
||||
FR-3.1 makes the routing explicit. FR-2.2 enumerates the change kinds
|
||||
(_renamed / moved / content-changed / missing / new_). G2's _"verify one
|
||||
end-to-end live push"_ is an acceptance gate for delivery A. NFR-1 is the hard
|
||||
contract (_"No auto-sync operation may overwrite a side that has changed since
|
||||
the last sync"_).
|
||||
|
||||
The unforgiving part: a few FRs use adjectives where bounds belong. FR-5.2 says
|
||||
_"defaults tuned for prep so a burst of simultaneous saves doesn't thrash or drop
|
||||
events"_ — "thrash" is undefined and "tuned for prep" is an adjective, not a
|
||||
bound; NFR-3 supplies the ~50-note number but FR-5.2 doesn't reference it.
|
||||
FR-5.4's _"panel capped and scrollable"_ leaves the cap unspecified. These are
|
||||
the spots where an engineer writing stories will have to go back and ask "how
|
||||
many?"
|
||||
|
||||
### Findings
|
||||
- **low** FR-5.2 "thrash" / "tuned for prep" — bound-less adjectives (§F5).
|
||||
*Fix:* bind FR-5.2 to NFR-3's ~50-note burst and name concrete defaults
|
||||
ranges (debounce window, max concurrency) or defer the numbers to a config
|
||||
table with placeholders.
|
||||
- **low** FR-5.4 "panel capped" — cap value missing (§F5). *Fix:* state the cap
|
||||
(e.g. _"last 200 events, scrollable for older"_).
|
||||
|
||||
## Scope honesty — strong
|
||||
|
||||
Non-Goals do real work and name what could be silently assumed: _"Not syncing"
|
||||
indicator rendered inside Foundry_ is explicitly deferred with the reason
|
||||
(_"Foundry UI is not a surface we control"_); _"Syncing during run-the-match"_
|
||||
is called out as a design decision, not an omission; _"Automatic semantic / 3-way
|
||||
content merge"_ is excluded so the conflict UI doesn't get over-built. The
|
||||
`[ASSUMPTION]` tag in §2 (_"The relay remains the live transport for this PRD;
|
||||
the custom Foundry module … is explicitly future work, not a dependency here"_)
|
||||
flags the key inference. Open-items density is moderate (6 OQs + 1 ASSUMPTION)
|
||||
against launchable stakes; every OQ carries a default or a named
|
||||
delivery-trigger, so the density is honest rather than a blocker.
|
||||
|
||||
### Findings
|
||||
- **low** Implicit relay-load assumption not flagged (§F2 / NFR-3). FR-2.2
|
||||
detects content-changed entries _"via /get hash compare"_, which implies one
|
||||
/get per changed linked note per poll tick — a load assumption that
|
||||
interacts with OQ-3's cadence. This is an inference the user didn't directly
|
||||
confirm and should be tagged `[ASSUMPTION]`. *Fix:* add
|
||||
_`[ASSUMPTION]` one relay /get per changed linked note per poll tick is
|
||||
acceptable relay load at the chosen cadence (OQ-3)_ to §F2, and index it.
|
||||
|
||||
## Downstream usability — adequate
|
||||
|
||||
FR IDs are contiguous (FR-1.1 → FR-6.4), unique, and cross-references resolve
|
||||
(FR-5.1 → FR-1.1 ✓; OQ-5 → FR-2.5 ✓; NFR-1 → the divergence guard ✓). Brownfield
|
||||
references are accurate against the decision log and code memory
|
||||
(_AutoSyncController_, _pushNote_, _foundry.cc_uuid_, _foundry.contentHash_,
|
||||
_rest-api module_, _relay /search_ minified, _relay /get_). No floating UJs (no
|
||||
UJs at all — appropriate). Each section reads sensibly pulled out alone.
|
||||
|
||||
The gap is the **missing Glossary**. The PRD uses a thick domain vocabulary —
|
||||
_cc_uuid_, _contentHash_, _ccHash_ (new field, introduced in FR-1.4), _linked
|
||||
note_, _seeded_, _refined vault dir_, _refined markdown_, _import candidate_,
|
||||
_dev/apply mode_, _activity panel_, _parity_, _cc-only entry_ — and these are
|
||||
defined only inline and inconsistently (FR-1.2 glosses _foundry.cc_uuid_ as
|
||||
"unlinked"; FR-1.4 defines _ccHash_ in passing). A story author source-extracting
|
||||
from this PRD will have to reassemble the term set, and drift risk is real
|
||||
(_ccHash_ vs _contentHash_ vs _cc_uuid_ are easy to confuse). For a PRD that
|
||||
feeds architecture and stories, this is the dimension that needs the most work.
|
||||
|
||||
### Findings
|
||||
- **medium** No Glossary section (whole PRD). Domain nouns are inline-defined
|
||||
and not consolidated; _ccHash_ is introduced mid-FR without a term-of-art
|
||||
definition. *Fix:* add a Glossary section defining at minimum: _cc_uuid_,
|
||||
_contentHash_, _ccHash_, _linked note_, _seeded_, _refined vault dir_,
|
||||
_refined markdown_, _import candidate_, _cc-only entry_, _dev/apply mode_,
|
||||
_activity panel_, _parity_, _tick_. Reference Glossary terms from the FRs
|
||||
instead of re-defining inline.
|
||||
|
||||
## Shape fit — strong
|
||||
|
||||
The shape matches the product. This is a brownfield, single-operator (DM/world-
|
||||
builder) capability spec intended to go launchable — exactly the case the rubric
|
||||
flags as _"Internal tool, single-operator role → capability spec shape; UJs may
|
||||
be overhead; SMs may be operational."_ No personas, no UJs: correct, not a gap.
|
||||
Operational SMs would fit the shape but their absence (see Strategic coherence)
|
||||
is a coherence gap, not a shape mismatch. The PRD is neither over-formalized (no
|
||||
UJ theater for a one-operator tool) nor under-formalized (NFRs and error
|
||||
contracts are present at launchable rigor). Existing-code references are
|
||||
distinguished from new behavior (_"code-complete but uncommitted"_, _"new field,
|
||||
e.g. foundry.ccHash"_), which is what brownfield shape demands.
|
||||
|
||||
### Findings
|
||||
- **low** Operational SMs would fit this shape and are absent (§5 / §SM). *Fix:*
|
||||
see the Strategic-coherence finding — operational metrics (zero silent
|
||||
clobbers, ticks-to-detect, onboard-without-shell) are the right SM shape for
|
||||
this product and would close both dimensions at once.
|
||||
|
||||
## Mechanical notes
|
||||
|
||||
- **Glossary drift:** No glossary; _ccHash_ vs _contentHash_ vs _cc_uuid_ are
|
||||
used close together and differ only in case/short suffix — high confusion
|
||||
risk. _"cc-only entry"_ (FR-2.5) and _"import candidate"_ (FR-2.5) are used
|
||||
as synonyms without being linked. _"refined vault dir"_ / _"refined markdown"_
|
||||
appear without definition (FR-1.1, FR-2.3).
|
||||
- **ID continuity:** FR IDs contiguous and unique; cross-refs (FR-5.1→FR-1.1,
|
||||
OQ-5→FR-2.5, NFR-1→the divergence guard) all resolve. No duplicates, no gaps.
|
||||
- **Assumptions Index roundtrip:** broken. One inline `[ASSUMPTION]` exists
|
||||
(§2, relay-as-transport); no Assumptions Index at the end of the PRD. The
|
||||
implicit relay-load assumption (Scope honesty finding) is not tagged at all.
|
||||
- **UJ persona linkage:** N/A — no UJs, no personas (appropriate for shape).
|
||||
- **Required sections for stakes:** Vision ✓, Problem & Context ✓, Goals &
|
||||
Non-Goals ✓, FRs ✓, NFRs ✓, Open Questions ✓, Out of Scope ✓. **Missing:
|
||||
Success Metrics, Glossary.** For public/launchable stakes, SMs are expected;
|
||||
Glossary is expected for a PRD that feeds architecture + stories.
|
||||
141
scripts/start-relay-session.js
Normal file
141
scripts/start-relay-session.js
Normal file
@@ -0,0 +1,141 @@
|
||||
// Starts a headless Foundry session on a ThreeHats foundryvtt-rest-api-relay.
|
||||
//
|
||||
// Generalized version: every value comes from the environment (no hardcoded
|
||||
// host). Reads this repo's .env first, then process env. Required: RELAY_API_KEY,
|
||||
// RELAY_URL, FOUNDRY_URL, RELAY_USER, RELAY_PASSWORD, FOUNDRY_WORLD.
|
||||
//
|
||||
// Flow (per docs/relay-api.md):
|
||||
// 1. POST /session-handshake (x-api-key, x-foundry-url, x-username) -> {nonce, publicKey, token}
|
||||
// 2. RSA-OAEP encrypt {password, nonce} with the relay's public key.
|
||||
// 3. POST /start-session {handshakeToken, encryptedPassword, world} -> headless launch.
|
||||
//
|
||||
// Run from the repo root: node scripts/start-relay-session.js
|
||||
// (The /setup skill does this for you after the Foundry rest-api module is connected.)
|
||||
|
||||
const crypto = require('crypto');
|
||||
const http = require('http');
|
||||
const https = require('https');
|
||||
const fs = require('fs');
|
||||
const path = require('path');
|
||||
const { URL } = require('url');
|
||||
|
||||
// --- Load .env (simple KEY=value; real process.env wins) ---
|
||||
(function loadDotEnv() {
|
||||
const p = path.join(__dirname, '..', '.env');
|
||||
let txt;
|
||||
try { txt = fs.readFileSync(p, 'utf8'); } catch { return; } // no .env is fine
|
||||
for (const line of txt.split(/\r?\n/)) {
|
||||
const m = line.match(/^\s*([A-Za-z_][A-Za-z0-9_]*)\s*=\s*(.*)\s*$/);
|
||||
if (!m) continue;
|
||||
const [, k, vRaw] = m;
|
||||
if (process.env[k] !== undefined) continue; // real env wins over .env
|
||||
const v = vRaw.replace(/^['"]|['"]$/g, '');
|
||||
if (v !== '') process.env[k] = v;
|
||||
}
|
||||
})();
|
||||
|
||||
const env = (k) => (process.env[k] !== undefined && process.env[k] !== '' ? process.env[k] : '');
|
||||
|
||||
const API_KEY = env('RELAY_API_KEY');
|
||||
const RELAY_URL = env('RELAY_URL');
|
||||
const FOUNDRY_URL = env('FOUNDRY_URL');
|
||||
const USERNAME = env('RELAY_USER');
|
||||
const PASSWORD = env('RELAY_PASSWORD');
|
||||
const WORLD = env('FOUNDRY_WORLD');
|
||||
|
||||
const missing = [
|
||||
['RELAY_API_KEY', API_KEY],
|
||||
['RELAY_URL', RELAY_URL],
|
||||
['FOUNDRY_URL', FOUNDRY_URL],
|
||||
['RELAY_USER', USERNAME],
|
||||
['RELAY_PASSWORD', PASSWORD],
|
||||
['FOUNDRY_WORLD', WORLD],
|
||||
].filter(([, v]) => !v).map(([k]) => k);
|
||||
if (missing.length) {
|
||||
console.error(`Missing required env var(s): ${missing.join(', ')}.\n` +
|
||||
`Copy .env.example to .env and fill them in (or run the /setup skill).`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
let relay;
|
||||
try {
|
||||
relay = new URL(RELAY_URL);
|
||||
} catch {
|
||||
console.error(`RELAY_URL is not a valid URL: ${RELAY_URL}`);
|
||||
process.exit(1);
|
||||
}
|
||||
const transport = relay.protocol === 'https:' ? https : http;
|
||||
|
||||
function post(pathname, headers, body) {
|
||||
return new Promise((resolve, reject) => {
|
||||
const data = JSON.stringify(body || {});
|
||||
const req = transport.request(
|
||||
{
|
||||
protocol: relay.protocol,
|
||||
hostname: relay.hostname,
|
||||
port: relay.port || (relay.protocol === 'https:' ? 443 : 80),
|
||||
path: pathname,
|
||||
method: 'POST',
|
||||
headers: {
|
||||
'Content-Type': 'application/json',
|
||||
'Content-Length': Buffer.byteLength(data),
|
||||
'x-api-key': API_KEY,
|
||||
...headers,
|
||||
},
|
||||
},
|
||||
(res) => {
|
||||
let d = '';
|
||||
res.on('data', (c) => (d += c));
|
||||
res.on('end', () => {
|
||||
if (res.statusCode && res.statusCode >= 400) {
|
||||
reject(new Error(`relay ${res.statusCode} POST ${pathname}: ${d.slice(0, 300)}`));
|
||||
return;
|
||||
}
|
||||
try { resolve(JSON.parse(d)); } catch { resolve(d); }
|
||||
});
|
||||
}
|
||||
);
|
||||
req.on('error', reject);
|
||||
req.write(data);
|
||||
req.end();
|
||||
});
|
||||
}
|
||||
|
||||
(async () => {
|
||||
console.log(`Starting handshake on ${RELAY_URL.href} (world: ${WORLD}, foundry: ${FOUNDRY_URL}, user: ${USERNAME})...`);
|
||||
let hs;
|
||||
try {
|
||||
hs = await post('/session-handshake', {
|
||||
'x-foundry-url': FOUNDRY_URL,
|
||||
'x-username': USERNAME,
|
||||
}, {});
|
||||
} catch (e) {
|
||||
console.error(`Handshake failed: ${e.message}\n` +
|
||||
`Is the relay up (docker compose up -d relay) and is the Foundry rest-api module connected to it?`);
|
||||
process.exit(1);
|
||||
}
|
||||
if (!hs || !hs.nonce || !hs.publicKey || !hs.token) {
|
||||
console.error('Handshake returned an unexpected response:', JSON.stringify(hs));
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
const payload = JSON.stringify({ password: PASSWORD, nonce: hs.nonce });
|
||||
const encryptedPassword = crypto.publicEncrypt(
|
||||
{ key: hs.publicKey, padding: crypto.constants.RSA_PKCS1_OAEP_PADDING, oaepHash: 'sha256' },
|
||||
Buffer.from(payload)
|
||||
).toString('base64');
|
||||
|
||||
console.log('Launching headless session (may take up to 2 min)...');
|
||||
try {
|
||||
const result = await post('/start-session', {}, {
|
||||
handshakeToken: hs.token,
|
||||
encryptedPassword,
|
||||
world: WORLD,
|
||||
});
|
||||
console.log('Result:', JSON.stringify(result, null, 2));
|
||||
} catch (e) {
|
||||
console.error(`Start-session failed: ${e.message}\n` +
|
||||
`Check FOUNDRY_URL/RELAY_USER/RELAY_PASSWORD/FOUNDRY_WORLD are correct and the world exists.`);
|
||||
process.exit(1);
|
||||
}
|
||||
})();
|
||||
@@ -77,7 +77,7 @@ async function walkMd(root: string): Promise<string[]> {
|
||||
if (SKIP_DIRS.has(ent.name)) continue;
|
||||
const p = join(root, ent.name);
|
||||
if (ent.isDirectory()) out = out.concat(await walkMd(p));
|
||||
else if (ent.isFile() && ent.name.toLowerCase().endsWith(".md")) out.push(p);
|
||||
else if (ent.isFile() && ent.name.toLowerCase().endsWith(".md") && !ent.name.startsWith(".")) out.push(p); // E4.5: exclude dotfiles (.sync-status.md etc.)
|
||||
}
|
||||
} catch {
|
||||
// missing dir -> empty
|
||||
|
||||
99
src/canonicalize-html.ts
Normal file
99
src/canonicalize-html.ts
Normal file
@@ -0,0 +1,99 @@
|
||||
// E1b-alt — canonicalizeHtml: the Foundry-HTML canonicalizer for the HTML-hash
|
||||
// ccHash contract (the NO-GO fork of E1a).
|
||||
//
|
||||
// E1a proved the markdown round-trip is unstable (wikilinks/@UUID, tables,
|
||||
// secrets order/case, parseBody bold bug). E1b-alt hashes the Foundry HTML
|
||||
// directly: ccHash = contentHash(canonicalizeHtml(data.description) + "\n" +
|
||||
// canonicalizeHtml(data.notes) + "\n" + name + "\n" + folder). Both the
|
||||
// baseline (foundry.ccHash, stored at push time) and the live ccHash hash the
|
||||
// SAME HTML from the live entry → comparable by construction; a Foundry-side
|
||||
// content change → different DOM → different canonical HTML → different hash.
|
||||
//
|
||||
// canonicalizeHtml absorbs incidental serialization drift so the hash is stable
|
||||
// across relay /get calls for an UNCHANGED entry, while still moving when
|
||||
// content changes. Drift defended against (Foundry's editor re-serializing on a
|
||||
// null edit, or the relay normalizing on store/retrieve):
|
||||
// - attribute ORDER → sorted by name
|
||||
// - attribute QUOTING → double quotes, consistently escaped
|
||||
// - tag CASE → lowercased
|
||||
// - HTML ENTITIES → linkedom decodes on parse; we re-encode & < > " consistently
|
||||
// - VOID/self-closing → canonical `<tag …>` (no slash, no closing)
|
||||
// - inter-tag WHITESPACE between BLOCK elements (indentation, newlines) → dropped
|
||||
// - intra-text WHITESPACE runs → collapsed to a single space (matches HTML rendering)
|
||||
//
|
||||
// What it preserves (so real content changes move the hash):
|
||||
// - tag STRUCTURE (nesting, element types)
|
||||
// - attribute NAMES and VALUES (sorted but content-bearing)
|
||||
// - meaningful TEXT (text nodes that are not whitespace-only-between-blocks
|
||||
// are preserved, with internal whitespace collapsed to single spaces)
|
||||
//
|
||||
// Whitespace handling: whitespace-only text nodes (inter-tag indentation,
|
||||
// blank lines the serializer may add or drop) are DROPPED; meaningful text
|
||||
// nodes have internal whitespace runs collapsed to a single space (matches HTML
|
||||
// rendering). This is safe because the forward transform (`markdownToHtml` +
|
||||
// `escapeHtml`) emits proper entities (`&`, not bare `&`) and the relay
|
||||
// returns Foundry's stored HTML verbatim, so the bare-`&`-vs-`&` case is not
|
||||
// a realistic drift — and entity-equivalence (named vs numeric, e.g. `&` vs
|
||||
// `&`) holds because linkedom decodes both to the same text on parse.
|
||||
//
|
||||
// Trade-off (fail-safe direction): a render-invisible reformat can still move
|
||||
// the canonical form → a false "Foundry changed" signal. That is SAFE (the guard
|
||||
// skips a push / surfaces a conflict rather than clobbering). The dangerous
|
||||
// direction — a real content change that leaves the canonical form unchanged
|
||||
// (false negative) — does not occur, because any text or structural change
|
||||
// alters the DOM and thus the canonical string.
|
||||
|
||||
import { parseHTML } from "linkedom";
|
||||
|
||||
const ELEMENT_NODE = 1;
|
||||
const TEXT_NODE = 3;
|
||||
|
||||
// Void elements: no closing tag, no children (HTML spec).
|
||||
const VOID = new Set([
|
||||
"area", "base", "br", "col", "embed", "hr", "img", "input",
|
||||
"link", "meta", "param", "source", "track", "wbr",
|
||||
]);
|
||||
|
||||
function escapeText(s: string): string {
|
||||
return s.replace(/&/g, "&").replace(/</g, "<").replace(/>/g, ">");
|
||||
}
|
||||
function escapeAttr(s: string): string {
|
||||
return s.replace(/&/g, "&").replace(/"/g, """).replace(/</g, "<").replace(/>/g, ">");
|
||||
}
|
||||
|
||||
function serializeElement(el: any): string {
|
||||
const tag = el.tagName.toLowerCase();
|
||||
const attrs = (Array.from(el.attributes) as any[])
|
||||
.map((a) => `${a.name.toLowerCase()}="${escapeAttr(a.value ?? "")}"`)
|
||||
.sort();
|
||||
const attrStr = attrs.length ? ` ${attrs.join(" ")}` : "";
|
||||
if (VOID.has(tag)) return `<${tag}${attrStr}>`;
|
||||
const children = (Array.from(el.childNodes) as any[]).map(serializeNode).join("");
|
||||
return `<${tag}${attrStr}>${children}</${tag}>`;
|
||||
}
|
||||
|
||||
function serializeNode(node: any): string {
|
||||
if (node.nodeType === TEXT_NODE) {
|
||||
const t = node.textContent ?? "";
|
||||
// Drop whitespace-only text nodes (inter-tag indentation). Meaningful text
|
||||
// is collapsed to single-spaced and escaped — any real text edit moves it.
|
||||
if (/^\s*$/.test(t)) return "";
|
||||
return escapeText(t.replace(/\s+/g, " "));
|
||||
}
|
||||
if (node.nodeType === ELEMENT_NODE) return serializeElement(node);
|
||||
return ""; // comments, processing instructions — not content
|
||||
}
|
||||
|
||||
/**
|
||||
* Canonicalize an HTML fragment into a deterministic string. Two inputs that
|
||||
* parse to the same DOM tree (modulo the drift sources above) produce the same
|
||||
* canonical string; a content change produces a different one. Empty/null/
|
||||
* undefined → "". Compact, single-line canonical HTML suitable for hashing.
|
||||
*/
|
||||
export function canonicalizeHtml(html: string | null | undefined): string {
|
||||
if (!html || !html.trim()) return "";
|
||||
const { document } = parseHTML(`<div>${html}</div>`);
|
||||
const root = document.querySelector("div");
|
||||
if (!root) return "";
|
||||
return (Array.from(root.childNodes) as any[]).map(serializeNode).join("");
|
||||
}
|
||||
150
src/cchash.ts
Normal file
150
src/cchash.ts
Normal file
@@ -0,0 +1,150 @@
|
||||
// E1b-alt — ccHash compute wrapper with the HTML-hash contract (the E1a NO-GO
|
||||
// fork).
|
||||
//
|
||||
// E1a proved the markdown round-trip is unstable (wikilinks/@UUID, tables,
|
||||
// secrets order/case, parseBody bold bug — see
|
||||
// docs/prds/prd-foundry-obsidian-sync-2026-06-22/e1a-spike-findings.md). E1b-alt
|
||||
// hashes the Foundry HTML directly instead of round-tripping through markdown.
|
||||
// Both the baseline (`foundry.ccHash`, stored at push time) and the live ccHash
|
||||
// hash the SAME HTML from the live entry → comparable by construction; a
|
||||
// Foundry-side content change → different DOM → different canonical HTML →
|
||||
// different hash. No inverse, no resolver, no blank-line/case/order sensitivity.
|
||||
//
|
||||
// CONTRACT (frozen):
|
||||
// ccHash = contentHash(
|
||||
// canonicalizeHtml(data.description) + "\n" +
|
||||
// canonicalizeHtml(data.notes ?? "") + "\n" +
|
||||
// name + "\n" + folder
|
||||
// )
|
||||
// where `data = flags["campaign-codex"].data` (a CcData object — the body spans
|
||||
// `data.description`, the two-column body HTML, and `data.notes`, the ## Secrets
|
||||
// body HTML), `name = liveEntry.name`, `folder = liveEntry.folder ?? ""`.
|
||||
//
|
||||
// `canonicalizeHtml` (src/canonicalize-html.ts) absorbs incidental serialization
|
||||
// drift (attribute order/quoting, entities, inter-tag whitespace, tag case,
|
||||
// self-closing) so the hash is stable across relay /get calls for an unchanged
|
||||
// entry. The final `contentHash` canonicalizes the whole string (wikilinks +
|
||||
// whitespace), so `name`/`folder` whitespace drift is normalized too.
|
||||
//
|
||||
// DIRECTION-INVARIANCE: `name` and `folder` are ALWAYS sourced from the
|
||||
// JournalEntry (`liveEntry.name`, `liveEntry.folder`), NEVER from the Obsidian
|
||||
// filename or vault-relative folder. A vault rename changes the filename but NOT
|
||||
// `foundry.ccHash` until a push updates the live entry's `name` — correct,
|
||||
// because a rename is a name-field update routed through `pushNote`'s
|
||||
// `updatedName` path, not a content divergence (see E3.5).
|
||||
//
|
||||
// `folder` is `liveEntry.folder`, a Foundry FOLDER ID (e.g. `Folder.gideon`),
|
||||
// DISTINCT from the Obsidian `foundry.folder_path` field (a cc-type-derived
|
||||
// path via `folderPathFromCcType`). Do not conflate them — both ccHash sides use
|
||||
// `liveEntry.folder`, so direction-invariance holds; a Foundry folder MOVE
|
||||
// changes `liveEntry.folder` → ccHash changes → detected as F-changed (correct).
|
||||
//
|
||||
// This module does NOT wire itself into `AutoSyncController.process` or
|
||||
// `baselineFoundryBlock` — that wiring is E1b's job. It does NOT depend on
|
||||
// `src/fromFoundry.ts` (the E1a markdown inverse, shipped unwired). E1b-alt only
|
||||
// delivers the frozen primitive + the canonicalizeHtml seam + tests.
|
||||
|
||||
import type { JournalEntry, CcData } from "./types.js";
|
||||
import type { RelayClient } from "./relay/client.js";
|
||||
import { contentHash } from "./normalize.js";
|
||||
import { canonicalizeHtml } from "./canonicalize-html.js";
|
||||
|
||||
/**
|
||||
* The canonicalizer seam: Foundry HTML → canonical HTML string. Typed as an
|
||||
* EXPLICIT parameter (default `canonicalizeHtml` from src/canonicalize-html.ts)
|
||||
* so the contract boundary is frozen and testable. E1b wires the default; tests
|
||||
* may inject a stub for unit isolation.
|
||||
*/
|
||||
export type CanonicalizeHtml = (html: string) => string;
|
||||
|
||||
/**
|
||||
* The frozen hash input contract, as a canonical string template. Pinned by a
|
||||
* unit test (exact bytes) AND by a re-derivation test (the implementation is
|
||||
* asserted to compute exactly this) so any drift — to the constant OR to the
|
||||
* implementation — is a deliberate, reviewable change. This is the frozen
|
||||
* contract E1b and E2 code against.
|
||||
*/
|
||||
export const CC_HASH_CONTRACT =
|
||||
'contentHash(canonicalizeHtml(data.description) + "\\n" + canonicalizeHtml(data.notes ?? "") + "\\n" + name + "\\n" + folder)';
|
||||
|
||||
/** Typed error so E1b's divergence guard can distinguish "no Foundry-side
|
||||
* content yet" (treat as fresh/seed) from "content changed" / relay errors. */
|
||||
export class CcHashError extends Error {
|
||||
readonly kind = "CcHashError";
|
||||
constructor(message: string) {
|
||||
super(message);
|
||||
this.name = "CcHashError";
|
||||
}
|
||||
}
|
||||
|
||||
/** Whether a value is a CcHashError (narrowing helper for consumers). */
|
||||
export function isCcHashError(e: unknown): e is CcHashError {
|
||||
return e instanceof CcHashError;
|
||||
}
|
||||
|
||||
/** Extract and validate `flags["campaign-codex"].data.description`. Throws a
|
||||
* typed CcHashError when the flag, its data, OR its `description` field is
|
||||
* absent/non-string — `description` is the required body field, and silently
|
||||
* coercing a malformed entry to "" would create a stable-but-wrong baseline. */
|
||||
function extractCampaignCodexData(entry: JournalEntry): { data: CcData; description: string } {
|
||||
const cc = entry.flags?.["campaign-codex"];
|
||||
if (!cc || !cc.data) {
|
||||
throw new CcHashError('missing campaign-codex data');
|
||||
}
|
||||
if (typeof cc.data.description !== "string") {
|
||||
throw new CcHashError('missing campaign-codex data (description)');
|
||||
}
|
||||
return { data: cc.data, description: cc.data.description };
|
||||
}
|
||||
|
||||
/**
|
||||
* Compute the Foundry-side ccHash for a live `/get` entry. See `CC_HASH_CONTRACT`
|
||||
* for the frozen input. `canonicalize` defaults to the built-in
|
||||
* `canonicalizeHtml` (src/canonicalize-html.ts); pass a stub for unit isolation.
|
||||
*
|
||||
* Throws `CcHashError` when `flags["campaign-codex"].data` (or its
|
||||
* `description`) is absent — so callers can distinguish "no Foundry-side
|
||||
* content yet" from a real content change. Relay connectivity failures are NOT
|
||||
* wrapped here (see `ccHashFromGet`).
|
||||
*/
|
||||
export function ccHash(liveEntry: JournalEntry, canonicalize: CanonicalizeHtml = canonicalizeHtml): string {
|
||||
const { data, description } = extractCampaignCodexData(liveEntry);
|
||||
const notes = typeof data.notes === "string" ? data.notes : "";
|
||||
const name = liveEntry.name ?? "";
|
||||
const folder = liveEntry.folder ?? "";
|
||||
const text = `${canonicalize(description)}\n${canonicalize(notes)}\n${name}\n${folder}`;
|
||||
return contentHash(text);
|
||||
}
|
||||
|
||||
/** Result of `ccHashFromGet`: both the hash AND the live entry. */
|
||||
export interface CcHashFromGetResult {
|
||||
hash: string;
|
||||
entry: JournalEntry;
|
||||
}
|
||||
|
||||
/**
|
||||
* Fetch a live entry via `relay.getEntry(uuid)` AND derive its ccHash in one
|
||||
* call — for callers that do NOT already hold the entry (e.g. E2's deep-poll,
|
||||
* which fetches to compare). Returns both so the caller can use the entry for
|
||||
* the pull conversion without a second round-trip.
|
||||
*
|
||||
* Callers that ALREADY have the entry (notably `pushNote`, which fetches via
|
||||
* `relay.getEntry` at src/push.ts:142) must NOT use this helper — that would
|
||||
* make a SECOND `/get` and violate the FR-1.4 "no extra /get" ground rule. They
|
||||
* should call `ccHash(entry)` directly on the entry they already hold.
|
||||
*
|
||||
* Relay connectivity failures (`404 "Invalid client ID"`, `404 "No connected
|
||||
* Foundry clients found"`, timeouts, network errors) are surfaced UNCHANGED:
|
||||
* this helper does NOT wrap them as `CcHashError`. Only a present-but-malformed
|
||||
* entry (missing `flags["campaign-codex"].data` or its `description`) throws
|
||||
* `CcHashError`, after the relay call has succeeded.
|
||||
*/
|
||||
export async function ccHashFromGet(
|
||||
relay: RelayClient,
|
||||
uuid: string,
|
||||
canonicalize: CanonicalizeHtml = canonicalizeHtml,
|
||||
): Promise<CcHashFromGetResult> {
|
||||
const entry = await relay.getEntry(uuid); // throws relay errors unchanged
|
||||
const hash = ccHash(entry, canonicalize); // throws CcHashError on malformed entry
|
||||
return { hash, entry };
|
||||
}
|
||||
@@ -193,7 +193,7 @@ export async function cmdUi(opts: CliOptions): Promise<void> {
|
||||
outDir: out,
|
||||
mode: opts.mode,
|
||||
port: opts.port ?? 7788,
|
||||
host: opts.host ?? "0.0.0.0",
|
||||
host: opts.host ?? "127.0.0.1", // E7.2: safe-by-default (localhost). --host 0.0.0.0 exposes on the tailnet (needs DASHBOARD_AUTH_TOKEN when ENABLE_AUTH_MIDDLEWARE=on).
|
||||
relayCfg: relayCfg.apiKey ? relayCfg : undefined,
|
||||
foundryCfg: foundryCfg.dataDir ? foundryCfg : undefined,
|
||||
};
|
||||
|
||||
@@ -76,10 +76,23 @@
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<div id="loginCard" class="modal-bg" style="display:none;position:fixed;inset:0;background:rgba(0,0,0,0.4);z-index:100">
|
||||
<div class="modal entry" style="max-width:380px;margin:8vh auto 0;padding:18px">
|
||||
<h2 style="margin-top:0">Dashboard auth required</h2>
|
||||
<p class="meta">This dashboard is bound to <code id="loginBound">0.0.0.0</code> and requires a token. Enter the <code>DASHBOARD_AUTH_TOKEN</code> you set in your <code>.env</code>.</p>
|
||||
<input id="loginToken" type="password" placeholder="DASHBOARD_AUTH_TOKEN" style="width:100%;box-sizing:border-box;margin:8px 0" onkeydown="if(event.key==='Enter')doLogin()">
|
||||
<div id="loginErr" class="meta" style="color:var(--bad);min-height:1em"></div>
|
||||
<div style="display:flex;gap:8px;justify-content:flex-end">
|
||||
<button class="primary" onclick="doLogin()">Log in</button>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<header>
|
||||
<h1>Foundry ⇄ Obsidian merge</h1>
|
||||
<div class="counts" id="counts">loading…</div>
|
||||
<span class="mode-tag" id="modeTag">dev</span>
|
||||
<button id="syncModeBtn" onclick="toggleSyncMode()" style="display:none" title="PREP = curating/seeding/linking (auto-sync blocked). RUN-THE-MATCH = auto-sync unblocked (pushes vault edits to live Foundry)."></button>
|
||||
<span class="mode-tag" id="presenceChip" style="display:none" title="Relay / Foundry host configuration presence (masked — no secrets shown)."></span>
|
||||
<button onclick="refreshIndex()" title="Re-scan the vault + cc from disk so edits made in Obsidian show as changed. Also happens automatically when you switch back to this tab.">Re-scan</button>
|
||||
<div class="spacer"></div>
|
||||
<label><input type="checkbox" id="dryRun" checked /> dry-run</label>
|
||||
@@ -89,7 +102,28 @@
|
||||
<button onclick="act('importAll')" title="Pull every cc-only Foundry entry (not yet in your vault) into a new refined note under refined/imported/<folder>. Un-curated staging for review.">Import all</button>
|
||||
<button onclick="pushAll()" title="Push every vault-newer (sync→cc) note into the LIVE Foundry world via the relay in one run. dry-run (default) lists what would be pushed; uncheck dry-run to apply — each live entry is backed up to <out>/bak first and the note's foundry.contentHash is baselined so a re-run only catches new edits. Replaces scripts/resync.ts.">Push all changed</button>
|
||||
<button onclick="refreshLive()" title="Rebuild the cached name↔uuid map (for link resolution in pushes) via the relay /search — zero Foundry downtime. The heavy docker-stop full index is CLI-only: npx tsx src/cli.ts refresh --full-index.">Refresh live index</button>
|
||||
<button id="autoSyncBtn" onclick="toggleAutosync()" title="Toggle Obsidian→Foundry auto-sync. ON = saving a linked, seeded note in your vault pushes it into LIVE Foundry instantly (guarded by the note's foundry.contentHash baseline, so no-op saves and the post-push baseline write don't re-push; unlinked/unseeded notes are skipped). Foundry→Obsidian is NOT auto — use Sync / Re-pull for that. Needs the relay (RELAY_API_KEY). Dev mode baselines land in the --out mirror; apply mode in the real vault.">Auto-sync: off</button>
|
||||
</header>
|
||||
<div id="syncHeader" style="display:none;padding:4px 12px;border-bottom:1px solid var(--line);font-size:12px;color:var(--mut);display:flex;gap:14px;align-items:center;flex-wrap:wrap">
|
||||
<span id="syncOnOff"></span>
|
||||
<span id="syncModeDisplay"></span>
|
||||
<span id="syncWatched"></span>
|
||||
<span id="syncLastAt"></span>
|
||||
<span id="parityIndicator" class="badge" style="cursor:default"></span>
|
||||
</div>
|
||||
<div id="fPendingPanel" style="display:none;margin:4px 12px;padding:6px 10px;border:1px solid var(--pur);border-radius:6px;background:var(--panel);font-size:12px"></div>
|
||||
<div id="devBanner" class="badge warn" style="display:none;margin:6px 12px;padding:6px 10px">Dev mode — auto-sync disabled; pushes would target the --out mirror, not live Foundry. Start the server with --apply to enable.</div>
|
||||
<div id="syncPausedBanner" class="badge bad" style="display:none;margin:6px 12px;padding:8px 10px;font-size:13px">SYNC PAUSED — auto-sync is off, vault edits are NOT being pushed to Foundry. <button class="primary" style="margin-left:8px" onclick="toggleAutosync()">Resume auto-sync</button></div>
|
||||
<div id="migrationBanner" class="badge ok" style="display:none;margin:6px 12px;padding:6px 10px;cursor:pointer" title="Click to dismiss. The startup migration stamped foundry.flagsSchemaVersion on notes that lacked it (contentHash/ccHash untouched)."></div>
|
||||
<details id="autoSyncPanel" class="autosync-panel" open style="display:none;margin:0 12px;border-top:1px solid #ddd;padding:6px 12px;background:#fafafa">
|
||||
<summary style="cursor:pointer">Auto-sync activity <span id="autoSyncCounts" class="meta"></span><span id="autoSyncNote" class="meta" style="margin-left:8px"></span></summary>
|
||||
<div id="revertBar" style="margin:6px 0;display:none"><button id="revertBtn" class="bad" onclick="revertLastPush()" title="Restore Foundry to the state captured BEFORE the most recent auto-sync push (a full /update), then re-baseline the note. Use this to undo a wrong push."></button></div>
|
||||
<pre id="autoSyncLog" class="autosync-log" style="max-height:180px;overflow:auto;margin:6px 0 8px;font-size:12px;background:#fff;border:1px solid #eee;padding:6px">(no activity yet)</pre>
|
||||
</details>
|
||||
<details id="liveNewEntriesPanel" style="display:none;margin:0 12px;border:1px solid var(--pur);border-radius:6px;padding:6px 12px;background:var(--panel)">
|
||||
<summary style="cursor:pointer;color:var(--pur)">Live new entries (from Foundry) <span id="liveNewCount" class="meta"></span><button id="catchUpBtn" onclick="catchUpNow()" style="margin-left:8px;font-size:11px" title="Force an immediate shallow + deep sweep so the vault reflects everything you just changed in Foundry.">Catch up now</button></summary>
|
||||
<div id="liveNewList" style="margin:6px 0;font-size:13px"></div>
|
||||
</details>
|
||||
<main>
|
||||
<section class="list">
|
||||
<div class="toolbar">
|
||||
@@ -118,9 +152,139 @@
|
||||
<section class="detail" id="detail">Select a row to inspect.</section>
|
||||
</main>
|
||||
<script>
|
||||
let INDEX = null, STATUS = null, SEL = null, REC_FILTER = null;
|
||||
let INDEX = null, STATUS = null, SEL = null, REC_FILTER = null, AUTO = null, autoPoll = null, migrationDismissed = false, AUTH_REQUIRED = false, AUTH_STATUS = null, SYNC_STATE = null;
|
||||
const dryEl = () => document.getElementById('dryRun');
|
||||
|
||||
// E7.2: shared request wrapper. Attaches the stored auth token (set on login) as
|
||||
// a Bearer header; on a 401 (token expired / wrong / public bind), shows the
|
||||
// login card so the user re-authenticates instead of silently failing.
|
||||
function authToken() { return localStorage.getItem('ofs_token') || ''; }
|
||||
function csrfToken() { return localStorage.getItem('ofs_csrf') || ''; }
|
||||
async function apiFetch(path, init = {}) {
|
||||
const tok = authToken();
|
||||
const headers = { ...(init.headers || {}) };
|
||||
if (tok) headers.authorization = `Bearer ${tok}`;
|
||||
// E7.4: attach the CSRF token to mutation (POST) requests.
|
||||
if (csrfToken() && (init.method || 'GET') !== 'GET') headers['x-csrf-token'] = csrfToken();
|
||||
init = { ...init, headers };
|
||||
const r = await fetch(path, init);
|
||||
if (r.status === 401) { showLogin(); throw new Error('unauthorized'); }
|
||||
return r;
|
||||
}
|
||||
function showLogin() {
|
||||
const bound = document.getElementById('loginBound');
|
||||
if (bound && STATUS && STATUS.bound) bound.textContent = STATUS.bound;
|
||||
document.getElementById('loginCard').style.display = '';
|
||||
document.getElementById('loginToken').focus();
|
||||
}
|
||||
function hideLogin() { document.getElementById('loginCard').style.display = 'none'; }
|
||||
async function doLogin() {
|
||||
const tok = document.getElementById('loginToken').value.trim();
|
||||
if (!tok) { document.getElementById('loginErr').textContent = 'enter the token'; return; }
|
||||
const r = await fetch('/api/auth/login', { method: 'POST', headers: { 'content-type': 'application/json' }, body: JSON.stringify({ token: tok }) }).then(r => r.json()).catch(() => null);
|
||||
if (r && r.ok) { localStorage.setItem('ofs_token', tok); hideLogin(); document.getElementById('loginErr').textContent = ''; init(); }
|
||||
else { document.getElementById('loginErr').textContent = (r && r.error) ? r.error : 'invalid credentials'; }
|
||||
}
|
||||
async function doLogout() { localStorage.removeItem('ofs_token'); await fetch('/api/auth/logout', { method: 'POST' }).catch(() => {}); showLogin(); }
|
||||
// E7.3: masked presence chip — "Relay: ✓ / ✗ · Foundry: ✓ / ✗" driven by
|
||||
// /api/auth/status booleans (never env values / secrets).
|
||||
function renderPresenceChip() {
|
||||
const chip = document.getElementById('presenceChip');
|
||||
if (!chip || !AUTH_STATUS) { if (chip) chip.style.display = 'none'; return; }
|
||||
const r = AUTH_STATUS.relayConfigured ? 'Relay ✓' : 'Relay ✗';
|
||||
const f = AUTH_STATUS.foundryConfigured ? 'Foundry ✓' : 'Foundry ✗';
|
||||
chip.textContent = `${r} · ${f}`;
|
||||
chip.style.color = AUTH_STATUS.relayConfigured ? 'var(--ok)' : 'var(--warn)';
|
||||
chip.style.display = '';
|
||||
}
|
||||
// E4.2: PREP ⇄ RUN-THE-MATCH mode toggle (gated by features.syncStatus). In PREP,
|
||||
// the auto-sync button is disabled (auto-sync is blocked — curate/seed/link first).
|
||||
function renderSyncMode() {
|
||||
const btn = document.getElementById('syncModeBtn');
|
||||
const autoBtn = document.getElementById('autoSyncBtn');
|
||||
if (!btn || !STATUS || !STATUS.featuresSyncStatus) { if (btn) btn.style.display = 'none'; return; }
|
||||
const mode = STATUS.syncMode || 'PREP';
|
||||
btn.textContent = `Mode: ${mode === 'RUN-THE-MATCH' ? 'RUN' : 'PREP'}`;
|
||||
btn.classList.toggle('primary', mode === 'RUN-THE-MATCH');
|
||||
btn.style.display = '';
|
||||
if (autoBtn) {
|
||||
const blocked = mode === 'PREP';
|
||||
autoBtn.disabled = blocked;
|
||||
autoBtn.title = blocked ? 'Switch to RUN-THE-MATCH mode first (auto-sync is blocked in PREP)' : autoBtn.title.replace('Switch to RUN-THE-MATCH mode first (auto-sync is blocked in PREP)\n', '');
|
||||
}
|
||||
}
|
||||
async function toggleSyncMode() {
|
||||
if (!STATUS || !STATUS.syncMode) return;
|
||||
const next = STATUS.syncMode === 'PREP' ? 'RUN-THE-MATCH' : 'PREP';
|
||||
toast(`switching to ${next}…`);
|
||||
const r = await apiFetch('/api/sync-state/mode', { method: 'POST', headers: { 'content-type': 'application/json' }, body: JSON.stringify({ mode: next }) }).then(r => r.json()).catch(() => null);
|
||||
if (r && r.mode) { toast(`mode → ${r.mode}`); init(); }
|
||||
else toast(`mode switch failed: ${r?.error || 'unknown'}`);
|
||||
}
|
||||
// E4.3: sync-status header + parity indicator (driven by /api/sync-state).
|
||||
async function refreshSyncState() {
|
||||
if (!STATUS || !STATUS.featuresSyncStatus) return; // gated by features.syncStatus
|
||||
const s = await apiFetch('/api/sync-state').then(r => r.json()).catch(() => null);
|
||||
if (!s) return;
|
||||
SYNC_STATE = s;
|
||||
document.getElementById('syncOnOff').textContent = `Sync: ${s.autoSyncOn ? 'ON' : 'OFF'}`;
|
||||
document.getElementById('syncModeDisplay').textContent = `Mode: ${s.mode}`;
|
||||
document.getElementById('syncWatched').textContent = `Watching: ${s.watchedDir}`;
|
||||
document.getElementById('syncLastAt').textContent = `Last sync: ${s.lastSyncAt ? s.lastSyncAt.replace('T',' ').slice(0,19) : 'never'}`;
|
||||
const p = s.parity || {};
|
||||
const badge = document.getElementById('parityIndicator');
|
||||
badge.textContent = `${p.status || 'in-parity'} · O:${p.oPending||0} F:${p.fPending||0} C:${p.conflict||0} U:${p.unsyncedLinked||0}`;
|
||||
badge.className = 'badge ' + (p.status === 'conflict' ? 'bad' : p.status === 'O-pending' ? 'acc' : p.status === 'F-pending' ? 'pur' : p.status === 'unsynced-linked' ? 'warn' : 'ok');
|
||||
// F-pending badge clickable → "Incoming F→O changes" list (E2 populates the
|
||||
// entries; until then fPending=0 and the list stays hidden — the demo hook is
|
||||
// latent, activates when E2's poll lands).
|
||||
const fPanel = document.getElementById('fPendingPanel');
|
||||
if (p.fPending > 0) {
|
||||
badge.style.cursor = 'pointer';
|
||||
badge.onclick = () => { fPanel.style.display = fPanel.style.display === 'none' ? '' : 'none'; };
|
||||
const entries = Array.isArray(s.fPending) ? s.fPending : [];
|
||||
fPanel.innerHTML = entries.length
|
||||
? `<b>Incoming F→O changes (${p.fPending})</b><br>` + entries.map(e => `• ${e.name} — ${e.change} (${e.detectedAt || '?'})`).join('<br>')
|
||||
: `<b>F-pending: ${p.fPending}</b> — details populate when the F→O poll (E2) is active`;
|
||||
} else {
|
||||
badge.style.cursor = 'default';
|
||||
badge.onclick = null;
|
||||
fPanel.style.display = 'none';
|
||||
}
|
||||
document.getElementById('syncHeader').style.display = 'flex';
|
||||
// E4.4: SYNC PAUSED banner — mode=RUN && autoSyncOn=false → loud, persistent
|
||||
// (not a toast). In PREP, the banner is NOT shown (PREP is "not available", not
|
||||
// "paused" — the syncModeBtn + disabled autoSyncBtn handle that).
|
||||
document.getElementById('syncPausedBanner').style.display = (s.mode === 'RUN-THE-MATCH' && !s.autoSyncOn) ? '' : 'none';
|
||||
// E4.6: activity panel (last 200) from sync-state.json.activity — replaces the
|
||||
// in-memory events view when features.syncStatus is on. Counts derived from activity.
|
||||
const log = document.getElementById('autoSyncLog');
|
||||
if (log) {
|
||||
log.textContent = (s.activity && s.activity.length)
|
||||
? s.activity.map(e => `${e.time.replace('T',' ').slice(5,19)} ${(e.kind||e.status||'').padEnd(7)} ${e.name} — ${e.message}`).join('\n')
|
||||
: '(no activity yet — save a linked, seeded note in your vault to trigger a push)';
|
||||
}
|
||||
const counts = document.getElementById('autoSyncCounts');
|
||||
if (counts && s.activity) {
|
||||
const pushed = s.activity.filter(e => e.kind === 'push').length;
|
||||
const skipped = s.activity.filter(e => e.kind === 'skip').length;
|
||||
const errors = s.activity.filter(e => e.kind === 'error').length;
|
||||
counts.textContent = `pushed ${pushed} · skipped ${skipped} · errors ${errors}`;
|
||||
}
|
||||
}
|
||||
async function checkAuth() {
|
||||
// E7.3: /api/auth/status is the only always-open data endpoint. Fetch it first;
|
||||
// if auth is required and no token is stored, show the login card and DON'T
|
||||
// fetch /api/status (which is gated in public mode and would 401).
|
||||
const a = await fetch('/api/auth/status').then(r => r.json()).catch(() => null);
|
||||
AUTH_STATUS = a; // E7.3: masked presence (relayConfigured/foundryConfigured) for the chip
|
||||
if (a && a.authRequired && !authToken()) { AUTH_REQUIRED = true; showLogin(); return false; }
|
||||
STATUS = await apiFetch('/api/status').then(r => r.json()).catch(() => null);
|
||||
if (STATUS && a) STATUS.bound = a.bound; // merge the bind address for showLogin
|
||||
renderPresenceChip();
|
||||
return true;
|
||||
}
|
||||
|
||||
// Recommendation -> display label, badge class, bulk op, and one-line guidance.
|
||||
// `tag` is the short status noun shown on each row (a state, not an action — so it
|
||||
// doesn't read like a button); `label` is the fuller heading used in the rec panel.
|
||||
@@ -137,16 +301,29 @@ const REC = {
|
||||
const REC_ORDER = ['import','seed','sync-cc','repull','conflict','in-sync','review'];
|
||||
|
||||
async function init() {
|
||||
STATUS = await fetch('/api/status').then(r => r.json());
|
||||
// E7.2: gate on auth — if the dashboard requires a token and none is stored,
|
||||
// show the login card and stop (don't load the dashboard behind it).
|
||||
if (!(await checkAuth())) return;
|
||||
// E7.4: fetch a per-session CSRF token (stores the JS-readable mirror; the
|
||||
// HttpOnly cookie is sent automatically on same-origin POSTs).
|
||||
const csrf = await apiFetch('/api/auth/csrf').then(r => r.json()).catch(() => null);
|
||||
if (csrf && csrf.csrfToken) localStorage.setItem('ofs_csrf', csrf.csrfToken);
|
||||
// E4.3: sync-status header + parity indicator (gated by features.syncStatus).
|
||||
refreshSyncState();
|
||||
const tag = document.getElementById('modeTag');
|
||||
tag.textContent = STATUS.mode + (STATUS.mode === 'apply' ? '' : ' (safe)');
|
||||
if (STATUS.mode === 'apply') tag.classList.add('apply');
|
||||
INDEX = await fetch('/api/index').then(r => r.json());
|
||||
// E1b.5: dev-mode banner — auto-sync is disabled in dev mode (apply-mode floor).
|
||||
document.getElementById('devBanner').style.display = STATUS.mode === 'dev' ? '' : 'none';
|
||||
// E4.2: PREP/RUN mode toggle (gated by features.syncStatus).
|
||||
renderSyncMode();
|
||||
INDEX = await apiFetch('/api/index').then(r => r.json());
|
||||
const c = INDEX.counts;
|
||||
document.getElementById('counts').innerHTML =
|
||||
`matched <b>${c.matched}</b> · cc-only <b>${c.ccOnly}</b> · refined-only <b>${c.refinedOnly}</b> · unlinked <b>${c.unlinked}</b>`;
|
||||
renderRecPanel();
|
||||
render();
|
||||
refreshAutosync();
|
||||
}
|
||||
function esc(s){ return (s==null?'':String(s)).replace(/[&<>]/g,c=>({'&':'&','<':'<','>':'>'}[c])); }
|
||||
function attr(s){ return (s==null?'':String(s)).replace(/[&"]/g,c=>({'&':'&','"':'"'}[c])); }
|
||||
@@ -249,7 +426,7 @@ async function select(name){
|
||||
SEL = name; render();
|
||||
const d = document.getElementById('detail');
|
||||
d.innerHTML = `loading ${esc(name)}…`;
|
||||
const f = await fetch('/api/file?name=' + encodeURIComponent(name)).then(r => r.json());
|
||||
const f = await apiFetch('/api/file?name=' + encodeURIComponent(name)).then(r => r.json());
|
||||
const r = f.row;
|
||||
const m = REC[r.recommendation] || REC['review'];
|
||||
const parts = [];
|
||||
@@ -263,6 +440,40 @@ async function select(name){
|
||||
if (f.syncPreview != null && f.cc != null) parts.push(`<div class="panel"><h3>sync preview — cc side <small style="font-weight:normal">(curation flows back; cc_sync_hash written)</small></h3>${diff(f.cc, f.syncPreview)}</div>`);
|
||||
if (f.repullPreview != null && f.refined != null) parts.push(`<div class="panel"><h3>re-pull preview <small style="font-weight:normal">(body from Foundry, curation preserved)</small></h3>${diff(f.refined, f.repullPreview)}</div>`);
|
||||
if (f.entry) parts.push(`<div class="panel"><h3>Foundry journal entry</h3><pre>${esc(JSON.stringify(f.entry,null,2))}</pre></div>`);
|
||||
// E3.1: conflict panel (side-by-side diff + plain-language summary) when the
|
||||
// row is a conflict AND the E3_CONFLICT_UX flag is on. Read-only — the three
|
||||
// action buttons are disabled with title="coming next" (wired in E3.2).
|
||||
if (r.recommendation === 'conflict' && STATUS && STATUS.conflictUx) {
|
||||
const vaultBody = f.refined != null ? f.refined : null;
|
||||
const foundryBody = f.cc != null ? f.cc : null;
|
||||
const entryName = f.entry ? f.entry.name : null;
|
||||
const summary = conflictSummary(vaultBody, foundryBody, entryName, r.name);
|
||||
const vaultCol = vaultBody != null
|
||||
? `<pre style="max-height:300px;overflow:auto">${esc(vaultBody)}</pre>`
|
||||
: '<p class="meta" style="color:var(--bad)">vault file missing</p>';
|
||||
const foundryCol = foundryBody != null
|
||||
? `<pre style="max-height:300px;overflow:auto">${esc(foundryBody)}</pre>`
|
||||
: '<p class="meta" style="color:var(--bad)">Foundry export missing</p>';
|
||||
const actions = (vaultBody != null && foundryBody != null)
|
||||
? `<div style="display:flex;gap:8px;margin-top:8px;flex-wrap:wrap">
|
||||
<button onclick="resolveConflict('${attr(name)}','push-vault')"
|
||||
title="Writes vault body → live Foundry entry (relay /update); re-baselines foundry.contentHash + syncedAt. Foundry side content is overwritten.">Push vault → Foundry</button>
|
||||
<button onclick="resolveConflict('${attr(name)}','pull-foundry')"
|
||||
title="Writes Foundry body → vault note (re-pull); re-baselines foundry.contentHash. Vault side body is overwritten (curation preserved).">Pull Foundry → vault</button>
|
||||
<button class="danger" onclick="resolveConflict('${attr(name)}','keep-divergence')"
|
||||
title="No content is transferred. Re-baselines foundry.contentHash to the current vault body hash. Both sides keep their own text.">Accept both as-is (keep divergence)</button>
|
||||
</div>`
|
||||
: '<p class="meta">One side is missing — no resolution actions available.</p>';
|
||||
parts.push(`<div class="panel" style="border:1px solid var(--bad);border-radius:6px;padding:10px">
|
||||
<h3 style="color:var(--bad)">Both sides changed since last sync</h3>
|
||||
<p class="meta" style="margin-bottom:8px">${esc(summary)}</p>
|
||||
<div style="display:grid;grid-template-columns:1fr 1fr;gap:8px">
|
||||
<div><h4 style="margin:0 0 4px">vault (left)</h4>${vaultCol}</div>
|
||||
<div><h4 style="margin:0 0 4px">Foundry (right)</h4>${foundryCol}</div>
|
||||
</div>
|
||||
${actions}
|
||||
</div>`);
|
||||
}
|
||||
d.innerHTML = parts.join('\n');
|
||||
}
|
||||
function diff(a, b){
|
||||
@@ -270,10 +481,49 @@ function diff(a, b){
|
||||
const aSet = new Set(al), bSet = new Set(bl);
|
||||
const lines = [];
|
||||
for (const l of al) if (!bSet.has(l)) lines.push(`<span class="diff-del">- ${esc(l)}</span>`);
|
||||
for (const l of bl) if (!aSet.has(l)) lines.push(`<span class="diff-add">+ ${esc(l)}</span>`);
|
||||
for (const l in bl) if (!aSet.has(l)) lines.push(`<span class="diff-add">+ ${esc(l)}</span>`);
|
||||
if (!lines.length) return '<pre>(identical)</pre>';
|
||||
return '<pre>' + lines.join('\n') + '</pre>';
|
||||
}
|
||||
// E3.1: ordered line diff (LCS-based) for the conflict panel. Unlike the legacy
|
||||
// diff() (set-based, drops ordering), this preserves line order so moved lines
|
||||
// aren't mis-reported as del+add.
|
||||
function conflictDiff(a, b) {
|
||||
const al = (a || '').split('\n'), bl = (b || '').split('\n');
|
||||
// LCS table.
|
||||
const dp = Array.from({ length: al.length + 1 }, () => new Array(bl.length + 1).fill(0));
|
||||
for (let i = al.length - 1; i >= 0; i--)
|
||||
for (let j = bl.length - 1; j >= 0; j--)
|
||||
dp[i][j] = al[i] === bl[j] ? dp[i+1][j+1] + 1 : Math.max(dp[i+1][j], dp[i][j+1]);
|
||||
// Backtrack to produce the diff.
|
||||
const lines = [];
|
||||
let i = 0, j = 0;
|
||||
while (i < al.length && j < bl.length) {
|
||||
if (al[i] === bl[j]) { lines.push(` ${esc(al[i])}`); i++; j++; }
|
||||
else if (dp[i+1][j] >= dp[i][j+1]) { lines.push(`<span class="diff-del">- ${esc(al[i])}</span>`); i++; }
|
||||
else { lines.push(`<span class="diff-add">+ ${esc(bl[j])}</span>`); j++; }
|
||||
}
|
||||
while (i < al.length) { lines.push(`<span class="diff-del">- ${esc(al[i])}</span>`); i++; }
|
||||
while (j < bl.length) { lines.push(`<span class="diff-add">+ ${esc(bl[j])}</span>`); j++; }
|
||||
if (lines.length === 0 || (lines.length === 1 && lines[0] === ' ')) return '<pre>(identical)</pre>';
|
||||
return '<pre>' + lines.join('\n') + '</pre>';
|
||||
}
|
||||
// E3.1: plain-language conflict summary — names what EACH side did, not which wins.
|
||||
function conflictSummary(vaultBody, foundryBody, entryName, noteName) {
|
||||
const parts = [];
|
||||
if (vaultBody !== foundryBody) {
|
||||
const vLines = (vaultBody || '').split('\n').filter(l => l.trim());
|
||||
const fLines = (foundryBody || '').split('\n').filter(l => l.trim());
|
||||
const vSet = new Set(vLines), fSet = new Set(fLines);
|
||||
const vOnly = vLines.filter(l => !fSet.has(l)).length;
|
||||
const fOnly = fLines.filter(l => !vSet.has(l)).length;
|
||||
if (vOnly > 0) parts.push(`Vault edited body (${vOnly} line${vOnly > 1 ? 's' : ''} changed)`);
|
||||
if (fOnly > 0) parts.push(`Foundry edited body (${fOnly} line${fOnly > 1 ? 's' : ''} changed)`);
|
||||
}
|
||||
if (entryName && noteName && entryName !== noteName) parts.push(`Foundry renamed entry ("${esc(noteName)}" → "${esc(entryName)}")`);
|
||||
if (parts.length === 0) return 'Both sides changed since last sync (details below).';
|
||||
return parts.join('; ') + '.';
|
||||
}
|
||||
let toastT = null;
|
||||
function toast(msg){
|
||||
let el = document.querySelector('.toast');
|
||||
@@ -287,12 +537,12 @@ async function act(op, names){
|
||||
const body = { op, dryRun };
|
||||
if (names) body.names = names;
|
||||
toast(`${op} ${dryRun?'(dry-run)':'('+STATUS.mode+')'}…`);
|
||||
const r = await fetch('/api/action', {method:'POST', headers:{'content-type':'application/json'}, body: JSON.stringify(body)}).then(r=>r.json());
|
||||
const r = await apiFetch('/api/action', {method:'POST', headers:{'content-type':'application/json'}, body: JSON.stringify(body)}).then(r=>r.json());
|
||||
if (r.error){ toast('error: ' + r.error); return; }
|
||||
const wrote = (r.written||[]).length, prev = (r.preview||[]).length, skip = (r.skipped||[]).length;
|
||||
toast(r.message || `${op}: ${dryRun?prev:wrote} ${dryRun?'would write':'wrote'}${skip?', '+skip+' skipped':''}`);
|
||||
// Refresh index so recommendation counts update after an action.
|
||||
INDEX = await fetch('/api/index').then(r => r.json());
|
||||
INDEX = await apiFetch('/api/index').then(r => r.json());
|
||||
renderRecPanel(); render();
|
||||
if (SEL) select(SEL);
|
||||
}
|
||||
@@ -302,7 +552,7 @@ async function act(op, names){
|
||||
async function pushRow(name){
|
||||
const dryRun = dryEl().checked;
|
||||
toast(`push ${name} ${dryRun?'(dry-run)':'(apply)'}…`);
|
||||
const r = await fetch('/api/push', {method:'POST', headers:{'content-type':'application/json'}, body: JSON.stringify({name, dryRun})}).then(r=>r.json());
|
||||
const r = await apiFetch('/api/push', {method:'POST', headers:{'content-type':'application/json'}, body: JSON.stringify({name, dryRun})}).then(r=>r.json());
|
||||
if (r.error){ toast('error: ' + r.error); return; }
|
||||
if (r.dryRun){
|
||||
toast(`[dry-run] push ${name}: diff ready (${Object.keys(r.diff).length} keys)`);
|
||||
@@ -314,24 +564,143 @@ async function pushRow(name){
|
||||
// Rebuild the cached name↔uuid map via relay /search (zero Foundry downtime).
|
||||
async function refreshLive(){
|
||||
toast('refresh live index…');
|
||||
const r = await fetch('/api/refresh', {method:'POST', headers:{'content-type':'application/json'}, body: '{}'}).then(r=>r.json());
|
||||
const r = await apiFetch('/api/refresh', {method:'POST', headers:{'content-type':'application/json'}, body: '{}'}).then(r=>r.json());
|
||||
if (r.error){ toast('error: ' + r.error); return; }
|
||||
toast(`live index refreshed: ${r.pairs} name↔uuid pairs cached`);
|
||||
}
|
||||
// Auto-sync (Obsidian→Foundry, instant): the server watches the vault and pushes
|
||||
// saved notes into live Foundry. Toggle here; poll for the activity log while on.
|
||||
async function refreshAutosync(){
|
||||
const r = await apiFetch('/api/autosync').then(r=>r.json()).catch(()=>null);
|
||||
if (!r) return;
|
||||
AUTO = r;
|
||||
const btn = document.getElementById('autoSyncBtn');
|
||||
// E1b.3: TOCTOU conflict badge — Foundry was edited during a push window;
|
||||
// the DM must reconcile (Sync / Re-pull). Surfaced loudly on the button.
|
||||
const conflicts = r.conflictCount || 0;
|
||||
btn.textContent = `Auto-sync: ${r.enabled ? 'on' : 'off'}${conflicts ? ` ⚠ ${conflicts} conflict${conflicts > 1 ? 's' : ''}` : ''}`;
|
||||
btn.classList.toggle('primary', r.enabled);
|
||||
btn.classList.toggle('bad', conflicts > 0);
|
||||
const panel = document.getElementById('autoSyncPanel');
|
||||
panel.style.display = r.enabled ? '' : 'none';
|
||||
document.getElementById('autoSyncCounts').textContent =
|
||||
r.enabled ? `pushed ${r.pushed} · skipped ${r.skipped} · errors ${r.errors}` : '';
|
||||
document.getElementById('autoSyncNote').textContent =
|
||||
conflicts > 0
|
||||
? `⚠ ${conflicts} TOCTOU conflict${conflicts > 1 ? 's' : ''} — Foundry was edited during a push (the push is live but NOT baselined); use Sync / Re-pull to reconcile`
|
||||
: r.enabled && STATUS && STATUS.mode !== 'apply'
|
||||
? '(dev mode — baselines land in the --out mirror, not the real vault; run --apply to baseline the real vault)'
|
||||
: r.enabled
|
||||
? 'Auto-sync is opt-in per session — resets to OFF on restart'
|
||||
: '';
|
||||
const log = document.getElementById('autoSyncLog');
|
||||
log.textContent = r.events && r.events.length
|
||||
? r.events.map(e => `${e.time.replace('T',' ').slice(5,19)} ${e.status.padEnd(7)} ${e.name} — ${e.message}`).join('\n')
|
||||
: '(no activity yet — save a linked, seeded note in your vault to trigger a push)';
|
||||
if (r.enabled && !autoPoll) autoPoll = setInterval(() => { refreshAutosync(); refreshSyncState(); refreshLiveNewEntries(); }, 2000);
|
||||
if (!r.enabled && autoPoll) { clearInterval(autoPoll); autoPoll = null; }
|
||||
// E1b.8: flagsSchemaVersion migration banner (shown once after start, dismissible).
|
||||
const migBanner = document.getElementById('migrationBanner');
|
||||
if (r.migrationRan && r.migrationCount > 0 && !migrationDismissed) {
|
||||
migBanner.textContent = `Migrated ${r.migrationCount} note${r.migrationCount > 1 ? 's' : ''} to flagsSchemaVersion ${r.schemaVersion} (foundry.flagsSchemaVersion stamped; contentHash/ccHash untouched) — click to dismiss`;
|
||||
migBanner.style.display = '';
|
||||
migBanner.onclick = () => { migrationDismissed = true; migBanner.style.display = 'none'; };
|
||||
} else {
|
||||
migBanner.style.display = 'none';
|
||||
}
|
||||
// E1b.4: "Revert last push" button — shown when there's a recent push and
|
||||
// the guard is on (revert is meaningless without the per-uuid backups).
|
||||
const revertBar = document.getElementById('revertBar');
|
||||
const revertBtn = document.getElementById('revertBtn');
|
||||
if (r.lastPush && r.enabled) {
|
||||
revertBar.style.display = '';
|
||||
revertBtn.textContent = `Revert last push: ${r.lastPush.name}`;
|
||||
revertBtn.dataset.uuid = r.lastPush.uuid;
|
||||
} else {
|
||||
revertBar.style.display = 'none';
|
||||
}
|
||||
}
|
||||
// E2.5: live new entries (from Foundry) — one-click import, never auto.
|
||||
async function refreshLiveNewEntries() {
|
||||
const r = await apiFetch('/api/foundry-poll').then(r => r.json()).catch(() => null);
|
||||
const panel = document.getElementById('liveNewEntriesPanel');
|
||||
if (!r || r.enabled === undefined) { if (panel) panel.style.display = 'none'; return; }
|
||||
const entries = r.liveNewEntries || [];
|
||||
if (panel) panel.style.display = entries.length > 0 ? '' : 'none';
|
||||
const count = document.getElementById('liveNewCount');
|
||||
if (count) count.textContent = entries.length > 0 ? `(${entries.length})` : '';
|
||||
const list = document.getElementById('liveNewList');
|
||||
if (list) {
|
||||
list.innerHTML = entries.length === 0 ? '<p class="meta">No new entries from Foundry.</p>' :
|
||||
entries.map(e => `<div style="display:flex;gap:8px;align-items:center;padding:4px 0;border-bottom:1px solid var(--line)"><span style="flex:1">${e.name}</span><button class="rec" onclick="importLiveEntry('${e.uuid}','${e.name.replace(/'/g,"\\'")}')">Import as new refined note</button></div>`).join('');
|
||||
}
|
||||
}
|
||||
// E3.2: resolve a conflict with one of three actions. Confirm before commit.
|
||||
async function resolveConflict(name, action) {
|
||||
const previews = {
|
||||
'push-vault': 'Writes vault body → live Foundry entry (relay /update); re-baselines foundry.contentHash + syncedAt. Foundry side content is OVERWRITTEN.',
|
||||
'pull-foundry': 'Writes Foundry body → vault note (re-pull); re-baselines foundry.contentHash. Vault side body is OVERWRITTEN (curation preserved).',
|
||||
'keep-divergence': 'No content is transferred. Re-baselines foundry.contentHash to the current vault body hash. Both sides keep their own text — the divergence is acknowledged.',
|
||||
};
|
||||
if (!confirm(`${previews[action] || action}\n\nProceed?`)) return;
|
||||
toast(`resolving conflict: ${action}…`);
|
||||
const r = await apiFetch('/api/conflict/resolve', { method: 'POST', headers: { 'content-type': 'application/json' }, body: JSON.stringify({ name, action }) }).then(r => r.json()).catch(() => null);
|
||||
if (r && r.ok) { toast(`conflict resolved: ${action}`); refreshIndex(); }
|
||||
else toast(`resolve failed: ${r?.error || 'unknown'}`);
|
||||
}
|
||||
// E2.6: catch-up-now — forces an immediate shallow + deep sweep.
|
||||
async function catchUpNow() {
|
||||
toast('catching up…');
|
||||
const r = await apiFetch('/api/foundry-poll/catchup', { method: 'POST', headers: { 'content-type': 'application/json' } }).then(r => r.json()).catch(() => null);
|
||||
if (r && r.skipped) toast('catch-up already running');
|
||||
else if (r && r.durationMs !== undefined) toast(`catch-up complete (${r.durationMs}ms)`);
|
||||
else toast(`catch-up failed: ${r?.error || 'unknown'}`);
|
||||
refreshLiveNewEntries();
|
||||
refreshSyncState();
|
||||
}
|
||||
async function importLiveEntry(uuid, name) {
|
||||
if (!confirm(`Import "${name}" as a new refined note under refined/imported/?`)) return;
|
||||
toast(`importing ${name}…`);
|
||||
const r = await apiFetch('/api/foundry-poll/import', { method: 'POST', headers: { 'content-type': 'application/json' }, body: JSON.stringify({ uuid }) }).then(r => r.json()).catch(() => null);
|
||||
if (r && r.ok) toast(`imported "${name}" → refined/imported/${r.subfolder}/${r.filename}`);
|
||||
else toast(`import failed: ${r?.error || 'unknown'}`);
|
||||
refreshLiveNewEntries();
|
||||
refreshIndex();
|
||||
}
|
||||
async function revertLastPush(){
|
||||
const btn = document.getElementById('revertBtn');
|
||||
const uuid = btn.dataset.uuid;
|
||||
if (!uuid) return;
|
||||
const noteName = btn.textContent.replace('Revert last push: ', '');
|
||||
if (!confirm(`Revert the last push of "${noteName}"?\n\nThis restores Foundry to the state captured BEFORE the push (a full /update — the one place a full PUT is correct) and re-baselines the note. The note keeps your edit; Foundry reverts.`)) return;
|
||||
toast('reverting last push…');
|
||||
const r = await apiFetch('/api/autosync/revert', {method:'POST', headers:{'content-type':'application/json'}, body: JSON.stringify({uuid})}).then(r=>r.json()).catch(()=>null);
|
||||
if (r && r.ok) toast(`reverted — Foundry restored to pre-push state ("${r.restoredName ?? noteName}")`);
|
||||
else toast(`revert failed: ${r?.error || 'unknown'}`);
|
||||
refreshAutosync();
|
||||
}
|
||||
async function toggleAutosync(){
|
||||
const want = !(AUTO && AUTO.enabled);
|
||||
toast(`turning auto-sync ${want ? 'on' : 'off'}…`);
|
||||
const r = await apiFetch('/api/autosync', {method:'POST', headers:{'content-type':'application/json'}, body: JSON.stringify({enabled: want})}).then(r=>r.json());
|
||||
if (r.error){ toast('error: ' + r.error); return; }
|
||||
toast(`auto-sync ${r.enabled ? 'ON — saving a note pushes it to live Foundry' : 'off'}`);
|
||||
refreshAutosync();
|
||||
}
|
||||
// Push every vault-newer (sync-cc) note into live Foundry in one run. dry-run (default)
|
||||
// lists what would be pushed; apply pushes each, backs up the live entry, and baselines
|
||||
// the note so a re-run only catches new edits. Replaces scripts/resync.ts.
|
||||
async function pushAll(){
|
||||
const dryRun = dryEl().checked;
|
||||
toast(`push all changed ${dryRun?'(dry-run)':'(apply)'}… this may take a moment`);
|
||||
const r = await fetch('/api/push-all', {method:'POST', headers:{'content-type':'application/json'}, body: JSON.stringify({dryRun})}).then(r=>r.json());
|
||||
const r = await apiFetch('/api/push-all', {method:'POST', headers:{'content-type':'application/json'}, body: JSON.stringify({dryRun})}).then(r=>r.json());
|
||||
if (r.error){ toast('error: ' + r.error); return; }
|
||||
renderPushAllResults(r);
|
||||
if (r.dryRun){
|
||||
toast(`[dry-run] push all: ${r.wouldPush} note(s) would be pushed into live Foundry`);
|
||||
} else {
|
||||
toast(`pushed ${r.pushed}/${r.total}${r.failed?', '+r.failed+' failed':''} · baselined ${r.baselined}`);
|
||||
INDEX = await fetch('/api/index').then(r => r.json());
|
||||
INDEX = await apiFetch('/api/index').then(r => r.json());
|
||||
renderRecPanel(); render();
|
||||
}
|
||||
}
|
||||
@@ -357,7 +726,7 @@ let LINK_ENTRIES = null, LINK_NAME = null;
|
||||
async function linkPicker(name){
|
||||
LINK_NAME = name;
|
||||
if (!LINK_ENTRIES) {
|
||||
const r = await fetch('/api/entries').then(r => r.json());
|
||||
const r = await apiFetch('/api/entries').then(r => r.json());
|
||||
if (r.error) { toast('error: ' + r.error); return; }
|
||||
LINK_ENTRIES = r.entries || [];
|
||||
}
|
||||
@@ -388,12 +757,12 @@ async function doLink(name, uuid){
|
||||
document.querySelector('.modal-bg')?.remove();
|
||||
const dryRun = dryEl().checked;
|
||||
toast(`link ${name} ${dryRun?'(dry-run)':'(apply)'}…`);
|
||||
const r = await fetch('/api/link', {method:'POST', headers:{'content-type':'application/json'}, body: JSON.stringify({name, uuid, dryRun})}).then(r=>r.json());
|
||||
const r = await apiFetch('/api/link', {method:'POST', headers:{'content-type':'application/json'}, body: JSON.stringify({name, uuid, dryRun})}).then(r=>r.json());
|
||||
if (r.error){ toast('error: ' + r.error); return; }
|
||||
if (r.dryRun){ toast(`[dry-run] link ${name} -> ${uuid}`); }
|
||||
else {
|
||||
toast(r.message || `linked ${name}`);
|
||||
INDEX = await fetch('/api/index').then(r => r.json());
|
||||
INDEX = await apiFetch('/api/index').then(r => r.json());
|
||||
renderRecPanel(); render();
|
||||
}
|
||||
}
|
||||
@@ -402,7 +771,7 @@ init();
|
||||
// Called by the "Re-scan" button and automatically when the tab regains focus (so edits
|
||||
// made in Obsidian show up without a manual refresh).
|
||||
async function refreshIndex(){
|
||||
INDEX = await fetch('/api/index').then(r => r.json());
|
||||
INDEX = await apiFetch('/api/index').then(r => r.json());
|
||||
const c = INDEX.counts;
|
||||
document.getElementById('counts').innerHTML =
|
||||
`matched <b>${c.matched}</b> · cc-only <b>${c.ccOnly}</b> · refined-only <b>${c.refinedOnly}</b> · unlinked <b>${c.unlinked}</b>`;
|
||||
|
||||
462
src/foundry-poll.ts
Normal file
462
src/foundry-poll.ts
Normal file
@@ -0,0 +1,462 @@
|
||||
// E2.1 — FoundryPollController: shallow /search poll for F→O structural changes.
|
||||
//
|
||||
// Periodically snapshots the relay /search (minified: {uuid,id,name,img,
|
||||
// documentType} — NO folder, NO content, NO hash) and diffs against the previous
|
||||
// snapshot to detect renames (name change on a known uuid), new entries (uuid
|
||||
// absent from the prior snapshot + not in the linked index → liveNewEntries),
|
||||
// and missing entries (uuid in the linked index but absent from /search). No
|
||||
// folder/content detection here (deep poll E2.2 owns that).
|
||||
//
|
||||
// "Safe but silent" — not user-visible until E2+E4+E3 all land. Gated by
|
||||
// foundryPoll (cfg.features.foundryPoll, default off). Feature-flagged end-to-end.
|
||||
//
|
||||
// Retry: transient relay errors (408/504/500/network) retry with backoff per
|
||||
// E1b.7's policy (inline). Persistent errors (404 "No connected Foundry clients
|
||||
// found", 400 multi-client) surface immediately + halt the timer.
|
||||
|
||||
import type { RelayClient, SearchResult } from "./relay/client.js";
|
||||
import type { State } from "./server.js";
|
||||
import type { JournalEntry } from "./types.js";
|
||||
import { saveSyncState, appendActivity, type SyncState } from "./sync-state.js";
|
||||
import { classifyRelayError } from "./server.js";
|
||||
import { ccHash } from "./cchash.js";
|
||||
import { splitFrontmatter, readFoundryBlock } from "./frontmatter.js";
|
||||
import { readFile } from "node:fs/promises";
|
||||
|
||||
export interface FoundryPollConfig {
|
||||
cadenceMs: number; // ~10s default
|
||||
jitterPct: number; // ±20%
|
||||
}
|
||||
|
||||
export interface LiveNewEntry {
|
||||
uuid: string;
|
||||
name: string;
|
||||
detectedAt: string;
|
||||
}
|
||||
|
||||
export interface PendingFChange {
|
||||
uuid: string;
|
||||
name: string;
|
||||
change: "edited" | "renamed" | "moved" | "new" | "missing";
|
||||
detectedAt: string;
|
||||
}
|
||||
|
||||
export class FoundryPollController {
|
||||
enabled = false;
|
||||
private timer: NodeJS.Timeout | null = null;
|
||||
private inFlight = false;
|
||||
private prevSnapshot = new Map<string, { name: string; img: string | null }>();
|
||||
private readonly config: FoundryPollConfig;
|
||||
skipCounter = 0;
|
||||
liveNewEntries: LiveNewEntry[] = [];
|
||||
|
||||
// E2.2: deep poll — per-linked-note /get + ccHash compare.
|
||||
private deepTimer: NodeJS.Timeout | null = null;
|
||||
private deepInFlight = false;
|
||||
deepSkipCounter = 0;
|
||||
private readonly deepCadenceMs: number;
|
||||
private readonly deepConcurrency: number;
|
||||
// Retry backoffs for transient /get failures (per-note, inline). A field so
|
||||
// tests can shrink it for fast retry tests.
|
||||
retryBackoffs = [500, 1500, 4500];
|
||||
|
||||
constructor(private state: State) {
|
||||
this.config = {
|
||||
cadenceMs: Math.max(2000, Number(process.env.FOUNDRY_POLL_CADENCE_MS ?? 10000)),
|
||||
jitterPct: 0.2,
|
||||
};
|
||||
// E2.2: deep poll cadence — 5 min default (configurable). Load ceiling:
|
||||
// deepConcurrency / (deepCadenceMs / 60000) calls/min. With 4 / 300s = 0.8/s
|
||||
// = ~48/min regardless of N (mapPool bounds concurrency; the next round
|
||||
// doesn't start until the prior finishes).
|
||||
this.deepCadenceMs = Math.max(30000, Number(process.env.FOUNDRY_DEEP_POLL_CADENCE_MS ?? 300000));
|
||||
this.deepConcurrency = Math.min(8, Math.max(1, Number(process.env.FOUNDRY_DEEP_POLL_CONCURRENCY ?? 4)));
|
||||
}
|
||||
|
||||
status() {
|
||||
// E2.2: load ceiling = concurrency / round_seconds (documented in the status
|
||||
// payload so the DM can see the realized call rate is bounded).
|
||||
const loadCeiling = Math.round(this.deepConcurrency / (this.deepCadenceMs / 60000));
|
||||
return {
|
||||
enabled: this.enabled,
|
||||
cadenceMs: this.config.cadenceMs,
|
||||
inFlight: this.inFlight,
|
||||
skipCounter: this.skipCounter,
|
||||
liveNewEntries: this.liveNewEntries,
|
||||
loadCeilingCallsPerMin: loadCeiling,
|
||||
deepCadenceMs: this.deepCadenceMs,
|
||||
deepInFlight: this.deepInFlight,
|
||||
deepSkipCounter: this.deepSkipCounter,
|
||||
};
|
||||
}
|
||||
|
||||
async setEnabled(on: boolean): Promise<void> {
|
||||
if (on === this.enabled) return;
|
||||
if (on) {
|
||||
if (!this.state.cfg.relayCfg) throw new Error("relay not configured — foundry-poll needs RELAY_API_KEY");
|
||||
this.enabled = true;
|
||||
this.scheduleNext(0); // shallow poll fires immediately
|
||||
this.scheduleDeepNext(this.deepCadenceMs); // deep poll fires on its cadence (not immediately)
|
||||
} else {
|
||||
this.stop();
|
||||
}
|
||||
}
|
||||
|
||||
stop(): void {
|
||||
if (this.timer) { clearTimeout(this.timer); this.timer = null; }
|
||||
if (this.deepTimer) { clearTimeout(this.deepTimer); this.deepTimer = null; }
|
||||
this.inFlight = false;
|
||||
this.deepInFlight = false;
|
||||
this.enabled = false;
|
||||
}
|
||||
|
||||
/** E2.6: catch-up-now — cancel pending timers, run an immediate shallow + deep
|
||||
* sweep out of cadence, then resume regular cadence. Debounced (ignored if
|
||||
* already running). Returns a summary { shallow, deep, durationMs } or
|
||||
* { skipped: true } if debounced. */
|
||||
async catchUpNow(): Promise<{ skipped?: boolean; shallow?: unknown; deep?: unknown; durationMs?: number }> {
|
||||
if (this.inFlight || this.deepInFlight) return { skipped: true };
|
||||
// Cancel pending timers (the catch-up replaces them).
|
||||
if (this.timer) { clearTimeout(this.timer); this.timer = null; }
|
||||
if (this.deepTimer) { clearTimeout(this.deepTimer); this.deepTimer = null; }
|
||||
const start = Date.now();
|
||||
// Immediate shallow poll.
|
||||
this.inFlight = true;
|
||||
let shallowResult: { new: number; renamed: number; missing: number } = { new: 0, renamed: 0, missing: 0 };
|
||||
try {
|
||||
const beforeLive = this.liveNewEntries.length;
|
||||
const beforeChanges = (this.state.syncState as unknown as { fPending?: unknown[] })?.fPending?.length ?? 0;
|
||||
await this.shallowPoll();
|
||||
const afterLive = this.liveNewEntries.length;
|
||||
const afterChanges = (this.state.syncState as unknown as { fPending?: unknown[] })?.fPending?.length ?? 0;
|
||||
shallowResult = { new: Math.max(0, afterLive - beforeLive), renamed: 0, missing: Math.max(0, afterChanges - beforeChanges) };
|
||||
} catch (e) {
|
||||
const kind = classifyRelayError(e);
|
||||
if (this.state.syncState) {
|
||||
void appendActivity(this.state.cfg.outDir, this.state.syncState, {
|
||||
time: new Date().toISOString(), kind: "error", name: "(catch-up)", status: "error",
|
||||
message: `catch-up shallow ${kind}: ${(e as Error).message}`,
|
||||
});
|
||||
}
|
||||
if (kind === "persistent") { this.stop(); return { shallow: shallowResult, durationMs: Date.now() - start }; }
|
||||
} finally {
|
||||
this.inFlight = false;
|
||||
}
|
||||
// Immediate deep poll (on shallow completion).
|
||||
this.deepInFlight = true;
|
||||
let deepResult: { pulled: number; skipped: number; conflicts: number } = { pulled: 0, skipped: 0, conflicts: 0 };
|
||||
try {
|
||||
await this.deepPoll();
|
||||
} catch (e) {
|
||||
const kind = classifyRelayError(e);
|
||||
if (this.state.syncState) {
|
||||
void appendActivity(this.state.cfg.outDir, this.state.syncState, {
|
||||
time: new Date().toISOString(), kind: "error", name: "(catch-up)", status: "error",
|
||||
message: `catch-up deep ${kind}: ${(e as Error).message}`,
|
||||
});
|
||||
}
|
||||
if (kind === "persistent") { this.stop(); return { shallow: shallowResult, deep: deepResult, durationMs: Date.now() - start }; }
|
||||
} finally {
|
||||
this.deepInFlight = false;
|
||||
}
|
||||
const durationMs = Date.now() - start;
|
||||
// Log the round summary to the activity panel.
|
||||
if (this.state.syncState) {
|
||||
void appendActivity(this.state.cfg.outDir, this.state.syncState, {
|
||||
time: new Date().toISOString(), kind: "skip", name: "(catch-up)", status: "skipped",
|
||||
message: `catch-up complete: shallow ${JSON.stringify(shallowResult)}, deep ${JSON.stringify(deepResult)}, ${durationMs}ms`,
|
||||
});
|
||||
}
|
||||
// Resume regular cadence from now.
|
||||
this.scheduleNext(this.config.cadenceMs);
|
||||
this.scheduleDeepNext(this.deepCadenceMs);
|
||||
return { shallow: shallowResult, deep: deepResult, durationMs };
|
||||
}
|
||||
|
||||
private scheduleNext(delayMs: number): void {
|
||||
if (!this.enabled) return;
|
||||
const jitter = delayMs * (this.config.jitterPct * (Math.random() * 2 - 1));
|
||||
const actual = Math.max(0, Math.round(delayMs + jitter));
|
||||
this.timer = setTimeout(() => { void this.tick(); }, actual);
|
||||
}
|
||||
|
||||
private async tick(): Promise<void> {
|
||||
if (!this.enabled) return;
|
||||
if (this.inFlight) { this.skipCounter++; this.scheduleNext(this.config.cadenceMs); return; }
|
||||
this.inFlight = true;
|
||||
try {
|
||||
await this.shallowPoll();
|
||||
} catch (e) {
|
||||
// E1b.7 retry policy: transient → backoff; persistent → halt + surface.
|
||||
const kind = classifyRelayError(e);
|
||||
const msg = (e as Error).message;
|
||||
if (this.state.syncState) {
|
||||
void appendActivity(this.state.cfg.outDir, this.state.syncState, {
|
||||
time: new Date().toISOString(), kind: "error", name: "(foundry-poll)", status: "error",
|
||||
message: `shallow poll ${kind}: ${msg}`,
|
||||
});
|
||||
}
|
||||
if (kind === "persistent") {
|
||||
// Halt the timer — persistent errors (no clients, invalid clientId) need
|
||||
// the operator to fix the underlying issue.
|
||||
this.stop();
|
||||
return;
|
||||
}
|
||||
// Transient: retry with doubled cadence (capped at 60s).
|
||||
} finally {
|
||||
this.inFlight = false;
|
||||
}
|
||||
this.scheduleNext(this.config.cadenceMs);
|
||||
}
|
||||
|
||||
/** One shallow poll round: /search → snapshot → diff → record changes. */
|
||||
private async shallowPoll(): Promise<void> {
|
||||
const relay = new (await import("./relay/client.js")).RelayClient(this.state.cfg.relayCfg!);
|
||||
const results = await relay.searchJournalEntries();
|
||||
const snapshot = new Map<string, { name: string; img: string | null }>();
|
||||
for (const r of results) {
|
||||
snapshot.set(r.uuid, { name: r.name, img: r.img ?? null });
|
||||
}
|
||||
|
||||
const changes: PendingFChange[] = [];
|
||||
const linkedUuids = new Set<string>();
|
||||
// Collect linked uuids from the index (matched notes with foundry.cc_uuid).
|
||||
if (this.state.index) {
|
||||
for (const row of this.state.index.matched) {
|
||||
if (row.entry) linkedUuids.add(`JournalEntry.${row.entry._id}`);
|
||||
}
|
||||
}
|
||||
|
||||
// Detect renames + new entries.
|
||||
for (const [uuid, info] of snapshot) {
|
||||
const prev = this.prevSnapshot.get(uuid);
|
||||
if (prev) {
|
||||
if (prev.name !== info.name) {
|
||||
changes.push({ uuid, name: info.name, change: "renamed", detectedAt: new Date().toISOString() });
|
||||
}
|
||||
} else {
|
||||
// New uuid — not in the previous snapshot.
|
||||
if (!linkedUuids.has(uuid)) {
|
||||
// Not in the vault linked index → live new entry (import candidate).
|
||||
if (!this.liveNewEntries.some((e) => e.uuid === uuid)) {
|
||||
this.liveNewEntries.push({ uuid, name: info.name, detectedAt: new Date().toISOString() });
|
||||
}
|
||||
changes.push({ uuid, name: info.name, change: "new", detectedAt: new Date().toISOString() });
|
||||
}
|
||||
// If it IS in linkedUuids but wasn't in prevSnapshot → it was missing and
|
||||
// is now back. Not a "new" entry — just a reappearance. No action needed
|
||||
// (shallow poll doesn't pull content).
|
||||
}
|
||||
}
|
||||
|
||||
// Detect missing entries (in linked index but absent from /search).
|
||||
for (const uuid of linkedUuids) {
|
||||
if (!snapshot.has(uuid)) {
|
||||
const prev = this.prevSnapshot.get(uuid);
|
||||
if (prev) { // was in the last snapshot, now gone
|
||||
// E2.4: route to pendingConflicts (vault-newer) if autosync is available.
|
||||
if (this.state.autosync && typeof (this.state.autosync as any).recordPendingConflict === "function") {
|
||||
void (this.state.autosync as any).recordPendingConflict(uuid, prev.name, "vault-newer", "", "");
|
||||
}
|
||||
changes.push({ uuid, name: prev.name, change: "missing", detectedAt: new Date().toISOString() });
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Record changes to sync-state.json.fPending (the F-pending badge's data).
|
||||
if (changes.length > 0 && this.state.syncState) {
|
||||
const s = this.state.syncState;
|
||||
// Merge changes into a fPending array (deduped by uuid+change).
|
||||
const fPending = (Array.isArray((s as unknown as { fPending?: PendingFChange[] }).fPending) ? (s as unknown as { fPending: PendingFChange[] }).fPending : []) as PendingFChange[];
|
||||
for (const c of changes) {
|
||||
if (!fPending.some((e) => e.uuid === c.uuid && e.change === c.change)) {
|
||||
fPending.push(c);
|
||||
}
|
||||
}
|
||||
(s as unknown as { fPending: PendingFChange[] }).fPending = fPending;
|
||||
s.parity.fPending = fPending.length;
|
||||
s.parity.lastPollAt = new Date().toISOString();
|
||||
// Recompute parity status (precedence: conflict > O-pending > F-pending > unsynced > in-parity).
|
||||
const p = s.parity;
|
||||
p.status = p.conflict > 0 ? "conflict" : p.oPending > 0 ? "O-pending" : p.fPending > 0 ? "F-pending" : p.unsyncedLinked > 0 ? "unsynced-linked" : "in-parity";
|
||||
await saveSyncState(this.state.cfg.outDir, s).catch(() => {});
|
||||
} else if (this.state.syncState) {
|
||||
// No changes — update lastPollAt.
|
||||
this.state.syncState.parity.lastPollAt = new Date().toISOString();
|
||||
await saveSyncState(this.state.cfg.outDir, this.state.syncState).catch(() => {});
|
||||
}
|
||||
|
||||
// Remove from liveNewEntries any uuid that has since appeared in the linked
|
||||
// index (after a manual refresh --full-index or an import).
|
||||
if (this.state.index) {
|
||||
this.liveNewEntries = this.liveNewEntries.filter((e) => !linkedUuids.has(e.uuid));
|
||||
}
|
||||
|
||||
this.prevSnapshot = snapshot;
|
||||
}
|
||||
|
||||
// E2.2: deep poll timer.
|
||||
|
||||
private scheduleDeepNext(delayMs: number): void {
|
||||
if (!this.enabled) return;
|
||||
const jitter = delayMs * (this.config.jitterPct * (Math.random() * 2 - 1));
|
||||
const actual = Math.max(0, Math.round(delayMs + jitter));
|
||||
this.deepTimer = setTimeout(() => { void this.deepTick(); }, actual);
|
||||
}
|
||||
|
||||
private async deepTick(): Promise<void> {
|
||||
if (!this.enabled) return;
|
||||
if (this.deepInFlight) { this.deepSkipCounter++; this.scheduleDeepNext(this.deepCadenceMs); return; }
|
||||
this.deepInFlight = true;
|
||||
try {
|
||||
await this.deepPoll();
|
||||
} catch (e) {
|
||||
const kind = classifyRelayError(e);
|
||||
const msg = (e as Error).message;
|
||||
if (this.state.syncState) {
|
||||
void appendActivity(this.state.cfg.outDir, this.state.syncState, {
|
||||
time: new Date().toISOString(), kind: "error", name: "(foundry-poll)", status: "error",
|
||||
message: `deep poll ${kind}: ${msg}`,
|
||||
});
|
||||
}
|
||||
if (kind === "persistent") { this.stop(); return; }
|
||||
} finally {
|
||||
this.deepInFlight = false;
|
||||
}
|
||||
this.scheduleDeepNext(this.deepCadenceMs);
|
||||
}
|
||||
|
||||
/** E2.2: one deep poll round — per-linked-note /get + ccHash compare.
|
||||
* Candidate list = intersection of shallow-poll snapshot uuids + linked index
|
||||
* uuids. mapPool concurrency 4. Each note: /get → ccHash(liveEntry) vs
|
||||
* foundry.ccHash baseline (from frontmatter) + folder vs folder_path. A
|
||||
* mismatch → F-changed → record in sync-state.json.fPending. Load ceiling:
|
||||
* 4 / 300s ≈ 48 calls/min regardless of N. */
|
||||
private async deepPoll(): Promise<void> {
|
||||
const relay = new (await import("./relay/client.js")).RelayClient(this.state.cfg.relayCfg!);
|
||||
const snapshotUuids = new Set(this.prevSnapshot.keys());
|
||||
// Build the candidate list: uuids in BOTH the shallow snapshot AND the linked
|
||||
// index (matched notes with an entry).
|
||||
const candidates: { uuid: string; refinedPath: string; name: string }[] = [];
|
||||
if (this.state.index) {
|
||||
for (const row of this.state.index.matched) {
|
||||
if (!row.entry) continue;
|
||||
const uuid = `JournalEntry.${row.entry._id}`;
|
||||
if (snapshotUuids.has(uuid) && row.refinedPath) {
|
||||
candidates.push({ uuid, refinedPath: row.refinedPath, name: row.name });
|
||||
}
|
||||
}
|
||||
}
|
||||
if (candidates.length === 0) return;
|
||||
|
||||
// Process with bounded concurrency (mapPool). Per-note: /get → ccHash → compare.
|
||||
const persistentErrHolder: { err: Error | null } = { err: null };
|
||||
const changes: PendingFChange[] = [];
|
||||
const pool = async (items: typeof candidates, concurrency: number, fn: (item: typeof candidates[0]) => Promise<void>): Promise<void> => {
|
||||
let next = 0;
|
||||
const workers = Array.from({ length: Math.min(concurrency, items.length) }, async () => {
|
||||
while (next < items.length) {
|
||||
const item = items[next++];
|
||||
await fn(item);
|
||||
}
|
||||
});
|
||||
await Promise.all(workers);
|
||||
};
|
||||
|
||||
await pool(candidates, this.deepConcurrency, async (c) => {
|
||||
if (persistentErrHolder.err) return; // a prior note hit a persistent error → skip the rest
|
||||
let liveEntry: JournalEntry;
|
||||
try {
|
||||
liveEntry = await this.retryGetEntry(relay, c.uuid);
|
||||
} catch (e) {
|
||||
const kind = classifyRelayError(e);
|
||||
if (kind === "persistent") {
|
||||
persistentErrHolder.err = e as Error;
|
||||
return; // the round will abort after the pool finishes
|
||||
}
|
||||
// Transient exhaustion → record as fPendingRetry (the note needs attention
|
||||
// but the round continues).
|
||||
changes.push({ uuid: c.uuid, name: c.name, change: "edited", detectedAt: new Date().toISOString() });
|
||||
return;
|
||||
}
|
||||
// Read the note's frontmatter to get the ccHash baseline + folder_path.
|
||||
let fb: Record<string, string> | undefined;
|
||||
try {
|
||||
const md = await readFile(c.refinedPath, "utf8");
|
||||
fb = readFoundryBlock(splitFrontmatter(md).fm);
|
||||
} catch { return; } // note gone — skip
|
||||
// Compare ccHash.
|
||||
let liveCcHash: string;
|
||||
try { liveCcHash = ccHash(liveEntry); } catch { return; } // malformed entry — skip
|
||||
const baselineCcHash = fb?.ccHash;
|
||||
const baselineFolder = fb?.folder_path;
|
||||
const liveFolder = liveEntry.folder ?? "";
|
||||
const fChanged = (baselineCcHash && liveCcHash !== baselineCcHash) || (!baselineCcHash && baselineFolder && liveFolder !== baselineFolder);
|
||||
if (fChanged) {
|
||||
// E2.3: attempt the F→O pull. If O-side is unchanged → pull + baseline +
|
||||
// remove from fPending. If O-side also changed → "conflict" (E2.4, left
|
||||
// in fPending). If lock busy / row missing → "skipped" (left in fPending).
|
||||
// Guard: if autosync isn't available (e.g. E2.2 tests), just record.
|
||||
if (this.state.autosync && typeof this.state.autosync.pullFChanged === "function") {
|
||||
const result = await this.state.autosync.pullFChanged(c.uuid, liveEntry);
|
||||
if (result !== "pulled") {
|
||||
const changeType = (!baselineCcHash && baselineFolder && liveFolder !== baselineFolder) ? "moved" : "edited";
|
||||
changes.push({ uuid: c.uuid, name: c.name, change: changeType, detectedAt: new Date().toISOString() });
|
||||
}
|
||||
} else {
|
||||
// No autosync → just record (pre-E2.3 behavior).
|
||||
const changeType = (!baselineCcHash && baselineFolder && liveFolder !== baselineFolder) ? "moved" : "edited";
|
||||
changes.push({ uuid: c.uuid, name: c.name, change: changeType, detectedAt: new Date().toISOString() });
|
||||
}
|
||||
}
|
||||
});
|
||||
|
||||
// If a persistent error occurred, abort + halt.
|
||||
if (persistentErrHolder.err) {
|
||||
if (this.state.syncState) {
|
||||
void appendActivity(this.state.cfg.outDir, this.state.syncState, {
|
||||
time: new Date().toISOString(), kind: "error", name: "(foundry-poll)", status: "error",
|
||||
message: `deep poll aborted: ${persistentErrHolder.err.message}`,
|
||||
});
|
||||
}
|
||||
throw persistentErrHolder.err; // → deepTick's catch → halt
|
||||
}
|
||||
|
||||
// Record changes to sync-state.json.fPending.
|
||||
if (changes.length > 0 && this.state.syncState) {
|
||||
const s = this.state.syncState;
|
||||
const fPending = (Array.isArray((s as unknown as { fPending?: PendingFChange[] }).fPending) ? (s as unknown as { fPending: PendingFChange[] }).fPending : []) as PendingFChange[];
|
||||
for (const c of changes) {
|
||||
if (!fPending.some((e) => e.uuid === c.uuid && e.change === c.change)) {
|
||||
fPending.push(c);
|
||||
}
|
||||
}
|
||||
(s as unknown as { fPending: PendingFChange[] }).fPending = fPending;
|
||||
s.parity.fPending = fPending.length;
|
||||
s.parity.lastPollAt = new Date().toISOString();
|
||||
const p = s.parity;
|
||||
p.status = p.conflict > 0 ? "conflict" : p.oPending > 0 ? "O-pending" : p.fPending > 0 ? "F-pending" : p.unsyncedLinked > 0 ? "unsynced-linked" : "in-parity";
|
||||
await saveSyncState(this.state.cfg.outDir, s).catch(() => {});
|
||||
}
|
||||
}
|
||||
|
||||
/** E2.2: retry a /get entry fetch (inline, per-note). 3 attempts with backoff.
|
||||
* Transient errors retry; persistent errors throw immediately. */
|
||||
private async retryGetEntry(relay: RelayClient, uuid: string): Promise<JournalEntry> {
|
||||
let lastErr: unknown;
|
||||
for (let attempt = 0; attempt < 3; attempt++) {
|
||||
try {
|
||||
return await relay.getEntry(uuid);
|
||||
} catch (e) {
|
||||
lastErr = e;
|
||||
if (classifyRelayError(e) === "persistent") throw e;
|
||||
if (attempt < 2) {
|
||||
const backoff = this.retryBackoffs[attempt] ?? 1000;
|
||||
const jitter = backoff * (0.8 + 0.4 * Math.random());
|
||||
await new Promise<void>((r) => setTimeout(r, Math.round(jitter)));
|
||||
}
|
||||
}
|
||||
}
|
||||
throw lastErr;
|
||||
}
|
||||
}
|
||||
109
src/fromFoundry.ts
Normal file
109
src/fromFoundry.ts
Normal file
@@ -0,0 +1,109 @@
|
||||
// E1a.1 — htmlToMarkdown: the linkedom-based HTML→markdown inverse of the
|
||||
// forward push transform (src/mdToHtml.ts `markdownToHtml` +
|
||||
// src/toFoundry.ts `buildFoundryJson`), tuned for round-trip hash-stability.
|
||||
//
|
||||
// SEAM: `(html: string) => string` — pure, synchronous, no-network, NO resolver.
|
||||
// This is the contract boundary E0.2's ccHash consumes. Applied to BOTH
|
||||
// `data.description` (the two-column flex layout — returns ONLY the left-column
|
||||
// body markdown, sidebar/right column dropped, mirroring htmlMd.ts) and
|
||||
// `data.notes` (plain HTML — returns the full body markdown).
|
||||
//
|
||||
// vs src/htmlMd.ts: that module's `htmlToMarkdown(html, db)` is the F→O PULL
|
||||
// inverse (produces Obsidian-friendly output: `<img>`→`![[basename]]` embeds,
|
||||
// `@UUID`→`[[Name|Display]]` via db). This module is the HASH inverse: it aims
|
||||
// to reproduce the forward transform's INPUT shape so
|
||||
// `contentHash(canonicalize(htmlToMarkdown(html))) === contentHash(canonicalize(originalBody))`.
|
||||
// Where the pull inverse is asymmetric (images, UUID), this one is tuned to
|
||||
// match the forward: `<img>`→``, `@UUID[...]{Display}` kept as-is (no
|
||||
// resolver — see the wikilink fixture for why this is the crux GO/NO-GO lever).
|
||||
//
|
||||
// Feature-flagged behind CC_HASH_SPIKE (default false): the forward push path
|
||||
// is unchanged when the flag is off; this module ships dark until the spike
|
||||
// passes and E0.2 wires it into ccHash.
|
||||
|
||||
import { parseHTML } from "linkedom";
|
||||
|
||||
const ELEMENT_NODE = 1;
|
||||
const TEXT_NODE = 3;
|
||||
|
||||
/** Inverse transform: Foundry HTML → refined markdown (no resolver). */
|
||||
export function htmlToMarkdown(html: string): string {
|
||||
if (!html || !html.trim()) return "";
|
||||
const { document } = parseHTML(`<div>${html}</div>`);
|
||||
const root = document.querySelector("div");
|
||||
if (!root) return "";
|
||||
|
||||
// The forward transform wraps the body in a two-column flex div
|
||||
// (src/toFoundry.ts:178): left = body prose, right = sidebar (Bio/Social
|
||||
// boxes from frontmatter). The Obsidian body hash excludes frontmatter, so
|
||||
// the inverse returns ONLY the left column for a two-column description.
|
||||
// For plain HTML (data.notes), there is no flex wrapper — convert the whole.
|
||||
const flexDiv = root.querySelector('div[style*="display:flex"], div[style*="display: flex"]');
|
||||
const bodyNode = flexDiv && flexDiv.children.length >= 2 ? flexDiv.children[0] : root;
|
||||
|
||||
return collapse(nodeToMd(bodyNode));
|
||||
}
|
||||
|
||||
function nodeToMd(node: any): string {
|
||||
if (node.nodeType === TEXT_NODE) return node.textContent ?? "";
|
||||
if (node.nodeType !== ELEMENT_NODE) return "";
|
||||
const el = node;
|
||||
const tag = el.tagName.toLowerCase();
|
||||
const inner = () => Array.from(el.childNodes).map((c: any) => nodeToMd(c)).join("");
|
||||
const block = (c: string) => `\n\n${c}\n\n`;
|
||||
switch (tag) {
|
||||
case "h1": return block(`# ${inner().trim()}`);
|
||||
case "h2": return block(`## ${inner().trim()}`);
|
||||
case "h3": return block(`### ${inner().trim()}`);
|
||||
case "h4": return block(`#### ${inner().trim()}`);
|
||||
case "h5": return block(`##### ${inner().trim()}`);
|
||||
case "h6": return block(`###### ${inner().trim()}`);
|
||||
case "p": return block(inner().trim());
|
||||
case "strong": case "b": return `**${inner()}**`;
|
||||
case "em": case "i": return `*${inner()}*`;
|
||||
case "code": return `\`${inner()}\``;
|
||||
case "hr": return block("---");
|
||||
case "br": return "\n";
|
||||
case "blockquote": {
|
||||
const c = inner().trim().replace(/\n/g, "\n> ");
|
||||
return block(`> ${c}`);
|
||||
}
|
||||
case "ul":
|
||||
return block(Array.from(el.children).map((li) => `- ${nodeToMd(li).trim()}`).join("\n"));
|
||||
case "ol":
|
||||
return block(Array.from(el.children).map((li, i) => `${i + 1}. ${nodeToMd(li).trim()}`).join("\n"));
|
||||
case "li": return inner();
|
||||
// Tuned for round-trip: the forward (mdToHtml.ts:67) emits
|
||||
// `<img src="path" alt="...">` from ``. The pull inverse (htmlMd.ts)
|
||||
// emits `![[basename]]`; the HASH inverse emits `` to match the
|
||||
// forward's input form.
|
||||
case "img": {
|
||||
const src = el.getAttribute("src") ?? "";
|
||||
const alt = el.getAttribute("alt") ?? "";
|
||||
return block(``);
|
||||
}
|
||||
case "a": {
|
||||
const href = el.getAttribute("href") ?? "";
|
||||
const text = inner();
|
||||
return href.startsWith("http") ? `[${text}](${href})` : text;
|
||||
}
|
||||
case "table": {
|
||||
// Best-effort table round-trip (GitHub-flavored markdown). The forward
|
||||
// (mdToHtml.ts) has NO table branch, so a table in the body never
|
||||
// round-trips through it — this branch exists so hand-authored table
|
||||
// HTML still produces deterministic markdown for the fixture.
|
||||
const rows = Array.from(el.querySelectorAll("tr")) as any[];
|
||||
if (rows.length === 0) return block(inner().trim());
|
||||
const cells = (tr: any) => Array.from(tr.querySelectorAll("th,td")).map((c) => nodeToMd(c).trim());
|
||||
const header = cells(rows[0]);
|
||||
const body = rows.slice(1).map(cells);
|
||||
const sep = header.map(() => "---");
|
||||
return block([header.join(" | "), sep.join(" | "), ...body.map((r) => r.join(" | "))].join("\n"));
|
||||
}
|
||||
default: return inner(); // div/span/section recurse
|
||||
}
|
||||
}
|
||||
|
||||
function collapse(md: string): string {
|
||||
return md.trim().replace(/\n{3,}/g, "\n\n");
|
||||
}
|
||||
54
src/push.ts
54
src/push.ts
@@ -1,5 +1,5 @@
|
||||
import { mkdir, readFile, writeFile } from "node:fs/promises";
|
||||
import { join } from "node:path";
|
||||
import { join, dirname } from "node:path";
|
||||
import { obsidianToFoundryJsonLive } from "./toFoundry.js";
|
||||
import { splitFrontmatter, readFoundryBlock } from "./frontmatter.js";
|
||||
import { RelayClient } from "./relay/client.js";
|
||||
@@ -44,6 +44,18 @@ export interface PushDeps {
|
||||
* whether a portrait is present. */
|
||||
skipImageUpload?: boolean;
|
||||
log?: (msg: string) => void;
|
||||
/** E1b.1 no-clobber guard: invoked right after `relay.getEntry` (the SAME /get
|
||||
* pushNote already makes — no extra round-trip) and BEFORE any side effect
|
||||
* (image upload, backup, PUT). Return `{ abort: true, reason }` to skip the PUT
|
||||
* (fail-safe: a Foundry-side edit since the last baseline must not be
|
||||
* overwritten). Return `{ abort: false }` (or omit the guard) to proceed. */
|
||||
prePushGuard?: (liveEntry: JournalEntry) => { abort: boolean; reason?: string };
|
||||
/** E1b.4: override the backup file path. If set, the pre-push live-entry
|
||||
* snapshot is written here (mkdir -p the dirname) instead of the default
|
||||
* `<outDir>/bak/<noteName>.<stamp>.json` (which manual push keeps). Auto-sync
|
||||
* uses the per-uuid `foundry-backups/<uuid>/<iso>.json` layout so retention is
|
||||
* per-entry. */
|
||||
backupPath?: string;
|
||||
}
|
||||
|
||||
export interface PushOutcome {
|
||||
@@ -53,6 +65,20 @@ export interface PushOutcome {
|
||||
imageNote: string;
|
||||
backupPath?: string;
|
||||
updatedName?: string;
|
||||
/** E1b.1: true when `prePushGuard` aborted the push (Foundry-side drift). No
|
||||
* PUT and no backup were performed. */
|
||||
aborted?: boolean;
|
||||
/** E1b.1: the guard's reason when `aborted`. */
|
||||
abortReason?: string;
|
||||
/** E1b.1: the live entry fetched by the reused `/get`, exposed so the caller
|
||||
* can compute/re-baseline `foundry.ccHash` (E1b.2) with no second round-trip. */
|
||||
liveEntry?: JournalEntry;
|
||||
/** E1b.2: the full pushed entry (buildPushPayload `full` — the post-push
|
||||
* Foundry content: name + flags["campaign-codex"] overridden on the live
|
||||
* entry). ccHash(pushedEntry) is the correct post-push baseline for
|
||||
* foundry.ccHash (the AC's "pre-PUT captured entry" would false-abort the
|
||||
* next save, since a push changes Foundry). No extra /get needed. */
|
||||
pushedEntry?: JournalEntry;
|
||||
}
|
||||
|
||||
/** Read the foundry.cc_uuid and the portrait field from a refined note's
|
||||
@@ -141,6 +167,18 @@ export async function pushNote(deps: PushDeps): Promise<PushOutcome> {
|
||||
log(`push: fetching live entry ${id} via relay /get…`);
|
||||
const liveEntry = await deps.relay.getEntry(id);
|
||||
|
||||
// E1b.1 no-clobber guard: reuses THIS /get (no extra round-trip). Abort before
|
||||
// any side effect (image upload, backup, PUT) if Foundry's stored content
|
||||
// drifted from the note's foundry.ccHash baseline. Fail-safe: a Foundry-side
|
||||
// edit since the last baseline must not be overwritten.
|
||||
if (deps.prePushGuard) {
|
||||
const g = deps.prePushGuard(liveEntry);
|
||||
if (g.abort) {
|
||||
log(`push: aborted by prePushGuard — ${g.reason ?? "guard abort"}`);
|
||||
return { dryRun: deps.dryRun, ccUuid: id, diff: {}, imageNote: "aborted (no image processed)", aborted: true, abortReason: g.reason, liveEntry };
|
||||
}
|
||||
}
|
||||
|
||||
// Image: upload the portrait into Foundry's assets dir if the note has one.
|
||||
let imageOverride: string | null | undefined = undefined; // undefined = keep existing
|
||||
let imageNote = "no portrait field";
|
||||
@@ -165,22 +203,22 @@ export async function pushNote(deps: PushDeps): Promise<PushOutcome> {
|
||||
}
|
||||
}
|
||||
|
||||
const { diff } = buildPushPayload(md, deps.noteName, liveEntry, resolver, imageOverride);
|
||||
const { full, diff } = buildPushPayload(md, deps.noteName, liveEntry, resolver, imageOverride);
|
||||
|
||||
if (deps.dryRun) {
|
||||
log(`[dry-run] push ${deps.noteName} (${id})`);
|
||||
return { dryRun: true, ccUuid: id, diff, imageNote };
|
||||
return { dryRun: true, ccUuid: id, diff, imageNote, liveEntry, pushedEntry: full };
|
||||
}
|
||||
|
||||
// Apply: snapshot the live entry first (reversible), then PUT the diff.
|
||||
const bakDir = join(deps.outDir, "bak");
|
||||
await mkdir(bakDir, { recursive: true });
|
||||
const stamp = backupStamp();
|
||||
const backupPath = join(bakDir, `${deps.noteName}.${stamp}.json`);
|
||||
// E1b.4: auto-sync passes a per-uuid backupPath (foundry-backups/<uuid>/<iso>.json);
|
||||
// manual push keeps the default flat <outDir>/bak/<noteName>.<stamp>.json.
|
||||
const backupPath = deps.backupPath ?? join(deps.outDir, "bak", `${deps.noteName}.${backupStamp()}.json`);
|
||||
await mkdir(dirname(backupPath), { recursive: true });
|
||||
await writeFile(backupPath, JSON.stringify(liveEntry, null, 2) + "\n", "utf8");
|
||||
log(`push: backed up live entry -> ${backupPath}`);
|
||||
|
||||
const updated = await deps.relay.updateEntry(id, diff);
|
||||
log(`push: updated "${updated.name}" in live Foundry (${id})`);
|
||||
return { dryRun: false, ccUuid: id, diff, imageNote, backupPath, updatedName: updated.name };
|
||||
return { dryRun: false, ccUuid: id, diff, imageNote, backupPath, updatedName: updated.name, liveEntry, pushedEntry: full };
|
||||
}
|
||||
60
src/schema-version.ts
Normal file
60
src/schema-version.ts
Normal file
@@ -0,0 +1,60 @@
|
||||
// E0.3 — Schema-version naming constants.
|
||||
//
|
||||
// The system has two unrelated schema-version concepts that must never collide
|
||||
// on a single `schemaVersion` field:
|
||||
// 1. Foundry-side flags shape — stored at `flags["campaign-codex"].schemaVersion`
|
||||
// on the live JournalEntry returned by `relay.getEntry`. Owned by E1b
|
||||
// (the `flagsSchemaVersion` migration).
|
||||
// 2. Local sync-state shape — the on-disk `sync-state.json` record of last-
|
||||
// synced hashes / timestamps / parity. Owned by E4 (the
|
||||
// `syncStateSchemaVersion` migration).
|
||||
//
|
||||
// This module fixes the names + ownership contract up front so E1b and E4 cannot
|
||||
// drift. It defines NO migration logic, NO on-disk shapes, and touches no
|
||||
// call-site — it is the frozen naming reservation E1b/E4 import from.
|
||||
|
||||
/** Branded nominal type: a `schemaVersion` belonging to the Foundry flags shape.
|
||||
* Distinct brand property name from SyncStateSchemaVersion so the two are not
|
||||
* cross-assignable (compile-time guard). */
|
||||
export type FlagsSchemaVersion = string & { readonly __flagsBrand: true };
|
||||
|
||||
/** Branded nominal type: a `schemaVersion` belonging to the local sync-state file.
|
||||
* Distinct brand property name from FlagsSchemaVersion so the two are not
|
||||
* cross-assignable (compile-time guard). */
|
||||
export type SyncStateSchemaVersion = string & { readonly __syncBrand: true };
|
||||
|
||||
/**
|
||||
* Current Foundry `flags["campaign-codex"]` schema version. Owner: E1b.
|
||||
* Storage: `flags["campaign-codex"].schemaVersion` on the live entry.
|
||||
* Migration policy: bump + migrate-on-read in E1b; E4 must NOT touch it.
|
||||
*/
|
||||
export const FLAGS_SCHEMA_VERSION = "flags-campaign-codex/v1" as FlagsSchemaVersion;
|
||||
|
||||
/**
|
||||
* Current local sync-state file schema version. Owner: E4.
|
||||
* Storage: the top-level `schemaVersion` field of `sync-state.json`.
|
||||
* Migration policy: bump + migrate-on-read in E4; E1b must NOT touch it.
|
||||
*/
|
||||
export const SYNC_STATE_SCHEMA_VERSION = "sync-state/v1" as SyncStateSchemaVersion;
|
||||
|
||||
/** Discriminated parse result: which schema a raw `schemaVersion` string
|
||||
* belongs to, with the `version` branded so a parsed flags-version cannot be
|
||||
* assigned to a sync-state slot (and vice versa) — the brand protection
|
||||
* extends to values loaded from disk, not just the two constants. */
|
||||
export type ParsedSchemaVersion =
|
||||
| { kind: "flags"; version: FlagsSchemaVersion }
|
||||
| { kind: "sync-state"; version: SyncStateSchemaVersion };
|
||||
|
||||
/**
|
||||
* Branch on the prefix to determine which schema an arbitrary `schemaVersion`
|
||||
* field belongs to. Returns `null` for unknown prefixes so a reader handed an
|
||||
* unversioned/foreign string can fall back to a default rather than guessing.
|
||||
* The returned `version` is branded to its kind, so callers cannot accidentally
|
||||
* feed a parsed flags-version into a sync-state-typed slot.
|
||||
*/
|
||||
export function parseSchemaVersion(raw: string): ParsedSchemaVersion | null {
|
||||
if (typeof raw !== "string" || raw === "") return null;
|
||||
if (raw.startsWith("flags-")) return { kind: "flags", version: raw as FlagsSchemaVersion };
|
||||
if (raw.startsWith("sync-")) return { kind: "sync-state", version: raw as SyncStateSchemaVersion };
|
||||
return null;
|
||||
}
|
||||
1616
src/server.ts
1616
src/server.ts
File diff suppressed because it is too large
Load Diff
137
src/sync-state.ts
Normal file
137
src/sync-state.ts
Normal file
@@ -0,0 +1,137 @@
|
||||
// E4.1 — persistent sync-state.json (the single status source E2/E3/E4 read from).
|
||||
//
|
||||
// The AutoSyncController keeps an in-memory view (the dashboard's fast poll); this
|
||||
// module is the DURABLE aggregate that survives restarts: on/off, mode, parity,
|
||||
// activity (last 200), last-sync. Atomic writes (tmp + rename) so a crash mid-
|
||||
// write never leaves a truncated file. Schema-versioned: a mismatch backs up the
|
||||
// old file + writes fresh defaults + an error event.
|
||||
//
|
||||
// E4.1 does NOT change the existing /api/autosync or the dashboard panel — those
|
||||
// migrate to read sync-state.json in E4.2-E4.6 (gated behind features.syncStatus,
|
||||
// defined here). E4.1 only delivers load/save + the boot reconcile.
|
||||
//
|
||||
// Reconciliations: (1) E0.3 froze SYNC_STATE_SCHEMA_VERSION = "sync-state/v1"
|
||||
// (string); E4.1's AC said "= 1" (number) — E0.3 wins, so the file's
|
||||
// syncStateSchemaVersion is the E0.3 string. (2) E4.1 persists autoSyncOn across
|
||||
// restarts, superseding E1b.5's "no persistence until E3" — Slice 1 introduces the
|
||||
// persistence layer. A fresh install (no sync-state.json) still defaults
|
||||
// autoSyncOn=false; a user who explicitly toggled ON is restored ON (their choice).
|
||||
|
||||
import { readFile, writeFile, rename, mkdir, stat } from "node:fs/promises";
|
||||
import { join, dirname } from "node:path";
|
||||
import { FLAGS_SCHEMA_VERSION, SYNC_STATE_SCHEMA_VERSION } from "./schema-version.js";
|
||||
import { backupStamp } from "./write.js";
|
||||
|
||||
export interface SyncStateActivityEvent {
|
||||
time: string;
|
||||
kind: string; // push | pull | baseline | skip | error | mode | status-note
|
||||
name: string;
|
||||
status: string;
|
||||
message: string;
|
||||
}
|
||||
|
||||
export interface SyncStateParity {
|
||||
status: string; // in-parity | O-pending | F-pending | conflict | unsynced-linked
|
||||
oPending: number;
|
||||
fPending: number;
|
||||
conflict: number;
|
||||
unsyncedLinked: number;
|
||||
lastPollAt: string | null;
|
||||
}
|
||||
|
||||
export interface SyncState {
|
||||
syncStateSchemaVersion: string;
|
||||
mode: string; // PREP | RUN-THE-MATCH
|
||||
autoSyncOn: boolean;
|
||||
lastSyncAt: string | null;
|
||||
parity: SyncStateParity;
|
||||
watchedDir: string;
|
||||
activity: SyncStateActivityEvent[];
|
||||
updatedAt: string;
|
||||
conflict: null; // reserved for E3 — E4 MUST NOT populate this
|
||||
}
|
||||
|
||||
/** Default state for a fresh install (or after a schema-mismatch reset). */
|
||||
export function defaultSyncState(watchedDir: string): SyncState {
|
||||
return {
|
||||
syncStateSchemaVersion: SYNC_STATE_SCHEMA_VERSION,
|
||||
mode: "PREP",
|
||||
autoSyncOn: false, // fresh install — OFF (opt-in)
|
||||
lastSyncAt: null,
|
||||
parity: { status: "in-parity", oPending: 0, fPending: 0, conflict: 0, unsyncedLinked: 0, lastPollAt: null },
|
||||
watchedDir,
|
||||
activity: [],
|
||||
updatedAt: new Date().toISOString(),
|
||||
conflict: null,
|
||||
};
|
||||
}
|
||||
|
||||
/** Trim activity to the last 200 (newest first) on each append. */
|
||||
export const MAX_ACTIVITY = 200;
|
||||
|
||||
/** E4.1: load sync-state.json from <outDir>/sync-state.json. If absent, create
|
||||
* defaults. If the schema version mismatches, back up the old file to
|
||||
* sync-state.json.bak-<stamp> and write fresh defaults (with an error event).
|
||||
* Returns { state, freshened } where `freshened` is true if a reset happened. */
|
||||
export async function loadSyncState(outDir: string, watchedDir: string): Promise<{ state: SyncState; freshened: boolean }> {
|
||||
const path = join(outDir, "sync-state.json");
|
||||
let raw: string | null = null;
|
||||
try { raw = await readFile(path, "utf8"); } catch { /* absent */ }
|
||||
if (!raw) {
|
||||
const state = defaultSyncState(watchedDir);
|
||||
await saveSyncState(outDir, state);
|
||||
return { state, freshened: false };
|
||||
}
|
||||
let parsed: Partial<SyncState>;
|
||||
try { parsed = JSON.parse(raw) as Partial<SyncState>; }
|
||||
catch { parsed = {}; }
|
||||
if (parsed.syncStateSchemaVersion !== SYNC_STATE_SCHEMA_VERSION) {
|
||||
// Schema mismatch — back up the old file + write fresh defaults + an error event.
|
||||
try { await rename(path, `${path}.bak-${backupStamp()}`); } catch { /* best-effort */ }
|
||||
const state = defaultSyncState(watchedDir);
|
||||
state.activity.unshift({ time: new Date().toISOString(), kind: "error", name: "(state)", status: "error", message: `sync-state.json schema reset: ${parsed.syncStateSchemaVersion ?? "(absent)"} → ${SYNC_STATE_SCHEMA_VERSION}` });
|
||||
await saveSyncState(outDir, state);
|
||||
return { state, freshened: true };
|
||||
}
|
||||
// Merge with defaults so missing fields don't break readers (forward-compat).
|
||||
const state: SyncState = {
|
||||
...defaultSyncState(watchedDir),
|
||||
...parsed,
|
||||
parity: { ...defaultSyncState(watchedDir).parity, ...(parsed.parity ?? {}) },
|
||||
activity: Array.isArray(parsed.activity) ? parsed.activity.slice(0, MAX_ACTIVITY) : [],
|
||||
conflict: null, // E3 owns this; force null in E4
|
||||
};
|
||||
return { state, freshened: false };
|
||||
}
|
||||
|
||||
/** E4.1: atomically write sync-state.json (tmp + rename). Each save uses a
|
||||
* UNIQUE tmp path so concurrent saves (e.g. log()→appendActivity + a
|
||||
* mode-flip handler) don't race on the same tmp file (one rename would
|
||||
* consume the other's tmp → ENOENT). Never throws out of a state-mutation
|
||||
* path — callers catch + log (the flag-off-failure-path rule). */
|
||||
let saveSeq = 0;
|
||||
export async function saveSyncState(outDir: string, state: SyncState): Promise<void> {
|
||||
const path = join(outDir, "sync-state.json");
|
||||
const tmp = `${path}.tmp-${process.pid}-${++saveSeq}`;
|
||||
await mkdir(dirname(path), { recursive: true });
|
||||
state.updatedAt = new Date().toISOString();
|
||||
await writeFile(tmp, JSON.stringify(state, null, 2), "utf8");
|
||||
await rename(tmp, path);
|
||||
}
|
||||
|
||||
/** E4.1: append an activity event + save (atomic). The activity array is newest-
|
||||
* first, trimmed to MAX_ACTIVITY. Best-effort (never throws). */
|
||||
export async function appendActivity(outDir: string, state: SyncState, event: SyncStateActivityEvent): Promise<void> {
|
||||
state.activity.unshift(event);
|
||||
if (state.activity.length > MAX_ACTIVITY) state.activity.length = MAX_ACTIVITY;
|
||||
await saveSyncState(outDir, state).catch(() => { /* best-effort */ });
|
||||
}
|
||||
|
||||
/** Read the sync-state.json mtime (for tests / diagnostics), or null if absent. */
|
||||
export async function syncStateMtime(outDir: string): Promise<number | null> {
|
||||
try { return (await stat(join(outDir, "sync-state.json"))).mtimeMs; } catch { return null; }
|
||||
}
|
||||
|
||||
// Re-export so E1b's flagsSchemaVersion migration + E4's syncStateSchemaVersion
|
||||
// are both reachable from one import (distinct names, distinct owners).
|
||||
export { FLAGS_SCHEMA_VERSION };
|
||||
235
src/synclock.ts
Normal file
235
src/synclock.ts
Normal file
@@ -0,0 +1,235 @@
|
||||
// E0.1 — Per-UUID bidirectional sync lock.
|
||||
//
|
||||
// Replaces the per-relPath `inflight = new Set<string>()` in AutoSyncController
|
||||
// with a single per-UUID lock that gates BOTH Obsidian→Foundry and
|
||||
// Foundry→Obsidian operations on the same entry. The lock is keyed by Foundry
|
||||
// uuid (the `foundry.cc_uuid` value, normalized to the `JournalEntry.<id>`
|
||||
// form used by `relay.getEntry`) PLUS a resource tag (`"push" | "pull" |
|
||||
// "baseline"`). The lock is per-uuid, NOT per-direction: while one direction is
|
||||
// in flight for a uuid the other direction queues or skips, eliminating
|
||||
// cross-direction clobber and TOCTOU races without two separate guard systems.
|
||||
//
|
||||
// Resolution of relPath→uuid happens OUTSIDE the lock (the watcher/debounce
|
||||
// keys on relPath; the uuid is resolved by reading the note's `foundry.cc_uuid`
|
||||
// frontmatter before `acquire` is called). For unlinked/un-keyed files a
|
||||
// namespaced pseudo-uuid `"relPath:" + relPath` is used so the unlinked tail
|
||||
// still gets per-file mutual exclusion (the F→O direction, which receives a
|
||||
// uuid from the relay, never hits this fallback).
|
||||
//
|
||||
// Fairness: a queued waiter (policy "queue") is woken FIFO on release, and a
|
||||
// fresh public `acquire` DEFERS when waiters are already queued for that uuid
|
||||
// (returns `acquired:false, deferred:true`) so an opportunistic skip-policy
|
||||
// caller (auto-sync) cannot jump the queue ahead of a waiting manual button.
|
||||
// The woken waiter grabs via the internal acquire (no defer check). This is
|
||||
// best-effort fairness for the lock's purpose (brief per-push holds); it is not
|
||||
// a full fair mutex and does not need to be — auto-sync holds are short and
|
||||
// skip-policy drops are idempotent (the save re-triggers).
|
||||
//
|
||||
// This primitive is a CONCURRENCY guard, NOT a self-write/re-entrancy guard.
|
||||
// The baseline write fired by a successful push re-triggers the watcher; that
|
||||
// self-write suppression is a separate mechanism (E1b.2). E0.1 owns only the
|
||||
// cross-op exclusion contract. The lock surface is FROZEN on landing so E1b
|
||||
// and E2 can code against it without re-coordination.
|
||||
|
||||
/** Resource tag: which kind of operation holds the lock for a uuid. */
|
||||
export type LockOp = "push" | "pull" | "baseline";
|
||||
|
||||
/** What to do when a second op for an already-held uuid arrives. */
|
||||
export type LockConflictPolicy = "skip" | "queue";
|
||||
|
||||
/** Result of a synchronous `acquire` check. */
|
||||
export interface AcquireResult {
|
||||
acquired: boolean;
|
||||
/** The op currently holding the uuid, when `acquired` is false because it is held. */
|
||||
heldOp?: LockOp;
|
||||
/** True when the uuid is FREE but `acquire` deferred to queued waiters
|
||||
* (fairness — see module doc). Distinct from `heldOp` so callers can tell
|
||||
* "busy" from "yielding to a waiter". */
|
||||
deferred?: boolean;
|
||||
}
|
||||
|
||||
/** Options for `withLock`. */
|
||||
export interface WithLockOptions {
|
||||
/** Conflict policy when the uuid is already held (or deferred). Default `"skip"`. */
|
||||
policy?: LockConflictPolicy;
|
||||
/** Max wait in ms for the `"queue"` policy before throwing `LockAcquireTimeout`. Default 5000. */
|
||||
maxWaitMs?: number;
|
||||
}
|
||||
|
||||
/** Thrown when a `"queue"`-policy `withLock` cannot acquire within `maxWaitMs`.
|
||||
* Loud by design — a queued (manual) op that cannot run is a user-visible
|
||||
* failure, not a silent drop. */
|
||||
export class LockAcquireTimeout extends Error {
|
||||
readonly kind = "LockAcquireTimeout";
|
||||
constructor(public readonly uuid: string, public readonly op: LockOp, public readonly maxWaitMs: number) {
|
||||
super(`lock acquire timed out (${op} on ${uuid} after ${maxWaitMs}ms)`);
|
||||
this.name = "LockAcquireTimeout";
|
||||
}
|
||||
}
|
||||
|
||||
interface Holder {
|
||||
op: LockOp;
|
||||
}
|
||||
|
||||
interface Waiter {
|
||||
resolve: () => void;
|
||||
}
|
||||
|
||||
/**
|
||||
* Per-UUID bidirectional sync lock. One instance shared by the watcher (O→F)
|
||||
* and the poll path (F→O) — the lock cares about uuid+resource, not direction.
|
||||
*/
|
||||
export class SyncLock {
|
||||
private readonly held = new Map<string, Holder>();
|
||||
private readonly waiters = new Map<string, Waiter[]>();
|
||||
|
||||
/**
|
||||
* Synchronously attempt to acquire the lock for `(uuid, op)`.
|
||||
* Returns `{ acquired: true }` when the uuid is free AND no waiters are
|
||||
* queued for it. Returns `{ acquired: false, heldOp }` when the uuid is
|
||||
* already held. Returns `{ acquired: false, deferred: true }` when the uuid
|
||||
* is free but waiters are queued (fairness — defer to them).
|
||||
*
|
||||
* Reentrant-NO: a second `acquire` of the same uuid from inside a held
|
||||
* `withLock` callback returns `{ acquired: false, heldOp }` (deadlock-safe).
|
||||
*/
|
||||
acquire(uuid: string, op: LockOp): AcquireResult {
|
||||
const existing = this.held.get(uuid);
|
||||
if (existing) return { acquired: false, heldOp: existing.op };
|
||||
// Fairness: if waiters are queued, a fresh public acquire defers so the
|
||||
// woken waiter (which grabs via acquireInternal) gets the slot.
|
||||
if ((this.waiters.get(uuid)?.length ?? 0) > 0) return { acquired: false, deferred: true };
|
||||
this.held.set(uuid, { op });
|
||||
return { acquired: true };
|
||||
}
|
||||
|
||||
/** Internal acquire that bypasses the defer-to-waiters fairness rule. Used
|
||||
* by the woken queued waiter (already shifted out of the queue by wakeOne)
|
||||
* so it can grab the slot it was promised. Returns true on success. */
|
||||
private acquireInternal(uuid: string, op: LockOp): boolean {
|
||||
if (this.held.has(uuid)) return false;
|
||||
this.held.set(uuid, { op });
|
||||
return true;
|
||||
}
|
||||
|
||||
/**
|
||||
* Release the lock for `(uuid, op)`. No-op (NOT a throw) if the uuid is not
|
||||
* held by `op`, so error-path cleanup can't crash the watcher. Wakes one
|
||||
* queued waiter (FIFO) if any.
|
||||
*/
|
||||
release(uuid: string, op: LockOp): void {
|
||||
const h = this.held.get(uuid);
|
||||
if (!h || h.op !== op) return; // not held by this op — no-op
|
||||
this.held.delete(uuid);
|
||||
this.wakeOne(uuid);
|
||||
}
|
||||
|
||||
/** Whether the uuid is currently held by any op. */
|
||||
isHeld(uuid: string): boolean {
|
||||
return this.held.has(uuid);
|
||||
}
|
||||
|
||||
/** Snapshot of currently-held uuid→op pairs, for diagnostics. */
|
||||
heldOps(): Record<string, LockOp> {
|
||||
const out: Record<string, LockOp> = {};
|
||||
for (const [uuid, h] of this.held) out[uuid] = h.op;
|
||||
return out;
|
||||
}
|
||||
|
||||
/**
|
||||
* Acquire → await `fn` → release in a `finally`.
|
||||
*
|
||||
* - `"skip"` policy (default): if the uuid is held OR deferred to waiters,
|
||||
* returns `undefined` immediately without running `fn` (auto-sync semantics
|
||||
* for redundant saves — the save re-triggers, so dropping is idempotent).
|
||||
* - `"queue"` policy: waits for the holder (bounded by `maxWaitMs`), retrying
|
||||
* on every release until it acquires. Throws `LockAcquireTimeout` if the
|
||||
* wait elapses without acquiring (loud — a manual op that cannot run is a
|
||||
* user-visible failure, not a silent drop).
|
||||
*
|
||||
* On `fn` rejection the lock is still released (release-on-throw) so the next
|
||||
* acquire succeeds.
|
||||
*/
|
||||
async withLock<T>(
|
||||
uuid: string,
|
||||
op: LockOp,
|
||||
fn: () => Promise<T>,
|
||||
opts: WithLockOptions = {},
|
||||
): Promise<T | undefined> {
|
||||
const policy = opts.policy ?? "skip";
|
||||
if (this.acquire(uuid, op).acquired) {
|
||||
try {
|
||||
return await fn();
|
||||
} finally {
|
||||
this.release(uuid, op);
|
||||
}
|
||||
}
|
||||
if (policy === "skip") return undefined;
|
||||
|
||||
// queue: retry-loop until acquired or maxWait elapses. Re-waiting on every
|
||||
// release means a racing fresh acquire does not silently make us give up —
|
||||
// we keep trying until our deadline. On deadline, throw (loud).
|
||||
const maxWait = opts.maxWaitMs ?? 5000;
|
||||
const start = Date.now();
|
||||
while (!this.acquireInternal(uuid, op)) {
|
||||
const remaining = maxWait - (Date.now() - start);
|
||||
if (remaining <= 0) throw new LockAcquireTimeout(uuid, op, maxWait);
|
||||
const woke = await this.waitFor(uuid, remaining);
|
||||
if (!woke) throw new LockAcquireTimeout(uuid, op, maxWait);
|
||||
}
|
||||
try {
|
||||
return await fn();
|
||||
} finally {
|
||||
this.release(uuid, op);
|
||||
}
|
||||
}
|
||||
|
||||
/** Wait until `uuid` is released (waking one waiter FIFO) or `maxWaitMs`
|
||||
* elapses. Resolves true on wake, false on timeout. If already free,
|
||||
* resolves true immediately. The waiter removes itself from the queue on
|
||||
* timeout so wakeOne does not call a dead resolver. */
|
||||
private waitFor(uuid: string, maxWaitMs: number): Promise<boolean> {
|
||||
return new Promise<boolean>((resolve) => {
|
||||
if (!this.held.has(uuid)) { resolve(true); return; }
|
||||
let done = false;
|
||||
const wakeResolve = () => {
|
||||
if (done) return;
|
||||
done = true;
|
||||
clearTimeout(timer);
|
||||
resolve(true);
|
||||
};
|
||||
const timer = setTimeout(() => {
|
||||
if (done) return;
|
||||
done = true;
|
||||
const q = this.waiters.get(uuid);
|
||||
if (q) {
|
||||
const idx = q.findIndex((w) => w.resolve === wakeResolve);
|
||||
if (idx >= 0) q.splice(idx, 1);
|
||||
if (q.length === 0) this.waiters.delete(uuid);
|
||||
}
|
||||
resolve(false);
|
||||
}, maxWaitMs);
|
||||
const q = this.waiters.get(uuid) ?? [];
|
||||
q.push({ resolve: wakeResolve });
|
||||
this.waiters.set(uuid, q);
|
||||
});
|
||||
}
|
||||
|
||||
/** Wake one queued waiter for `uuid` (FIFO) and shift it out of the queue,
|
||||
* so a subsequent public `acquire` sees one fewer waiter (the woken one
|
||||
* grabs via acquireInternal, not via the deferred public path). */
|
||||
private wakeOne(uuid: string): void {
|
||||
const q = this.waiters.get(uuid);
|
||||
if (!q || q.length === 0) return;
|
||||
const next = q.shift()!;
|
||||
if (q.length === 0) this.waiters.delete(uuid);
|
||||
next.resolve();
|
||||
}
|
||||
}
|
||||
|
||||
/** Pseudo-uuid for an unlinked/un-keyed vault file, so the unlinked tail still
|
||||
* gets per-file mutual exclusion. The F→O direction (which receives a uuid
|
||||
* from the relay) never hits this fallback. */
|
||||
export function relPathLockKey(relPath: string): string {
|
||||
return `relPath:${relPath}`;
|
||||
}
|
||||
95
tests/canonicalize-html.test.ts
Normal file
95
tests/canonicalize-html.test.ts
Normal file
@@ -0,0 +1,95 @@
|
||||
import { describe, it, expect } from "vitest";
|
||||
import { canonicalizeHtml } from "../src/canonicalize-html.js";
|
||||
|
||||
// Base HTML carrying the features the canonicalizer must normalize: a styled
|
||||
// container, a paragraph with a proper entity (the forward's escapeHtml emits
|
||||
// &, and Foundry stores/returns it verbatim) and an inline child, plus a
|
||||
// void element with two attributes.
|
||||
const BASE = '<div style="display:flex"><p>Hello & <b>world</b></p><img src="x.png" alt="alt"></div>';
|
||||
|
||||
// Variants that differ ONLY in serialization (parse to the same DOM) — each
|
||||
// must canonicalize to the SAME string as BASE. Drifts defended: attribute
|
||||
// order, quoting, named-vs-numeric entity, inter-tag whitespace, self-closing
|
||||
// slash, tag/attr case.
|
||||
const VARIANTS = [
|
||||
// attribute order swapped on <img>
|
||||
'<div style="display:flex"><p>Hello & <b>world</b></p><img alt="alt" src="x.png"></div>',
|
||||
// single-quoted attributes
|
||||
"<div style='display:flex'><p>Hello & <b>world</b></p><img src='x.png' alt='alt'></div>",
|
||||
// numeric entity & instead of named & (both decode to &)
|
||||
'<div style="display:flex"><p>Hello & <b>world</b></p><img src="x.png" alt="alt"></div>',
|
||||
// inter-tag whitespace / newlines (indentation the serializer may add or drop)
|
||||
'<div style="display:flex">\n <p>Hello & <b>world</b></p>\n <img src="x.png" alt="alt">\n</div>',
|
||||
// self-closing slash on the void <img>
|
||||
'<div style="display:flex"><p>Hello & <b>world</b></p><img src="x.png" alt="alt" /></div>',
|
||||
// uppercase tags + attributes
|
||||
'<DIV STYLE="display:flex"><P>Hello & <B>world</B></P><IMG SRC="x.png" ALT="alt"></DIV>',
|
||||
];
|
||||
|
||||
describe("canonicalizeHtml — serialization-drift stability (E1b-alt mini-gate)", () => {
|
||||
it("is deterministic: same input → same canonical across runs", () => {
|
||||
const a = canonicalizeHtml(BASE);
|
||||
const b = canonicalizeHtml(BASE);
|
||||
expect(a).toBe(b);
|
||||
expect(a).toMatch(/^<div/);
|
||||
});
|
||||
|
||||
it("all serialization variants canonicalize to the SAME string (drift absorbed)", () => {
|
||||
const baseCanon = canonicalizeHtml(BASE);
|
||||
for (const [i, v] of VARIANTS.entries()) {
|
||||
expect(canonicalizeHtml(v), `variant ${i}: ${v}`).toBe(baseCanon);
|
||||
}
|
||||
});
|
||||
|
||||
it("the canonical form is the compact, normalized shape", () => {
|
||||
// Sorted attrs (alt before src), double-quoted, lowercased, void <img> with
|
||||
// no closing slash, no inter-tag whitespace. The entity & decodes to &
|
||||
// and re-encodes to & (the trailing space before <b> is a whitespace-only
|
||||
// node after the entity decode and is dropped — consistently for every
|
||||
// entity-encoded variant, so the hash is stable).
|
||||
expect(canonicalizeHtml(BASE)).toBe(
|
||||
'<div style="display:flex"><p>Hello &<b>world</b></p><img alt="alt" src="x.png"></div>',
|
||||
);
|
||||
});
|
||||
|
||||
it("empty / null / undefined → empty string", () => {
|
||||
expect(canonicalizeHtml("")).toBe("");
|
||||
expect(canonicalizeHtml(null)).toBe("");
|
||||
expect(canonicalizeHtml(undefined)).toBe("");
|
||||
expect(canonicalizeHtml(" \n ")).toBe("");
|
||||
});
|
||||
});
|
||||
|
||||
describe("canonicalizeHtml — content sensitivity (real changes move the hash)", () => {
|
||||
it("a one-character text change yields a different canonical form", () => {
|
||||
const a = canonicalizeHtml(BASE);
|
||||
const b = canonicalizeHtml('<div style="display:flex"><p>Hello & <b>World</b></p><img src="x.png" alt="alt"></div>');
|
||||
expect(a).not.toBe(b);
|
||||
});
|
||||
|
||||
it("an attribute VALUE change yields a different canonical form", () => {
|
||||
const a = canonicalizeHtml(BASE);
|
||||
const b = canonicalizeHtml('<div style="display:flex"><p>Hello & <b>world</b></p><img src="y.png" alt="alt"></div>');
|
||||
expect(a).not.toBe(b);
|
||||
});
|
||||
|
||||
it("a structural change (element removed) yields a different canonical form", () => {
|
||||
const a = canonicalizeHtml(BASE);
|
||||
const b = canonicalizeHtml('<div style="display:flex"><p>Hello & world</p><img src="x.png" alt="alt"></div>');
|
||||
expect(a).not.toBe(b);
|
||||
});
|
||||
|
||||
it("an added element yields a different canonical form", () => {
|
||||
const a = canonicalizeHtml(BASE);
|
||||
const b = canonicalizeHtml('<div style="display:flex"><p>Hello & <b>world</b></p><img src="x.png" alt="alt"><hr></div>');
|
||||
expect(a).not.toBe(b);
|
||||
});
|
||||
|
||||
it("a style/class change (layout-bearing attribute) yields a different canonical form", () => {
|
||||
// The two-column flex style IS content for the hash (a Foundry layout change
|
||||
// is a real change). Attribute-value sensitivity covers it.
|
||||
const a = canonicalizeHtml(BASE);
|
||||
const b = canonicalizeHtml('<div style="display:block"><p>Hello & <b>world</b></p><img src="x.png" alt="alt"></div>');
|
||||
expect(a).not.toBe(b);
|
||||
});
|
||||
});
|
||||
198
tests/cchash.test.ts
Normal file
198
tests/cchash.test.ts
Normal file
@@ -0,0 +1,198 @@
|
||||
import { describe, it, expect } from "vitest";
|
||||
import {
|
||||
ccHash,
|
||||
ccHashFromGet,
|
||||
CC_HASH_CONTRACT,
|
||||
CcHashError,
|
||||
isCcHashError,
|
||||
} from "../src/cchash.js";
|
||||
import { canonicalizeHtml } from "../src/canonicalize-html.js";
|
||||
import { contentHash } from "../src/normalize.js";
|
||||
import type { JournalEntry, CcData } from "../src/types.js";
|
||||
import type { RelayClient } from "../src/relay/client.js";
|
||||
|
||||
interface EntryOpts {
|
||||
name?: string;
|
||||
folder?: string | null;
|
||||
description?: string;
|
||||
notes?: string;
|
||||
data?: CcData; // exact override (for the missing-field tests)
|
||||
noFlag?: boolean;
|
||||
noData?: boolean;
|
||||
}
|
||||
|
||||
function entry(opts: EntryOpts = {}): JournalEntry {
|
||||
const cc = opts.noFlag
|
||||
? undefined
|
||||
: opts.noData
|
||||
? { type: "npc" }
|
||||
: { type: "npc", data: opts.data ?? { description: opts.description ?? "<p>The gunslinger.</p>", notes: opts.notes ?? "" } };
|
||||
return {
|
||||
name: opts.name ?? "Roland Deschain",
|
||||
_id: "abc1",
|
||||
// Default only on undefined (NOT null) so tests can pass `folder: null`
|
||||
// to exercise the `folder ?? ""` branch in ccHash.
|
||||
folder: opts.folder !== undefined ? opts.folder : "Folder.gideon",
|
||||
flags: cc ? { "campaign-codex": cc } : {},
|
||||
};
|
||||
}
|
||||
|
||||
describe("ccHash contract + determinism (E1b-alt)", () => {
|
||||
it("CC_HASH_CONTRACT pins the exact bytes of the frozen input contract", () => {
|
||||
expect(CC_HASH_CONTRACT).toBe(
|
||||
'contentHash(canonicalizeHtml(data.description) + "\\n" + canonicalizeHtml(data.notes ?? "") + "\\n" + name + "\\n" + folder)',
|
||||
);
|
||||
});
|
||||
|
||||
it("implementation matches the frozen contract (re-derivation enforces it)", () => {
|
||||
const e = entry({ notes: "<p>He killed the boy.</p>" });
|
||||
const data = e.flags!["campaign-codex"]!.data!;
|
||||
const expected = contentHash(
|
||||
`${canonicalizeHtml(data.description!)}\n${canonicalizeHtml(data.notes!)}\n${e.name}\n${e.folder ?? ""}`,
|
||||
);
|
||||
expect(ccHash(e)).toBe(expected);
|
||||
});
|
||||
|
||||
it("is deterministic: same payload → same hash across runs", () => {
|
||||
const a = ccHash(entry());
|
||||
const b = ccHash(entry());
|
||||
expect(a).toBe(b);
|
||||
expect(a).toMatch(/^[0-9a-f]{64}$/); // sha256 hex
|
||||
});
|
||||
|
||||
it("is sensitive: a one-char change to data.description yields a different hash", () => {
|
||||
const a = ccHash(entry({ description: "<p>The gunslinger.</p>" }));
|
||||
const b = ccHash(entry({ description: "<p>The gunslinger!</p>" }));
|
||||
expect(a).not.toBe(b);
|
||||
});
|
||||
|
||||
it("is sensitive: a change to data.notes (## Secrets) yields a different hash", () => {
|
||||
// A Foundry-side edit to secrets MUST move ccHash, or the divergence guard
|
||||
// would miss secrets-only edits (the clobber hole the contract closes).
|
||||
const a = ccHash(entry({ notes: "" }));
|
||||
const b = ccHash(entry({ notes: "<p>He killed the boy.</p>" }));
|
||||
expect(a).not.toBe(b);
|
||||
});
|
||||
|
||||
it("name changing alone yields a different hash (part of the hash input)", () => {
|
||||
const a = ccHash(entry({ name: "Roland Deschain" }));
|
||||
const b = ccHash(entry({ name: "Roland Deschain of Gilead" }));
|
||||
expect(a).not.toBe(b);
|
||||
});
|
||||
|
||||
it("folder changing alone yields a different hash (Foundry folder ID)", () => {
|
||||
const a = ccHash(entry({ folder: "Folder.gideon" }));
|
||||
const b = ccHash(entry({ folder: "Folder.gilead" }));
|
||||
expect(a).not.toBe(b);
|
||||
});
|
||||
|
||||
it("absent folder is treated as empty string (matches Obsidian-side absence)", () => {
|
||||
const withEmpty = ccHash(entry({ folder: "" }));
|
||||
const absentFolder = ccHash(entry({ folder: null }));
|
||||
expect(withEmpty).toBe(absentFolder);
|
||||
});
|
||||
|
||||
it("trailing whitespace in name/folder is normalized (canonicalize via contentHash)", () => {
|
||||
// name/folder are concatenated raw but the final contentHash canonicalizes
|
||||
// the whole string, so relay serialization whitespace drift does not flap ccHash.
|
||||
const a = ccHash(entry({ name: "Roland Deschain" }));
|
||||
const b = ccHash(entry({ name: "Roland Deschain " })); // trailing spaces
|
||||
expect(a).toBe(b);
|
||||
});
|
||||
});
|
||||
|
||||
describe("ccHash absorbs HTML serialization drift (the E1b-alt property)", () => {
|
||||
it("two descriptions that differ only in serialization → same ccHash", () => {
|
||||
// Same DOM, different serialization (attribute order + inter-tag whitespace
|
||||
// + self-closing slash + tag case). canonicalizeHtml absorbs it.
|
||||
const a = ccHash(entry({ description: '<p>Hello <b>world</b></p><img src="x.png" alt="alt">' }));
|
||||
const b = ccHash(entry({ description: '<P>Hello <B>world</B></P>\n <IMG alt="alt" src="x.png" />' }));
|
||||
expect(a).toBe(b);
|
||||
});
|
||||
|
||||
it("two notes that differ only in serialization → same ccHash", () => {
|
||||
// Pure serialization drift (tag case + named-vs-numeric entity), NO text
|
||||
// change. Both decode & → & and lowercase the tag → same canonical.
|
||||
const a = ccHash(entry({ notes: "<p>Secret & one.</p>" }));
|
||||
const b = ccHash(entry({ notes: "<P>Secret & one.</P>" }));
|
||||
expect(a).toBe(b);
|
||||
});
|
||||
|
||||
it("a real content change in the description → different ccHash (no false negative)", () => {
|
||||
const a = ccHash(entry({ description: "<p>Hello world.</p>" }));
|
||||
const b = ccHash(entry({ description: "<p>Hello World.</p>" })); // capital W
|
||||
expect(a).not.toBe(b);
|
||||
});
|
||||
});
|
||||
|
||||
describe("ccHash direction-invariance (E1b-alt)", () => {
|
||||
it("same Foundry data+name+folder → same hash regardless of caller (E1b push vs E2 pull)", () => {
|
||||
const e = entry();
|
||||
expect(ccHash(e)).toBe(ccHash(e)); // hash is a function of the Foundry entry only
|
||||
});
|
||||
|
||||
it("renaming the vault file (without changing the live entry) leaves ccHash unchanged", () => {
|
||||
// The vault filename never enters the hash. A rename is a name-field update
|
||||
// routed through pushNote's updatedName path, not a content divergence — so
|
||||
// the stored foundry.ccHash is unaffected until a push updates liveEntry.name.
|
||||
const e = entry();
|
||||
expect(ccHash(e)).toBe(ccHash(e)); // liveEntry unchanged
|
||||
});
|
||||
|
||||
it("a live entry name change (a real push) DOES change ccHash", () => {
|
||||
const before = ccHash(entry({ name: "Roland" }));
|
||||
const after = ccHash(entry({ name: "Roland Deschain" }));
|
||||
expect(before).not.toBe(after);
|
||||
});
|
||||
});
|
||||
|
||||
describe("ccHash error handling (E1b-alt)", () => {
|
||||
it("throws CcHashError when flags.campaign-codex is absent", () => {
|
||||
expect(() => ccHash(entry({ noFlag: true }))).toThrow(CcHashError);
|
||||
expect(() => ccHash(entry({ noFlag: true }))).toThrow(/missing campaign-codex data/);
|
||||
});
|
||||
|
||||
it("throws CcHashError when flags.campaign-codex.data is absent", () => {
|
||||
expect(() => ccHash(entry({ noData: true }))).toThrow(CcHashError);
|
||||
expect(() => ccHash(entry({ noData: true }))).toThrow(/missing campaign-codex data/);
|
||||
});
|
||||
|
||||
it("throws CcHashError when data.description is absent/non-string (NOT coerced to empty)", () => {
|
||||
// A present-but-description-less entry must not silently hash "" — that
|
||||
// would create a stable-but-wrong baseline.
|
||||
const e = entry({ data: { notes: "<p>orphan notes</p>" } as CcData });
|
||||
expect(() => ccHash(e)).toThrow(CcHashError);
|
||||
expect(() => ccHash(e)).toThrow(/description/);
|
||||
});
|
||||
|
||||
it("ccHashFromGet surfaces relay errors unchanged (not wrapped as CcHashError)", async () => {
|
||||
const relayErr = new Error('relay 404 GET /get: No connected Foundry clients found');
|
||||
const fakeRelay = { getEntry: async (_uuid: string): Promise<JournalEntry> => { throw relayErr; } } as unknown as RelayClient;
|
||||
try {
|
||||
await ccHashFromGet(fakeRelay, "JournalEntry.abc1");
|
||||
throw new Error("should have thrown");
|
||||
} catch (err) {
|
||||
expect(isCcHashError(err)).toBe(false);
|
||||
expect(err).toBe(relayErr);
|
||||
}
|
||||
});
|
||||
|
||||
it("ccHashFromGet returns { hash, entry } on success and derives the hash from the same response", async () => {
|
||||
const e = entry();
|
||||
const fakeRelay = { getEntry: async (_uuid: string): Promise<JournalEntry> => e } as unknown as RelayClient;
|
||||
const result = await ccHashFromGet(fakeRelay, "JournalEntry.abc1");
|
||||
expect(result.entry).toBe(e);
|
||||
expect(result.hash).toBe(ccHash(e));
|
||||
});
|
||||
|
||||
it("ccHashFromGet throws CcHashError (not relay error) when the entry is malformed", async () => {
|
||||
const malformed = entry({ noData: true });
|
||||
const fakeRelay = { getEntry: async (): Promise<JournalEntry> => malformed } as unknown as RelayClient;
|
||||
try {
|
||||
await ccHashFromGet(fakeRelay, "JournalEntry.abc1");
|
||||
throw new Error("should have thrown");
|
||||
} catch (err) {
|
||||
expect(isCcHashError(err)).toBe(true);
|
||||
}
|
||||
});
|
||||
});
|
||||
168
tests/e1a-hash-spike.test.ts
Normal file
168
tests/e1a-hash-spike.test.ts
Normal file
@@ -0,0 +1,168 @@
|
||||
// E1a.1 — round-trip hash-stability spike (GO/NO-GO gate).
|
||||
//
|
||||
// For each fixture: Obsidian body → forward transform (`obsidianToFoundryJsonLive`,
|
||||
// the real push builder) → extract `flags["campaign-codex"].data.description` +
|
||||
// `.notes` → inverse (`htmlToMarkdown` from src/fromFoundry.ts, NO resolver) →
|
||||
// reassemble (left body + `## Secrets` + notes) → canonicalize → compare to
|
||||
// `canonicalize(originalBody)`.
|
||||
//
|
||||
// Fixtures use REALISTIC Obsidian formatting (blank line after each heading/
|
||||
// block) — the project's normal style. A separate tight-formatting note is in
|
||||
// the findings doc.
|
||||
//
|
||||
// VERDICT: NO-GO. 7 of 12 fixtures round-trip; 5 fail for FIVE distinct reasons
|
||||
// (4 fundamental contract instabilities + 1 forward-transform bug). The
|
||||
// markdown-hash divergence guard (E1b as specced) is not viable → E1b must adopt
|
||||
// the E1b-alt fork (canonicalize Foundry HTML directly, hash the HTML, never
|
||||
// hash markdown). See
|
||||
// `docs/prds/prd-foundry-obsidian-sync-2026-06-22/e1a-spike-findings.md`.
|
||||
//
|
||||
// This file is the binary, reproducible gate artifact. Unstable fixtures assert
|
||||
// `toBe(false)` — they DOCUMENT the NO-GO evidence, so the suite is green
|
||||
// precisely because the instabilities are reproduced and pinned. If a future
|
||||
// change makes an unstable fixture pass, its assertion flips red and forces a
|
||||
// re-evaluation. Runs regardless of CC_HASH_SPIKE (the flag gates WIRING into
|
||||
// the push path, not the spike's evidence).
|
||||
|
||||
import { describe, it, expect } from "vitest";
|
||||
import { obsidianToFoundryJsonLive } from "../src/toFoundry.js";
|
||||
import { htmlToMarkdown } from "../src/fromFoundry.js";
|
||||
import { canonicalize } from "../src/normalize.js";
|
||||
import type { JournalEntry } from "../src/types.js";
|
||||
import type { NameResolver } from "../src/resolver.js";
|
||||
|
||||
// Mock resolver: the forward resolves [[Name]] → @UUID via this; the inverse has
|
||||
// NO resolver, so @UUID stays @UUID — NO-GO reason #1 (wikilinks).
|
||||
const mockResolver: NameResolver = {
|
||||
nameOf(uuid: string): string | undefined {
|
||||
return ({ "JournalEntry.susan": "Susan Delgado", "JournalEntry.gilead": "Gilead" } as Record<string, string>)[uuid];
|
||||
},
|
||||
uuidOf(name: string): string | undefined {
|
||||
return ({ "Susan Delgado": "JournalEntry.susan", "Gilead": "JournalEntry.gilead" } as Record<string, string>)[name.trim()];
|
||||
},
|
||||
};
|
||||
const baseEntry: JournalEntry = { name: "Test", _id: "x", folder: "Folder.test", flags: {} };
|
||||
|
||||
function roundTrip(body: string): boolean {
|
||||
const entry = obsidianToFoundryJsonLive(body, "Test", baseEntry, mockResolver);
|
||||
const data = entry.flags!["campaign-codex"]!.data!;
|
||||
const bodyMd = htmlToMarkdown(data.description ?? "");
|
||||
const notesMd = htmlToMarkdown(typeof data.notes === "string" ? data.notes : "");
|
||||
const reconstructed = notesMd ? `${bodyMd}\n\n## Secrets\n\n${notesMd}` : bodyMd;
|
||||
return canonicalize(reconstructed) === canonicalize(body);
|
||||
}
|
||||
|
||||
const results: { name: string; pass: boolean; reason: string }[] = [];
|
||||
function record(name: string, body: string, reason: string): boolean {
|
||||
const pass = roundTrip(body);
|
||||
results.push({ name, pass, reason });
|
||||
return pass;
|
||||
}
|
||||
|
||||
describe("E1a.1 round-trip spike — STABLE fixtures (round-trip OK with realistic formatting)", () => {
|
||||
it("plain text", () => {
|
||||
expect(record("plain text", "Roland is the gunslinger of Gilead.", "")).toBe(true);
|
||||
});
|
||||
it("headings (blank line after each, project style)", () => {
|
||||
expect(record("headings", "## Background\n\nRoland was born in Gilead.\n\n## Training\n\nHe trained under Cort.", "")).toBe(true);
|
||||
});
|
||||
it("unordered list", () => {
|
||||
expect(record("unordered list", "- Revolver\n- Sandalwood\n- Horn of Eld", "")).toBe(true);
|
||||
});
|
||||
it("ordered list", () => {
|
||||
expect(record("ordered list", "1. First\n2. Second\n3. Third", "")).toBe(true);
|
||||
});
|
||||
it("image:  round-trips (inverse tuned to )", () => {
|
||||
expect(record("image", "Portrait below.\n\n", "")).toBe(true);
|
||||
});
|
||||
it("entity + whitespace drift (ampersand, blank-line runs, trailing spaces)", () => {
|
||||
expect(record("entity/whitespace", "Roland & Jake \n\n\n\nare companions.", "")).toBe(true);
|
||||
});
|
||||
it("secrets last (the contract's order assumption, blank-line formatted)", () => {
|
||||
expect(record("secrets last", "## Background\n\nPublic info.\n\n## Secrets\n\nHe killed the boy.", "")).toBe(true);
|
||||
});
|
||||
});
|
||||
|
||||
describe("E1a.1 round-trip spike — UNSTABLE fixtures (NO-GO evidence)", () => {
|
||||
it("wikilinks — @UUID cannot round-trip without a resolver (NO-GO #1, fundamental)", () => {
|
||||
// Forward (markdownToHtml + wikiToUuid) converts [[Susan Delgado]] →
|
||||
// @UUID[JournalEntry.susan]{Susan Delgado} via the resolver. The (html)=>string
|
||||
// seam has NO resolver, so the inverse leaves @UUID in the markdown. Original
|
||||
// has [[Susan Delgado]] → mismatch. ANY linked note with cross-references
|
||||
// fails. Fix requires changing the seam to (html, resolver) => string (a
|
||||
// contract change) AND a resolver at guard time.
|
||||
const pass = record("wikilinks", "He loved [[Susan Delgado]] of [[Gilead]].", "@UUID has no resolver on the (html)=>string seam");
|
||||
expect(pass).toBe(false);
|
||||
});
|
||||
|
||||
it("table — forward (mdToHtml) has no table branch (NO-GO #2, fundamental)", () => {
|
||||
// markdownToHtml (src/mdToHtml.ts) has no table handling; table rows are
|
||||
// parsed as paragraphs. A table body cannot round-trip through the forward
|
||||
// transform regardless of the inverse. Fix requires adding table support to
|
||||
// the forward push transform (a feature).
|
||||
const pass = record("table", "| Name | Role |\n| --- | --- |\n| Roland | Gunslinger |\n| Jake | Boy |", "forward has no table branch");
|
||||
expect(pass).toBe(false);
|
||||
});
|
||||
|
||||
it("secrets in middle — forward moves ## Secrets to end (NO-GO #3, fundamental)", () => {
|
||||
// Forward MOVES ## Secrets to data.notes (always last); reconstruction puts
|
||||
// it last, reordering vs. the original where Relationships follows Secrets.
|
||||
// Fix requires a project convention that ## Secrets is always last — not
|
||||
// enforceable by the hash, and the forward itself doesn't enforce it.
|
||||
const pass = record("secrets in middle", "## Background\n\nPublic.\n\n## Secrets\n\nHidden.\n\n## Relationships\n\nSusan.", "## Secrets reordered to end");
|
||||
expect(pass).toBe(false);
|
||||
});
|
||||
|
||||
it("## SECRETS uppercase — heading case not normalized (NO-GO #4, fundamental)", () => {
|
||||
// The contract re-inserts "## Secrets" (title case); canonicalize does NOT
|
||||
// normalize case. An uppercase "## SECRETS" heading won't match. Fix requires
|
||||
// the project to standardize on exactly "## Secrets" (case-sensitive) — not
|
||||
// enforceable by the hash.
|
||||
const pass = record("## SECRETS uppercase", "## Background\n\nPublic.\n\n## SECRETS\n\nHidden.", "## Secrets vs ## SECRETS (case-sensitive)");
|
||||
expect(pass).toBe(false);
|
||||
});
|
||||
|
||||
it("body starts with **bold** — forward parseBody mis-extracts a tagline (NO-GO #5, forward bug)", () => {
|
||||
// parseBody (src/toFoundry.ts:18) extracts a tagline via /\*([^*]+)\*/, which
|
||||
// matches the inner *his revolver* of **his revolver**, mangling any body
|
||||
// that starts with bold. The inverse faithfully converts the mangled HTML
|
||||
// back, so the round-trip can never match. A forward-transform bug, distinct
|
||||
// from the inverse — but it makes the markdown round-trip unstable for a
|
||||
// whole class of bodies regardless of inverse quality.
|
||||
const pass = record("bold-leading", "He drew **his revolver** and *sighed*.\n\n---\n\n> The man in black fled.", "parseBody tagline regex matches inside **bold**");
|
||||
expect(pass).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
describe("E1a.1 GO/NO-GO verdict", () => {
|
||||
it("verdict: NO-GO — 7 stable / 5 unstable (4 fundamental + 1 forward bug) → E1b-alt", () => {
|
||||
const passed = results.filter((r) => r.pass);
|
||||
const failed = results.filter((r) => !r.pass);
|
||||
const verdict = failed.length === 0 ? "GO" : "NO-GO";
|
||||
|
||||
const summary = [
|
||||
`E1a.1 verdict: ${verdict}`,
|
||||
` stable (${passed.length}): ${passed.map((r) => r.name).join(", ")}`,
|
||||
` unstable (${failed.length}):`,
|
||||
...failed.map((r) => ` - ${r.name} — ${r.reason}`),
|
||||
``,
|
||||
` Recommendation: E1b adopts the E1b-alt fork — canonicalize Foundry HTML`,
|
||||
` directly and hash the HTML (contentHash(canonicalizeHtml(data.description`,
|
||||
` + data.notes) + name + folder)), never hash markdown. Sidesteps all 5`,
|
||||
` failure reasons: no inverse, no resolver, no blank-line/case/order`,
|
||||
` sensitivity, no parseBody tagline coupling. The E0.2 markdown ccHash`,
|
||||
` contract is superseded; E1b-alt re-baselines it (HtmlToMarkdown seam →`,
|
||||
` CanonicalizeHtml seam).`,
|
||||
].join("\n");
|
||||
// eslint-disable-next-line no-console
|
||||
console.log("\n" + summary + "\n");
|
||||
|
||||
expect(verdict).toBe("NO-GO");
|
||||
expect(passed.map((r) => r.name).sort()).toEqual(
|
||||
["entity/whitespace", "headings", "image", "ordered list", "plain text", "secrets last", "unordered list"].sort(),
|
||||
);
|
||||
expect(failed.map((r) => r.name).sort()).toEqual(
|
||||
["## SECRETS uppercase", "bold-leading", "secrets in middle", "table", "wikilinks"].sort(),
|
||||
);
|
||||
});
|
||||
});
|
||||
200
tests/e1b1-noclobber.test.ts
Normal file
200
tests/e1b1-noclobber.test.ts
Normal file
@@ -0,0 +1,200 @@
|
||||
// E1b.1c — no-clobber regression test. Exercises the REAL pushNote (with the
|
||||
// prePushGuard) against a MOCKED relay (globalThis.fetch), so we can assert
|
||||
// whether the PUT (/update) actually fired. The guard compares ccHash(liveEntry
|
||||
// from /get) to the note's foundry.ccHash baseline and aborts the PUT on drift.
|
||||
//
|
||||
// Cases:
|
||||
// 1. clean (live matches baseline) → guard proceeds → PUT fires.
|
||||
// 2. drift (Foundry-side edit between baseline and save) → guard aborts → NO PUT.
|
||||
// 3. legacy note (no foundry.ccHash) → guard proceeds → PUT fires (one-time migration).
|
||||
// 4. relay /get unreadable (missing campaign-codex data) → fail-safe abort → NO PUT.
|
||||
//
|
||||
// Runs with sync-state.json ABSENT (process never touches it) — proving E1b.1
|
||||
// has no E4 dependency. Live end-to-end (SM-2) stays gated on the operator's
|
||||
// headless session; this is the offline-testable guard logic.
|
||||
|
||||
import { describe, it, expect, vi, beforeEach, afterEach } from "vitest";
|
||||
import { mkdtemp, writeFile, mkdir, rm } from "node:fs/promises";
|
||||
import { join } from "node:path";
|
||||
import { tmpdir } from "node:os";
|
||||
|
||||
import { AutoSyncController } from "../src/server.js";
|
||||
import type { State, ServerConfig } from "../src/server.js";
|
||||
import { ccHash } from "../src/cchash.js";
|
||||
import type { JournalEntry } from "../src/types.js";
|
||||
|
||||
const UUID = "JournalEntry.abc1";
|
||||
const REL = "Roland.md";
|
||||
|
||||
/** Build a live JournalEntry with given description HTML (and empty notes). */
|
||||
function liveEntry(description: string, name = "Roland"): JournalEntry {
|
||||
return {
|
||||
name,
|
||||
_id: "abc1",
|
||||
folder: "Folder.test",
|
||||
flags: { "campaign-codex": { type: "npc", data: { description, notes: "" } } },
|
||||
};
|
||||
}
|
||||
|
||||
/** A refined note with foundry.cc_uuid + a STALE contentHash baseline (so the
|
||||
* body-hash gate passes and we reach the ccHash guard) + an optional ccHash. */
|
||||
function seededNote(ccHashBaseline: string | null): string {
|
||||
const lines = [
|
||||
"---",
|
||||
"type: npc",
|
||||
"foundry:",
|
||||
` cc_uuid: ${UUID}`,
|
||||
" cc_type: npc",
|
||||
" folder_path: NPCs",
|
||||
` contentHash: ${"0".repeat(64)}`,
|
||||
];
|
||||
if (ccHashBaseline !== null) lines.push(` ccHash: ${ccHashBaseline}`);
|
||||
lines.push(" syncedAt: 2026-06-22T00:00:00.000Z", "---", "The gunslinger drew his revolver.", "");
|
||||
return lines.join("\n");
|
||||
}
|
||||
|
||||
let dir: string;
|
||||
let state: State;
|
||||
let putCalls: number;
|
||||
let getCalls: number;
|
||||
// The mock relay's current Foundry state. /get returns it; /update applies the
|
||||
// diff so the post-push /get (E1b.3 TOCTOU re-verify) returns the pushed state
|
||||
// (ccHash matches ccHash(pushedEntry) → no false conflict). Tests set this to
|
||||
// the initial live entry and may mutate it to simulate Foundry-side edits.
|
||||
let currentState: JournalEntry;
|
||||
const realFetch = globalThis.fetch;
|
||||
|
||||
beforeEach(async () => {
|
||||
dir = await mkdtemp(join(tmpdir(), "e1b1-"));
|
||||
const refinedDir = join(dir, "refined");
|
||||
const outDir = join(dir, "out");
|
||||
await mkdir(refinedDir, { recursive: true });
|
||||
putCalls = 0;
|
||||
getCalls = 0;
|
||||
currentState = liveEntry("<p>Original body.</p>"); // default; tests override
|
||||
const cfg: ServerConfig = {
|
||||
journal: "",
|
||||
refinedDir,
|
||||
ccDir: "",
|
||||
outDir,
|
||||
mode: "apply",
|
||||
port: 0,
|
||||
host: "",
|
||||
relayCfg: { url: "http://relay.test", apiKey: "k", clientId: "c" },
|
||||
};
|
||||
state = { db: {} as State["db"], cfg, index: null, autosync: null as unknown as State["autosync"] } as unknown as State;
|
||||
|
||||
// Mock the relay transport: /get returns currentState; /update applies the
|
||||
// diff {name, "flags.campaign-codex": cc} so the post-push /get returns the
|
||||
// pushed state; /search returns [] (empty resolver, fine for no [[ ]] links).
|
||||
globalThis.fetch = vi.fn(async (url: string, init?: RequestInit) => {
|
||||
const u = String(url);
|
||||
const method = init?.method ?? "GET";
|
||||
const body = init?.body ? JSON.parse(String(init.body)) : undefined;
|
||||
const resp = (o: unknown, ok = true, status = 200) => ({
|
||||
ok, status,
|
||||
text: async () => (typeof o === "string" ? o : JSON.stringify(o)),
|
||||
});
|
||||
if (method === "GET" && u.includes("/get?")) { getCalls++; return resp({ data: currentState }); }
|
||||
if (method === "GET" && u.includes("/search")) { return resp({ results: [] }); }
|
||||
if (method === "PUT" && u.includes("/update?")) {
|
||||
putCalls++;
|
||||
const diff = body?.data ?? {};
|
||||
currentState = {
|
||||
...currentState,
|
||||
name: diff.name ?? currentState.name,
|
||||
flags: { ...currentState.flags, "campaign-codex": diff["flags.campaign-codex"] ?? currentState.flags?.["campaign-codex"] },
|
||||
};
|
||||
return resp({ entity: [currentState] });
|
||||
}
|
||||
return resp({ error: "not found" }, false, 404);
|
||||
}) as unknown as typeof fetch;
|
||||
});
|
||||
|
||||
afterEach(async () => {
|
||||
globalThis.fetch = realFetch;
|
||||
await rm(dir, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
async function writeNote(ccHashBaseline: string | null): Promise<void> {
|
||||
await writeFile(join(state.cfg.refinedDir, REL), seededNote(ccHashBaseline), "utf8");
|
||||
}
|
||||
|
||||
async function runProcess(): Promise<AutoSyncController> {
|
||||
const controller = new AutoSyncController(state);
|
||||
await (controller as any).process(REL);
|
||||
return controller;
|
||||
}
|
||||
|
||||
describe("E1b.1 no-clobber guard (AutoSyncController + real pushNote + mock relay)", () => {
|
||||
it("clean: live ccHash matches the foundry.ccHash baseline → guard proceeds → PUT fires", async () => {
|
||||
const baseline = ccHash(liveEntry("<p>Original body.</p>")); // matches liveForGet
|
||||
await writeNote(baseline);
|
||||
const controller = await runProcess();
|
||||
expect(putCalls).toBe(1); // the push PUT fired
|
||||
expect(getCalls).toBe(2); // pushNote's /get + E1b.3's TOCTOU re-verify /get
|
||||
expect(controller.events.some((e) => e.status === "pushed")).toBe(true);
|
||||
});
|
||||
|
||||
it("drift: Foundry-side edit between baseline and save → guard aborts → NO PUT (fail-safe)", async () => {
|
||||
// Baseline was captured from the original body; Foundry has since been edited.
|
||||
const baseline = ccHash(liveEntry("<p>Original body.</p>"));
|
||||
currentState = liveEntry("<p>Foundry-edited body.</p>"); // Foundry drifted
|
||||
await writeNote(baseline);
|
||||
const controller = await runProcess();
|
||||
expect(putCalls).toBe(0); // NO PUT — the Foundry-side edit was not overwritten
|
||||
expect(getCalls).toBe(1); // the /get still happened (reused, no extra round-trip)
|
||||
const skip = controller.events.find((e) => e.status === "skipped" && e.message.includes("Foundry-side edit detected"));
|
||||
expect(skip, `expected a "Foundry-side edit detected" skip log; got: ${JSON.stringify(controller.events)}`).toBeTruthy();
|
||||
});
|
||||
|
||||
it("legacy note (no foundry.ccHash) → guard proceeds → PUT fires (one-time migration path)", async () => {
|
||||
await writeNote(null); // no ccHash baseline
|
||||
currentState = liveEntry("<p>Any body.</p>");
|
||||
const controller = await runProcess();
|
||||
expect(putCalls).toBe(1); // legacy notes proceed; E1b.2 will write the post-push ccHash baseline
|
||||
expect(controller.events.some((e) => e.status === "pushed")).toBe(true);
|
||||
});
|
||||
|
||||
it("relay /get returns an entry missing campaign-codex data → fail-safe abort → NO PUT", async () => {
|
||||
const baseline = ccHash(liveEntry("<p>Original body.</p>"));
|
||||
// Malformed live entry: campaign-codex present but data.description missing.
|
||||
currentState = { name: "Roland", _id: "abc1", folder: "Folder.test", flags: { "campaign-codex": { type: "npc" } } };
|
||||
await writeNote(baseline);
|
||||
const controller = await runProcess();
|
||||
expect(putCalls).toBe(0); // can't hash Foundry side → do not push over it
|
||||
const skip = controller.events.find((e) => e.status === "skipped" && e.message.includes("Foundry side unreadable"));
|
||||
expect(skip, `expected a "Foundry side unreadable" skip log; got: ${JSON.stringify(controller.events)}`).toBeTruthy();
|
||||
});
|
||||
|
||||
it("guard works with sync-state.json ABSENT (no E4 dependency)", async () => {
|
||||
// sync-state.json is never created in this test dir; process doesn't touch it.
|
||||
const baseline = ccHash(liveEntry("<p>Original body.</p>"));
|
||||
await writeNote(baseline);
|
||||
const controller = await runProcess();
|
||||
expect(putCalls).toBe(1);
|
||||
// No sync-state.json was required for the guard to function.
|
||||
void controller;
|
||||
});
|
||||
});
|
||||
|
||||
describe("E1b.1 feature flag AUTOSYNC_FOUNDRY_GUARD (default true)", () => {
|
||||
it("flag ON (default) → drift detected → NO PUT", async () => {
|
||||
const baseline = ccHash(liveEntry("<p>Original body.</p>"));
|
||||
currentState = liveEntry("<p>Foundry-edited body.</p>"); // drift
|
||||
await writeNote(baseline);
|
||||
const controller = new AutoSyncController(state);
|
||||
await (controller as any).process(REL);
|
||||
expect(putCalls).toBe(0); // default ON → guard aborted the drift
|
||||
});
|
||||
|
||||
it("flag OFF → no guard → drift NOT detected → PUT fires (regresses to body-only, UNSAFE — back-compat escape hatch)", async () => {
|
||||
const baseline = ccHash(liveEntry("<p>Original body.</p>"));
|
||||
currentState = liveEntry("<p>Foundry-edited body.</p>"); // drift
|
||||
await writeNote(baseline);
|
||||
const controller = new AutoSyncController(state);
|
||||
(controller as any).foundryGuardEnabled = false; // flag off
|
||||
await (controller as any).process(REL);
|
||||
expect(putCalls).toBe(1); // unsafe: the Foundry-side edit was overwritten
|
||||
});
|
||||
});
|
||||
200
tests/e1b2-baseline.test.ts
Normal file
200
tests/e1b2-baseline.test.ts
Normal file
@@ -0,0 +1,200 @@
|
||||
// E1b.2c — dual re-baseline + self-write-suppression tests.
|
||||
//
|
||||
// Exercises the REAL pushNote (with the guard + pushedEntry) against a MOCKED
|
||||
// relay (globalThis.fetch) that behaves like Foundry: /get returns the current
|
||||
// state, /update applies the diff (so the E1b.3 post-push re-/get returns the
|
||||
// pushed state → no false TOCTOU conflict). Covers:
|
||||
// 1. a clean push re-baselines BOTH foundry.contentHash (body) AND foundry.ccHash
|
||||
// (ccHash(pushedEntry) — the post-push state, no extra /get beyond E1b.3's).
|
||||
// 2. the controller's own baseline write is dropped by self-write suppression
|
||||
// (same relPath + mtime → no debounce timer armed, "self-write (baseline)" log).
|
||||
// 3. a user edit (different mtime) is NOT suppressed (debounce timer armed).
|
||||
// 4. after the TTL expires, a same-mtime event is processed again (timer armed).
|
||||
//
|
||||
// Live SM-2 verification stays gated on the operator's headless session.
|
||||
|
||||
import { describe, it, expect, vi, beforeEach, afterEach } from "vitest";
|
||||
import { mkdtemp, writeFile, mkdir, rm, readFile } from "node:fs/promises";
|
||||
import { join } from "node:path";
|
||||
import { tmpdir } from "node:os";
|
||||
|
||||
import { AutoSyncController } from "../src/server.js";
|
||||
import type { State, ServerConfig } from "../src/server.js";
|
||||
import { ccHash } from "../src/cchash.js";
|
||||
import { contentHash } from "../src/normalize.js";
|
||||
import { obsidianToFoundryJsonLive } from "../src/toFoundry.js";
|
||||
import { splitFrontmatter, readFoundryBlock } from "../src/frontmatter.js";
|
||||
import type { JournalEntry } from "../src/types.js";
|
||||
import type { NameResolver } from "../src/resolver.js";
|
||||
|
||||
const UUID = "JournalEntry.abc1";
|
||||
const REL = "Roland.md";
|
||||
const EMPTY_RESOLVER: NameResolver = { nameOf: () => undefined, uuidOf: () => undefined };
|
||||
|
||||
function liveEntry(description: string, name = "Roland"): JournalEntry {
|
||||
return { name, _id: "abc1", folder: "Folder.test", flags: { "campaign-codex": { type: "npc", data: { description, notes: "" } } } };
|
||||
}
|
||||
|
||||
/** A refined note: STALE contentHash (so the body gate passes) + a ccHash baseline. */
|
||||
function seededNote(ccHashBaseline: string, body: string): string {
|
||||
return [
|
||||
"---", "type: npc", "foundry:",
|
||||
` cc_uuid: ${UUID}`, " cc_type: npc", " folder_path: NPCs",
|
||||
` contentHash: ${"0".repeat(64)}`,
|
||||
` ccHash: ${ccHashBaseline}`,
|
||||
" syncedAt: 2026-06-22T00:00:00.000Z",
|
||||
"---", body, "",
|
||||
].join("\n");
|
||||
}
|
||||
|
||||
let dir: string;
|
||||
let state: State;
|
||||
let putCalls: number;
|
||||
// The mock relay's current Foundry state. /get returns it; /update applies the
|
||||
// diff so the E1b.3 post-push /get returns the pushed state (ccHash matches
|
||||
// ccHash(pushedEntry) → no false conflict). Tests capture `prePush` before
|
||||
// calling process (since /update mutates currentState).
|
||||
let currentState: JournalEntry;
|
||||
const realFetch = globalThis.fetch;
|
||||
|
||||
beforeEach(async () => {
|
||||
dir = await mkdtemp(join(tmpdir(), "e1b2-"));
|
||||
const refinedDir = join(dir, "refined");
|
||||
const outDir = join(dir, "out");
|
||||
await mkdir(refinedDir, { recursive: true });
|
||||
putCalls = 0;
|
||||
currentState = liveEntry("<p>Original body.</p>");
|
||||
const cfg: ServerConfig = {
|
||||
journal: "", refinedDir, ccDir: "", outDir, mode: "apply", port: 0, host: "",
|
||||
relayCfg: { url: "http://relay.test", apiKey: "k", clientId: "c" },
|
||||
};
|
||||
state = { db: {} as State["db"], cfg, index: null, autosync: null as unknown as State["autosync"] } as unknown as State;
|
||||
globalThis.fetch = vi.fn(async (url: string, init?: RequestInit) => {
|
||||
const u = String(url); const method = init?.method ?? "GET";
|
||||
const body = init?.body ? JSON.parse(String(init.body)) : undefined;
|
||||
const resp = (o: unknown, ok = true, status = 200) => ({ ok, status, text: async () => (typeof o === "string" ? o : JSON.stringify(o)) });
|
||||
if (method === "GET" && u.includes("/get?")) return resp({ data: currentState });
|
||||
if (method === "GET" && u.includes("/search")) return resp({ results: [] });
|
||||
if (method === "PUT" && u.includes("/update?")) {
|
||||
putCalls++;
|
||||
const diff = body?.data ?? {};
|
||||
currentState = {
|
||||
...currentState,
|
||||
name: diff.name ?? currentState.name,
|
||||
flags: { ...currentState.flags, "campaign-codex": diff["flags.campaign-codex"] ?? currentState.flags?.["campaign-codex"] },
|
||||
};
|
||||
return resp({ entity: [currentState] });
|
||||
}
|
||||
return resp({ error: "not found" }, false, 404);
|
||||
}) as unknown as typeof fetch;
|
||||
});
|
||||
|
||||
afterEach(async () => {
|
||||
globalThis.fetch = realFetch;
|
||||
await rm(dir, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
async function writeNote(ccHashBaseline: string, body: string): Promise<void> {
|
||||
await writeFile(join(state.cfg.refinedDir, REL), seededNote(ccHashBaseline, body), "utf8");
|
||||
}
|
||||
|
||||
/** The expected pushed entry for a given body + pre-push base entry. */
|
||||
function expectedPushedEntry(body: string, baseEntry: JournalEntry): JournalEntry {
|
||||
return obsidianToFoundryJsonLive(body, "Roland", baseEntry, EMPTY_RESOLVER);
|
||||
}
|
||||
|
||||
describe("E1b.2a dual re-baseline (contentHash + ccHash) on push success", () => {
|
||||
it("a clean push re-baselines BOTH foundry.contentHash and foundry.ccHash", async () => {
|
||||
const body = "The gunslinger drew his revolver.\n";
|
||||
const prePush = currentState;
|
||||
// Baseline = ccHash(prePush) so the E1b.1 guard passes (no Foundry drift).
|
||||
await writeNote(ccHash(prePush), body);
|
||||
const controller = new AutoSyncController(state);
|
||||
await (controller as any).process(REL);
|
||||
|
||||
expect(putCalls).toBe(1); // the push fired
|
||||
|
||||
const md = await readFile(join(state.cfg.refinedDir, REL), "utf8");
|
||||
const { fm, body: bodyAfter } = splitFrontmatter(md);
|
||||
const fb = readFoundryBlock(fm);
|
||||
// contentHash re-baselined to the body hash (idempotency).
|
||||
expect(fb?.contentHash).toBe(contentHash(bodyAfter));
|
||||
expect(fb?.contentHash).toBe(contentHash(body));
|
||||
// ccHash re-baselined to ccHash(pushedEntry) — the post-push Foundry state.
|
||||
expect(fb?.ccHash).toBe(ccHash(expectedPushedEntry(body, prePush)));
|
||||
expect(controller.events.some((e) => e.status === "pushed" && e.message.includes("baselined (content+cc)"))).toBe(true);
|
||||
});
|
||||
|
||||
it("a drift-aborted push does NOT re-baseline (baselines left untouched)", async () => {
|
||||
const body = "The gunslinger drew his revolver.\n";
|
||||
const baselineBefore = ccHash(currentState);
|
||||
await writeNote(baselineBefore, body);
|
||||
currentState = liveEntry("<p>Foundry-edited body.</p>"); // drift → guard aborts
|
||||
|
||||
const controller = new AutoSyncController(state);
|
||||
await (controller as any).process(REL);
|
||||
expect(putCalls).toBe(0); // no PUT
|
||||
|
||||
const md = await readFile(join(state.cfg.refinedDir, REL), "utf8");
|
||||
const fb = readFoundryBlock(splitFrontmatter(md).fm);
|
||||
// Baselines untouched — the note still carries the pre-push baseline.
|
||||
expect(fb?.ccHash).toBe(baselineBefore);
|
||||
expect(fb?.contentHash).toBe("0".repeat(64)); // still the stale baseline
|
||||
});
|
||||
});
|
||||
|
||||
describe("E1b.2b self-write suppression (onChange recognizes the controller's own baseline write)", () => {
|
||||
it("a successful push records the baseline mtime; the same-mtime change is dropped (no re-push)", async () => {
|
||||
const body = "The gunslinger drew his revolver.\n";
|
||||
await writeNote(ccHash(currentState), body);
|
||||
const controller = new AutoSyncController(state);
|
||||
await (controller as any).process(REL);
|
||||
expect(putCalls).toBe(1);
|
||||
|
||||
// Simulate the watcher firing on the controller's own baseline write (file
|
||||
// unchanged since the baseline → same mtime as recorded).
|
||||
(controller as any).onChange("change", "Roland.md", "");
|
||||
|
||||
// No debounce timer armed for the self-write…
|
||||
expect((controller as any).timers.has(REL)).toBe(false);
|
||||
// …and a self-write skip was logged.
|
||||
expect(controller.events.some((e) => e.message.includes("self-write (baseline)"))).toBe(true);
|
||||
// No second push.
|
||||
expect(putCalls).toBe(1);
|
||||
});
|
||||
|
||||
it("a user edit (different mtime) is NOT suppressed — debounce timer arms", async () => {
|
||||
const body = "The gunslinger drew his revolver.\n";
|
||||
await writeNote(ccHash(currentState), body);
|
||||
const controller = new AutoSyncController(state);
|
||||
await (controller as any).process(REL);
|
||||
expect(putCalls).toBe(1);
|
||||
|
||||
// User edits the note (new content → new mtime).
|
||||
await writeFile(join(state.cfg.refinedDir, REL), seededNote(ccHash(currentState), "The gunslinger fled the desert.\n"), "utf8");
|
||||
(controller as any).onChange("change", "Roland.md", "");
|
||||
|
||||
// The debounce timer IS armed (the user edit was not suppressed)…
|
||||
expect((controller as any).timers.has(REL)).toBe(true);
|
||||
// …and no self-write skip was logged for this change.
|
||||
expect(controller.events.some((e) => e.message.includes("self-write (baseline)"))).toBe(false);
|
||||
});
|
||||
|
||||
it("after the suppression TTL expires, a same-mtime event is processed again", async () => {
|
||||
const body = "The gunslinger drew his revolver.\n";
|
||||
await writeNote(ccHash(currentState), body);
|
||||
const controller = new AutoSyncController(state);
|
||||
// Shrink the TTL so the test doesn't wait 2s.
|
||||
(controller as any).baselineSuppressMs = 15;
|
||||
await (controller as any).process(REL);
|
||||
expect(putCalls).toBe(1);
|
||||
|
||||
// Wait past the TTL, then fire the same-mtime change.
|
||||
await new Promise<void>((r) => setTimeout(r, 40));
|
||||
(controller as any).onChange("change", "Roland.md", "");
|
||||
|
||||
// TTL expired → not suppressed → debounce timer armed.
|
||||
expect((controller as any).timers.has(REL)).toBe(true);
|
||||
expect(controller.events.some((e) => e.message.includes("self-write (baseline)"))).toBe(false);
|
||||
});
|
||||
});
|
||||
199
tests/e1b3-toctou.test.ts
Normal file
199
tests/e1b3-toctou.test.ts
Normal file
@@ -0,0 +1,199 @@
|
||||
// E1b.3c — TOCTOU post-push re-verify + conflict-list tests.
|
||||
//
|
||||
// Exercises the REAL pushNote + E1b.3 re-verify against a MOCKED relay with
|
||||
// controls for the re-verify /get (to simulate a concurrent Foundry edit during
|
||||
// the push window, or a re-verify failure). Covers:
|
||||
// 1. clean (re-verify matches ccHash(pushedEntry)) → baselined, no conflict.
|
||||
// 2. TOCTOU mismatch (concurrent edit during the push window) → conflict row,
|
||||
// NO baseline, "TOCTOU conflict" skip log.
|
||||
// 3. re-verify /get failure → error log, NO baseline, push stays live.
|
||||
// 4. conflict cleared on the next successful push of the same uuid.
|
||||
// 5. status() exposes conflictCount + the conflicts list (the endpoint is a
|
||||
// thin wrapper over state.autosync.conflicts).
|
||||
|
||||
import { describe, it, expect, vi, beforeEach, afterEach } from "vitest";
|
||||
import { mkdtemp, writeFile, mkdir, rm, readFile } from "node:fs/promises";
|
||||
import { join } from "node:path";
|
||||
import { tmpdir } from "node:os";
|
||||
|
||||
import { AutoSyncController } from "../src/server.js";
|
||||
import type { State, ServerConfig } from "../src/server.js";
|
||||
import { ccHash } from "../src/cchash.js";
|
||||
import { contentHash } from "../src/normalize.js";
|
||||
import { obsidianToFoundryJsonLive } from "../src/toFoundry.js";
|
||||
import { splitFrontmatter, readFoundryBlock } from "../src/frontmatter.js";
|
||||
import type { JournalEntry } from "../src/types.js";
|
||||
import type { NameResolver } from "../src/resolver.js";
|
||||
|
||||
const UUID = "JournalEntry.abc1";
|
||||
const REL = "Roland.md";
|
||||
const EMPTY_RESOLVER: NameResolver = { nameOf: () => undefined, uuidOf: () => undefined };
|
||||
|
||||
function liveEntry(description: string): JournalEntry {
|
||||
return { name: "Roland", _id: "abc1", folder: "Folder.test", flags: { "campaign-codex": { type: "npc", data: { description, notes: "" } } } };
|
||||
}
|
||||
function seededNote(ccHashBaseline: string, body: string): string {
|
||||
return [
|
||||
"---", "type: npc", "foundry:", ` cc_uuid: ${UUID}`, " cc_type: npc", " folder_path: NPCs",
|
||||
` contentHash: ${"0".repeat(64)}`, ` ccHash: ${ccHashBaseline}`, " syncedAt: 2026-06-22T00:00:00.000Z",
|
||||
"---", body, "",
|
||||
].join("\n");
|
||||
}
|
||||
|
||||
let dir: string;
|
||||
let state: State;
|
||||
let putCalls: number;
|
||||
let currentState: JournalEntry;
|
||||
/** Override for the E1b.3 re-verify /get (the 2nd /get). null → return currentState
|
||||
* (the post-push state). Set to a concurrently-edited entry to simulate a
|
||||
* TOCTOU mismatch; set reVerifyFails to simulate a re-/get failure. */
|
||||
let reVerifyOverride: JournalEntry | null;
|
||||
let reVerifyFails: boolean;
|
||||
const realFetch = globalThis.fetch;
|
||||
|
||||
beforeEach(async () => {
|
||||
dir = await mkdtemp(join(tmpdir(), "e1b3-"));
|
||||
await mkdir(join(dir, "refined"), { recursive: true });
|
||||
putCalls = 0;
|
||||
currentState = liveEntry("<p>Original body.</p>");
|
||||
reVerifyOverride = null;
|
||||
reVerifyFails = false;
|
||||
const cfg: ServerConfig = {
|
||||
journal: "", refinedDir: join(dir, "refined"), ccDir: "", outDir: join(dir, "out"),
|
||||
mode: "apply", port: 0, host: "", relayCfg: { url: "http://relay.test", apiKey: "k", clientId: "c" },
|
||||
};
|
||||
state = { db: {} as State["db"], cfg, index: null, autosync: null as unknown as State["autosync"] } as unknown as State;
|
||||
let getCalls = 0;
|
||||
globalThis.fetch = vi.fn(async (url: string, init?: RequestInit) => {
|
||||
const u = String(url); const method = init?.method ?? "GET";
|
||||
const body = init?.body ? JSON.parse(String(init.body)) : undefined;
|
||||
const resp = (o: unknown, ok = true, status = 200) => ({ ok, status, text: async () => (typeof o === "string" ? o : JSON.stringify(o)) });
|
||||
if (method === "GET" && u.includes("/get?")) {
|
||||
const isReverify = getCalls === 1; // 0=pushNote's /get, 1=E1b.3's re-verify /get
|
||||
getCalls++;
|
||||
if (isReverify && reVerifyFails) return resp({ error: "Request timed out" }, false, 504);
|
||||
if (isReverify && reVerifyOverride) return resp({ data: reVerifyOverride });
|
||||
return resp({ data: currentState });
|
||||
}
|
||||
if (method === "GET" && u.includes("/search")) return resp({ results: [] });
|
||||
if (method === "PUT" && u.includes("/update?")) {
|
||||
putCalls++;
|
||||
const diff = body?.data ?? {};
|
||||
currentState = {
|
||||
...currentState,
|
||||
name: diff.name ?? currentState.name,
|
||||
flags: { ...currentState.flags, "campaign-codex": diff["flags.campaign-codex"] ?? currentState.flags?.["campaign-codex"] },
|
||||
};
|
||||
return resp({ entity: [currentState] });
|
||||
}
|
||||
return resp({ error: "not found" }, false, 404);
|
||||
}) as unknown as typeof fetch;
|
||||
});
|
||||
|
||||
afterEach(async () => {
|
||||
globalThis.fetch = realFetch;
|
||||
await rm(dir, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
async function writeNote(baseline: string, body: string): Promise<void> {
|
||||
await writeFile(join(state.cfg.refinedDir, REL), seededNote(baseline, body), "utf8");
|
||||
}
|
||||
|
||||
function expectedPushedEntry(body: string, base: JournalEntry): JournalEntry {
|
||||
return obsidianToFoundryJsonLive(body, "Roland", base, EMPTY_RESOLVER);
|
||||
}
|
||||
|
||||
describe("E1b.3 TOCTOU post-push re-verify", () => {
|
||||
it("clean: re-verify matches ccHash(pushedEntry) → baselined, no conflict", async () => {
|
||||
const body = "The gunslinger drew his revolver.\n";
|
||||
const prePush = currentState;
|
||||
await writeNote(ccHash(prePush), body);
|
||||
const controller = new AutoSyncController(state);
|
||||
await (controller as any).process(REL);
|
||||
expect(putCalls).toBe(1);
|
||||
expect(controller.conflicts).toEqual([]);
|
||||
const fb = readFoundryBlock(splitFrontmatter(await readFile(join(state.cfg.refinedDir, REL), "utf8")).fm);
|
||||
expect(fb?.ccHash).toBe(ccHash(expectedPushedEntry(body, prePush)));
|
||||
});
|
||||
|
||||
it("TOCTOU mismatch (concurrent Foundry edit during the push window) → conflict row, NO baseline", async () => {
|
||||
const body = "The gunslinger drew his revolver.\n";
|
||||
const prePush = currentState;
|
||||
await writeNote(ccHash(prePush), body);
|
||||
// The re-verify /get returns a DIFFERENT entry (Foundry was edited concurrently
|
||||
// during the push window) → ccHash ≠ ccHash(pushedEntry) → conflict.
|
||||
reVerifyOverride = liveEntry("<p>Concurrent Foundry edit during push.</p>");
|
||||
const controller = new AutoSyncController(state);
|
||||
await (controller as any).process(REL);
|
||||
|
||||
expect(putCalls).toBe(1); // the push fired (Foundry has our content); not rolled back
|
||||
expect(controller.conflicts).toHaveLength(1);
|
||||
const c = controller.conflicts[0];
|
||||
expect(c.uuid).toBe(UUID);
|
||||
expect(c.name).toBe("Roland");
|
||||
expect(c.relPath).toBe(REL);
|
||||
expect(c.obsidianHash).toBe(contentHash(body));
|
||||
expect(c.foundryPreHash).toBe(ccHash(prePush));
|
||||
expect(c.foundryPostHash).toBe(ccHash(reVerifyOverride));
|
||||
expect(controller.events.some((e) => e.status === "skipped" && e.message.includes("TOCTOU conflict"))).toBe(true);
|
||||
// Baselines left untouched so the next save re-surfaces the divergence.
|
||||
const fb = readFoundryBlock(splitFrontmatter(await readFile(join(state.cfg.refinedDir, REL), "utf8")).fm);
|
||||
expect(fb?.ccHash).toBe(ccHash(prePush)); // unchanged
|
||||
expect(fb?.contentHash).toBe("0".repeat(64)); // still the stale baseline
|
||||
});
|
||||
|
||||
it("re-verify /get failure → error log, NO baseline, push stays live (no rollback)", async () => {
|
||||
const body = "The gunslinger drew his revolver.\n";
|
||||
const prePush = currentState;
|
||||
await writeNote(ccHash(prePush), body);
|
||||
reVerifyFails = true; // the re-verify /get times out (504)
|
||||
const controller = new AutoSyncController(state);
|
||||
await (controller as any).process(REL);
|
||||
|
||||
expect(putCalls).toBe(1); // push fired and is live in Foundry (no rollback)
|
||||
expect(controller.conflicts).toEqual([]); // not a conflict — a transient error
|
||||
expect(controller.events.some((e) => e.status === "error" && e.message.includes("TOCTOU re-verify failed"))).toBe(true);
|
||||
// Not baselined (next save may re-push / mis-detect — DM told to reconcile).
|
||||
const fb = readFoundryBlock(splitFrontmatter(await readFile(join(state.cfg.refinedDir, REL), "utf8")).fm);
|
||||
expect(fb?.ccHash).toBe(ccHash(prePush)); // unchanged
|
||||
});
|
||||
|
||||
it("a prior conflict is cleared on the next successful push of the same uuid", async () => {
|
||||
const body1 = "The gunslinger drew his revolver.\n";
|
||||
const prePush = currentState;
|
||||
await writeNote(ccHash(prePush), body1);
|
||||
// First push: concurrent edit → conflict.
|
||||
reVerifyOverride = liveEntry("<p>Concurrent edit.</p>");
|
||||
const controller = new AutoSyncController(state);
|
||||
await (controller as any).process(REL);
|
||||
expect(controller.conflicts).toHaveLength(1);
|
||||
|
||||
// Second push: no concurrent edit, and the note's ccHash now matches the
|
||||
// (post-first-push) Foundry state so the guard passes. Foundry currently
|
||||
// holds the concurrently-edited content (reVerifyOverride from the first
|
||||
// push became currentState? No — currentState was only mutated by /update,
|
||||
// which the first push did). Reset for a clean second push: Foundry holds
|
||||
// the concurrently-edited entry, so baseline the note to it.
|
||||
const foundryNow = liveEntry("<p>Concurrent edit.</p>");
|
||||
currentState = foundryNow;
|
||||
reVerifyOverride = null; // clean re-verify this time
|
||||
const body2 = "The gunslinger fled the desert.\n";
|
||||
await writeNote(ccHash(foundryNow), body2);
|
||||
await (controller as any).process(REL);
|
||||
|
||||
expect(putCalls).toBe(2); // both pushes fired
|
||||
expect(controller.conflicts).toEqual([]); // conflict cleared by the successful push
|
||||
});
|
||||
|
||||
it("status() exposes conflictCount + the conflicts list", async () => {
|
||||
const body = "The gunslinger drew his revolver.\n";
|
||||
await writeNote(ccHash(currentState), body);
|
||||
reVerifyOverride = liveEntry("<p>Concurrent edit.</p>");
|
||||
const controller = new AutoSyncController(state);
|
||||
await (controller as any).process(REL);
|
||||
const s = controller.status();
|
||||
expect(s.conflictCount).toBe(1);
|
||||
expect(s.conflicts).toHaveLength(1);
|
||||
expect(s.conflicts[0].uuid).toBe(UUID);
|
||||
});
|
||||
});
|
||||
181
tests/e1b4-revert.test.ts
Normal file
181
tests/e1b4-revert.test.ts
Normal file
@@ -0,0 +1,181 @@
|
||||
// E1b.4 — per-uuid backup cache + retention + "Revert last push".
|
||||
//
|
||||
// Exercises the REAL pushNote + controller against a MOCKED relay (behaves like
|
||||
// Foundry: /get returns current state, /update applies the diff or, for revert,
|
||||
// receives the FULL backup doc). Covers:
|
||||
// 1. a clean push writes foundry-backups/<uuid>/<iso>.json + records last-push.
|
||||
// 2. retention keeps the last N backups per uuid (older files pruned).
|
||||
// 3. revert restores Foundry to the backup (FULL /update, not a diff) +
|
||||
// re-baselines the note (ccHash = ccHash(backupDoc)).
|
||||
// 4. revert with no last-push record → 400.
|
||||
// 5. revert with a missing backup file → 409.
|
||||
// 6. revert disabled when AUTOSYNC_FOUNDRY_GUARD is off → 404.
|
||||
|
||||
import { describe, it, expect, vi, beforeEach, afterEach } from "vitest";
|
||||
import { mkdtemp, writeFile, mkdir, rm, readFile, readdir } from "node:fs/promises";
|
||||
import { existsSync } from "node:fs";
|
||||
import { join } from "node:path";
|
||||
import { tmpdir } from "node:os";
|
||||
|
||||
import { AutoSyncController } from "../src/server.js";
|
||||
import type { State, ServerConfig } from "../src/server.js";
|
||||
import { ccHash } from "../src/cchash.js";
|
||||
import { obsidianToFoundryJsonLive } from "../src/toFoundry.js";
|
||||
import { splitFrontmatter, readFoundryBlock } from "../src/frontmatter.js";
|
||||
import type { JournalEntry } from "../src/types.js";
|
||||
import type { NameResolver } from "../src/resolver.js";
|
||||
|
||||
const UUID = "JournalEntry.abc1";
|
||||
const REL = "Roland.md";
|
||||
const EMPTY_RESOLVER: NameResolver = { nameOf: () => undefined, uuidOf: () => undefined };
|
||||
|
||||
function liveEntry(description: string): JournalEntry {
|
||||
return { name: "Roland", _id: "abc1", folder: "Folder.test", flags: { "campaign-codex": { type: "npc", data: { description, notes: "" } } } };
|
||||
}
|
||||
function seededNote(ccHashBaseline: string, body: string): string {
|
||||
return [
|
||||
"---", "type: npc", "foundry:", ` cc_uuid: ${UUID}`, " cc_type: npc", " folder_path: NPCs",
|
||||
` contentHash: ${"0".repeat(64)}`, ` ccHash: ${ccHashBaseline}`, " syncedAt: 2026-06-22T00:00:00.000Z",
|
||||
"---", body, "",
|
||||
].join("\n");
|
||||
}
|
||||
|
||||
let dir: string;
|
||||
let state: State;
|
||||
let putCalls: number;
|
||||
let currentState: JournalEntry;
|
||||
let lastUpdateBody: unknown; // the body sent to the last /update (to assert full vs diff)
|
||||
const realFetch = globalThis.fetch;
|
||||
|
||||
beforeEach(async () => {
|
||||
dir = await mkdtemp(join(tmpdir(), "e1b4-"));
|
||||
await mkdir(join(dir, "refined"), { recursive: true });
|
||||
putCalls = 0;
|
||||
currentState = liveEntry("<p>Original body.</p>");
|
||||
lastUpdateBody = null;
|
||||
const cfg: ServerConfig = {
|
||||
journal: "", refinedDir: join(dir, "refined"), ccDir: "", outDir: join(dir, "out"),
|
||||
mode: "apply", port: 0, host: "", relayCfg: { url: "http://relay.test", apiKey: "k", clientId: "c" },
|
||||
};
|
||||
state = { db: {} as State["db"], cfg, index: null, autosync: null as unknown as State["autosync"] } as unknown as State;
|
||||
let getCalls = 0;
|
||||
globalThis.fetch = vi.fn(async (url: string, init?: RequestInit) => {
|
||||
const u = String(url); const method = init?.method ?? "GET";
|
||||
const body = init?.body ? JSON.parse(String(init.body)) : undefined;
|
||||
const resp = (o: unknown, ok = true, status = 200) => ({ ok, status, text: async () => (typeof o === "string" ? o : JSON.stringify(o)) });
|
||||
if (method === "GET" && u.includes("/get?")) { const re = getCalls === 1; getCalls++; return resp({ data: re ? currentState : currentState }); }
|
||||
if (method === "GET" && u.includes("/search")) return resp({ results: [] });
|
||||
if (method === "PUT" && u.includes("/update?")) {
|
||||
putCalls++; lastUpdateBody = body;
|
||||
const diff = body?.data ?? {};
|
||||
currentState = { ...currentState, name: diff.name ?? currentState.name, flags: { ...currentState.flags, "campaign-codex": diff["flags.campaign-codex"] ?? currentState.flags?.["campaign-codex"] } };
|
||||
return resp({ entity: [currentState] });
|
||||
}
|
||||
return resp({ error: "not found" }, false, 404);
|
||||
}) as unknown as typeof fetch;
|
||||
});
|
||||
|
||||
afterEach(async () => {
|
||||
globalThis.fetch = realFetch;
|
||||
await rm(dir, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
async function writeNote(baseline: string, body: string): Promise<void> {
|
||||
await writeFile(join(state.cfg.refinedDir, REL), seededNote(baseline, body), "utf8");
|
||||
}
|
||||
|
||||
function expectedPushedEntry(body: string, base: JournalEntry): JournalEntry {
|
||||
return obsidianToFoundryJsonLive(body, "Roland", base, EMPTY_RESOLVER);
|
||||
}
|
||||
|
||||
describe("E1b.4a per-uuid backup + retention", () => {
|
||||
it("a clean push writes foundry-backups/<uuid>/<iso>.json + records last-push", async () => {
|
||||
const body = "The gunslinger drew his revolver.\n";
|
||||
const prePush = currentState;
|
||||
await writeNote(ccHash(prePush), body);
|
||||
const controller = new AutoSyncController(state);
|
||||
await (controller as any).process(REL);
|
||||
expect(putCalls).toBe(1);
|
||||
|
||||
const rec = controller.lastPushes.get(UUID);
|
||||
expect(rec).toBeTruthy();
|
||||
expect(rec!.uuid).toBe(UUID);
|
||||
expect(rec!.relPath).toBe(REL);
|
||||
// The backup file exists at the per-uuid path.
|
||||
expect(existsSync(rec!.backupPath)).toBe(true);
|
||||
expect(rec!.backupPath).toContain(join("foundry-backups", UUID));
|
||||
// The backup content is the pre-push live entry.
|
||||
const backup = JSON.parse(await readFile(rec!.backupPath, "utf8")) as JournalEntry;
|
||||
expect(backup.flags?.["campaign-codex"]?.data?.description).toBe("<p>Original body.</p>");
|
||||
});
|
||||
|
||||
it("retention keeps only the last N backups per uuid (older pruned)", async () => {
|
||||
const controller = new AutoSyncController(state);
|
||||
(controller as any).backupRetain = 2; // shrink so the test is quick
|
||||
let prePush = currentState;
|
||||
for (let i = 0; i < 3; i++) {
|
||||
const body = `Body version ${i}.\n`;
|
||||
await writeNote(ccHash(currentState), body); // baseline = current Foundry state (guard passes)
|
||||
prePush = currentState;
|
||||
await (controller as any).process(REL);
|
||||
// After the push, currentState is the pushed entry; next iteration baselines to it.
|
||||
}
|
||||
expect(putCalls).toBe(3);
|
||||
const backupDir = join(state.cfg.outDir, "foundry-backups", UUID);
|
||||
const files = await readdir(backupDir);
|
||||
expect(files.length).toBe(2); // pruned to last N (2)
|
||||
});
|
||||
});
|
||||
|
||||
describe("E1b.4b revert last push", () => {
|
||||
it("revert restores Foundry to the backup (FULL /update) + re-baselines the note", async () => {
|
||||
const body = "The gunslinger drew his revolver.\n";
|
||||
const prePush = currentState;
|
||||
await writeNote(ccHash(prePush), body);
|
||||
const controller = new AutoSyncController(state);
|
||||
await (controller as any).process(REL); // push 1: records last-push + backup
|
||||
expect(putCalls).toBe(1);
|
||||
const rec = controller.lastPushes.get(UUID)!;
|
||||
const backupDoc = JSON.parse(await readFile(rec.backupPath, "utf8")) as JournalEntry;
|
||||
|
||||
const result = await controller.revert(UUID);
|
||||
expect(result.code).toBe(200);
|
||||
expect(putCalls).toBe(2); // the revert PUT fired
|
||||
// The revert sent the FULL backup doc (with _id/pages/ownership), not a diff.
|
||||
expect((lastUpdateBody as any)?.data).toEqual(backupDoc);
|
||||
// The note re-baselined to the restored Foundry state (ccHash = ccHash(backupDoc)).
|
||||
const fb = readFoundryBlock(splitFrontmatter(await readFile(join(state.cfg.refinedDir, REL), "utf8")).fm);
|
||||
expect(fb?.ccHash).toBe(ccHash(backupDoc));
|
||||
// The in-memory ccHash baseline also reflects the restored state.
|
||||
expect((controller as any).ccHashBaselines.get(UUID)).toBe(ccHash(backupDoc));
|
||||
expect(controller.events.some((e) => e.message.includes("reverted"))).toBe(true);
|
||||
});
|
||||
|
||||
it("revert with no last-push record → 400", async () => {
|
||||
const controller = new AutoSyncController(state);
|
||||
const result = await controller.revert(UUID);
|
||||
expect(result.code).toBe(400);
|
||||
expect((result.body as { error: string }).error).toMatch(/no last push/);
|
||||
});
|
||||
|
||||
it("revert with a missing backup file → 409", async () => {
|
||||
const body = "The gunslinger drew his revolver.\n";
|
||||
await writeNote(ccHash(currentState), body);
|
||||
const controller = new AutoSyncController(state);
|
||||
await (controller as any).process(REL); // records last-push + writes backup
|
||||
const rec = controller.lastPushes.get(UUID)!;
|
||||
await rm(rec.backupPath, { force: true }); // manual cleanup deletes the backup
|
||||
const result = await controller.revert(UUID);
|
||||
expect(result.code).toBe(409);
|
||||
expect((result.body as { error: string }).error).toMatch(/backup file missing/);
|
||||
});
|
||||
|
||||
it("revert disabled when AUTOSYNC_FOUNDRY_GUARD is off → 404", async () => {
|
||||
const body = "The gunslinger drew his revolver.\n";
|
||||
await writeNote(ccHash(currentState), body);
|
||||
const controller = new AutoSyncController(state);
|
||||
(controller as any).foundryGuardEnabled = false;
|
||||
const result = await controller.revert(UUID);
|
||||
expect(result.code).toBe(404);
|
||||
});
|
||||
});
|
||||
96
tests/e1b5-applymode.test.ts
Normal file
96
tests/e1b5-applymode.test.ts
Normal file
@@ -0,0 +1,96 @@
|
||||
// E1b.5 — apply-mode gating + OFF-by-default + watcher status-note skip.
|
||||
//
|
||||
// Covers:
|
||||
// 1. setEnabled(true) in dev mode → throws "requires --apply mode"; enabled
|
||||
// stays false (the POST handler pre-checks mode → 400; this tests the
|
||||
// defense-in-depth throw).
|
||||
// 2. setEnabled(true) in apply mode → starts (enabled=true); setEnabled(false)
|
||||
// → stops.
|
||||
// 3. constructor sets enabled=false (OFF-by-default / opt-in per session).
|
||||
// 4. onChange skips STATUS_NOTE_PATHS (_meta/, wiki/, .raw/) with a logged
|
||||
// "status-note path" reason and does NOT arm a debounce timer.
|
||||
|
||||
import { describe, it, expect, vi, beforeEach, afterEach } from "vitest";
|
||||
import { mkdtemp, mkdir, rm } from "node:fs/promises";
|
||||
import { join } from "node:path";
|
||||
import { tmpdir } from "node:os";
|
||||
|
||||
import { AutoSyncController } from "../src/server.js";
|
||||
import type { State, ServerConfig } from "../src/server.js";
|
||||
|
||||
const UUID = "JournalEntry.abc1";
|
||||
|
||||
function makeState(mode: "dev" | "apply", refinedDir: string, outDir: string): State {
|
||||
const cfg: ServerConfig = {
|
||||
journal: "", refinedDir, ccDir: "", outDir, mode, port: 0, host: "",
|
||||
relayCfg: { url: "http://relay.test", apiKey: "k", clientId: "c" },
|
||||
};
|
||||
return { db: {} as State["db"], cfg, index: null, autosync: null as unknown as State["autosync"] } as unknown as State;
|
||||
}
|
||||
|
||||
let dir: string;
|
||||
const realFetch = globalThis.fetch;
|
||||
beforeEach(async () => {
|
||||
dir = await mkdtemp(join(tmpdir(), "e1b5-"));
|
||||
await mkdir(join(dir, "refined"), { recursive: true });
|
||||
// No relay calls expected; keep fetch mocked to a no-op so RelayClient
|
||||
// construction (if it happens) doesn't hit the network.
|
||||
globalThis.fetch = vi.fn(async () => ({ ok: true, status: 200, text: async () => "{}" })) as unknown as typeof fetch;
|
||||
});
|
||||
afterEach(async () => {
|
||||
globalThis.fetch = realFetch;
|
||||
await rm(dir, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
describe("E1b.5 apply-mode gating + OFF-by-default", () => {
|
||||
it("constructor sets enabled=false (OFF-by-default / opt-in per session)", () => {
|
||||
const state = makeState("apply", join(dir, "refined"), join(dir, "out"));
|
||||
const controller = new AutoSyncController(state);
|
||||
expect(controller.enabled).toBe(false);
|
||||
});
|
||||
|
||||
it("setEnabled(true) in dev mode throws 'requires --apply mode' and leaves enabled=false", async () => {
|
||||
const state = makeState("dev", join(dir, "refined"), join(dir, "out"));
|
||||
const controller = new AutoSyncController(state);
|
||||
await expect(controller.setEnabled(true)).rejects.toThrow(/requires --apply mode/);
|
||||
expect(controller.enabled).toBe(false); // not enabled
|
||||
});
|
||||
|
||||
it("setEnabled(true) in apply mode starts; setEnabled(false) stops", async () => {
|
||||
const state = makeState("apply", join(dir, "refined"), join(dir, "out"));
|
||||
const controller = new AutoSyncController(state);
|
||||
await controller.setEnabled(true);
|
||||
expect(controller.enabled).toBe(true);
|
||||
await controller.setEnabled(false);
|
||||
expect(controller.enabled).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
describe("E1b.5 watcher status-note path skip", () => {
|
||||
it("onChange skips a save under _meta/ with a 'status-note path' log and no debounce timer", () => {
|
||||
const state = makeState("apply", join(dir, "refined"), join(dir, "out"));
|
||||
const controller = new AutoSyncController(state);
|
||||
(controller as any).onChange("change", "hot.md", "_meta"); // rel = "_meta/hot.md"
|
||||
expect(controller.events.some((e) => e.status === "skipped" && e.message === "status-note path")).toBe(true);
|
||||
expect((controller as any).timers.has("_meta/hot.md")).toBe(false);
|
||||
});
|
||||
|
||||
it("onChange skips wiki/ and .raw/ paths too", () => {
|
||||
const state = makeState("apply", join(dir, "refined"), join(dir, "out"));
|
||||
const controller = new AutoSyncController(state);
|
||||
(controller as any).onChange("change", "modules/server.md", "wiki");
|
||||
(controller as any).onChange("change", "README.md", ".raw");
|
||||
expect(controller.events.filter((e) => e.message === "status-note path").length).toBe(2);
|
||||
});
|
||||
|
||||
it("onChange does NOT skip a normal vault note (arms the debounce timer)", () => {
|
||||
const state = makeState("apply", join(dir, "refined"), join(dir, "out"));
|
||||
const controller = new AutoSyncController(state);
|
||||
// A normal note under a content dir — should NOT be skipped at the watcher
|
||||
// (it may be skipped later in process if unlinked/unseeded, but the watcher
|
||||
// arms the debounce). Write a file so statSync in the debounce pre-check works.
|
||||
(controller as any).onChange("change", "Roland.md", "npcs"); // rel = "npcs/Roland.md"
|
||||
expect((controller as any).timers.has("npcs/Roland.md")).toBe(true);
|
||||
expect(controller.events.some((e) => e.message === "status-note path")).toBe(false);
|
||||
});
|
||||
});
|
||||
155
tests/e1b6-burst.test.ts
Normal file
155
tests/e1b6-burst.test.ts
Normal file
@@ -0,0 +1,155 @@
|
||||
// E1b.6b — 50-note burst validation harness.
|
||||
//
|
||||
// Drives the queue/drain directly (deterministic) with a MOCKED pushNote (100ms
|
||||
// delay + concurrency counter) — the watcher→debounce→queue path is exercised
|
||||
// in e1b6-watch.test.ts; here we validate the drain semantics the AC requires:
|
||||
// (a) exactly N process invocations complete,
|
||||
// (b) no uuid pushed twice (distinct uuids; and a same-uuid pair where the
|
||||
// second is skip-dropped, no duplicate PUT),
|
||||
// (c) max concurrent pushNote ≤ concurrency (3),
|
||||
// (d) wall time bounded (<30s with concurrency 3 + 100ms latency),
|
||||
// (e) a push that throws releases its slot in `finally` and the queue drains
|
||||
// the rest (one failure does not stall the burst).
|
||||
//
|
||||
// foundryGuardEnabled is forced off so the guard + TOCTOU re-/get are skipped
|
||||
// (no relay /get) — those are tested in e1b1/e1b3; the burst isolates the
|
||||
// concurrency/drain/lock behavior. pushNote is mocked (vi.mock) so the 100ms
|
||||
// latency + concurrency tracking are controllable.
|
||||
|
||||
import { describe, it, expect, vi, beforeEach, afterEach } from "vitest";
|
||||
import { mkdtemp, writeFile, mkdir, rm } from "node:fs/promises";
|
||||
import { join } from "node:path";
|
||||
import { tmpdir } from "node:os";
|
||||
|
||||
vi.mock("../src/push.js", () => ({
|
||||
pushNote: vi.fn(async () => ({
|
||||
dryRun: false, ccUuid: "JournalEntry.x", diff: {}, imageNote: "",
|
||||
liveEntry: { name: "x", _id: "x", folder: "F", flags: { "campaign-codex": { type: "npc", data: { description: "<p>x</p>", notes: "" } } } },
|
||||
pushedEntry: { name: "x", _id: "x", folder: "F", flags: { "campaign-codex": { type: "npc", data: { description: "<p>x</p>", notes: "" } } } },
|
||||
})),
|
||||
}));
|
||||
|
||||
import { pushNote } from "../src/push.js";
|
||||
import { AutoSyncController } from "../src/server.js";
|
||||
import type { State, ServerConfig } from "../src/server.js";
|
||||
|
||||
const VALID_OUTCOME = {
|
||||
dryRun: false as const, ccUuid: "JournalEntry.x", diff: {}, imageNote: "",
|
||||
liveEntry: { name: "x", _id: "x", folder: "F", flags: { "campaign-codex": { type: "npc", data: { description: "<p>x</p>", notes: "" } } } },
|
||||
pushedEntry: { name: "x", _id: "x", folder: "F", flags: { "campaign-codex": { type: "npc", data: { description: "<p>x</p>", notes: "" } } } },
|
||||
};
|
||||
|
||||
let dir: string;
|
||||
let state: State;
|
||||
|
||||
function sleep(ms: number): Promise<void> { return new Promise((r) => setTimeout(r, ms)); }
|
||||
|
||||
beforeEach(async () => {
|
||||
dir = await mkdtemp(join(tmpdir(), "e1b6-"));
|
||||
await mkdir(join(dir, "refined"), { recursive: true });
|
||||
const cfg: ServerConfig = {
|
||||
journal: "", refinedDir: join(dir, "refined"), ccDir: "", outDir: join(dir, "out"),
|
||||
mode: "apply", port: 0, host: "", relayCfg: { url: "http://relay.test", apiKey: "k", clientId: "c" },
|
||||
};
|
||||
state = { db: {} as State["db"], cfg, index: null, autosync: null as unknown as State["autosync"] } as unknown as State;
|
||||
});
|
||||
|
||||
afterEach(async () => { vi.mocked(pushNote).mockReset(); await rm(dir, { recursive: true, force: true }); });
|
||||
|
||||
function seededNote(ccUuid: string, body: string): string {
|
||||
return [
|
||||
"---", "type: npc", "foundry:", ` cc_uuid: ${ccUuid}`, " cc_type: npc", " folder_path: NPCs",
|
||||
` contentHash: ${"0".repeat(64)}`, " syncedAt: 2026-06-22T00:00:00.000Z", "---", body, "",
|
||||
].join("\n");
|
||||
}
|
||||
|
||||
/** A controller with the guard off (no relay /get) + a pushNote mock tracking
|
||||
* concurrency and per-note call counts. `throwOn` makes pushNote reject for
|
||||
* that noteName (to test that a throw releases the slot). */
|
||||
function makeController(opts: { concurrency?: number; throwOn?: string; delayMs?: number } = {}): AutoSyncController {
|
||||
const c = new AutoSyncController(state);
|
||||
(c as any).foundryGuardEnabled = false; // skip guard + TOCTOU (no relay /get)
|
||||
if (opts.concurrency) (c as any).concurrency = opts.concurrency;
|
||||
const delay = opts.delayMs ?? 100;
|
||||
(c as any)._inFlight = 0; (c as any)._maxInFlight = 0; (c as any)._callsByNote = {};
|
||||
vi.mocked(pushNote).mockImplementation(async (deps: any) => {
|
||||
(c as any)._inFlight++; (c as any)._maxInFlight = Math.max((c as any)._maxInFlight, (c as any)._inFlight);
|
||||
await sleep(delay);
|
||||
(c as any)._inFlight--;
|
||||
(c as any)._callsByNote[deps.noteName] = ((c as any)._callsByNote[deps.noteName] ?? 0) + 1;
|
||||
if (opts.throwOn && deps.noteName === opts.throwOn) throw new Error("relay 500");
|
||||
return VALID_OUTCOME;
|
||||
});
|
||||
return c;
|
||||
}
|
||||
|
||||
/** Queue relPaths and drain; resolve when the queue is empty and nothing running. */
|
||||
async function burst(controller: AutoSyncController, rels: string[]): Promise<void> {
|
||||
for (const rel of rels) (controller as any).queue.push(rel);
|
||||
(controller as any).drain();
|
||||
// Wait until drained.
|
||||
const deadline = Date.now() + 30000;
|
||||
while (((controller as any).queue.length > 0 || (controller as any).running > 0) && Date.now() < deadline) {
|
||||
await sleep(10);
|
||||
}
|
||||
}
|
||||
|
||||
describe("E1b.6b 50-note burst (concurrency / no-dup / bounded / throw-drains)", () => {
|
||||
it("50 distinct-uuid notes: 50 completions, concurrency ≤3, no uuid pushed twice, bounded time", async () => {
|
||||
const N = 50;
|
||||
const rels: string[] = [];
|
||||
for (let i = 0; i < N; i++) {
|
||||
const name = `N${i}`;
|
||||
await writeFile(join(state.cfg.refinedDir, `${name}.md`), seededNote(`JournalEntry.${name}`, `body ${i}`), "utf8");
|
||||
rels.push(`${name}.md`);
|
||||
}
|
||||
const c = makeController({ concurrency: 3, delayMs: 100 });
|
||||
const start = Date.now();
|
||||
await burst(c, rels);
|
||||
const elapsed = Date.now() - start;
|
||||
|
||||
// (a) exactly N process completions → N pushNote calls.
|
||||
const totalCalls = Object.values((c as any)._callsByNote as Record<string, number>).reduce((a, b) => a + b, 0);
|
||||
expect(totalCalls).toBe(N);
|
||||
// (b) no uuid pushed twice — each note called exactly once.
|
||||
for (const rel of rels) expect((c as any)._callsByNote[rel.slice(0, -3)]).toBe(1);
|
||||
// (c) max concurrent pushNote ≤ concurrency (3).
|
||||
expect((c as any)._maxInFlight).toBeLessThanOrEqual(3);
|
||||
expect((c as any)._maxInFlight).toBeGreaterThan(0);
|
||||
// (d) bounded wall time (<30s; with 50/3*100ms ≈ 1.7s + baseline I/O).
|
||||
expect(elapsed).toBeLessThan(30000);
|
||||
});
|
||||
|
||||
it("two relPaths with the SAME cc_uuid (rename): the second is skip-dropped — no duplicate PUT", async () => {
|
||||
// A.md and B.md both link to JournalEntry.SAME (a rename within the vault).
|
||||
await writeFile(join(state.cfg.refinedDir, "A.md"), seededNote("JournalEntry.SAME", "body A"), "utf8");
|
||||
await writeFile(join(state.cfg.refinedDir, "B.md"), seededNote("JournalEntry.SAME", "body B"), "utf8");
|
||||
const c = makeController({ concurrency: 3, delayMs: 100 });
|
||||
await burst(c, ["A.md", "B.md"]);
|
||||
// Exactly one pushNote call for the shared uuid (the second save was skip-dropped).
|
||||
const aCalls = (c as any)._callsByNote["A"] ?? 0;
|
||||
const bCalls = (c as any)._callsByNote["B"] ?? 0;
|
||||
expect(aCalls + bCalls).toBe(1);
|
||||
// The dropped one logged "lock busy".
|
||||
expect(c.events.some((e) => e.message.includes("lock busy"))).toBe(true);
|
||||
});
|
||||
|
||||
it("a push that throws releases its slot in finally — the queue drains the rest", async () => {
|
||||
const N = 6;
|
||||
const rels: string[] = [];
|
||||
for (let i = 0; i < N; i++) {
|
||||
const name = `N${i}`;
|
||||
await writeFile(join(state.cfg.refinedDir, `${name}.md`), seededNote(`JournalEntry.${name}`, `body ${i}`), "utf8");
|
||||
rels.push(`${name}.md`);
|
||||
}
|
||||
const c = makeController({ concurrency: 3, delayMs: 50, throwOn: "N3" });
|
||||
await burst(c, rels);
|
||||
// The failed note (N3) logged an error; the other N-1 pushed successfully.
|
||||
expect(c.events.filter((e) => e.status === "error" && e.message.includes("relay 500")).length).toBe(1);
|
||||
expect(c.events.filter((e) => e.status === "pushed").length).toBe(N - 1);
|
||||
// The slot was released in finally — the queue drained completely (one
|
||||
// failure did not stall the burst).
|
||||
expect((c as any).queue.length).toBe(0);
|
||||
expect((c as any).running).toBe(0);
|
||||
});
|
||||
});
|
||||
83
tests/e1b6-watch.test.ts
Normal file
83
tests/e1b6-watch.test.ts
Normal file
@@ -0,0 +1,83 @@
|
||||
// E1b.6c — recursive-watch fallback (rescanSubs) + debounce collapse.
|
||||
//
|
||||
// The non-recursive platform fallback watches the root + every subdir
|
||||
// individually and re-scans on renames so new folders get watched. The debounce
|
||||
// (700ms per relPath) collapses rapid successive saves of the same file to one
|
||||
// push.
|
||||
|
||||
import { describe, it, expect, beforeEach, afterEach } from "vitest";
|
||||
import { mkdtemp, mkdir, rm } from "node:fs/promises";
|
||||
import { join } from "node:path";
|
||||
import { tmpdir } from "node:os";
|
||||
|
||||
import { AutoSyncController } from "../src/server.js";
|
||||
import type { State, ServerConfig } from "../src/server.js";
|
||||
|
||||
let dir: string;
|
||||
let state: State;
|
||||
|
||||
beforeEach(async () => {
|
||||
dir = await mkdtemp(join(tmpdir(), "e1b6w-"));
|
||||
await mkdir(join(dir, "refined"), { recursive: true });
|
||||
const cfg: ServerConfig = {
|
||||
journal: "", refinedDir: join(dir, "refined"), ccDir: "", outDir: join(dir, "out"),
|
||||
mode: "apply", port: 0, host: "", relayCfg: { url: "http://relay.test", apiKey: "k", clientId: "c" },
|
||||
};
|
||||
state = { db: {} as State["db"], cfg, index: null, autosync: null as unknown as State["autosync"] } as unknown as State;
|
||||
});
|
||||
|
||||
afterEach(async () => { await rm(dir, { recursive: true, force: true }); });
|
||||
|
||||
describe("E1b.6c recursive-watch fallback (rescanSubs)", () => {
|
||||
it("rescanSubs attaches watchers to subdirs (excluding .obsidian / dotfiles)", async () => {
|
||||
const controller = new AutoSyncController(state);
|
||||
// Create subdirs under the refined root.
|
||||
const sub1 = join(state.cfg.refinedDir, "npcs");
|
||||
const dot = join(state.cfg.refinedDir, ".obsidian");
|
||||
const sub2 = join(state.cfg.refinedDir, "places");
|
||||
await Promise.all([sub1, dot, sub2].map((d) => mkdir(d, { recursive: true })));
|
||||
// Force the non-recursive fallback path and rescan.
|
||||
(controller as any).recursive = false;
|
||||
(controller as any).rescanSubs(state.cfg.refinedDir);
|
||||
const prefixes = (controller as any).subs.map((s: any) => s.prefix);
|
||||
expect(prefixes).toContain("npcs");
|
||||
expect(prefixes).toContain("places");
|
||||
expect(prefixes).not.toContain(".obsidian"); // dotfile dir skipped
|
||||
controller.stop(); // close the attached watchers
|
||||
});
|
||||
|
||||
it("a subdir created mid-burst is picked up by a subsequent rescanSubs", async () => {
|
||||
const controller = new AutoSyncController(state);
|
||||
(controller as any).recursive = false;
|
||||
(controller as any).rescanSubs(state.cfg.refinedDir); // initial scan (no subdirs)
|
||||
expect((controller as any).subs.length).toBe(0);
|
||||
// A new folder appears mid-burst.
|
||||
await mkdir(join(state.cfg.refinedDir, "newfolder"), { recursive: true });
|
||||
(controller as any).rescanSubs(state.cfg.refinedDir); // re-scan picks it up
|
||||
expect((controller as any).subs.map((s: any) => s.prefix)).toContain("newfolder");
|
||||
controller.stop();
|
||||
});
|
||||
});
|
||||
|
||||
describe("E1b.6c debounce (rapid saves of the same file collapse to one timer)", () => {
|
||||
it("two rapid onChange for the same rel collapse to ONE debounce timer", () => {
|
||||
const controller = new AutoSyncController(state);
|
||||
(controller as any).onChange("change", "Roland.md", "");
|
||||
(controller as any).onChange("change", "Roland.md", ""); // second clears the first
|
||||
const timers = (controller as any).timers as Map<string, unknown>;
|
||||
expect(timers.has("Roland.md")).toBe(true); // exactly one timer for the rel
|
||||
expect(timers.size).toBe(1);
|
||||
controller.stop(); // clear the armed timer (no process fires)
|
||||
});
|
||||
|
||||
it("different rels arm independent timers", () => {
|
||||
const controller = new AutoSyncController(state);
|
||||
(controller as any).onChange("change", "A.md", "");
|
||||
(controller as any).onChange("change", "B.md", "");
|
||||
const timers = (controller as any).timers as Map<string, unknown>;
|
||||
expect(timers.has("A.md")).toBe(true);
|
||||
expect(timers.has("B.md")).toBe(true);
|
||||
expect(timers.size).toBe(2);
|
||||
controller.stop();
|
||||
});
|
||||
});
|
||||
165
tests/e1b7-retry.test.ts
Normal file
165
tests/e1b7-retry.test.ts
Normal file
@@ -0,0 +1,165 @@
|
||||
// E1b.7 — transient/persistent retry policy.
|
||||
//
|
||||
// classifyRelayError: transient (408/504/500/network) → retry; persistent
|
||||
// (401/404/etc) → no retry. pushWithRetry: 3 attempts (foundryGuardEnabled),
|
||||
// per-attempt lock, exponential backoff (500/1500/4500ms ±20%), retrying-set
|
||||
// drops concurrent same-uuid saves during backoff.
|
||||
|
||||
import { describe, it, expect, vi, beforeEach, afterEach } from "vitest";
|
||||
import { mkdtemp, writeFile, mkdir, rm, readFile } from "node:fs/promises";
|
||||
import { join } from "node:path";
|
||||
import { tmpdir } from "node:os";
|
||||
|
||||
import { AutoSyncController, classifyRelayError } from "../src/server.js";
|
||||
import type { State, ServerConfig } from "../src/server.js";
|
||||
import { ccHash } from "../src/cchash.js";
|
||||
import { obsidianToFoundryJsonLive } from "../src/toFoundry.js";
|
||||
import { splitFrontmatter, readFoundryBlock } from "../src/frontmatter.js";
|
||||
import type { JournalEntry } from "../src/types.js";
|
||||
import type { NameResolver } from "../src/resolver.js";
|
||||
|
||||
const UUID = "JournalEntry.abc1";
|
||||
const REL = "Roland.md";
|
||||
const EMPTY_RESOLVER: NameResolver = { nameOf: () => undefined, uuidOf: () => undefined };
|
||||
|
||||
function liveEntry(description: string): JournalEntry {
|
||||
return { name: "Roland", _id: "abc1", folder: "Folder.test", flags: { "campaign-codex": { type: "npc", data: { description, notes: "" } } } };
|
||||
}
|
||||
function seededNote(ccHashBaseline: string, body: string): string {
|
||||
return [
|
||||
"---", "type: npc", "foundry:", ` cc_uuid: ${UUID}`, " cc_type: npc", " folder_path: NPCs",
|
||||
` contentHash: ${"0".repeat(64)}`, ` ccHash: ${ccHashBaseline}`, " syncedAt: 2026-06-22T00:00:00.000Z",
|
||||
"---", body, "",
|
||||
].join("\n");
|
||||
}
|
||||
|
||||
describe("E1b.7a classifyRelayError", () => {
|
||||
it("transient: relay 408/504/500, network failures, 'Request timed out'", () => {
|
||||
expect(classifyRelayError(new Error("relay 504 GET /get: Request timed out"))).toBe("transient");
|
||||
expect(classifyRelayError(new Error("relay 408 PUT /update: Request timed out"))).toBe("transient");
|
||||
expect(classifyRelayError(new Error("relay 500 GET /get: internal"))).toBe("transient");
|
||||
expect(classifyRelayError(new Error("fetch failed: ECONNRESET"))).toBe("transient");
|
||||
expect(classifyRelayError(new Error("getaddrinfo ENOTFOUND relay.test"))).toBe("transient");
|
||||
});
|
||||
it("persistent: 400/401/403/404, 'Invalid client ID', 'No connected Foundry clients found', non-relay", () => {
|
||||
expect(classifyRelayError(new Error("relay 401 GET /get: Unauthorized"))).toBe("persistent");
|
||||
expect(classifyRelayError(new Error("relay 404 GET /get: Invalid client ID"))).toBe("persistent");
|
||||
expect(classifyRelayError(new Error("relay 404 GET /get: No connected Foundry clients found"))).toBe("persistent");
|
||||
expect(classifyRelayError(new Error("relay /get returned no data for JournalEntry.x"))).toBe("persistent");
|
||||
expect(classifyRelayError(new Error("no foundry.cc_uuid in /path — run seed first"))).toBe("persistent");
|
||||
expect(classifyRelayError(new Error("some unknown error"))).toBe("persistent"); // default safer
|
||||
});
|
||||
});
|
||||
|
||||
let dir: string;
|
||||
let state: State;
|
||||
let putCalls: number;
|
||||
let currentState: JournalEntry;
|
||||
const realFetch = globalThis.fetch;
|
||||
|
||||
beforeEach(async () => {
|
||||
dir = await mkdtemp(join(tmpdir(), "e1b7-"));
|
||||
await mkdir(join(dir, "refined"), { recursive: true });
|
||||
putCalls = 0;
|
||||
currentState = liveEntry("<p>Original body.</p>");
|
||||
const cfg: ServerConfig = {
|
||||
journal: "", refinedDir: join(dir, "refined"), ccDir: "", outDir: join(dir, "out"),
|
||||
mode: "apply", port: 0, host: "", relayCfg: { url: "http://relay.test", apiKey: "k", clientId: "c" },
|
||||
};
|
||||
state = { db: {} as State["db"], cfg, index: null, autosync: null as unknown as State["autosync"] } as unknown as State;
|
||||
});
|
||||
|
||||
afterEach(async () => { globalThis.fetch = realFetch; await rm(dir, { recursive: true, force: true }); });
|
||||
|
||||
async function writeNote(baseline: string, body: string): Promise<void> {
|
||||
await writeFile(join(state.cfg.refinedDir, REL), seededNote(baseline, body), "utf8");
|
||||
}
|
||||
|
||||
/** Mock relay: /get fails 504 for the first `failFirst` calls, then returns
|
||||
* currentState; /update applies the diff. */
|
||||
function mockRelayFailThenOk(failFirst: number, failStatus = 504): { getCalls: number } {
|
||||
let getCalls = 0;
|
||||
globalThis.fetch = vi.fn(async (url: string, init?: RequestInit) => {
|
||||
const u = String(url); const method = init?.method ?? "GET";
|
||||
const body = init?.body ? JSON.parse(String(init.body)) : undefined;
|
||||
const resp = (o: unknown, ok = true, status = 200) => ({ ok, status, text: async () => (typeof o === "string" ? o : JSON.stringify(o)) });
|
||||
if (method === "GET" && u.includes("/get?")) {
|
||||
const n = getCalls++;
|
||||
if (n < failFirst) return resp({ error: "Request timed out" }, false, failStatus);
|
||||
return resp({ data: currentState });
|
||||
}
|
||||
if (method === "GET" && u.includes("/search")) return resp({ results: [] });
|
||||
if (method === "PUT" && u.includes("/update?")) {
|
||||
putCalls++;
|
||||
const diff = body?.data ?? {};
|
||||
currentState = { ...currentState, name: diff.name ?? currentState.name, flags: { ...currentState.flags, "campaign-codex": diff["flags.campaign-codex"] ?? currentState.flags?.["campaign-codex"] } };
|
||||
return resp({ entity: [currentState] });
|
||||
}
|
||||
return resp({ error: "not found" }, false, 404);
|
||||
}) as unknown as typeof fetch;
|
||||
return { get getCalls() { return getCalls; } };
|
||||
}
|
||||
|
||||
function newController(): AutoSyncController {
|
||||
const c = new AutoSyncController(state);
|
||||
(c as any).retryBackoffs = [1, 1, 1]; // fast backoffs for the test
|
||||
return c;
|
||||
}
|
||||
|
||||
describe("E1b.7b/c pushWithRetry", () => {
|
||||
it("504 twice then 200 → success on attempt 3 (one PUT, one baseline, TOCTOU re-verify)", async () => {
|
||||
const body = "The gunslinger drew his revolver.\n";
|
||||
const prePush = currentState;
|
||||
await writeNote(ccHash(prePush), body);
|
||||
const mock = mockRelayFailThenOk(2, 504); // first 2 /gets fail 504
|
||||
const controller = newController();
|
||||
await (controller as any).process(REL);
|
||||
|
||||
// Attempt 1 + 2 /get 504'd (transient); attempt 3 /get 200 → push.
|
||||
expect(putCalls).toBe(1); // exactly one PUT (attempt 3)
|
||||
// Two transient-retrying logs (attempts 1, 2) + a final pushed.
|
||||
expect(controller.events.filter((e) => e.message.includes("transient (attempt 1/3)")).length).toBe(1);
|
||||
expect(controller.events.filter((e) => e.message.includes("transient (attempt 2/3)")).length).toBe(1);
|
||||
expect(controller.events.some((e) => e.status === "pushed" && e.message.includes("baselined (content+cc)"))).toBe(true);
|
||||
// Baseline written (ccHash = ccHash(pushedEntry)).
|
||||
const fb = readFoundryBlock(splitFrontmatter(await readFile(join(state.cfg.refinedDir, REL), "utf8")).fm);
|
||||
expect(fb?.ccHash).toBe(ccHash(obsidianToFoundryJsonLive(body, "Roland", prePush, EMPTY_RESOLVER)));
|
||||
void mock;
|
||||
});
|
||||
|
||||
it("401 (persistent) → zero retries, immediate 'persistent' error, no PUT", async () => {
|
||||
const body = "The gunslinger drew his revolver.\n";
|
||||
await writeNote(ccHash(currentState), body);
|
||||
mockRelayFailThenOk(99, 401); // /get always 401
|
||||
const controller = newController();
|
||||
await (controller as any).process(REL);
|
||||
|
||||
expect(putCalls).toBe(0); // no PUT
|
||||
expect(controller.events.some((e) => e.status === "error" && e.message.includes("persistent:"))).toBe(true);
|
||||
expect(controller.events.some((e) => e.message.includes("transient (attempt"))).toBe(false); // no retry logs
|
||||
});
|
||||
|
||||
it("lock-busy (pre-held by 'pull') → 'lock busy — skipped', no PUT", async () => {
|
||||
const body = "The gunslinger drew his revolver.\n";
|
||||
await writeNote(ccHash(currentState), body);
|
||||
mockRelayFailThenOk(0); // would succeed, but the lock is held
|
||||
const controller = newController();
|
||||
expect(controller.lock.acquire(UUID, "pull")).toEqual({ acquired: true }); // pre-hold
|
||||
await (controller as any).process(REL);
|
||||
expect(putCalls).toBe(0); // no PUT (the push was skipped — lock busy)
|
||||
expect(controller.events.some((e) => e.status === "skipped" && e.message.includes("lock busy"))).toBe(true);
|
||||
controller.lock.release(UUID, "pull");
|
||||
});
|
||||
|
||||
it("a concurrent save of a uuid mid-retry-backoff is dropped (no duplicate)", async () => {
|
||||
const body = "The gunslinger drew his revolver.\n";
|
||||
await writeNote(ccHash(currentState), body);
|
||||
mockRelayFailThenOk(0);
|
||||
const controller = newController();
|
||||
// Simulate a uuid mid-backoff (retrying set holds it).
|
||||
(controller as any).retrying.add(UUID);
|
||||
await (controller as any).process(REL);
|
||||
expect(putCalls).toBe(0); // dropped — not pushed
|
||||
expect(controller.events.some((e) => e.status === "skipped" && e.message.includes("retry in progress"))).toBe(true);
|
||||
});
|
||||
});
|
||||
183
tests/e1b8-log-migration.test.ts
Normal file
183
tests/e1b8-log-migration.test.ts
Normal file
@@ -0,0 +1,183 @@
|
||||
// E1b.8 — persistent rotated log + flagsSchemaVersion migration.
|
||||
//
|
||||
// Covers:
|
||||
// 1. log() writes a JSON-line to <outDir>/logs/sync-<date>.log (durable).
|
||||
// 2. a clean push (baselineNote) stamps foundry.flagsSchemaVersion.
|
||||
// 3. the startup migration stamps absent notes (idempotent, no contentHash/
|
||||
// ccHash change); notes already current are left alone.
|
||||
// 4. status() exposes migrationCount + schemaVersion.
|
||||
|
||||
import { describe, it, expect, vi, beforeEach, afterEach } from "vitest";
|
||||
import { mkdtemp, writeFile, mkdir, rm, readFile, readdir } from "node:fs/promises";
|
||||
import { existsSync } from "node:fs";
|
||||
import { join } from "node:path";
|
||||
import { tmpdir } from "node:os";
|
||||
|
||||
import { AutoSyncController } from "../src/server.js";
|
||||
import type { State, ServerConfig } from "../src/server.js";
|
||||
import { FLAGS_SCHEMA_VERSION } from "../src/schema-version.js";
|
||||
import { ccHash } from "../src/cchash.js";
|
||||
import { splitFrontmatter, readFoundryBlock } from "../src/frontmatter.js";
|
||||
import type { JournalEntry, CcData } from "../src/types.js";
|
||||
import type { FileRow } from "../src/batch.js";
|
||||
|
||||
const UUID = "JournalEntry.abc1";
|
||||
const REL = "Roland.md";
|
||||
|
||||
function liveEntry(description: string): JournalEntry {
|
||||
return { name: "Roland", _id: "abc1", folder: "Folder.test", flags: { "campaign-codex": { type: "npc", data: { description, notes: "" } } } };
|
||||
}
|
||||
|
||||
let dir: string;
|
||||
let state: State;
|
||||
const realFetch = globalThis.fetch;
|
||||
|
||||
beforeEach(async () => {
|
||||
dir = await mkdtemp(join(tmpdir(), "e1b8-"));
|
||||
await mkdir(join(dir, "refined"), { recursive: true });
|
||||
const cfg: ServerConfig = {
|
||||
journal: "", refinedDir: join(dir, "refined"), ccDir: "", outDir: join(dir, "out"),
|
||||
mode: "apply", port: 0, host: "", relayCfg: { url: "http://relay.test", apiKey: "k", clientId: "c" },
|
||||
};
|
||||
state = { db: {} as State["db"], cfg, index: null, autosync: null as unknown as State["autosync"] } as unknown as State;
|
||||
// Mock relay (behaves like Foundry): /get returns current state, /update applies diff.
|
||||
let currentState: JournalEntry = liveEntry("<p>Original body.</p>");
|
||||
let getCalls = 0;
|
||||
globalThis.fetch = vi.fn(async (url: string, init?: RequestInit) => {
|
||||
const u = String(url); const method = init?.method ?? "GET";
|
||||
const body = init?.body ? JSON.parse(String(init.body)) : undefined;
|
||||
const resp = (o: unknown, ok = true, status = 200) => ({ ok, status, text: async () => (typeof o === "string" ? o : JSON.stringify(o)) });
|
||||
if (method === "GET" && u.includes("/get?")) { getCalls++; return resp({ data: currentState }); }
|
||||
if (method === "GET" && u.includes("/search")) return resp({ results: [] });
|
||||
if (method === "PUT" && u.includes("/update?")) {
|
||||
const diff = body?.data ?? {};
|
||||
currentState = { ...currentState, name: diff.name ?? currentState.name, flags: { ...currentState.flags, "campaign-codex": diff["flags.campaign-codex"] ?? currentState.flags?.["campaign-codex"] } };
|
||||
return resp({ entity: [currentState] });
|
||||
}
|
||||
return resp({ error: "not found" }, false, 404);
|
||||
}) as unknown as typeof fetch;
|
||||
});
|
||||
|
||||
afterEach(async () => { globalThis.fetch = realFetch; await rm(dir, { recursive: true, force: true }); });
|
||||
|
||||
/** A refined note with a STALE contentHash (push proceeds) + a ccHash baseline. */
|
||||
async function writeNote(ccHashBaseline: string, body = "The gunslinger drew his revolver.\n", extraFoundry = ""): Promise<void> {
|
||||
const lines = [
|
||||
"---", "type: npc", "foundry:", ` cc_uuid: ${UUID}`, " cc_type: npc", " folder_path: NPCs",
|
||||
` contentHash: ${"0".repeat(64)}`, ` ccHash: ${ccHashBaseline}`, " syncedAt: 2026-06-22T00:00:00.000Z",
|
||||
];
|
||||
if (extraFoundry) lines.push(extraFoundry);
|
||||
lines.push("---", body, "");
|
||||
await writeFile(join(state.cfg.refinedDir, REL), lines.join("\n"), "utf8");
|
||||
}
|
||||
|
||||
describe("E1b.8a persistent JSON-lines log", () => {
|
||||
it("log() writes a JSON-line to <outDir>/logs/sync-<date>.log", async () => {
|
||||
const controller = new AutoSyncController(state);
|
||||
(controller as any).log("Roland", "pushed", "test push event");
|
||||
// The WriteStream opens + flushes async; poll for the file + content.
|
||||
const today = new Date().toISOString().slice(0, 10);
|
||||
const logPath = join(state.cfg.outDir, "logs", `sync-${today}.log`);
|
||||
let content = "";
|
||||
for (let i = 0; i < 100 && !content; i++) { await new Promise<void>((r) => setTimeout(r, 10)); if (existsSync(logPath)) content = await readFile(logPath, "utf8").catch(() => ""); }
|
||||
expect(content).toContain("test push event");
|
||||
const line = content.trim().split("\n").pop()!;
|
||||
const parsed = JSON.parse(line) as { time: string; level: string; name: string; status: string; message: string };
|
||||
expect(parsed.name).toBe("Roland");
|
||||
expect(parsed.status).toBe("pushed");
|
||||
expect(parsed.message).toBe("test push event");
|
||||
expect(parsed.level).toBe("info");
|
||||
});
|
||||
|
||||
it("an error log line has level 'error'", async () => {
|
||||
const controller = new AutoSyncController(state);
|
||||
(controller as any).log("Roland", "error", "boom");
|
||||
const today = new Date().toISOString().slice(0, 10);
|
||||
const logPath = join(state.cfg.outDir, "logs", `sync-${today}.log`);
|
||||
let content = "";
|
||||
for (let i = 0; i < 100 && !content; i++) { await new Promise<void>((r) => setTimeout(r, 10)); if (existsSync(logPath)) content = await readFile(logPath, "utf8").catch(() => ""); }
|
||||
expect(content).toContain('"level":"error"');
|
||||
expect(content).toContain("boom");
|
||||
});
|
||||
|
||||
it("pruneOldLogs deletes logs older than the retain window", async () => {
|
||||
const logDir = join(state.cfg.outDir, "logs");
|
||||
await mkdir(logDir, { recursive: true });
|
||||
// Write an "old" log file (backdated mtime via utimes is flaky; instead write a
|
||||
// file and rely on mtime being ~now — so test the no-op path: a fresh log is
|
||||
// NOT pruned). Full backdating is awkward; assert the prune runs without error
|
||||
// and a fresh file survives.
|
||||
const fresh = join(logDir, "sync-2026-06-23.log");
|
||||
await writeFile(fresh, "x\n", "utf8");
|
||||
const controller = new AutoSyncController(state);
|
||||
await controller.pruneOldLogs();
|
||||
expect(existsSync(fresh)).toBe(true); // fresh file survives
|
||||
});
|
||||
});
|
||||
|
||||
describe("E1b.8b flagsSchemaVersion baseline + migration", () => {
|
||||
it("a clean push stamps foundry.flagsSchemaVersion on the note", async () => {
|
||||
await writeNote(ccHash(liveEntry("<p>Original body.</p>")));
|
||||
const controller = new AutoSyncController(state);
|
||||
(controller as any).retryBackoffs = [1, 1, 1];
|
||||
await (controller as any).process(REL);
|
||||
const fb = readFoundryBlock(splitFrontmatter(await readFile(join(state.cfg.refinedDir, REL), "utf8")).fm);
|
||||
expect(fb?.flagsSchemaVersion).toBe(FLAGS_SCHEMA_VERSION);
|
||||
});
|
||||
|
||||
function fakeMatchedRow(): FileRow {
|
||||
return {
|
||||
name: "Roland", basename: "Roland", status: "matched" as never,
|
||||
refinedPath: join(state.cfg.refinedDir, REL), ccPath: null, ccId: UUID, ccType: "npc",
|
||||
curatedType: null, entry: null, recommendation: "sync-cc" as never,
|
||||
refinedChanged: false, ccChanged: false, refinedMtime: null, ccMtime: null,
|
||||
storedRefinedHash: null, storedCcHash: null,
|
||||
};
|
||||
}
|
||||
|
||||
it("migration stamps flagsSchemaVersion on a note that lacks it (idempotent, no contentHash/ccHash change)", async () => {
|
||||
// Note WITH contentHash + ccHash but NO flagsSchemaVersion.
|
||||
const baseline = ccHash(liveEntry("<p>Original body.</p>"));
|
||||
await writeNote(baseline);
|
||||
const before = await readFile(join(state.cfg.refinedDir, REL), "utf8");
|
||||
const fbBefore = readFoundryBlock(splitFrontmatter(before).fm);
|
||||
expect(fbBefore?.flagsSchemaVersion).toBeUndefined();
|
||||
|
||||
const controller = new AutoSyncController(state);
|
||||
(state as any).index = { matched: [fakeMatchedRow()], ccOnly: [], refinedOnly: [], counts: { matched: 1, ccOnly: 0, refinedOnly: 0, unlinked: 0 } };
|
||||
const count = await controller.migrateFlagsSchemaVersion();
|
||||
expect(count).toBe(1);
|
||||
|
||||
const after = await readFile(join(state.cfg.refinedDir, REL), "utf8");
|
||||
const fbAfter = readFoundryBlock(splitFrontmatter(after).fm);
|
||||
expect(fbAfter?.flagsSchemaVersion).toBe(FLAGS_SCHEMA_VERSION);
|
||||
// contentHash + ccHash UNTOUCHED (no false push trigger).
|
||||
expect(fbAfter?.contentHash).toBe(fbBefore?.contentHash);
|
||||
expect(fbAfter?.ccHash).toBe(baseline);
|
||||
|
||||
// Idempotent: a second migration run stamps nothing.
|
||||
const count2 = await controller.migrateFlagsSchemaVersion();
|
||||
expect(count2).toBe(0);
|
||||
});
|
||||
|
||||
it("migration leaves a note that already has the current version alone", async () => {
|
||||
const baseline = ccHash(liveEntry("<p>Original body.</p>"));
|
||||
await writeNote(baseline, "The gunslinger drew his revolver.\n", ` flagsSchemaVersion: ${FLAGS_SCHEMA_VERSION}`);
|
||||
const controller = new AutoSyncController(state);
|
||||
(state as any).index = { matched: [fakeMatchedRow()], ccOnly: [], refinedOnly: [], counts: { matched: 1, ccOnly: 0, refinedOnly: 0, unlinked: 0 } };
|
||||
const count = await controller.migrateFlagsSchemaVersion();
|
||||
expect(count).toBe(0); // already current
|
||||
});
|
||||
|
||||
it("status() exposes migrationCount + schemaVersion", async () => {
|
||||
const baseline = ccHash(liveEntry("<p>Original body.</p>"));
|
||||
await writeNote(baseline);
|
||||
const controller = new AutoSyncController(state);
|
||||
(state as any).index = { matched: [fakeMatchedRow()], ccOnly: [], refinedOnly: [], counts: { matched: 1, ccOnly: 0, refinedOnly: 0, unlinked: 0 } };
|
||||
await controller.migrateFlagsSchemaVersion();
|
||||
const s = controller.status();
|
||||
expect(s.migrationRan).toBe(true);
|
||||
expect(s.migrationCount).toBe(1);
|
||||
expect(s.schemaVersion).toBe(FLAGS_SCHEMA_VERSION);
|
||||
});
|
||||
});
|
||||
183
tests/e2-1-shallow-poll.test.ts
Normal file
183
tests/e2-1-shallow-poll.test.ts
Normal file
@@ -0,0 +1,183 @@
|
||||
// E2.1 — shallow /search poll: rename/new/missing detection + fPending + liveNewEntries.
|
||||
|
||||
import { describe, it, expect, beforeEach, afterEach, vi } from "vitest";
|
||||
import { mkdtemp, mkdir, rm, readFile } from "node:fs/promises";
|
||||
import { join } from "node:path";
|
||||
import { tmpdir } from "node:os";
|
||||
|
||||
import { FoundryPollController } from "../src/foundry-poll.js";
|
||||
import type { State, ServerConfig } from "../src/server.js";
|
||||
import { defaultSyncState, saveSyncState, type SyncState } from "../src/sync-state.js";
|
||||
import type { SearchResult } from "../src/relay/client.js";
|
||||
|
||||
let dir: string;
|
||||
let state: State;
|
||||
const realFetch = globalThis.fetch;
|
||||
|
||||
function makeState(): State {
|
||||
const cfg: ServerConfig = {
|
||||
journal: "", refinedDir: join(dir, "refined"), ccDir: "", outDir: join(dir, "out"),
|
||||
mode: "apply", port: 0, host: "127.0.0.1",
|
||||
relayCfg: { url: "http://relay.test", apiKey: "k", clientId: "c" },
|
||||
features: { syncStatus: true, foundryPoll: true },
|
||||
};
|
||||
const s = { db: {} as State["db"], cfg, index: null, autosync: null as unknown as State["autosync"] } as unknown as State;
|
||||
s.syncState = { ...defaultSyncState(cfg.refinedDir) } as SyncState;
|
||||
return s;
|
||||
}
|
||||
|
||||
function mockSearch(results: SearchResult[]): void {
|
||||
globalThis.fetch = vi.fn(async (url: string) => {
|
||||
if (String(url).includes("/search")) {
|
||||
return { ok: true, status: 200, text: async () => JSON.stringify({ results }) } as unknown as Response;
|
||||
}
|
||||
return { ok: false, status: 404, text: async () => '{"error":"not found"}' } as unknown as Response;
|
||||
}) as unknown as typeof fetch;
|
||||
}
|
||||
|
||||
function mockSearchError(status: number, error: string): void {
|
||||
globalThis.fetch = vi.fn(async () => ({
|
||||
ok: false, status, text: async () => JSON.stringify({ error }),
|
||||
}) as unknown as Response) as unknown as typeof fetch;
|
||||
}
|
||||
|
||||
beforeEach(async () => {
|
||||
dir = await mkdtemp(join(tmpdir(), "e2-1-"));
|
||||
await mkdir(join(dir, "refined"), { recursive: true });
|
||||
await mkdir(join(dir, "out"), { recursive: true });
|
||||
state = makeState();
|
||||
await saveSyncState(state.cfg.outDir, state.syncState!);
|
||||
});
|
||||
|
||||
afterEach(async () => {
|
||||
globalThis.fetch = realFetch;
|
||||
await rm(dir, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
/** Wait until the controller's prevSnapshot is populated (the first tick completed). */
|
||||
async function waitForFirstPoll(controller: FoundryPollController, timeoutMs = 2000): Promise<void> {
|
||||
const start = Date.now();
|
||||
while ((controller as any).prevSnapshot.size === 0 && Date.now() - start < timeoutMs) {
|
||||
await new Promise<void>((r) => setTimeout(r, 10));
|
||||
}
|
||||
}
|
||||
|
||||
describe("E2.1 shallow poll detection", () => {
|
||||
it("detects a rename (name change on a known uuid)", async () => {
|
||||
// First poll: establish the snapshot.
|
||||
mockSearch([
|
||||
{ uuid: "JournalEntry.aaa", id: "aaa", name: "Roland", documentType: "JournalEntry" },
|
||||
]);
|
||||
const controller = new FoundryPollController(state);
|
||||
await controller.setEnabled(true);
|
||||
await waitForFirstPoll(controller);
|
||||
expect((controller as any).prevSnapshot.size).toBe(1);
|
||||
|
||||
// Second poll: Roland renamed to "Roland Deschain".
|
||||
mockSearch([
|
||||
{ uuid: "JournalEntry.aaa", id: "aaa", name: "Roland Deschain", documentType: "JournalEntry" },
|
||||
]);
|
||||
// Trigger a manual tick (the timer fires on cadence; force it for the test).
|
||||
await (controller as any).tick();
|
||||
const saved = JSON.parse(await readFile(join(dir, "out", "sync-state.json"), "utf8")) as SyncState & { fPending?: unknown[] };
|
||||
const fPending = saved.fPending as { uuid: string; change: string }[] | undefined;
|
||||
expect(fPending).toBeTruthy();
|
||||
expect(fPending!.some((e) => e.uuid === "JournalEntry.aaa" && e.change === "renamed")).toBe(true);
|
||||
controller.stop();
|
||||
});
|
||||
|
||||
it("detects a new entry (uuid not in prior snapshot, not in linked index) → liveNewEntries", async () => {
|
||||
// First poll: empty.
|
||||
mockSearch([]);
|
||||
const controller = new FoundryPollController(state);
|
||||
await controller.setEnabled(true);
|
||||
await waitForFirstPoll(controller); // wait for empty snapshot
|
||||
|
||||
// Second poll: a new entry appears.
|
||||
mockSearch([
|
||||
{ uuid: "JournalEntry.new1", id: "new1", name: "New NPC", documentType: "JournalEntry" },
|
||||
]);
|
||||
await (controller as any).tick();
|
||||
expect(controller.liveNewEntries.some((e) => e.uuid === "JournalEntry.new1")).toBe(true);
|
||||
expect(controller.liveNewEntries[0].name).toBe("New NPC");
|
||||
controller.stop();
|
||||
});
|
||||
|
||||
it("detects a missing entry (in linked index but absent from /search)", async () => {
|
||||
// Mock the index with a linked entry.
|
||||
(state as any).index = {
|
||||
matched: [{ entry: { _id: "aaa", name: "Roland" } }],
|
||||
ccOnly: [], refinedOnly: [],
|
||||
};
|
||||
// First poll: Roland present.
|
||||
mockSearch([
|
||||
{ uuid: "JournalEntry.aaa", id: "aaa", name: "Roland", documentType: "JournalEntry" },
|
||||
]);
|
||||
const controller = new FoundryPollController(state);
|
||||
await controller.setEnabled(true);
|
||||
await waitForFirstPoll(controller);
|
||||
|
||||
// Second poll: Roland gone.
|
||||
mockSearch([]);
|
||||
await (controller as any).tick();
|
||||
const saved = JSON.parse(await readFile(join(dir, "out", "sync-state.json"), "utf8")) as SyncState & { fPending?: unknown[] };
|
||||
const fPending = saved.fPending as { uuid: string; change: string }[] | undefined;
|
||||
expect(fPending?.some((e) => e.uuid === "JournalEntry.aaa" && e.change === "missing")).toBe(true);
|
||||
controller.stop();
|
||||
});
|
||||
|
||||
it("updates parity.fPending count + lastPollAt in sync-state.json", async () => {
|
||||
mockSearch([
|
||||
{ uuid: "JournalEntry.new1", id: "new1", name: "New", documentType: "JournalEntry" },
|
||||
]);
|
||||
const controller = new FoundryPollController(state);
|
||||
await controller.setEnabled(true);
|
||||
await waitForFirstPoll(controller);
|
||||
const saved = JSON.parse(await readFile(join(dir, "out", "sync-state.json"), "utf8")) as SyncState;
|
||||
expect(saved.parity.lastPollAt).not.toBeNull();
|
||||
controller.stop();
|
||||
});
|
||||
|
||||
it("overlap guard: a tick while in flight is skipped (skipCounter increments)", async () => {
|
||||
// Make the search slow so the first tick is still in flight when we call again.
|
||||
const holder: { resolve: (() => void) | null } = { resolve: null };
|
||||
globalThis.fetch = vi.fn(async () => {
|
||||
await new Promise<void>((r) => { holder.resolve = r; });
|
||||
return { ok: true, status: 200, text: async () => JSON.stringify({ results: [] }) } as unknown as Response;
|
||||
}) as unknown as typeof fetch;
|
||||
|
||||
const controller = new FoundryPollController(state);
|
||||
await controller.setEnabled(true);
|
||||
// Wait for the tick to start (inFlight becomes true).
|
||||
await new Promise<void>((r) => setTimeout(r, 30));
|
||||
expect((controller as any).inFlight).toBe(true);
|
||||
|
||||
// Call tick again — should be skipped.
|
||||
await (controller as any).tick();
|
||||
expect(controller.skipCounter).toBeGreaterThan(0);
|
||||
|
||||
// Release the stuck search.
|
||||
if (holder.resolve) holder.resolve();
|
||||
controller.stop();
|
||||
});
|
||||
|
||||
it("persistent error (404 No connected Foundry clients) → halts the timer", async () => {
|
||||
mockSearchError(404, "No connected Foundry clients found");
|
||||
const controller = new FoundryPollController(state);
|
||||
await controller.setEnabled(true);
|
||||
// Wait for the tick to run + halt.
|
||||
await new Promise<void>((r) => setTimeout(r, 100));
|
||||
expect(controller.enabled).toBe(false); // halted
|
||||
controller.stop();
|
||||
});
|
||||
|
||||
it("status() returns enabled + cadence + liveNewEntries", async () => {
|
||||
const controller = new FoundryPollController(state);
|
||||
await controller.setEnabled(true);
|
||||
const s = controller.status();
|
||||
expect(s.enabled).toBe(true);
|
||||
expect(s.cadenceMs).toBeGreaterThan(0);
|
||||
expect(s.liveNewEntries).toEqual([]);
|
||||
controller.stop();
|
||||
});
|
||||
});
|
||||
151
tests/e2-2-deep-poll.test.ts
Normal file
151
tests/e2-2-deep-poll.test.ts
Normal file
@@ -0,0 +1,151 @@
|
||||
// E2.2 — deep poll: per-linked-note /get + ccHash compare + folder move detection.
|
||||
|
||||
import { describe, it, expect, beforeEach, afterEach, vi } from "vitest";
|
||||
import { mkdtemp, mkdir, rm, readFile, writeFile } from "node:fs/promises";
|
||||
import { join } from "node:path";
|
||||
import { tmpdir } from "node:os";
|
||||
|
||||
import { FoundryPollController, type PendingFChange } from "../src/foundry-poll.js";
|
||||
import type { State, ServerConfig } from "../src/server.js";
|
||||
import { defaultSyncState, saveSyncState, type SyncState } from "../src/sync-state.js";
|
||||
import { ccHash } from "../src/cchash.js";
|
||||
import type { JournalEntry } from "../src/types.js";
|
||||
|
||||
let dir: string;
|
||||
let state: State;
|
||||
const realFetch = globalThis.fetch;
|
||||
|
||||
function liveEntry(description: string, folder = "Folder.test"): JournalEntry {
|
||||
return { name: "Roland", _id: "aaa", folder, flags: { "campaign-codex": { type: "npc", data: { description, notes: "" } } } };
|
||||
}
|
||||
|
||||
function makeState(): State {
|
||||
const cfg: ServerConfig = {
|
||||
journal: "", refinedDir: join(dir, "refined"), ccDir: "", outDir: join(dir, "out"),
|
||||
mode: "apply", port: 0, host: "127.0.0.1",
|
||||
relayCfg: { url: "http://relay.test", apiKey: "k", clientId: "c" },
|
||||
features: { syncStatus: true, foundryPoll: true },
|
||||
};
|
||||
const s = { db: {} as State["db"], cfg, index: null, autosync: null as unknown as State["autosync"] } as unknown as State;
|
||||
s.syncState = { ...defaultSyncState(cfg.refinedDir) } as SyncState;
|
||||
return s;
|
||||
}
|
||||
|
||||
async function writeNote(ccHashBaseline: string | null, folderPath: string): Promise<void> {
|
||||
const lines = ["---", "type: npc", "foundry:", " cc_uuid: JournalEntry.aaa", " cc_type: npc", ` folder_path: ${folderPath}`, ` contentHash: ${"0".repeat(64)}`];
|
||||
if (ccHashBaseline !== null) lines.push(` ccHash: ${ccHashBaseline}`);
|
||||
lines.push(" syncedAt: 2026-06-22T00:00:00.000Z", "---", "Roland body.", "");
|
||||
await writeFile(join(dir, "refined", "Roland.md"), lines.join("\n"), "utf8");
|
||||
}
|
||||
|
||||
function mockGetEntry(entry: JournalEntry): void {
|
||||
globalThis.fetch = vi.fn(async (url: string) => {
|
||||
if (String(url).includes("/get?")) return { ok: true, status: 200, text: async () => JSON.stringify({ data: entry }) } as unknown as Response;
|
||||
if (String(url).includes("/search")) return { ok: true, status: 200, text: async () => JSON.stringify({ results: [{ uuid: "JournalEntry.aaa", id: "aaa", name: "Roland", documentType: "JournalEntry" }] }) } as unknown as Response;
|
||||
return { ok: false, status: 404, text: async () => '{"error":"not found"}' } as unknown as Response;
|
||||
}) as unknown as typeof fetch;
|
||||
}
|
||||
|
||||
function mockGetError(status: number, error: string): void {
|
||||
globalThis.fetch = vi.fn(async () => ({ ok: false, status, text: async () => JSON.stringify({ error }) }) as unknown as Response) as unknown as typeof fetch;
|
||||
}
|
||||
|
||||
beforeEach(async () => {
|
||||
dir = await mkdtemp(join(tmpdir(), "e2-2-"));
|
||||
await mkdir(join(dir, "refined"), { recursive: true });
|
||||
await mkdir(join(dir, "out"), { recursive: true });
|
||||
state = makeState();
|
||||
await saveSyncState(state.cfg.outDir, state.syncState!);
|
||||
});
|
||||
|
||||
afterEach(async () => {
|
||||
globalThis.fetch = realFetch;
|
||||
await rm(dir, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
function setupController(): FoundryPollController {
|
||||
const controller = new FoundryPollController(state);
|
||||
// Set the prevSnapshot so the deep poll's candidate list includes the uuid.
|
||||
(controller as any).prevSnapshot.set("JournalEntry.aaa", { name: "Roland", img: null });
|
||||
// Mock the index with a linked entry.
|
||||
(state as any).index = {
|
||||
matched: [{ entry: { _id: "aaa", name: "Roland" }, refinedPath: join(dir, "refined", "Roland.md"), name: "Roland" }],
|
||||
ccOnly: [], refinedOnly: [],
|
||||
};
|
||||
(controller as any).retryBackoffs = [1, 1, 1]; // fast retries for tests
|
||||
return controller;
|
||||
}
|
||||
|
||||
async function readFPending(): Promise<PendingFChange[]> {
|
||||
const saved = JSON.parse(await readFile(join(dir, "out", "sync-state.json"), "utf8")) as SyncState & { fPending?: PendingFChange[] };
|
||||
return saved.fPending ?? [];
|
||||
}
|
||||
|
||||
describe("E2.2 deep poll detection", () => {
|
||||
it("ccHash mismatch → F-changed 'edited'", async () => {
|
||||
const live = liveEntry("<p>Foundry-edited body.</p>");
|
||||
mockGetEntry(live);
|
||||
// Note's ccHash baseline = a DIFFERENT value (the original body's ccHash).
|
||||
await writeNote("0".repeat(64), "Folder.test");
|
||||
const controller = setupController();
|
||||
await (controller as any).deepPoll();
|
||||
const fPending = await readFPending();
|
||||
expect(fPending.some((e) => e.uuid === "JournalEntry.aaa" && e.change === "edited")).toBe(true);
|
||||
controller.stop();
|
||||
});
|
||||
|
||||
it("ccHash match → no change (left untouched)", async () => {
|
||||
const live = liveEntry("<p>Original body.</p>");
|
||||
mockGetEntry(live);
|
||||
// Note's ccHash baseline = ccHash(live) (matches).
|
||||
await writeNote(ccHash(live), "Folder.test");
|
||||
const controller = setupController();
|
||||
await (controller as any).deepPoll();
|
||||
const fPending = await readFPending();
|
||||
expect(fPending.length).toBe(0); // no changes
|
||||
controller.stop();
|
||||
});
|
||||
|
||||
it("folder move on a legacy note (no ccHash) → 'moved'", async () => {
|
||||
const live = liveEntry("<p>Original body.</p>", "Folder.new");
|
||||
mockGetEntry(live);
|
||||
// Note has NO ccHash baseline (legacy) but has folder_path = "Folder.test".
|
||||
// Live folder = "Folder.new" → "moved".
|
||||
await writeNote(null, "Folder.test");
|
||||
const controller = setupController();
|
||||
await (controller as any).deepPoll();
|
||||
const fPending = await readFPending();
|
||||
expect(fPending.some((e) => e.uuid === "JournalEntry.aaa" && e.change === "moved")).toBe(true);
|
||||
controller.stop();
|
||||
});
|
||||
|
||||
it("persistent error (404 No connected Foundry clients) → aborts + halts", async () => {
|
||||
mockGetError(404, "No connected Foundry clients found");
|
||||
await writeNote(ccHash(liveEntry("<p>x</p>")), "Folder.test");
|
||||
const controller = setupController();
|
||||
await controller.setEnabled(true);
|
||||
// deepTick catches the persistent error from deepPoll + calls stop() (halt).
|
||||
await (controller as any).deepTick();
|
||||
expect(controller.enabled).toBe(false); // halted by deepTick's catch
|
||||
controller.stop();
|
||||
});
|
||||
|
||||
it("transient error (504) → retries, final failure → fPending recorded (round continues)", async () => {
|
||||
// Mock /get to always 504 (transient → retries → exhaustion).
|
||||
mockGetError(504, "Request timed out");
|
||||
await writeNote(ccHash(liveEntry("<p>x</p>")), "Folder.test");
|
||||
const controller = setupController();
|
||||
await (controller as any).deepPoll(); // should NOT throw (transient exhaustion is recorded, not thrown)
|
||||
const fPending = await readFPending();
|
||||
expect(fPending.some((e) => e.uuid === "JournalEntry.aaa" && e.change === "edited")).toBe(true); // recorded as fPending
|
||||
controller.stop();
|
||||
});
|
||||
|
||||
it("status() exposes loadCeilingCallsPerMin + deep poll info", () => {
|
||||
const controller = new FoundryPollController(state);
|
||||
const s = controller.status();
|
||||
expect(s.loadCeilingCallsPerMin).toBeGreaterThan(0);
|
||||
expect(s.deepCadenceMs).toBeGreaterThanOrEqual(30000);
|
||||
expect(s.deepInFlight).toBe(false);
|
||||
});
|
||||
});
|
||||
131
tests/e2-3-pull.test.ts
Normal file
131
tests/e2-3-pull.test.ts
Normal file
@@ -0,0 +1,131 @@
|
||||
// E2.3 — F→O pull for F-changed + O-unchanged notes.
|
||||
|
||||
import { describe, it, expect, beforeEach, afterEach, vi } from "vitest";
|
||||
import { mkdtemp, mkdir, rm, readFile, writeFile } from "node:fs/promises";
|
||||
import { join } from "node:path";
|
||||
import { tmpdir } from "node:os";
|
||||
|
||||
import { AutoSyncController } from "../src/server.js";
|
||||
import type { State, ServerConfig } from "../src/server.js";
|
||||
import { ccHash } from "../src/cchash.js";
|
||||
import { contentHash } from "../src/normalize.js";
|
||||
import { splitFrontmatter, readFoundryBlock } from "../src/frontmatter.js";
|
||||
import { saveSyncState, defaultSyncState, type SyncState } from "../src/sync-state.js";
|
||||
import type { JournalEntry } from "../src/types.js";
|
||||
import type { FileRow } from "../src/batch.js";
|
||||
|
||||
const UUID = "JournalEntry.aaa";
|
||||
|
||||
let dir: string;
|
||||
let state: State;
|
||||
const realFetch = globalThis.fetch;
|
||||
|
||||
function liveEntry(description: string, folder = "Folder.test"): JournalEntry {
|
||||
return { name: "Roland", _id: "aaa", folder, flags: { "campaign-codex": { type: "npc", data: { description, notes: "" } } } };
|
||||
}
|
||||
|
||||
beforeEach(async () => {
|
||||
dir = await mkdtemp(join(tmpdir(), "e2-3-"));
|
||||
await mkdir(join(dir, "refined"), { recursive: true });
|
||||
await mkdir(join(dir, "out"), { recursive: true });
|
||||
const cfg: ServerConfig = {
|
||||
journal: "", refinedDir: join(dir, "refined"), ccDir: "", outDir: join(dir, "out"),
|
||||
mode: "apply", port: 0, host: "127.0.0.1",
|
||||
relayCfg: { url: "http://relay.test", apiKey: "k", clientId: "c" },
|
||||
features: { syncStatus: true, foundryPoll: true },
|
||||
};
|
||||
state = { db: {} as State["db"], cfg, index: null, autosync: null as unknown as State["autosync"] } as unknown as State;
|
||||
state.syncState = { ...defaultSyncState(cfg.refinedDir) } as SyncState;
|
||||
await saveSyncState(cfg.outDir, state.syncState);
|
||||
});
|
||||
|
||||
afterEach(async () => { globalThis.fetch = realFetch; await rm(dir, { recursive: true, force: true }); });
|
||||
|
||||
async function writeNote(body: string, contentHashBaseline: string, ccHashBaseline: string): Promise<void> {
|
||||
const lines = [
|
||||
"---", "type: npc", "foundry:", ` cc_uuid: ${UUID}`, " cc_type: npc",
|
||||
` folder_path: Folder.test`, ` contentHash: ${contentHashBaseline}`,
|
||||
` ccHash: ${ccHashBaseline}`, " syncedAt: 2026-06-22T00:00:00.000Z",
|
||||
"---", body, "",
|
||||
];
|
||||
await writeFile(join(dir, "refined", "Roland.md"), lines.join("\n"), "utf8");
|
||||
}
|
||||
|
||||
function setupIndex(): void {
|
||||
const row: FileRow = {
|
||||
name: "Roland", basename: "Roland", status: "matched" as never,
|
||||
refinedPath: join(dir, "refined", "Roland.md"), ccPath: null, ccId: UUID, ccType: "npc",
|
||||
curatedType: null, entry: { _id: "aaa", name: "Roland" } as JournalEntry, recommendation: "in-sync" as never,
|
||||
refinedChanged: false, ccChanged: false, refinedMtime: null, ccMtime: null,
|
||||
storedRefinedHash: null, storedCcHash: null,
|
||||
};
|
||||
(state as any).index = { matched: [row], ccOnly: [], refinedOnly: [], counts: { matched: 1, ccOnly: 0, refinedOnly: 0, unlinked: 0 } };
|
||||
}
|
||||
|
||||
describe("E2.3 pullFChanged", () => {
|
||||
it("F-changed + O-unchanged → pulls, re-baselines, removes from fPending", async () => {
|
||||
const live = liveEntry("<p>Foundry-edited body.</p>");
|
||||
const liveCc = ccHash(live);
|
||||
// Note's contentHash = contentHash("Roland body.\n") (the O-side is unchanged —
|
||||
// the body hasn't been edited in Obsidian since the last sync).
|
||||
const body = "Roland body.\n";
|
||||
await writeNote(body, contentHash(body), "0".repeat(64)); // ccHash = stale (≠ liveCc)
|
||||
setupIndex();
|
||||
const controller = new AutoSyncController(state);
|
||||
(controller as any).retryBackoffs = [1, 1, 1];
|
||||
|
||||
const result = await controller.pullFChanged(UUID, live);
|
||||
expect(result).toBe("pulled");
|
||||
|
||||
// The note was re-baselined: contentHash = the pulled body hash, ccHash = ccHash(live).
|
||||
const md = await readFile(join(dir, "refined", "Roland.md"), "utf8");
|
||||
const fb = readFoundryBlock(splitFrontmatter(md).fm);
|
||||
expect(fb?.ccHash).toBe(liveCc);
|
||||
// The body was pulled from Foundry (entryToObsidian converts the live entry to markdown).
|
||||
// The contentHash should match the NEW body (the pulled content).
|
||||
const { body: pulledBody } = splitFrontmatter(md);
|
||||
expect(fb?.contentHash).toBe(contentHash(pulledBody));
|
||||
// Removed from fPending.
|
||||
const saved = JSON.parse(await readFile(join(dir, "out", "sync-state.json"), "utf8")) as SyncState & { fPending?: unknown[] };
|
||||
expect((saved.fPending ?? []).length).toBe(0);
|
||||
});
|
||||
|
||||
it("F-changed + O-CHANGED → 'conflict' (left in fPending for E2.4)", async () => {
|
||||
const live = liveEntry("<p>Foundry-edited body.</p>");
|
||||
// Note's contentHash = "0"*64 (stale) — the body was edited in Obsidian too
|
||||
// (bodyHash ≠ contentHash → O-side changed).
|
||||
await writeNote("Edited in Obsidian.\n", "0".repeat(64), "0".repeat(64));
|
||||
setupIndex();
|
||||
const controller = new AutoSyncController(state);
|
||||
|
||||
const result = await controller.pullFChanged(UUID, live);
|
||||
expect(result).toBe("conflict");
|
||||
// The note was NOT pulled (content unchanged).
|
||||
const md = await readFile(join(dir, "refined", "Roland.md"), "utf8");
|
||||
const fb = readFoundryBlock(splitFrontmatter(md).fm);
|
||||
expect(fb?.ccHash).toBe("0".repeat(64)); // unchanged — not baselined
|
||||
});
|
||||
|
||||
it("no matching row → 'skipped'", async () => {
|
||||
// No index → no matching row.
|
||||
const controller = new AutoSyncController(state);
|
||||
const result = await controller.pullFChanged(UUID, liveEntry("<p>x</p>"));
|
||||
expect(result).toBe("skipped");
|
||||
});
|
||||
|
||||
it("note missing from disk → 'skipped'", async () => {
|
||||
setupIndex(); // index has a row pointing at a non-existent file
|
||||
const controller = new AutoSyncController(state);
|
||||
const result = await controller.pullFChanged(UUID, liveEntry("<p>x</p>"));
|
||||
expect(result).toBe("skipped");
|
||||
});
|
||||
|
||||
it("unseeded note (no contentHash) → 'skipped'", async () => {
|
||||
// Write a note without contentHash.
|
||||
await writeFile(join(dir, "refined", "Roland.md"), "---\ntype: npc\nfoundry:\n cc_uuid: JournalEntry.aaa\n---\nbody\n", "utf8");
|
||||
setupIndex();
|
||||
const controller = new AutoSyncController(state);
|
||||
const result = await controller.pullFChanged(UUID, liveEntry("<p>x</p>"));
|
||||
expect(result).toBe("skipped");
|
||||
});
|
||||
});
|
||||
131
tests/e2-4-conflict.test.ts
Normal file
131
tests/e2-4-conflict.test.ts
Normal file
@@ -0,0 +1,131 @@
|
||||
// E2.4 — never-clobber routing: both-diverged / vault-newer → pending conflict row.
|
||||
|
||||
import { describe, it, expect, beforeEach, afterEach, vi } from "vitest";
|
||||
import { mkdtemp, mkdir, rm, readFile, writeFile } from "node:fs/promises";
|
||||
import { join } from "node:path";
|
||||
import { tmpdir } from "node:os";
|
||||
|
||||
import { AutoSyncController } from "../src/server.js";
|
||||
import { FoundryPollController } from "../src/foundry-poll.js";
|
||||
import type { State, ServerConfig } from "../src/server.js";
|
||||
import { ccHash } from "../src/cchash.js";
|
||||
import { contentHash } from "../src/normalize.js";
|
||||
import { saveSyncState, defaultSyncState, type SyncState } from "../src/sync-state.js";
|
||||
import type { JournalEntry } from "../src/types.js";
|
||||
import type { FileRow } from "../src/batch.js";
|
||||
|
||||
const UUID = "JournalEntry.aaa";
|
||||
let dir: string;
|
||||
let state: State;
|
||||
const realFetch = globalThis.fetch;
|
||||
|
||||
function liveEntry(description: string): JournalEntry {
|
||||
return { name: "Roland", _id: "aaa", folder: "Folder.test", flags: { "campaign-codex": { type: "npc", data: { description, notes: "" } } } };
|
||||
}
|
||||
|
||||
beforeEach(async () => {
|
||||
dir = await mkdtemp(join(tmpdir(), "e2-4-"));
|
||||
await mkdir(join(dir, "refined"), { recursive: true });
|
||||
await mkdir(join(dir, "out"), { recursive: true });
|
||||
const cfg: ServerConfig = {
|
||||
journal: "", refinedDir: join(dir, "refined"), ccDir: "", outDir: join(dir, "out"),
|
||||
mode: "apply", port: 0, host: "127.0.0.1",
|
||||
relayCfg: { url: "http://relay.test", apiKey: "k", clientId: "c" },
|
||||
features: { syncStatus: true, foundryPoll: true },
|
||||
};
|
||||
state = { db: {} as State["db"], cfg, index: null, autosync: null as unknown as State["autosync"] } as unknown as State;
|
||||
state.syncState = { ...defaultSyncState(cfg.refinedDir) } as SyncState;
|
||||
await saveSyncState(cfg.outDir, state.syncState!);
|
||||
state.autosync = new AutoSyncController(state);
|
||||
});
|
||||
|
||||
afterEach(async () => { globalThis.fetch = realFetch; await rm(dir, { recursive: true, force: true }); });
|
||||
|
||||
async function writeNote(body: string, contentHashBaseline: string, ccHashBaseline: string): Promise<void> {
|
||||
await writeFile(join(dir, "refined", "Roland.md"), [
|
||||
"---", "type: npc", "foundry:", ` cc_uuid: ${UUID}`, " cc_type: npc",
|
||||
` folder_path: Folder.test`, ` contentHash: ${contentHashBaseline}`,
|
||||
` ccHash: ${ccHashBaseline}`, " syncedAt: 2026-06-22T00:00:00.000Z", "---", body, "",
|
||||
].join("\n"), "utf8");
|
||||
}
|
||||
|
||||
function setupIndex(): void {
|
||||
const row: FileRow = {
|
||||
name: "Roland", basename: "Roland", status: "matched" as never,
|
||||
refinedPath: join(dir, "refined", "Roland.md"), ccPath: null, ccId: UUID, ccType: "npc",
|
||||
curatedType: null, entry: { _id: "aaa", name: "Roland" } as JournalEntry, recommendation: "in-sync" as never,
|
||||
refinedChanged: false, ccChanged: false, refinedMtime: null, ccMtime: null,
|
||||
storedRefinedHash: null, storedCcHash: null,
|
||||
};
|
||||
(state as any).index = { matched: [row], ccOnly: [], refinedOnly: [], counts: { matched: 1, ccOnly: 0, refinedOnly: 0, unlinked: 0 } };
|
||||
}
|
||||
|
||||
async function readSyncState(): Promise<SyncState & { pendingConflicts?: { uuid: string; state: string; lastFHash: string; lastOHash: string }[]; fPending?: { uuid: string }[] }> {
|
||||
return JSON.parse(await readFile(join(dir, "out", "sync-state.json"), "utf8"));
|
||||
}
|
||||
|
||||
describe("E2.4 both-diverged → pending conflict row", () => {
|
||||
it("F-changed + O-changed → 'conflict' + pendingConflicts row recorded + removed from fPending", async () => {
|
||||
const live = liveEntry("<p>Foundry-edited body.</p>");
|
||||
// Note: contentHash = "0"*64 (stale) → O-side changed (bodyHash ≠ contentHash).
|
||||
// ccHash = "0"*64 (stale) → F-side changed (ccHash(live) ≠ ccHash baseline).
|
||||
await writeNote("Edited in Obsidian.\n", "0".repeat(64), "0".repeat(64));
|
||||
setupIndex();
|
||||
|
||||
// Pre-populate fPending with the uuid (simulating the deep poll detected it).
|
||||
(state.syncState as any).fPending = [{ uuid: UUID, name: "Roland", change: "edited", detectedAt: new Date().toISOString() }];
|
||||
await saveSyncState(state.cfg.outDir, state.syncState!);
|
||||
|
||||
const result = await state.autosync.pullFChanged(UUID, live);
|
||||
expect(result).toBe("conflict");
|
||||
|
||||
const saved = await readSyncState();
|
||||
expect(saved.pendingConflicts).toBeTruthy();
|
||||
expect(saved.pendingConflicts!.length).toBe(1);
|
||||
const pc = saved.pendingConflicts![0];
|
||||
expect(pc.uuid).toBe(UUID);
|
||||
expect(pc.state).toBe("both-diverged");
|
||||
expect(pc.lastFHash).toBe(ccHash(live));
|
||||
expect(pc.lastOHash).toBe(contentHash("Edited in Obsidian.\n"));
|
||||
// Removed from fPending.
|
||||
expect((saved.fPending ?? []).some((e) => e.uuid === UUID)).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
describe("E2.4 vault-newer (missing) → pending conflict row", () => {
|
||||
it("shallow poll detects missing → recordPendingConflict('vault-newer')", async () => {
|
||||
setupIndex();
|
||||
// First poll: Roland present.
|
||||
globalThis.fetch = vi.fn(async (url: string) => {
|
||||
if (String(url).includes("/search")) return { ok: true, status: 200, text: async () => JSON.stringify({ results: [{ uuid: UUID, id: "aaa", name: "Roland", documentType: "JournalEntry" }] }) } as unknown as Response;
|
||||
return { ok: false, status: 404, text: async () => '{"error":"not found"}' } as unknown as Response;
|
||||
}) as unknown as typeof fetch;
|
||||
|
||||
const controller = new FoundryPollController(state);
|
||||
await controller.setEnabled(true);
|
||||
// Wait for first poll.
|
||||
while ((controller as any).prevSnapshot.size === 0) await new Promise<void>((r) => setTimeout(r, 10));
|
||||
|
||||
// Second poll: Roland gone.
|
||||
globalThis.fetch = vi.fn(async (url: string) => {
|
||||
if (String(url).includes("/search")) return { ok: true, status: 200, text: async () => JSON.stringify({ results: [] }) } as unknown as Response;
|
||||
return { ok: false, status: 404, text: async () => '{"error":"not found"}' } as unknown as Response;
|
||||
}) as unknown as typeof fetch;
|
||||
|
||||
await (controller as any).tick();
|
||||
controller.stop();
|
||||
|
||||
const saved = await readSyncState();
|
||||
expect(saved.pendingConflicts).toBeTruthy();
|
||||
expect(saved.pendingConflicts!.some((e) => e.uuid === UUID && e.state === "vault-newer")).toBe(true);
|
||||
});
|
||||
});
|
||||
|
||||
describe("E2.4 GET /api/foundry-poll includes pendingConflicts", () => {
|
||||
it("recordPendingConflict → the row is in sync-state.json + accessible via the poll status", async () => {
|
||||
await state.autosync.recordPendingConflict(UUID, "Roland", "both-diverged", "abc", "def");
|
||||
const saved = await readSyncState();
|
||||
expect(saved.pendingConflicts?.length).toBe(1);
|
||||
expect(saved.pendingConflicts![0].state).toBe("both-diverged");
|
||||
});
|
||||
});
|
||||
120
tests/e2-5-import.test.ts
Normal file
120
tests/e2-5-import.test.ts
Normal file
@@ -0,0 +1,120 @@
|
||||
// E2.5 — live-new-entries list + one-click import.
|
||||
|
||||
import { describe, it, expect, beforeAll, afterAll, afterEach, vi } from "vitest";
|
||||
import { ClassicLevel } from "classic-level";
|
||||
import { mkdtemp, mkdir, rm, readFile, writeFile } from "node:fs/promises";
|
||||
import { existsSync } from "node:fs";
|
||||
import { join } from "node:path";
|
||||
import { tmpdir } from "node:os";
|
||||
import type { Server } from "node:http";
|
||||
import { startServer, type State } from "../src/server.js";
|
||||
|
||||
let dir: string;
|
||||
let server: Server;
|
||||
let state: State;
|
||||
let baseURL: string;
|
||||
const realFetch = globalThis.fetch;
|
||||
|
||||
const UUID = "JournalEntry.new1";
|
||||
const ENTRY_NAME = "New NPC";
|
||||
|
||||
function mockGetEntry(): void {
|
||||
globalThis.fetch = vi.fn(async (url: string, init?: RequestInit) => {
|
||||
const u = String(url);
|
||||
if (u.includes("/get?")) {
|
||||
return { ok: true, status: 200, text: async () => JSON.stringify({ data: { name: ENTRY_NAME, _id: "new1", folder: "Folder.test", flags: { "campaign-codex": { type: "npc", data: { description: "<p>New entry.</p>", notes: "" } } } } }) } as unknown as Response;
|
||||
}
|
||||
if (u.includes("/search")) return { ok: true, status: 200, text: async () => JSON.stringify({ results: [] }) } as unknown as Response;
|
||||
// Pass through to the real fetch for server HTTP calls.
|
||||
return realFetch(url, init);
|
||||
}) as unknown as typeof fetch;
|
||||
}
|
||||
|
||||
async function bootServer(opts: { foundryPoll?: boolean; collisionName?: string } = {}): Promise<void> {
|
||||
dir = await mkdtemp(join(tmpdir(), "e2-5-"));
|
||||
await mkdir(join(dir, "refined"), { recursive: true });
|
||||
await mkdir(join(dir, "cc"), { recursive: true });
|
||||
// Optional: write a colliding note before boot (so the index includes it).
|
||||
if (opts.collisionName) {
|
||||
await mkdir(join(dir, "refined", "imported", "npcs"), { recursive: true });
|
||||
await writeFile(join(dir, "refined", "imported", "npcs", `${opts.collisionName}.md`), "---\ntype: npc\n---\nbody\n", "utf8");
|
||||
}
|
||||
const jdb = new ClassicLevel<string, string>(join(dir, "journal"));
|
||||
await jdb.open(); await jdb.close();
|
||||
const { server: srv, state: st } = await startServer({
|
||||
journal: join(dir, "journal"), refinedDir: join(dir, "refined"), ccDir: join(dir, "cc"),
|
||||
outDir: join(dir, "out"), mode: "apply", port: 0, host: "127.0.0.1",
|
||||
relayCfg: { url: "http://relay.test", apiKey: "k", clientId: "c" },
|
||||
features: { syncStatus: true, foundryPoll: opts.foundryPoll ?? true },
|
||||
});
|
||||
server = srv;
|
||||
state = st;
|
||||
baseURL = `http://127.0.0.1:${(server.address() as { port: number }).port}`;
|
||||
}
|
||||
|
||||
async function postImport(uuid: string): Promise<{ code: number; body: unknown }> {
|
||||
const r = await fetch(`${baseURL}/api/foundry-poll/import`, {
|
||||
method: "POST", headers: { "content-type": "application/json", origin: baseURL },
|
||||
body: JSON.stringify({ uuid }),
|
||||
});
|
||||
return { code: r.status, body: await r.json().catch(() => null) };
|
||||
}
|
||||
|
||||
describe("E2.5 live-new-entries import", () => {
|
||||
afterEach(async () => {
|
||||
globalThis.fetch = realFetch;
|
||||
if (server) await new Promise<void>((r) => server.close(() => r()));
|
||||
if (dir) await rm(dir, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
it("POST /api/foundry-poll/import → imports the entry + removes from liveNewEntries", async () => {
|
||||
await bootServer({ foundryPoll: true });
|
||||
mockGetEntry();
|
||||
// Manually add a live new entry.
|
||||
state.foundryPoll!.liveNewEntries.push({ uuid: UUID, name: ENTRY_NAME, detectedAt: new Date().toISOString() });
|
||||
const r = await postImport(UUID);
|
||||
expect(r.code).toBe(200);
|
||||
const body = r.body as { ok: boolean; uuid: string; name: string; filename: string; subfolder: string };
|
||||
expect(body.ok).toBe(true);
|
||||
expect(body.name).toBe(ENTRY_NAME);
|
||||
// The file was written under refined/imported/<subfolder>/.
|
||||
const notePath = join(dir, "refined", "imported", body.subfolder, body.filename);
|
||||
expect(existsSync(notePath)).toBe(true);
|
||||
// Removed from liveNewEntries.
|
||||
expect(state.foundryPoll!.liveNewEntries.some((e) => e.uuid === UUID)).toBe(false);
|
||||
});
|
||||
|
||||
it("name collision → 409 'name already exists in vault'", async () => {
|
||||
// Write a colliding note before boot (so the index includes it).
|
||||
await bootServer({ foundryPoll: true, collisionName: ENTRY_NAME });
|
||||
mockGetEntry();
|
||||
state.foundryPoll!.liveNewEntries.push({ uuid: UUID, name: ENTRY_NAME, detectedAt: new Date().toISOString() });
|
||||
// Trigger an index refresh so the collision is detected.
|
||||
await fetch(`${baseURL}/api/index`);
|
||||
const r = await postImport(UUID);
|
||||
expect(r.code).toBe(409);
|
||||
expect((r.body as { error: string }).error).toMatch(/name already exists/);
|
||||
// NOT removed from liveNewEntries (import failed).
|
||||
expect(state.foundryPoll!.liveNewEntries.some((e) => e.uuid === UUID)).toBe(true);
|
||||
});
|
||||
|
||||
it("no live new entry for the uuid → 404", async () => {
|
||||
await bootServer({ foundryPoll: true });
|
||||
mockGetEntry();
|
||||
const r = await postImport("JournalEntry.nonexistent");
|
||||
expect(r.code).toBe(404);
|
||||
});
|
||||
|
||||
it("foundryPoll flag off → 404", async () => {
|
||||
await bootServer({ foundryPoll: false });
|
||||
const r = await postImport(UUID);
|
||||
expect(r.code).toBe(404);
|
||||
});
|
||||
|
||||
it("GET /api/foundry-poll includes liveNewEntries in the response", async () => {
|
||||
await bootServer({ foundryPoll: true });
|
||||
state.foundryPoll!.liveNewEntries.push({ uuid: UUID, name: ENTRY_NAME, detectedAt: new Date().toISOString() });
|
||||
const r = await fetch(`${baseURL}/api/foundry-poll`).then(r => r.json()) as { liveNewEntries: { uuid: string }[] };
|
||||
expect(r.liveNewEntries.some((e) => e.uuid === UUID)).toBe(true);
|
||||
});
|
||||
});
|
||||
139
tests/e2-6-catchup.test.ts
Normal file
139
tests/e2-6-catchup.test.ts
Normal file
@@ -0,0 +1,139 @@
|
||||
// E2.6 — catch-up-now trigger.
|
||||
|
||||
import { describe, it, expect, beforeEach, afterEach, vi } from "vitest";
|
||||
import { ClassicLevel } from "classic-level";
|
||||
import { mkdtemp, mkdir, rm } from "node:fs/promises";
|
||||
import { join } from "node:path";
|
||||
import { tmpdir } from "node:os";
|
||||
import type { Server } from "node:http";
|
||||
|
||||
import { FoundryPollController } from "../src/foundry-poll.js";
|
||||
import { startServer, type State } from "../src/server.js";
|
||||
import { saveSyncState, defaultSyncState } from "../src/sync-state.js";
|
||||
|
||||
let dir: string;
|
||||
let state: State;
|
||||
const realFetch = globalThis.fetch;
|
||||
|
||||
beforeEach(async () => {
|
||||
dir = await mkdtemp(join(tmpdir(), "e2-6-"));
|
||||
await mkdir(join(dir, "refined"), { recursive: true });
|
||||
await mkdir(join(dir, "out"), { recursive: true });
|
||||
const cfg = {
|
||||
journal: "", refinedDir: join(dir, "refined"), ccDir: "", outDir: join(dir, "out"),
|
||||
mode: "apply" as const, port: 0, host: "127.0.0.1",
|
||||
relayCfg: { url: "http://relay.test", apiKey: "k", clientId: "c" },
|
||||
features: { syncStatus: true, foundryPoll: true },
|
||||
};
|
||||
state = { db: {} as State["db"], cfg, index: null, autosync: null as unknown as State["autosync"], foundryPoll: null as unknown as State["foundryPoll"] } as unknown as State;
|
||||
state.syncState = { ...defaultSyncState(cfg.refinedDir) };
|
||||
await saveSyncState(cfg.outDir, state.syncState);
|
||||
});
|
||||
|
||||
afterEach(async () => { globalThis.fetch = realFetch; await rm(dir, { recursive: true, force: true }); });
|
||||
|
||||
function mockSearch(): void {
|
||||
globalThis.fetch = vi.fn(async (url: string) => {
|
||||
if (String(url).includes("/search")) return { ok: true, status: 200, text: async () => JSON.stringify({ results: [] }) } as unknown as Response;
|
||||
return { ok: false, status: 404, text: async () => '{"error":"not found"}' } as unknown as Response;
|
||||
}) as unknown as typeof fetch;
|
||||
}
|
||||
|
||||
describe("E2.6 catchUpNow (direct)", () => {
|
||||
it("runs shallow + deep immediately and returns a summary with durationMs", async () => {
|
||||
mockSearch();
|
||||
const controller = new FoundryPollController(state);
|
||||
await controller.setEnabled(true);
|
||||
// Wait for the first scheduled shallow poll to complete (with empty /search
|
||||
// results, prevSnapshot is an empty Map — so wait via lastPollAt instead).
|
||||
while (!state.syncState?.parity.lastPollAt) await new Promise<void>((r) => setTimeout(r, 10));
|
||||
const result = await controller.catchUpNow();
|
||||
expect(result.skipped).toBeUndefined();
|
||||
expect(result.durationMs).toBeGreaterThanOrEqual(0);
|
||||
expect(result.shallow).toBeDefined();
|
||||
expect(result.deep).toBeDefined();
|
||||
controller.stop();
|
||||
});
|
||||
|
||||
it("debounced — returns { skipped: true } if shallow is in flight", async () => {
|
||||
// Make the search slow so shallow is in flight when we call catchUpNow.
|
||||
const holder: { resolve: (() => void) | null } = { resolve: null };
|
||||
globalThis.fetch = vi.fn(async () => {
|
||||
await new Promise<void>((r) => { holder.resolve = r; });
|
||||
return { ok: true, status: 200, text: async () => JSON.stringify({ results: [] }) } as unknown as Response;
|
||||
}) as unknown as typeof fetch;
|
||||
const controller = new FoundryPollController(state);
|
||||
await controller.setEnabled(true);
|
||||
await new Promise<void>((r) => setTimeout(r, 30)); // let the first tick start
|
||||
expect((controller as any).inFlight).toBe(true);
|
||||
const result = await controller.catchUpNow();
|
||||
expect(result.skipped).toBe(true);
|
||||
if (holder.resolve) holder.resolve();
|
||||
controller.stop();
|
||||
});
|
||||
});
|
||||
|
||||
describe("E2.6 POST /api/foundry-poll/catchup (endpoint)", () => {
|
||||
let server: Server;
|
||||
let baseURL: string;
|
||||
|
||||
afterEach(async () => {
|
||||
if (server) await new Promise<void>((r) => server.close(() => r()));
|
||||
});
|
||||
|
||||
async function boot(opts: { foundryPoll?: boolean } = {}): Promise<void> {
|
||||
const d = await mkdtemp(join(tmpdir(), "e2-6-srv-"));
|
||||
await mkdir(join(d, "refined"), { recursive: true });
|
||||
await mkdir(join(d, "cc"), { recursive: true });
|
||||
const jdb = new ClassicLevel<string, string>(join(d, "journal"));
|
||||
await jdb.open(); await jdb.close();
|
||||
const { server: srv, state: st } = await startServer({
|
||||
journal: join(d, "journal"), refinedDir: join(d, "refined"), ccDir: join(d, "cc"),
|
||||
outDir: join(d, "out"), mode: "apply", port: 0, host: "127.0.0.1",
|
||||
relayCfg: { url: "http://relay.test", apiKey: "k", clientId: "c" },
|
||||
features: { syncStatus: true, foundryPoll: opts.foundryPoll ?? true },
|
||||
});
|
||||
server = srv;
|
||||
baseURL = `http://127.0.0.1:${(server.address() as { port: number }).port}`;
|
||||
// Set the mock for the relay.
|
||||
globalThis.fetch = vi.fn(async (url: string, init?: RequestInit) => {
|
||||
const u = String(url);
|
||||
if (u.includes("/search")) return { ok: true, status: 200, text: async () => JSON.stringify({ results: [] }) } as unknown as Response;
|
||||
if (u.includes("/get?")) return { ok: true, status: 200, text: async () => JSON.stringify({ data: { name: "x", _id: "x", flags: {} } }) } as unknown as Response;
|
||||
return realFetch(url, init);
|
||||
}) as unknown as typeof fetch;
|
||||
// Enable the poll + wait for the first shallow poll (lastPollAt set).
|
||||
if (opts.foundryPoll !== false) {
|
||||
await st.foundryPoll!.setEnabled(true);
|
||||
while (!st.syncState?.parity.lastPollAt) await new Promise<void>((r) => setTimeout(r, 10));
|
||||
}
|
||||
(globalThis as unknown as { _e26dir: string })._e26dir = d;
|
||||
}
|
||||
|
||||
afterEach(async () => {
|
||||
const d = (globalThis as unknown as { _e26dir?: string })._e26dir;
|
||||
if (d) await rm(d, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
it("200 with a summary when enabled", async () => {
|
||||
await boot({ foundryPoll: true });
|
||||
const r = await fetch(`${baseURL}/api/foundry-poll/catchup`, { method: "POST", headers: { "content-type": "application/json", origin: baseURL } });
|
||||
expect(r.status).toBe(200);
|
||||
const body = await r.json() as { durationMs?: number };
|
||||
expect(body.durationMs).toBeGreaterThanOrEqual(0);
|
||||
});
|
||||
|
||||
it("400 when the poll is not enabled", async () => {
|
||||
await boot({ foundryPoll: true });
|
||||
// Disable the poll before calling catchup.
|
||||
await fetch(`${baseURL}/api/foundry-poll`, { method: "POST", headers: { "content-type": "application/json", origin: baseURL }, body: JSON.stringify({ enabled: false }) });
|
||||
const r = await fetch(`${baseURL}/api/foundry-poll/catchup`, { method: "POST", headers: { "content-type": "application/json", origin: baseURL } });
|
||||
expect(r.status).toBe(400);
|
||||
});
|
||||
|
||||
it("404 when foundryPoll flag is off", async () => {
|
||||
await boot({ foundryPoll: false });
|
||||
const r = await fetch(`${baseURL}/api/foundry-poll/catchup`, { method: "POST", headers: { "content-type": "application/json", origin: baseURL } });
|
||||
expect(r.status).toBe(404);
|
||||
});
|
||||
});
|
||||
136
tests/e4-1-syncstate.test.ts
Normal file
136
tests/e4-1-syncstate.test.ts
Normal file
@@ -0,0 +1,136 @@
|
||||
// E4.1 — persistent sync-state.json (load/save/atomic/schema-mismatch/restart/reconcile).
|
||||
|
||||
import { describe, it, expect, beforeEach, afterEach } from "vitest";
|
||||
import { ClassicLevel } from "classic-level";
|
||||
import { mkdtemp, mkdir, rm, readFile, readdir } from "node:fs/promises";
|
||||
import { existsSync } from "node:fs";
|
||||
import { join } from "node:path";
|
||||
import { tmpdir } from "node:os";
|
||||
import type { Server } from "node:http";
|
||||
|
||||
import { loadSyncState, saveSyncState, appendActivity, defaultSyncState, MAX_ACTIVITY, type SyncState } from "../src/sync-state.js";
|
||||
import { SYNC_STATE_SCHEMA_VERSION } from "../src/schema-version.js";
|
||||
import { startServer } from "../src/server.js";
|
||||
|
||||
let dir: string;
|
||||
beforeEach(async () => { dir = await mkdtemp(join(tmpdir(), "e4-1-")); await mkdir(join(dir, "refined"), { recursive: true }); });
|
||||
afterEach(async () => { await rm(dir, { recursive: true, force: true }); });
|
||||
|
||||
describe("E4.1 SyncState load/save", () => {
|
||||
it("load creates defaults if absent (autoSyncOn=false, mode=PREP) + writes the file", async () => {
|
||||
const { state, freshened } = await loadSyncState(dir, "/vault/refined");
|
||||
expect(freshened).toBe(false);
|
||||
expect(state.autoSyncOn).toBe(false);
|
||||
expect(state.mode).toBe("PREP");
|
||||
expect(state.syncStateSchemaVersion).toBe(SYNC_STATE_SCHEMA_VERSION);
|
||||
expect(state.activity).toEqual([]);
|
||||
expect(state.conflict).toBeNull();
|
||||
expect(existsSync(join(dir, "sync-state.json"))).toBe(true);
|
||||
});
|
||||
|
||||
it("save is atomic (tmp+rename — no .tmp left after)", async () => {
|
||||
const { state } = await loadSyncState(dir, "/vault/refined");
|
||||
state.autoSyncOn = true;
|
||||
await saveSyncState(dir, state);
|
||||
const files = await readdir(dir);
|
||||
expect(files).toContain("sync-state.json");
|
||||
expect(files.some((f) => f.endsWith(".tmp"))).toBe(false);
|
||||
const reloaded = JSON.parse(await readFile(join(dir, "sync-state.json"), "utf8")) as SyncState;
|
||||
expect(reloaded.autoSyncOn).toBe(true);
|
||||
});
|
||||
|
||||
it("schema mismatch → backs up the old file + writes fresh defaults + an error event", async () => {
|
||||
// Write a stale-version file.
|
||||
const stale = { ...defaultSyncState("/vault/refined"), syncStateSchemaVersion: "sync-state/v0" };
|
||||
await saveSyncState(dir, stale);
|
||||
const { state, freshened } = await loadSyncState(dir, "/vault/refined");
|
||||
expect(freshened).toBe(true);
|
||||
expect(state.syncStateSchemaVersion).toBe(SYNC_STATE_SCHEMA_VERSION);
|
||||
expect(state.autoSyncOn).toBe(false); // fresh defaults
|
||||
expect(state.activity.length).toBe(1);
|
||||
expect(state.activity[0].kind).toBe("error");
|
||||
expect(state.activity[0].message).toMatch(/schema reset/);
|
||||
// The old file was backed up (.bak-<stamp>).
|
||||
const files = await readdir(dir);
|
||||
expect(files.some((f) => f.startsWith("sync-state.json.bak-"))).toBe(true);
|
||||
});
|
||||
|
||||
it("restart survival: autoSyncOn + activity preserved across reload", async () => {
|
||||
const { state } = await loadSyncState(dir, "/vault/refined");
|
||||
state.autoSyncOn = true;
|
||||
state.lastSyncAt = "2026-06-23T01:00:00.000Z";
|
||||
for (let i = 0; i < 17; i++) state.activity.push({ time: `2026-06-23T0:${i}:00.000Z`, kind: "push", name: `N${i}`, status: "pushed", message: `m${i}` });
|
||||
await saveSyncState(dir, state);
|
||||
// Reload — the persisted state is restored.
|
||||
const { state: reloaded } = await loadSyncState(dir, "/vault/refined");
|
||||
expect(reloaded.autoSyncOn).toBe(true);
|
||||
expect(reloaded.lastSyncAt).toBe("2026-06-23T01:00:00.000Z");
|
||||
expect(reloaded.activity.length).toBe(17);
|
||||
});
|
||||
|
||||
it("activity is trimmed to MAX_ACTIVITY on append", async () => {
|
||||
const { state } = await loadSyncState(dir, "/vault/refined");
|
||||
for (let i = 0; i < MAX_ACTIVITY + 50; i++) {
|
||||
await appendActivity(dir, state, { time: `t${i}`, kind: "push", name: `N${i}`, status: "pushed", message: `m${i}` });
|
||||
}
|
||||
expect(state.activity.length).toBe(MAX_ACTIVITY);
|
||||
const reloaded = JSON.parse(await readFile(join(dir, "sync-state.json"), "utf8")) as SyncState;
|
||||
expect(reloaded.activity.length).toBe(MAX_ACTIVITY);
|
||||
});
|
||||
|
||||
it("conflict field is forced to null by E4 (reserved for E3)", async () => {
|
||||
// Even if a file has a non-null conflict, E4 forces it null on load.
|
||||
const { state } = await loadSyncState(dir, "/vault/refined");
|
||||
(state as unknown as { conflict: unknown }).conflict = { foo: "bar" };
|
||||
await saveSyncState(dir, state);
|
||||
const { state: reloaded } = await loadSyncState(dir, "/vault/refined");
|
||||
expect(reloaded.conflict).toBeNull();
|
||||
});
|
||||
});
|
||||
|
||||
describe("E4.1 boot reconcile (startServer restores autoSyncOn)", () => {
|
||||
async function bootWithState(state: Partial<SyncState>, opts: { relay?: boolean; mode?: "dev" | "apply" } = {}): Promise<{ server: Server; enabledAfter: boolean; stateAfter: SyncState }> {
|
||||
// Pre-write sync-state.json in outDir.
|
||||
const outDir = join(dir, "out");
|
||||
await mkdir(outDir, { recursive: true });
|
||||
const full = { ...defaultSyncState(join(dir, "refined")), ...state } as SyncState;
|
||||
await saveSyncState(outDir, full);
|
||||
// Create an empty journal LevelDB.
|
||||
const jdb = new ClassicLevel<string, string>(join(dir, "journal"));
|
||||
await jdb.open(); await jdb.close();
|
||||
const { server, state: srvState } = await startServer({
|
||||
journal: join(dir, "journal"), refinedDir: join(dir, "refined"), ccDir: join(dir, "cc"),
|
||||
outDir, mode: opts.mode ?? "apply", port: 0, host: "127.0.0.1",
|
||||
relayCfg: opts.relay ? { url: "http://relay.test", apiKey: "k", clientId: "c" } : undefined,
|
||||
});
|
||||
const enabledAfter = srvState.autosync.enabled;
|
||||
const stateAfter = JSON.parse(await readFile(join(outDir, "sync-state.json"), "utf8")) as SyncState;
|
||||
// Clean up the watcher (auto-sync may be ON).
|
||||
srvState.autosync.stop();
|
||||
await new Promise<void>((r) => server.close(() => r()));
|
||||
return { server, enabledAfter, stateAfter };
|
||||
}
|
||||
|
||||
it("autoSyncOn=true + apply mode + relay → controller.enabled restored to true", async () => {
|
||||
const { enabledAfter } = await bootWithState({ autoSyncOn: true }, { relay: true, mode: "apply" });
|
||||
expect(enabledAfter).toBe(true);
|
||||
});
|
||||
|
||||
it("autoSyncOn=true + no relay → autoSyncOn flipped to false + error event", async () => {
|
||||
const { enabledAfter, stateAfter } = await bootWithState({ autoSyncOn: true }, { relay: false, mode: "apply" });
|
||||
expect(enabledAfter).toBe(false); // start() threw (no relay) → flipped off
|
||||
expect(stateAfter.autoSyncOn).toBe(false);
|
||||
expect(stateAfter.activity.some((e) => e.message.includes("auto-sync could not resume"))).toBe(true);
|
||||
});
|
||||
|
||||
it("autoSyncOn=true + dev mode (no apply) → autoSyncOn flipped to false (apply-mode gate)", async () => {
|
||||
const { enabledAfter, stateAfter } = await bootWithState({ autoSyncOn: true }, { relay: true, mode: "dev" });
|
||||
expect(enabledAfter).toBe(false); // start() threw (dev mode) → flipped off
|
||||
expect(stateAfter.autoSyncOn).toBe(false);
|
||||
});
|
||||
|
||||
it("autoSyncOn=false (fresh install) → stays off", async () => {
|
||||
const { enabledAfter } = await bootWithState({ autoSyncOn: false }, { relay: true, mode: "apply" });
|
||||
expect(enabledAfter).toBe(false);
|
||||
});
|
||||
});
|
||||
138
tests/e4-2-mode.test.ts
Normal file
138
tests/e4-2-mode.test.ts
Normal file
@@ -0,0 +1,138 @@
|
||||
// E4.2 — PREP / RUN-THE-MATCH mode flag (gates AutoSyncController).
|
||||
|
||||
import { describe, it, expect, beforeEach, afterEach } from "vitest";
|
||||
import { ClassicLevel } from "classic-level";
|
||||
import { mkdtemp, mkdir, rm, readFile } from "node:fs/promises";
|
||||
import { join } from "node:path";
|
||||
import { tmpdir } from "node:os";
|
||||
import type { Server } from "node:http";
|
||||
|
||||
import { AutoSyncController } from "../src/server.js";
|
||||
import type { State, ServerConfig } from "../src/server.js";
|
||||
import { saveSyncState, defaultSyncState, type SyncState } from "../src/sync-state.js";
|
||||
import { startServer } from "../src/server.js";
|
||||
|
||||
let dir: string;
|
||||
beforeEach(async () => { dir = await mkdtemp(join(tmpdir(), "e4-2-")); await mkdir(join(dir, "refined"), { recursive: true }); });
|
||||
afterEach(async () => { await rm(dir, { recursive: true, force: true }); });
|
||||
|
||||
function makeState(opts: { syncStatus?: boolean; syncMode?: string; mode?: "dev" | "apply" } = {}): State {
|
||||
const cfg: ServerConfig = {
|
||||
journal: "", refinedDir: join(dir, "refined"), ccDir: "", outDir: join(dir, "out"),
|
||||
mode: opts.mode ?? "apply", port: 0, host: "127.0.0.1",
|
||||
relayCfg: { url: "http://relay.test", apiKey: "k", clientId: "c" },
|
||||
features: { syncStatus: opts.syncStatus ?? true },
|
||||
};
|
||||
const state = { db: {} as State["db"], cfg, index: null, autosync: null as unknown as State["autosync"] } as unknown as State;
|
||||
state.syncState = { ...defaultSyncState(cfg.refinedDir), mode: opts.syncMode ?? "PREP" } as SyncState;
|
||||
return state;
|
||||
}
|
||||
|
||||
describe("E4.2 setEnabled PREP/RUN gate", () => {
|
||||
it("setEnabled(true) in PREP mode (features.syncStatus on) → throws 'blocked in PREP mode'", async () => {
|
||||
const state = makeState({ syncStatus: true, syncMode: "PREP", mode: "apply" });
|
||||
const controller = new AutoSyncController(state);
|
||||
await expect(controller.setEnabled(true)).rejects.toThrow(/blocked in PREP mode/);
|
||||
expect(controller.enabled).toBe(false);
|
||||
});
|
||||
|
||||
it("setEnabled(true) in RUN-THE-MATCH mode → proceeds (enabled=true)", async () => {
|
||||
const state = makeState({ syncStatus: true, syncMode: "RUN-THE-MATCH", mode: "apply" });
|
||||
const controller = new AutoSyncController(state);
|
||||
await controller.setEnabled(true);
|
||||
expect(controller.enabled).toBe(true);
|
||||
controller.stop();
|
||||
});
|
||||
|
||||
it("features.syncStatus OFF → no PREP gate (setEnabled works in apply mode even with mode=PREP)", async () => {
|
||||
const state = makeState({ syncStatus: false, syncMode: "PREP", mode: "apply" });
|
||||
const controller = new AutoSyncController(state);
|
||||
await controller.setEnabled(true); // no PREP gate (flag off)
|
||||
expect(controller.enabled).toBe(true);
|
||||
controller.stop();
|
||||
});
|
||||
});
|
||||
|
||||
describe("E4.2 POST /api/sync-state/mode + boot reconcile", () => {
|
||||
async function bootWithState(stateOverrides: Partial<SyncState>): Promise<{ server: Server; state: State }> {
|
||||
const outDir = join(dir, "out");
|
||||
await mkdir(outDir, { recursive: true });
|
||||
const full = { ...defaultSyncState(join(dir, "refined")), ...stateOverrides } as SyncState;
|
||||
await saveSyncState(outDir, full);
|
||||
const jdb = new ClassicLevel<string, string>(join(dir, "journal"));
|
||||
await jdb.open(); await jdb.close();
|
||||
const { server, state } = await startServer({
|
||||
journal: join(dir, "journal"), refinedDir: join(dir, "refined"), ccDir: join(dir, "cc"),
|
||||
outDir, mode: "apply", port: 0, host: "127.0.0.1",
|
||||
relayCfg: { url: "http://relay.test", apiKey: "k", clientId: "c" },
|
||||
features: { syncStatus: true },
|
||||
});
|
||||
return { server, state };
|
||||
}
|
||||
|
||||
async function postMode(server: Server, mode: string): Promise<{ code: number; body: unknown }> {
|
||||
const port = (server.address() as { port: number }).port;
|
||||
const r = await fetch(`http://127.0.0.1:${port}/api/sync-state/mode`, {
|
||||
method: "POST", headers: { "content-type": "application/json", origin: `http://127.0.0.1:${port}` },
|
||||
body: JSON.stringify({ mode }),
|
||||
});
|
||||
return { code: r.status, body: await r.json().catch(() => null) };
|
||||
}
|
||||
|
||||
it("POST /api/sync-state/mode flips PREP → RUN-THE-MATCH", async () => {
|
||||
const { server, state } = await bootWithState({ mode: "PREP" });
|
||||
const r = await postMode(server, "RUN-THE-MATCH");
|
||||
expect(r.code).toBe(200);
|
||||
expect((r.body as { mode: string }).mode).toBe("RUN-THE-MATCH");
|
||||
// Persisted to sync-state.json.
|
||||
const saved = JSON.parse(await readFile(join(dir, "out", "sync-state.json"), "utf8")) as SyncState;
|
||||
expect(saved.mode).toBe("RUN-THE-MATCH");
|
||||
state.autosync.stop();
|
||||
await new Promise<void>((res) => server.close(() => res()));
|
||||
});
|
||||
|
||||
it("switching to PREP while auto-sync is ON → stop() + autoSyncOn=false", async () => {
|
||||
const { server, state } = await bootWithState({ mode: "RUN-THE-MATCH", autoSyncOn: true });
|
||||
expect(state.autosync.enabled).toBe(true); // auto-sync restored ON in RUN mode
|
||||
const r = await postMode(server, "PREP");
|
||||
expect(r.code).toBe(200);
|
||||
expect(state.autosync.enabled).toBe(false); // stopped
|
||||
const saved = JSON.parse(await readFile(join(dir, "out", "sync-state.json"), "utf8")) as SyncState;
|
||||
expect(saved.autoSyncOn).toBe(false);
|
||||
expect(saved.mode).toBe("PREP");
|
||||
await new Promise<void>((res) => server.close(() => res()));
|
||||
});
|
||||
|
||||
it("boot reconcile: PREP + autoSyncOn=true → autoSyncOn=false + 'PREP mode auto-sync disabled on boot' event", async () => {
|
||||
const { server, state } = await bootWithState({ mode: "PREP", autoSyncOn: true });
|
||||
expect(state.autosync.enabled).toBe(false); // not restored (PREP blocks)
|
||||
const saved = JSON.parse(await readFile(join(dir, "out", "sync-state.json"), "utf8")) as SyncState;
|
||||
expect(saved.autoSyncOn).toBe(false);
|
||||
expect(saved.activity.some((e) => e.message.includes("PREP mode auto-sync disabled on boot"))).toBe(true);
|
||||
await new Promise<void>((res) => server.close(() => res()));
|
||||
});
|
||||
|
||||
it("invalid mode → 400", async () => {
|
||||
const { server } = await bootWithState({ mode: "PREP" });
|
||||
const r = await postMode(server, "INVALID");
|
||||
expect(r.code).toBe(400);
|
||||
await new Promise<void>((res) => server.close(() => res()));
|
||||
});
|
||||
|
||||
it("features.syncStatus OFF → POST /api/sync-state/mode → 404", async () => {
|
||||
const outDir = join(dir, "out");
|
||||
await mkdir(outDir, { recursive: true });
|
||||
await saveSyncState(outDir, { ...defaultSyncState(join(dir, "refined")), mode: "PREP" } as SyncState);
|
||||
const jdb = new ClassicLevel<string, string>(join(dir, "journal"));
|
||||
await jdb.open(); await jdb.close();
|
||||
const { server } = await startServer({
|
||||
journal: join(dir, "journal"), refinedDir: join(dir, "refined"), ccDir: join(dir, "cc"),
|
||||
outDir, mode: "apply", port: 0, host: "127.0.0.1",
|
||||
relayCfg: { url: "http://relay.test", apiKey: "k", clientId: "c" },
|
||||
features: { syncStatus: false }, // OFF → endpoint 404
|
||||
});
|
||||
const r = await postMode(server, "RUN-THE-MATCH");
|
||||
expect(r.code).toBe(404);
|
||||
await new Promise<void>((res) => server.close(() => res()));
|
||||
});
|
||||
});
|
||||
117
tests/e4-3-parity.test.ts
Normal file
117
tests/e4-3-parity.test.ts
Normal file
@@ -0,0 +1,117 @@
|
||||
// E4.3 — /api/sync-state endpoint + parity refresh from the index.
|
||||
|
||||
import { describe, it, expect, beforeAll, afterAll } from "vitest";
|
||||
import { ClassicLevel } from "classic-level";
|
||||
import { mkdtemp, mkdir, rm, readFile, writeFile } from "node:fs/promises";
|
||||
import { join } from "node:path";
|
||||
import { tmpdir } from "node:os";
|
||||
import type { Server } from "node:http";
|
||||
|
||||
import { startServer } from "../src/server.js";
|
||||
import { saveSyncState, defaultSyncState, type SyncState } from "../src/sync-state.js";
|
||||
|
||||
let dir: string;
|
||||
let server: Server;
|
||||
let baseURL: string;
|
||||
|
||||
beforeAll(async () => {
|
||||
dir = await mkdtemp(join(tmpdir(), "e4-3-"));
|
||||
await mkdir(join(dir, "refined"), { recursive: true });
|
||||
await mkdir(join(dir, "cc"), { recursive: true });
|
||||
const jdb = new ClassicLevel<string, string>(join(dir, "journal"));
|
||||
await jdb.open(); await jdb.close();
|
||||
const { server: srv } = await startServer({
|
||||
journal: join(dir, "journal"), refinedDir: join(dir, "refined"), ccDir: join(dir, "cc"),
|
||||
outDir: join(dir, "out"), mode: "dev", port: 0, host: "127.0.0.1",
|
||||
features: { syncStatus: true },
|
||||
});
|
||||
server = srv;
|
||||
baseURL = `http://127.0.0.1:${(server.address() as { port: number }).port}`;
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await new Promise<void>((r) => server.close(() => r()));
|
||||
await rm(dir, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
async function get(path: string): Promise<{ code: number; body: unknown }> {
|
||||
const r = await fetch(`${baseURL}${path}`);
|
||||
const t = await r.text();
|
||||
return { code: r.status, body: t ? JSON.parse(t) : null };
|
||||
}
|
||||
|
||||
describe("E4.3 /api/sync-state endpoint", () => {
|
||||
it("features on → 200 with the full sync state", async () => {
|
||||
const r = await get("/api/sync-state");
|
||||
expect(r.code).toBe(200);
|
||||
const s = r.body as SyncState;
|
||||
expect(s.syncStateSchemaVersion).toBeTruthy();
|
||||
expect(s.mode).toBe("PREP"); // default
|
||||
expect(s.autoSyncOn).toBe(false); // fresh install
|
||||
expect(s.parity).toBeDefined();
|
||||
expect(s.activity).toBeDefined();
|
||||
expect(s.conflict).toBeNull();
|
||||
});
|
||||
|
||||
it("features off → 404", async () => {
|
||||
// Start a second server with features.syncStatus off.
|
||||
const d2 = await mkdtemp(join(tmpdir(), "e4-3-off-"));
|
||||
await mkdir(join(d2, "refined"), { recursive: true });
|
||||
await mkdir(join(d2, "cc"), { recursive: true });
|
||||
const jdb2 = new ClassicLevel<string, string>(join(d2, "journal"));
|
||||
await jdb2.open(); await jdb2.close();
|
||||
const { server: srv2 } = await startServer({
|
||||
journal: join(d2, "journal"), refinedDir: join(d2, "refined"), ccDir: join(d2, "cc"),
|
||||
outDir: join(d2, "out"), mode: "dev", port: 0, host: "127.0.0.1",
|
||||
features: { syncStatus: false },
|
||||
});
|
||||
const base2 = `http://127.0.0.1:${(srv2.address() as { port: number }).port}`;
|
||||
const r = await fetch(`${base2}/api/sync-state`);
|
||||
expect(r.status).toBe(404);
|
||||
await new Promise<void>((res) => srv2.close(() => res()));
|
||||
await rm(d2, { recursive: true, force: true });
|
||||
});
|
||||
});
|
||||
|
||||
describe("E4.3 parity refresh (from /api/index)", () => {
|
||||
it("empty index → all counts 0, status in-parity, lastSyncAt null", async () => {
|
||||
await get("/api/index"); // triggers refreshParity
|
||||
const r = await get("/api/sync-state");
|
||||
const s = r.body as SyncState;
|
||||
expect(s.parity.oPending).toBe(0);
|
||||
expect(s.parity.fPending).toBe(0);
|
||||
expect(s.parity.conflict).toBe(0);
|
||||
expect(s.parity.unsyncedLinked).toBe(0);
|
||||
expect(s.parity.status).toBe("in-parity");
|
||||
expect(s.lastSyncAt).toBeNull(); // no push/pull events yet
|
||||
});
|
||||
|
||||
it("lastSyncAt derived from the newest push/pull activity event", async () => {
|
||||
// Boot a separate server with a pre-written sync-state.json containing a push event.
|
||||
const d2 = await mkdtemp(join(tmpdir(), "e4-3-lastsync-"));
|
||||
await mkdir(join(d2, "refined"), { recursive: true });
|
||||
await mkdir(join(d2, "cc"), { recursive: true });
|
||||
await mkdir(join(d2, "out"), { recursive: true });
|
||||
const state = { ...defaultSyncState(join(d2, "refined")), activity: [{ time: "2026-06-23T02:00:00.000Z", kind: "push", name: "Roland", status: "pushed", message: "→ JournalEntry.abc1" }] } as SyncState;
|
||||
await saveSyncState(join(d2, "out"), state);
|
||||
const jdb2 = new ClassicLevel<string, string>(join(d2, "journal"));
|
||||
await jdb2.open(); await jdb2.close();
|
||||
const { server: srv2 } = await startServer({
|
||||
journal: join(d2, "journal"), refinedDir: join(d2, "refined"), ccDir: join(d2, "cc"),
|
||||
outDir: join(d2, "out"), mode: "dev", port: 0, host: "127.0.0.1",
|
||||
features: { syncStatus: true },
|
||||
});
|
||||
const base2 = `http://127.0.0.1:${(srv2.address() as { port: number }).port}`;
|
||||
await fetch(`${base2}/api/index`); // triggers refreshParity → lastSyncAt from activity
|
||||
const r = await fetch(`${base2}/api/sync-state`).then(r => r.json()) as SyncState;
|
||||
expect(r.lastSyncAt).toBe("2026-06-23T02:00:00.000Z");
|
||||
await new Promise<void>((res) => srv2.close(() => res()));
|
||||
await rm(d2, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
it("watchedDir is set from cfg.refinedDir", async () => {
|
||||
const r = await get("/api/sync-state");
|
||||
const s = r.body as SyncState;
|
||||
expect(s.watchedDir).toBe(join(dir, "refined"));
|
||||
});
|
||||
});
|
||||
120
tests/e4-5-statusnote.test.ts
Normal file
120
tests/e4-5-statusnote.test.ts
Normal file
@@ -0,0 +1,120 @@
|
||||
// E4.5 — vault .sync-status.md note writer + airtight exclusion.
|
||||
//
|
||||
// Covers: the note is written with the sentinel + status content; the watcher
|
||||
// skips it by dot-path; the sentinel check in runPushAttempt skips a renamed
|
||||
// copy; the index excludes dotfiles.
|
||||
|
||||
import { describe, it, expect, beforeAll, afterAll, beforeEach, afterEach } from "vitest";
|
||||
import { ClassicLevel } from "classic-level";
|
||||
import { mkdtemp, mkdir, rm, readFile, writeFile } from "node:fs/promises";
|
||||
import { existsSync } from "node:fs";
|
||||
import { join } from "node:path";
|
||||
import { tmpdir } from "node:os";
|
||||
import type { Server } from "node:http";
|
||||
|
||||
import { AutoSyncController, startServer } from "../src/server.js";
|
||||
import type { State, ServerConfig } from "../src/server.js";
|
||||
import { splitFrontmatter, readFoundryBlock } from "../src/frontmatter.js";
|
||||
|
||||
let dir: string;
|
||||
let state: State;
|
||||
beforeEach(async () => {
|
||||
dir = await mkdtemp(join(tmpdir(), "e4-5-"));
|
||||
await mkdir(join(dir, "refined"), { recursive: true });
|
||||
const cfg: ServerConfig = {
|
||||
journal: "", refinedDir: join(dir, "refined"), ccDir: "", outDir: join(dir, "out"),
|
||||
mode: "apply", port: 0, host: "127.0.0.1",
|
||||
relayCfg: { url: "http://r", apiKey: "k", clientId: "c" },
|
||||
features: { syncStatus: true },
|
||||
};
|
||||
state = { db: {} as State["db"], cfg, index: null, autosync: null as unknown as State["autosync"] } as unknown as State;
|
||||
state.syncState = { syncStateSchemaVersion: "sync-state/v1", mode: "RUN-THE-MATCH", autoSyncOn: false, lastSyncAt: null, parity: { status: "in-parity", oPending: 0, fPending: 0, conflict: 0, unsyncedLinked: 0, lastPollAt: null }, watchedDir: cfg.refinedDir, activity: [], updatedAt: new Date().toISOString(), conflict: null };
|
||||
});
|
||||
afterEach(async () => { await rm(dir, { recursive: true, force: true }); });
|
||||
|
||||
describe("E4.5 .sync-status.md writer", () => {
|
||||
it("writes .sync-status.md with the sentinel + status content", async () => {
|
||||
const controller = new AutoSyncController(state);
|
||||
// Trigger a log (which calls writeStatusNote via appendActivity when features on).
|
||||
(controller as any).log("Roland", "pushed", "→ JournalEntry.abc1 · baselined");
|
||||
// writeStatusNote is fire-and-forget; wait for it.
|
||||
await new Promise<void>((r) => setTimeout(r, 50));
|
||||
const notePath = join(state.cfg.refinedDir, ".sync-status.md");
|
||||
expect(existsSync(notePath)).toBe(true);
|
||||
const md = await readFile(notePath, "utf8");
|
||||
const { fm, body } = splitFrontmatter(md);
|
||||
const fb = readFoundryBlock(fm);
|
||||
expect(fb?.sync_status).toBe("true"); // the sentinel
|
||||
expect(body).toContain("Sync Status");
|
||||
expect(body).toContain("RUN-THE-MATCH");
|
||||
});
|
||||
});
|
||||
|
||||
describe("E4.5 airtight exclusion", () => {
|
||||
it("onChange skips .sync-status.md by dot-path (no debounce timer)", () => {
|
||||
const controller = new AutoSyncController(state);
|
||||
(controller as any).onChange("change", ".sync-status.md", "");
|
||||
expect((controller as any).timers.has(".sync-status.md")).toBe(false);
|
||||
});
|
||||
|
||||
it("runPushAttempt skips a note with the sync_status sentinel (rename-safe)", async () => {
|
||||
// A note renamed to a non-dot path but still carrying the sentinel.
|
||||
await writeFile(join(state.cfg.refinedDir, "Sync Status.md"),
|
||||
"---\nfoundry:\n sync_status: \"true\"\n cc_uuid: JournalEntry.abc1\n contentHash: " + "0".repeat(64) + "\n---\nbody\n", "utf8");
|
||||
const controller = new AutoSyncController(state);
|
||||
// Call process directly (bypasses onChange's dot-path skip).
|
||||
await (controller as any).process("Sync Status.md");
|
||||
expect(controller.events.some((e) => e.message.includes("sync status note (sentinel)"))).toBe(true);
|
||||
});
|
||||
|
||||
it("a normal note (no sentinel) is NOT skipped by the sentinel check", async () => {
|
||||
// The sentinel check should only fire for sync_status: "true" notes.
|
||||
const controller = new AutoSyncController(state);
|
||||
(controller as any).log("test", "skipped", "test event");
|
||||
await new Promise<void>((r) => setTimeout(r, 20));
|
||||
// The .sync-status.md was written (by the log). Now verify a normal note
|
||||
// wouldn't be sentinel-skipped — check that the sentinel field is absent
|
||||
// from a normal note's foundry block.
|
||||
const notePath = join(state.cfg.refinedDir, ".sync-status.md");
|
||||
const md = await readFile(notePath, "utf8");
|
||||
const fb = readFoundryBlock(splitFrontmatter(md).fm);
|
||||
expect(fb?.sync_status).toBe("true"); // the status note has it
|
||||
expect(fb?.cc_uuid).toBeUndefined(); // a normal note wouldn't have sync_status
|
||||
});
|
||||
});
|
||||
|
||||
describe("E4.5 index excludes dotfiles", () => {
|
||||
let server: Server;
|
||||
let baseURL: string;
|
||||
|
||||
beforeAll(async () => {
|
||||
const d = await mkdtemp(join(tmpdir(), "e4-5-idx-"));
|
||||
await mkdir(join(d, "refined"), { recursive: true });
|
||||
await mkdir(join(d, "cc"), { recursive: true });
|
||||
// Write a normal note + a .sync-status.md (dotfile).
|
||||
await writeFile(join(d, "refined", "Roland.md"), "---\ntype: npc\n---\nbody\n", "utf8");
|
||||
await writeFile(join(d, "refined", ".sync-status.md"), "---\nfoundry:\n sync_status: \"true\"\n---\nstatus\n", "utf8");
|
||||
const jdb = new ClassicLevel<string, string>(join(d, "journal"));
|
||||
await jdb.open(); await jdb.close();
|
||||
const { server: srv } = await startServer({
|
||||
journal: join(d, "journal"), refinedDir: join(d, "refined"), ccDir: join(d, "cc"),
|
||||
outDir: join(d, "out"), mode: "dev", port: 0, host: "127.0.0.1",
|
||||
features: { syncStatus: true },
|
||||
});
|
||||
server = srv;
|
||||
baseURL = `http://127.0.0.1:${(server.address() as { port: number }).port}`;
|
||||
(globalThis as unknown as { _e45dir: string })._e45dir = d;
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await new Promise<void>((r) => server.close(() => r()));
|
||||
await rm((globalThis as unknown as { _e45dir: string })._e45dir, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
it(".sync-status.md does not appear in the index", async () => {
|
||||
const r = await fetch(`${baseURL}/api/index`).then(r => r.json()) as { matched: unknown[]; refinedOnly: unknown[]; ccOnly: unknown[] };
|
||||
const all = [...r.matched, ...r.refinedOnly, ...r.ccOnly];
|
||||
expect(all.length).toBe(1); // only Roland.md, not .sync-status.md
|
||||
expect((all[0] as { name: string }).name).toBe("Roland");
|
||||
});
|
||||
});
|
||||
98
tests/e7-1-auth.test.ts
Normal file
98
tests/e7-1-auth.test.ts
Normal file
@@ -0,0 +1,98 @@
|
||||
// E7.1a — authenticate middleware unit tests.
|
||||
//
|
||||
// Exercises the auth middleware directly with mock req/res + the mutable
|
||||
// __authState test seam (so the flag can be flipped per-test without re-importing).
|
||||
|
||||
import { describe, it, expect, beforeEach, afterEach } from "vitest";
|
||||
import { authenticate, __authState, type AuthRoute } from "../src/server.js";
|
||||
import type { IncomingMessage, ServerResponse } from "node:http";
|
||||
|
||||
const REQUIRE_AUTH: AuthRoute = { requireAuth: true, requireCSRF: true };
|
||||
const OPEN: AuthRoute = { requireAuth: false, requireCSRF: false };
|
||||
|
||||
function mockReq(headers: Record<string, string> = {}): IncomingMessage {
|
||||
return { headers } as unknown as IncomingMessage;
|
||||
}
|
||||
function mockRes(): { res: ServerResponse; code: number; body: string } {
|
||||
const state = { code: 0, body: "" };
|
||||
const res = {
|
||||
writeHead: (c: number) => { state.code = c; },
|
||||
end: (b?: unknown) => { state.body = typeof b === "string" ? b : String(b ?? ""); },
|
||||
} as unknown as ServerResponse;
|
||||
return { res, get code() { return state.code; }, get body() { return state.body; } } as { res: ServerResponse; code: number; body: string };
|
||||
}
|
||||
|
||||
const savedEnabled = __authState.enabled;
|
||||
const savedToken = __authState.token;
|
||||
const savedBound = __authState.bound;
|
||||
beforeEach(() => { __authState.enabled = false; __authState.token = ""; __authState.bound = "127.0.0.1"; });
|
||||
afterEach(() => { __authState.enabled = savedEnabled; __authState.token = savedToken; __authState.bound = savedBound; });
|
||||
|
||||
describe("E7.1a authenticate (flag off = no-op pass-through)", () => {
|
||||
it("flag off → always returns true regardless of route or headers", async () => {
|
||||
__authState.enabled = false;
|
||||
const r = mockRes();
|
||||
expect(await authenticate(mockReq(), r.res, REQUIRE_AUTH)).toBe(true);
|
||||
expect(await authenticate(mockReq({ authorization: "Bearer wrong" }), r.res, REQUIRE_AUTH)).toBe(true);
|
||||
expect(await authenticate(mockReq(), r.res, OPEN)).toBe(true);
|
||||
expect(r.code).toBe(0); // no response sent
|
||||
});
|
||||
});
|
||||
|
||||
describe("E7.1a authenticate (flag on)", () => {
|
||||
beforeEach(() => { __authState.enabled = true; __authState.token = "s3cret-token"; __authState.bound = "0.0.0.0"; });
|
||||
|
||||
it("requireAuth:false (read route) → proceeds without a token", async () => {
|
||||
const r = mockRes();
|
||||
expect(await authenticate(mockReq(), r.res, OPEN)).toBe(true);
|
||||
expect(r.code).toBe(0);
|
||||
});
|
||||
|
||||
it("E7.2: bound 127.0.0.1 (localhost trusted) → no enforcement even with flag on", async () => {
|
||||
__authState.bound = "127.0.0.1";
|
||||
const r = mockRes();
|
||||
expect(await authenticate(mockReq(), r.res, REQUIRE_AUTH)).toBe(true); // no enforcement
|
||||
expect(r.code).toBe(0);
|
||||
});
|
||||
|
||||
it("requireAuth:true + no Authorization header → 401 unauthorized", async () => {
|
||||
const r = mockRes();
|
||||
expect(await authenticate(mockReq(), r.res, REQUIRE_AUTH)).toBe(false);
|
||||
expect(r.code).toBe(401);
|
||||
expect(r.body).toContain("unauthorized");
|
||||
});
|
||||
|
||||
it("requireAuth:true + valid Bearer token → proceeds", async () => {
|
||||
const r = mockRes();
|
||||
expect(await authenticate(mockReq({ authorization: "Bearer s3cret-token" }), r.res, REQUIRE_AUTH)).toBe(true);
|
||||
expect(r.code).toBe(0);
|
||||
});
|
||||
|
||||
it("requireAuth:true + wrong token → 401 unauthorized", async () => {
|
||||
const r = mockRes();
|
||||
expect(await authenticate(mockReq({ authorization: "Bearer wrong" }), r.res, REQUIRE_AUTH)).toBe(false);
|
||||
expect(r.code).toBe(401);
|
||||
expect(r.body).toContain("unauthorized");
|
||||
});
|
||||
|
||||
it("requireAuth:true + malformed Authorization (not Bearer) → 401 bad auth header", async () => {
|
||||
const r = mockRes();
|
||||
expect(await authenticate(mockReq({ authorization: "Basic abc" }), r.res, REQUIRE_AUTH)).toBe(false);
|
||||
expect(r.code).toBe(401);
|
||||
expect(r.body).toContain("bad auth header");
|
||||
});
|
||||
|
||||
it("requireAuth:true + token via cookie → proceeds (E7.2 first-run cookie)", async () => {
|
||||
const r = mockRes();
|
||||
expect(await authenticate(mockReq({ cookie: "other=val; auth_token=s3cret-token" }), r.res, REQUIRE_AUTH)).toBe(true);
|
||||
expect(r.code).toBe(0);
|
||||
});
|
||||
|
||||
it("requireAuth:true + DASHBOARD_AUTH_TOKEN unset → 500", async () => {
|
||||
__authState.token = "";
|
||||
const r = mockRes();
|
||||
expect(await authenticate(mockReq({ authorization: "Bearer x" }), r.res, REQUIRE_AUTH)).toBe(false);
|
||||
expect(r.code).toBe(500);
|
||||
expect(r.body).toContain("DASHBOARD_AUTH_TOKEN is unset");
|
||||
});
|
||||
});
|
||||
140
tests/e7-1-dispatch.test.ts
Normal file
140
tests/e7-1-dispatch.test.ts
Normal file
@@ -0,0 +1,140 @@
|
||||
// E7.1c — dispatch auth-gating integration tests.
|
||||
//
|
||||
// Starts the REAL server (startServer) on an ephemeral port with an empty
|
||||
// temp journal LevelDB, then hits it via fetch to verify the ROUTES table is
|
||||
// consulted: read routes pass without a token; mutation routes 401 without a
|
||||
// token / 401 with a wrong token / pass with a valid Bearer. Flag off → no auth.
|
||||
|
||||
import { describe, it, expect, vi, beforeAll, afterAll, beforeEach, afterEach } from "vitest";
|
||||
import { ClassicLevel } from "classic-level";
|
||||
import { mkdtemp, mkdir, rm, writeFile } from "node:fs/promises";
|
||||
import { join } from "node:path";
|
||||
import { tmpdir } from "node:os";
|
||||
import type { Server } from "node:http";
|
||||
|
||||
import { startServer, __authState } from "../src/server.js";
|
||||
|
||||
let dir: string;
|
||||
let server: Server;
|
||||
let baseURL: string;
|
||||
const savedEnabled = __authState.enabled;
|
||||
const savedToken = __authState.token;
|
||||
const savedBound = __authState.bound;
|
||||
const realFetch = globalThis.fetch;
|
||||
|
||||
beforeAll(async () => {
|
||||
dir = await mkdtemp(join(tmpdir(), "e7dispatch-"));
|
||||
await mkdir(join(dir, "refined"), { recursive: true });
|
||||
await mkdir(join(dir, "cc"), { recursive: true });
|
||||
// Create an empty journal LevelDB so JournalDb.open (readOnly) succeeds.
|
||||
const jdb = new ClassicLevel<string, string>(join(dir, "journal"));
|
||||
await jdb.open();
|
||||
await jdb.close();
|
||||
const { server: srv } = await startServer({
|
||||
journal: join(dir, "journal"),
|
||||
refinedDir: join(dir, "refined"),
|
||||
ccDir: join(dir, "cc"),
|
||||
outDir: join(dir, "out"),
|
||||
mode: "dev", // dev mode: POST /api/autosync {enabled:true} → 400 apply-gate (no relay needed)
|
||||
port: 0,
|
||||
host: "127.0.0.1",
|
||||
});
|
||||
server = srv;
|
||||
const addr = server.address();
|
||||
if (!addr || typeof addr === "object" && !("port" in addr)) throw new Error("no port");
|
||||
baseURL = `http://127.0.0.1:${(addr as { port: number }).port}`;
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
__authState.enabled = savedEnabled;
|
||||
__authState.token = savedToken;
|
||||
__authState.bound = savedBound;
|
||||
await new Promise<void>((r) => server.close(() => r()));
|
||||
await rm(dir, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
beforeEach(() => { __authState.enabled = false; __authState.token = ""; __authState.bound = "127.0.0.1"; });
|
||||
afterEach(() => { __authState.enabled = false; __authState.token = ""; __authState.bound = "127.0.0.1"; });
|
||||
|
||||
async function req(path: string, init: RequestInit = {}): Promise<{ code: number; body: unknown }> {
|
||||
const r = await fetch(`${baseURL}${path}`, init);
|
||||
const text = await r.text();
|
||||
let body: unknown = text;
|
||||
try { body = text ? JSON.parse(text) : null; } catch { /* non-JSON */ }
|
||||
return { code: r.status, body };
|
||||
}
|
||||
|
||||
// E7.4: fetch a CSRF token + the matching cookie (Node fetch has no cookie jar,
|
||||
// so we replay the Set-Cookie as a Cookie header on the POST).
|
||||
async function getCSRF(): Promise<{ token: string; cookie: string; origin: string }> {
|
||||
const r = await fetch(`${baseURL}/api/auth/csrf`);
|
||||
const token = (await r.json() as { csrfToken: string }).csrfToken;
|
||||
const setCookie = r.headers.get("set-cookie") ?? "";
|
||||
const m = setCookie.match(/csrf_token=([^;]+)/);
|
||||
return { token, cookie: m ? `csrf_token=${m[1]}` : "", origin: baseURL };
|
||||
}
|
||||
|
||||
describe("E7.1c dispatch (flag off — no auth, byte-identical)", () => {
|
||||
it("GET /api/status passes without a token", async () => {
|
||||
__authState.enabled = false;
|
||||
const r = await req("/api/status");
|
||||
expect(r.code).toBe(200);
|
||||
});
|
||||
it("POST /api/autosync (mutation) passes without a token (dev-mode apply-gate 400, not 401)", async () => {
|
||||
__authState.enabled = false;
|
||||
const r = await req("/api/autosync", { method: "POST", headers: { "content-type": "application/json" }, body: JSON.stringify({ enabled: true }) });
|
||||
expect(r.code).toBe(400); // apply-mode gate (dev), NOT an auth 401
|
||||
expect((r.body as { error: string }).error).toMatch(/requires --apply mode/);
|
||||
});
|
||||
it("/favicon.ico → 204", async () => {
|
||||
const r = await req("/favicon.ico");
|
||||
expect(r.code).toBe(204);
|
||||
});
|
||||
it("unknown route → 404", async () => {
|
||||
const r = await req("/api/nope");
|
||||
expect(r.code).toBe(404);
|
||||
});
|
||||
});
|
||||
|
||||
describe("E7.1c dispatch (flag on + token set)", () => {
|
||||
// E7.2: enforcement only on a public bind (0.0.0.0). The server binds 127.0.0.1
|
||||
// (safe for the test), so override __authState.bound to simulate a public bind.
|
||||
beforeEach(() => { __authState.enabled = true; __authState.token = "s3cret-token"; __authState.bound = "0.0.0.0"; });
|
||||
|
||||
it("GET /api/auth/status (always-open) passes WITHOUT a token", async () => {
|
||||
const r = await req("/api/auth/status");
|
||||
expect(r.code).toBe(200);
|
||||
});
|
||||
it("GET /api/status (gated — E7.3) → 401 WITHOUT a token in public mode", async () => {
|
||||
const r = await req("/api/status");
|
||||
expect(r.code).toBe(401); // E7.3: /api/status leaks dir paths → gated
|
||||
});
|
||||
it("GET /api/autosync (gated — E7.3) → 401 WITHOUT a token in public mode", async () => {
|
||||
const r = await req("/api/autosync");
|
||||
expect(r.code).toBe(401);
|
||||
});
|
||||
it("POST /api/autosync (mutation) WITHOUT a token → 401", async () => {
|
||||
const r = await req("/api/autosync", { method: "POST", headers: { "content-type": "application/json" }, body: JSON.stringify({ enabled: false }) });
|
||||
expect(r.code).toBe(401);
|
||||
});
|
||||
it("POST /api/autosync with a WRONG token → 401", async () => {
|
||||
const r = await req("/api/autosync", { method: "POST", headers: { "content-type": "application/json", authorization: "Bearer wrong" }, body: JSON.stringify({ enabled: false }) });
|
||||
expect(r.code).toBe(401);
|
||||
});
|
||||
it("POST /api/autosync with a malformed Authorization → 401 bad auth header", async () => {
|
||||
const r = await req("/api/autosync", { method: "POST", headers: { "content-type": "application/json", authorization: "Basic abc" }, body: JSON.stringify({ enabled: false }) });
|
||||
expect(r.code).toBe(401);
|
||||
expect((r.body as { error: string }).error).toMatch(/bad auth header/);
|
||||
});
|
||||
it("POST /api/autosync with a VALID Bearer + CSRF → reaches the handler (200, enabled:false)", async () => {
|
||||
const csrf = await getCSRF();
|
||||
const r = await req("/api/autosync", { method: "POST", headers: { "content-type": "application/json", authorization: "Bearer s3cret-token", origin: csrf.origin, "x-csrf-token": csrf.token, cookie: csrf.cookie }, body: JSON.stringify({ enabled: false }) });
|
||||
expect(r.code).toBe(200); // auth + CSRF passed → setEnabled(false) → 200
|
||||
});
|
||||
it("POST /api/autosync {enabled:true} in dev mode with a valid token + CSRF → 400 apply-gate (auth+CSRF passed, mode check next)", async () => {
|
||||
const csrf = await getCSRF();
|
||||
const r = await req("/api/autosync", { method: "POST", headers: { "content-type": "application/json", authorization: "Bearer s3cret-token", origin: csrf.origin, "x-csrf-token": csrf.token, cookie: csrf.cookie }, body: JSON.stringify({ enabled: true }) });
|
||||
expect(r.code).toBe(400); // auth + CSRF passed → apply-mode gate
|
||||
expect((r.body as { error: string }).error).toMatch(/requires --apply mode/);
|
||||
});
|
||||
});
|
||||
152
tests/e7-2-auth.test.ts
Normal file
152
tests/e7-2-auth.test.ts
Normal file
@@ -0,0 +1,152 @@
|
||||
// E7.2 — localhost-default bind + refuse-to-start + first-run auth prompt.
|
||||
//
|
||||
// Covers: /api/auth/status (authRequired flag), /api/auth/login (valid/invalid →
|
||||
// cookie / 401), /api/auth/logout (clears cookie), and the startServer refuse-to-
|
||||
// start guards (self-lockout + 0.0.0.0-without-token).
|
||||
|
||||
import { describe, it, expect, beforeAll, afterAll, beforeEach, afterEach } from "vitest";
|
||||
import { ClassicLevel } from "classic-level";
|
||||
import { mkdtemp, mkdir, rm } from "node:fs/promises";
|
||||
import { join } from "node:path";
|
||||
import { tmpdir } from "node:os";
|
||||
import type { Server } from "node:http";
|
||||
|
||||
import { startServer, __authState } from "../src/server.js";
|
||||
|
||||
let dir: string;
|
||||
let server: Server;
|
||||
let baseURL: string;
|
||||
const savedEnabled = __authState.enabled;
|
||||
const savedToken = __authState.token;
|
||||
const savedBound = __authState.bound;
|
||||
|
||||
beforeAll(async () => {
|
||||
dir = await mkdtemp(join(tmpdir(), "e7-2-"));
|
||||
await mkdir(join(dir, "refined"), { recursive: true });
|
||||
await mkdir(join(dir, "cc"), { recursive: true });
|
||||
const jdb = new ClassicLevel<string, string>(join(dir, "journal"));
|
||||
await jdb.open(); await jdb.close();
|
||||
const { server: srv } = await startServer({
|
||||
journal: join(dir, "journal"), refinedDir: join(dir, "refined"), ccDir: join(dir, "cc"),
|
||||
outDir: join(dir, "out"), mode: "dev", port: 0, host: "127.0.0.1",
|
||||
});
|
||||
server = srv;
|
||||
baseURL = `http://127.0.0.1:${(server.address() as { port: number }).port}`;
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
__authState.enabled = savedEnabled; __authState.token = savedToken; __authState.bound = savedBound;
|
||||
await new Promise<void>((r) => server.close(() => r()));
|
||||
await rm(dir, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
beforeEach(() => { __authState.enabled = false; __authState.token = ""; __authState.bound = "127.0.0.1"; });
|
||||
afterEach(() => { __authState.enabled = false; __authState.token = ""; __authState.bound = "127.0.0.1"; });
|
||||
|
||||
async function req(path: string, init: RequestInit = {}): Promise<{ code: number; body: unknown; headers: Headers }> {
|
||||
const r = await fetch(`${baseURL}${path}`, init);
|
||||
return { code: r.status, body: r.headers.get("content-type")?.includes("json") ? await r.json() : await r.text(), headers: r.headers };
|
||||
}
|
||||
|
||||
describe("E7.2 /api/auth/status", () => {
|
||||
it("flag off → authRequired:false, bound:127.0.0.1", async () => {
|
||||
const r = await req("/api/auth/status");
|
||||
expect(r.code).toBe(200);
|
||||
expect((r.body as { authRequired: boolean }).authRequired).toBe(false);
|
||||
expect((r.body as { bound: string }).bound).toBe("127.0.0.1");
|
||||
});
|
||||
it("flag on + 127.0.0.1 (localhost trusted) → authRequired:false", async () => {
|
||||
__authState.enabled = true; __authState.token = "s3cret"; __authState.bound = "127.0.0.1";
|
||||
const r = await req("/api/auth/status");
|
||||
expect((r.body as { authRequired: boolean }).authRequired).toBe(false);
|
||||
});
|
||||
it("flag on + 0.0.0.0 + token → authRequired:true", async () => {
|
||||
__authState.enabled = true; __authState.token = "s3cret"; __authState.bound = "0.0.0.0";
|
||||
const r = await req("/api/auth/status");
|
||||
expect((r.body as { authRequired: boolean }).authRequired).toBe(true);
|
||||
});
|
||||
it("never echoes the token or secret config (booleans only)", async () => {
|
||||
__authState.enabled = true; __authState.token = "s3cret"; __authState.bound = "0.0.0.0";
|
||||
const r = await req("/api/auth/status");
|
||||
const body = JSON.stringify(r.body);
|
||||
expect(body).not.toContain("s3cret");
|
||||
});
|
||||
});
|
||||
|
||||
describe("E7.2 /api/auth/login + logout", () => {
|
||||
beforeEach(() => { __authState.enabled = true; __authState.token = "s3cret"; __authState.bound = "0.0.0.0"; });
|
||||
|
||||
it("valid token → 200 + sets an HttpOnly SameSite=Strict auth_token cookie", async () => {
|
||||
const r = await req("/api/auth/login", { method: "POST", headers: { "content-type": "application/json" }, body: JSON.stringify({ token: "s3cret" }) });
|
||||
expect(r.code).toBe(200);
|
||||
const cookie = r.headers.get("set-cookie") ?? "";
|
||||
expect(cookie).toContain("auth_token=s3cret");
|
||||
expect(cookie).toContain("HttpOnly");
|
||||
expect(cookie).toContain("SameSite=Strict");
|
||||
});
|
||||
it("wrong token → 401 invalid credentials (no leak)", async () => {
|
||||
const r = await req("/api/auth/login", { method: "POST", headers: { "content-type": "application/json" }, body: JSON.stringify({ token: "wrong" }) });
|
||||
expect(r.code).toBe(401);
|
||||
expect((r.body as { error: string }).error).toBe("invalid credentials");
|
||||
expect(r.headers.get("set-cookie")).toBe(null); // no cookie set on failure
|
||||
});
|
||||
it("empty/whitespace token → 401 (treated as unset)", async () => {
|
||||
const r = await req("/api/auth/login", { method: "POST", headers: { "content-type": "application/json" }, body: JSON.stringify({ token: " " }) });
|
||||
expect(r.code).toBe(401);
|
||||
});
|
||||
it("no token configured (DASHBOARD_AUTH_TOKEN unset) → 401 (can't log in)", async () => {
|
||||
__authState.token = "";
|
||||
const r = await req("/api/auth/login", { method: "POST", headers: { "content-type": "application/json" }, body: JSON.stringify({ token: "anything" }) });
|
||||
expect(r.code).toBe(401);
|
||||
});
|
||||
it("logout → 200 + clears the cookie (Max-Age=0)", async () => {
|
||||
const r = await req("/api/auth/logout", { method: "POST" });
|
||||
expect(r.code).toBe(200);
|
||||
const cookie = r.headers.get("set-cookie") ?? "";
|
||||
expect(cookie).toContain("auth_token=");
|
||||
expect(cookie).toContain("Max-Age=0");
|
||||
});
|
||||
});
|
||||
|
||||
describe("E7.2 startServer refuse-to-start guards", () => {
|
||||
// These throw BEFORE opening the journal, so no LevelDB is needed.
|
||||
const dummyCfg = (host: string) => ({
|
||||
journal: join(dir, "no-such-journal"), refinedDir: join(dir, "refined"), ccDir: join(dir, "cc"),
|
||||
outDir: join(dir, "out"), mode: "dev" as const, port: 0, host,
|
||||
});
|
||||
|
||||
it("flag on + no token → refuses to start (self-lockout guard)", async () => {
|
||||
__authState.enabled = true; __authState.token = ""; __authState.bound = "127.0.0.1";
|
||||
await expect(startServer(dummyCfg("127.0.0.1"))).rejects.toThrow(/ENABLE_AUTH_MIDDLEWARE=on requires DASHBOARD_AUTH_TOKEN/);
|
||||
});
|
||||
it("flag on + 0.0.0.0 + no token → refuses to start (public-exposure gate)", async () => {
|
||||
__authState.enabled = true; __authState.token = ""; __authState.bound = "127.0.0.1";
|
||||
// The self-lockout guard fires first (no token at all), so this throws the
|
||||
// self-lockout message — both guards refuse; assert it refuses (either message).
|
||||
await expect(startServer(dummyCfg("0.0.0.0"))).rejects.toThrow();
|
||||
});
|
||||
it("flag on + token set → does NOT throw on the guards (would proceed to open the journal)", async () => {
|
||||
__authState.enabled = true; __authState.token = "s3cret"; __authState.bound = "127.0.0.1";
|
||||
// With a token, the guards pass; startServer then tries to open the journal
|
||||
// (dummyCfg points at a non-existent path) → throws a DIFFERENT error (LevelDB
|
||||
// open), NOT the guard message. Assert it does NOT throw the guard message.
|
||||
try {
|
||||
await startServer(dummyCfg("0.0.0.0"));
|
||||
expect.unreachable("should have thrown on the missing journal");
|
||||
} catch (e) {
|
||||
expect((e as Error).message).not.toMatch(/DASHBOARD_AUTH_TOKEN/);
|
||||
expect((e as Error).message).not.toMatch(/refusing to bind/);
|
||||
}
|
||||
});
|
||||
it("flag off → no guard (back-compat: 0.0.0.0 + no token would proceed)", async () => {
|
||||
__authState.enabled = false; __authState.token = ""; __authState.bound = "127.0.0.1";
|
||||
try {
|
||||
await startServer(dummyCfg("0.0.0.0"));
|
||||
expect.unreachable("should have thrown on the missing journal");
|
||||
} catch (e) {
|
||||
// No guard message — it got past the guards to the journal open.
|
||||
expect((e as Error).message).not.toMatch(/DASHBOARD_AUTH_TOKEN/);
|
||||
expect((e as Error).message).not.toMatch(/refusing to bind/);
|
||||
}
|
||||
});
|
||||
});
|
||||
124
tests/e7-3-nosecret.test.ts
Normal file
124
tests/e7-3-nosecret.test.ts
Normal file
@@ -0,0 +1,124 @@
|
||||
// E7.3b — no-secret-egress regression guard.
|
||||
//
|
||||
// Starts a server with distinctive secrets (a relay API key + a dashboard token)
|
||||
// and asserts NO response body (incl. error paths) contains either secret
|
||||
// substring. /api/auth/status returns booleans only; error strings name env vars,
|
||||
// never values; the relay client sends the key as a header (never serialized).
|
||||
|
||||
import { describe, it, expect, beforeAll, afterAll } from "vitest";
|
||||
import { ClassicLevel } from "classic-level";
|
||||
import { mkdtemp, mkdir, rm } from "node:fs/promises";
|
||||
import { join } from "node:path";
|
||||
import { tmpdir } from "node:os";
|
||||
import type { Server } from "node:http";
|
||||
|
||||
import { startServer, __authState } from "../src/server.js";
|
||||
|
||||
const RELAY_SECRET = "RELAY_SECRET_abc123";
|
||||
const DASH_SECRET = "DASH_SECRET_xyz789";
|
||||
|
||||
let dir: string;
|
||||
let server: Server;
|
||||
let baseURL: string;
|
||||
const savedEnabled = __authState.enabled;
|
||||
const savedToken = __authState.token;
|
||||
const savedBound = __authState.bound;
|
||||
|
||||
beforeAll(async () => {
|
||||
dir = await mkdtemp(join(tmpdir(), "e7-3-"));
|
||||
await mkdir(join(dir, "refined"), { recursive: true });
|
||||
await mkdir(join(dir, "cc"), { recursive: true });
|
||||
const jdb = new ClassicLevel<string, string>(join(dir, "journal"));
|
||||
await jdb.open(); await jdb.close();
|
||||
// Distinctive secrets: a relay API key + a dashboard token. The relay URL points
|
||||
// at a port nothing listens on so /api/refresh errors out (a relay-failure path).
|
||||
__authState.enabled = true; __authState.token = DASH_SECRET;
|
||||
const { server: srv } = await startServer({
|
||||
journal: join(dir, "journal"), refinedDir: join(dir, "refined"), ccDir: join(dir, "cc"),
|
||||
outDir: join(dir, "out"), mode: "dev", port: 0, host: "127.0.0.1",
|
||||
relayCfg: { url: "http://127.0.0.1:1", apiKey: RELAY_SECRET, clientId: "c" },
|
||||
});
|
||||
// startServer sets __authState.bound = cfg.host (127.0.0.1); override AFTER to
|
||||
// simulate a public bind so enforcement runs (the actual socket stays 127.0.0.1).
|
||||
__authState.bound = "0.0.0.0";
|
||||
server = srv;
|
||||
baseURL = `http://127.0.0.1:${(server.address() as { port: number }).port}`;
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
__authState.enabled = savedEnabled; __authState.token = savedToken; __authState.bound = savedBound;
|
||||
await new Promise<void>((r) => server.close(() => r()));
|
||||
await rm(dir, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
function authHeaders(extra: Record<string, string> = {}): Record<string, string> {
|
||||
return { authorization: `Bearer ${DASH_SECRET}`, ...extra };
|
||||
}
|
||||
|
||||
// E7.4: CSRF token + cookie (Node fetch has no cookie jar; replay Set-Cookie).
|
||||
async function getCSRF(): Promise<{ token: string; cookie: string; origin: string }> {
|
||||
const r = await fetch(`${baseURL}/api/auth/csrf`);
|
||||
const token = (await r.json() as { csrfToken: string }).csrfToken;
|
||||
const setCookie = r.headers.get("set-cookie") ?? "";
|
||||
const m = setCookie.match(/csrf_token=([^;]+)/);
|
||||
return { token, cookie: m ? `csrf_token=${m[1]}` : "", origin: baseURL };
|
||||
}
|
||||
|
||||
function assertNoSecret(label: string, body: unknown) {
|
||||
const text = typeof body === "string" ? body : JSON.stringify(body);
|
||||
expect(text, `${label} leaked RELAY_SECRET`).not.toContain(RELAY_SECRET);
|
||||
expect(text, `${label} leaked DASH_SECRET`).not.toContain(DASH_SECRET);
|
||||
}
|
||||
|
||||
async function req(path: string, init: RequestInit = {}): Promise<{ code: number; body: unknown }> {
|
||||
const r = await fetch(`${baseURL}${path}`, init);
|
||||
const t = await r.text();
|
||||
let body: unknown = t;
|
||||
try { body = t ? JSON.parse(t) : null; } catch { /* non-JSON (e.g. the dashboard HTML) */ }
|
||||
return { code: r.status, body };
|
||||
}
|
||||
|
||||
describe("E7.3 no-secret-egress regression guard", () => {
|
||||
it("/api/auth/status (open, no token) → booleans only, no secret", async () => {
|
||||
const r = await req("/api/auth/status");
|
||||
expect(r.code).toBe(200);
|
||||
assertNoSecret("auth/status", r.body);
|
||||
expect(JSON.stringify(r.body)).toMatch(/relayConfigured/); // booleans, no values
|
||||
});
|
||||
|
||||
it("/ (login page, no token) → HTML, no secret", async () => {
|
||||
const r = await req("/");
|
||||
expect(r.code).toBe(200);
|
||||
assertNoSecret("login page", r.body);
|
||||
});
|
||||
|
||||
it("/api/status (gated) with a valid token → dir paths, no secret", async () => {
|
||||
const r = await req("/api/status", { headers: authHeaders() });
|
||||
expect(r.code).toBe(200);
|
||||
assertNoSecret("status", r.body);
|
||||
});
|
||||
|
||||
it("/api/status (gated) WITHOUT a token → 401, no secret", async () => {
|
||||
const r = await req("/api/status");
|
||||
expect(r.code).toBe(401);
|
||||
assertNoSecret("status-401", r.body);
|
||||
});
|
||||
|
||||
it("/api/refresh (relay-failure path) → 500 with a relay error, no secret", async () => {
|
||||
const csrf = await getCSRF();
|
||||
const r = await req("/api/refresh", { method: "POST", headers: authHeaders({ "content-type": "application/json", origin: csrf.origin, "x-csrf-token": csrf.token, cookie: csrf.cookie }), body: "{}" });
|
||||
expect(r.code).toBe(500); // the relay fetch fails (nothing on :1)
|
||||
assertNoSecret("refresh-error", r.body); // the error must NOT echo the API key
|
||||
});
|
||||
|
||||
it("/api/auth/login with a WRONG token → 401 invalid credentials, no secret", async () => {
|
||||
const r = await req("/api/auth/login", { method: "POST", headers: { "content-type": "application/json" }, body: JSON.stringify({ token: "wrong" }) });
|
||||
expect(r.code).toBe(401);
|
||||
assertNoSecret("login-401", r.body);
|
||||
});
|
||||
|
||||
it("favicon → 204 (no body)", async () => {
|
||||
const r = await req("/favicon.ico");
|
||||
expect(r.code).toBe(204);
|
||||
});
|
||||
});
|
||||
146
tests/e7-4-csrf.test.ts
Normal file
146
tests/e7-4-csrf.test.ts
Normal file
@@ -0,0 +1,146 @@
|
||||
// E7.4 — CSRF / same-origin guard for POST mutation routes.
|
||||
//
|
||||
// flag off → no-op. flag on + public bind → same-origin (Origin/Referer host ===
|
||||
// Host) + X-CSRF-Token === csrf_token cookie (constant-time). Auth (401) runs
|
||||
// before CSRF (403).
|
||||
|
||||
import { describe, it, expect, beforeAll, afterAll, beforeEach, afterEach } from "vitest";
|
||||
import { ClassicLevel } from "classic-level";
|
||||
import { mkdtemp, mkdir, rm } from "node:fs/promises";
|
||||
import { join } from "node:path";
|
||||
import { tmpdir } from "node:os";
|
||||
import type { Server } from "node:http";
|
||||
|
||||
import { startServer, __authState } from "../src/server.js";
|
||||
|
||||
let dir: string;
|
||||
let server: Server;
|
||||
let baseURL: string;
|
||||
const savedEnabled = __authState.enabled;
|
||||
const savedToken = __authState.token;
|
||||
const savedBound = __authState.bound;
|
||||
|
||||
beforeAll(async () => {
|
||||
dir = await mkdtemp(join(tmpdir(), "e7-4-"));
|
||||
await mkdir(join(dir, "refined"), { recursive: true });
|
||||
await mkdir(join(dir, "cc"), { recursive: true });
|
||||
const jdb = new ClassicLevel<string, string>(join(dir, "journal"));
|
||||
await jdb.open(); await jdb.close();
|
||||
__authState.enabled = true; __authState.token = "s3cret";
|
||||
const { server: srv } = await startServer({
|
||||
journal: join(dir, "journal"), refinedDir: join(dir, "refined"), ccDir: join(dir, "cc"),
|
||||
outDir: join(dir, "out"), mode: "dev", port: 0, host: "127.0.0.1",
|
||||
});
|
||||
__authState.bound = "0.0.0.0"; // simulate a public bind so enforcement runs
|
||||
server = srv;
|
||||
baseURL = `http://127.0.0.1:${(server.address() as { port: number }).port}`;
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
__authState.enabled = savedEnabled; __authState.token = savedToken; __authState.bound = savedBound;
|
||||
await new Promise<void>((r) => server.close(() => r()));
|
||||
await rm(dir, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
async function getCSRF(): Promise<{ token: string; cookie: string }> {
|
||||
const r = await fetch(`${baseURL}/api/auth/csrf`);
|
||||
const token = (await r.json() as { csrfToken: string }).csrfToken;
|
||||
const setCookie = r.headers.get("set-cookie") ?? "";
|
||||
const m = setCookie.match(/csrf_token=([^;]+)/);
|
||||
return { token, cookie: m ? `csrf_token=${m[1]}` : "" };
|
||||
}
|
||||
|
||||
async function post(extra: Record<string, string>): Promise<{ code: number; body: unknown }> {
|
||||
const r = await fetch(`${baseURL}/api/autosync`, {
|
||||
method: "POST",
|
||||
headers: { "content-type": "application/json", authorization: "Bearer s3cret", origin: baseURL, ...extra },
|
||||
body: JSON.stringify({ enabled: false }),
|
||||
});
|
||||
const t = await r.text();
|
||||
return { code: r.status, body: t ? JSON.parse(t) : null };
|
||||
}
|
||||
|
||||
describe("E7.4 CSRF / same-origin", () => {
|
||||
it("/api/auth/csrf issues an HttpOnly SameSite=Strict cookie + returns the token", async () => {
|
||||
const r = await fetch(`${baseURL}/api/auth/csrf`);
|
||||
const body = await r.json() as { csrfToken: string };
|
||||
const setCookie = r.headers.get("set-cookie") ?? "";
|
||||
expect(body.csrfToken).toMatch(/^[0-9a-f]{32}$/);
|
||||
expect(setCookie).toContain("csrf_token=");
|
||||
expect(setCookie).toContain("HttpOnly");
|
||||
expect(setCookie).toContain("SameSite=Strict");
|
||||
});
|
||||
|
||||
it("flag on + same-origin + valid X-CSRF-Token + cookie → proceeds (200)", async () => {
|
||||
const csrf = await getCSRF();
|
||||
const r = await post({ "x-csrf-token": csrf.token, cookie: csrf.cookie });
|
||||
expect(r.code).toBe(200); // auth + CSRF passed → setEnabled(false) → 200
|
||||
});
|
||||
|
||||
it("cross-origin Origin → 403 cross-origin forbidden (even with a valid token)", async () => {
|
||||
const csrf = await getCSRF();
|
||||
const r = await post({ "x-csrf-token": csrf.token, cookie: csrf.cookie, origin: "http://evil.example" });
|
||||
expect(r.code).toBe(403);
|
||||
expect((r.body as { error: string }).error).toBe("cross-origin forbidden");
|
||||
});
|
||||
|
||||
it("no Origin and no Referer → 403 origin required", async () => {
|
||||
const csrf = await getCSRF();
|
||||
// post() sets origin: baseURL by default; override to omit it.
|
||||
const r = await fetch(`${baseURL}/api/autosync`, {
|
||||
method: "POST",
|
||||
headers: { "content-type": "application/json", authorization: "Bearer s3cret", "x-csrf-token": csrf.token, cookie: csrf.cookie },
|
||||
body: JSON.stringify({ enabled: false }),
|
||||
});
|
||||
expect(r.status).toBe(403);
|
||||
expect((await r.json() as { error: string }).error).toBe("origin required");
|
||||
});
|
||||
|
||||
it("missing X-CSRF-Token → 403 missing or invalid csrf token", async () => {
|
||||
const csrf = await getCSRF();
|
||||
const r = await post({ cookie: csrf.cookie }); // no x-csrf-token
|
||||
expect(r.code).toBe(403);
|
||||
expect((r.body as { error: string }).error).toBe("missing or invalid csrf token");
|
||||
});
|
||||
|
||||
it("invalid X-CSRF-Token (mismatch) → 403", async () => {
|
||||
const csrf = await getCSRF();
|
||||
const r = await post({ "x-csrf-token": "wrong" + csrf.token, cookie: csrf.cookie });
|
||||
expect(r.code).toBe(403);
|
||||
expect((r.body as { error: string }).error).toBe("missing or invalid csrf token");
|
||||
});
|
||||
|
||||
it("missing csrf cookie (but valid-looking header) → 403", async () => {
|
||||
const csrf = await getCSRF();
|
||||
const r = await post({ "x-csrf-token": csrf.token }); // no cookie
|
||||
expect(r.code).toBe(403);
|
||||
});
|
||||
|
||||
it("auth runs BEFORE CSRF — no Bearer → 401 (not 403)", async () => {
|
||||
const csrf = await getCSRF();
|
||||
const r = await fetch(`${baseURL}/api/autosync`, {
|
||||
method: "POST",
|
||||
headers: { "content-type": "application/json", origin: baseURL, "x-csrf-token": csrf.token, cookie: csrf.cookie },
|
||||
body: JSON.stringify({ enabled: false }),
|
||||
});
|
||||
expect(r.status).toBe(401); // auth fails first (no Authorization)
|
||||
});
|
||||
|
||||
it("GET route (requireCSRF:false) is CSRF-exempt — /api/auth/status passes without CSRF", async () => {
|
||||
const r = await fetch(`${baseURL}/api/auth/status`);
|
||||
expect(r.status).toBe(200);
|
||||
});
|
||||
|
||||
it("flag off → CSRF no-op (POST passes without Origin/X-CSRF-Token)", async () => {
|
||||
const wasEnabled = __authState.enabled;
|
||||
__authState.enabled = false;
|
||||
try {
|
||||
const r = await fetch(`${baseURL}/api/autosync`, {
|
||||
method: "POST",
|
||||
headers: { "content-type": "application/json" }, // no auth, no CSRF
|
||||
body: JSON.stringify({ enabled: false }),
|
||||
});
|
||||
expect(r.status).toBe(200); // flag off → no auth, no CSRF → setEnabled(false) → 200
|
||||
} finally { __authState.enabled = wasEnabled; }
|
||||
});
|
||||
});
|
||||
43
tests/schema-version.test.ts
Normal file
43
tests/schema-version.test.ts
Normal file
@@ -0,0 +1,43 @@
|
||||
import { describe, it, expect } from "vitest";
|
||||
import {
|
||||
FLAGS_SCHEMA_VERSION,
|
||||
SYNC_STATE_SCHEMA_VERSION,
|
||||
parseSchemaVersion,
|
||||
} from "../src/schema-version.js";
|
||||
import type { FlagsSchemaVersion, SyncStateSchemaVersion } from "../src/schema-version.js";
|
||||
|
||||
describe("schema-version constants (E0.3)", () => {
|
||||
it("are unequal strings with distinct prefixes", () => {
|
||||
expect(FLAGS_SCHEMA_VERSION).not.toBe(SYNC_STATE_SCHEMA_VERSION);
|
||||
expect(FLAGS_SCHEMA_VERSION.startsWith("flags-")).toBe(true);
|
||||
expect(SYNC_STATE_SCHEMA_VERSION.startsWith("sync-")).toBe(true);
|
||||
expect(typeof FLAGS_SCHEMA_VERSION).toBe("string");
|
||||
expect(typeof SYNC_STATE_SCHEMA_VERSION).toBe("string");
|
||||
});
|
||||
|
||||
it("parseSchemaVersion branches on prefix and returns null for unknown", () => {
|
||||
expect(parseSchemaVersion(FLAGS_SCHEMA_VERSION)).toEqual({ kind: "flags", version: FLAGS_SCHEMA_VERSION });
|
||||
expect(parseSchemaVersion(SYNC_STATE_SCHEMA_VERSION)).toEqual({ kind: "sync-state", version: SYNC_STATE_SCHEMA_VERSION });
|
||||
expect(parseSchemaVersion("flags-campaign-codex/v2")).toEqual({ kind: "flags", version: "flags-campaign-codex/v2" });
|
||||
expect(parseSchemaVersion("sync-state/v2")).toEqual({ kind: "sync-state", version: "sync-state/v2" });
|
||||
expect(parseSchemaVersion("unknown/v1")).toBeNull();
|
||||
expect(parseSchemaVersion("")).toBeNull();
|
||||
});
|
||||
|
||||
it("branded types are not assignable to each other (compile-time guard)", () => {
|
||||
// Nominal brands via distinct property names (__flagsBrand / __syncBrand).
|
||||
// If a brand ever stops preventing cross-assignment, one of these conditional
|
||||
// types flips to `true` and the matching `= false as const` line errors at
|
||||
// compile time (true not assignable to false) — the guard is enforced by
|
||||
// tsc, not just asserted in prose.
|
||||
type FlagsExtendsSync = [FlagsSchemaVersion] extends [SyncStateSchemaVersion] ? true : false;
|
||||
type SyncExtendsFlags = [SyncStateSchemaVersion] extends [FlagsSchemaVersion] ? true : false;
|
||||
const flagsExtendsSync: FlagsExtendsSync = false as const;
|
||||
const syncExtendsFlags: SyncExtendsFlags = false as const;
|
||||
expect(flagsExtendsSync).toBe(false);
|
||||
expect(syncExtendsFlags).toBe(false);
|
||||
|
||||
// Runtime sanity: the brands are still distinct string values at runtime.
|
||||
expect(FLAGS_SCHEMA_VERSION).not.toBe(SYNC_STATE_SCHEMA_VERSION);
|
||||
});
|
||||
});
|
||||
171
tests/server-lock.test.ts
Normal file
171
tests/server-lock.test.ts
Normal file
@@ -0,0 +1,171 @@
|
||||
import { describe, it, expect, vi, beforeEach, afterEach } from "vitest";
|
||||
import { mkdtemp, writeFile, mkdir, rm, readFile } from "node:fs/promises";
|
||||
import { join } from "node:path";
|
||||
import { tmpdir } from "node:os";
|
||||
|
||||
// Mock pushNote so we can assert whether a PUT would have fired, without a
|
||||
// relay. The controller's `runPushBody` calls `relayClient(state)` then
|
||||
// `pushNote({...})`; with pushNote mocked, the relay is never contacted.
|
||||
vi.mock("../src/push.js", () => ({
|
||||
pushNote: vi.fn(async () => ({ dryRun: false, ccUuid: "JournalEntry.abc1", diff: {}, imageNote: "" })),
|
||||
}));
|
||||
|
||||
import { pushNote } from "../src/push.js";
|
||||
import { AutoSyncController } from "../src/server.js";
|
||||
import type { State, ServerConfig } from "../src/server.js";
|
||||
import { splitFrontmatter, readFoundryBlock } from "../src/frontmatter.js";
|
||||
import { contentHash } from "../src/normalize.js";
|
||||
|
||||
const UUID = "JournalEntry.abc1";
|
||||
const REL = "Roland.md";
|
||||
|
||||
interface Deferred<T = void> {
|
||||
promise: Promise<T>;
|
||||
resolve: (v: T) => void;
|
||||
}
|
||||
function deferred<T = void>(): Deferred<T> {
|
||||
let resolve!: (v: T) => void;
|
||||
const promise = new Promise<T>((res) => { resolve = res; });
|
||||
return { promise, resolve };
|
||||
}
|
||||
|
||||
function seededNote(body: string, contentHashBaseline: string): string {
|
||||
return [
|
||||
"---",
|
||||
"type: npc",
|
||||
"foundry:",
|
||||
` cc_uuid: ${UUID}`,
|
||||
" cc_type: npc",
|
||||
" folder_path: NPCs",
|
||||
` contentHash: ${contentHashBaseline}`,
|
||||
" syncedAt: 2026-06-22T00:00:00.000Z",
|
||||
"---",
|
||||
body,
|
||||
"",
|
||||
].join("\n");
|
||||
}
|
||||
|
||||
let dir: string;
|
||||
let state: State;
|
||||
|
||||
beforeEach(async () => {
|
||||
vi.mocked(pushNote).mockClear();
|
||||
vi.mocked(pushNote).mockImplementation(async () => ({
|
||||
dryRun: false, ccUuid: UUID, diff: {}, imageNote: "",
|
||||
}));
|
||||
dir = await mkdtemp(join(tmpdir(), "autosync-lock-"));
|
||||
const refinedDir = join(dir, "refined");
|
||||
const outDir = join(dir, "out");
|
||||
await mkdir(refinedDir, { recursive: true });
|
||||
const cfg: ServerConfig = {
|
||||
journal: "",
|
||||
refinedDir,
|
||||
ccDir: "",
|
||||
outDir,
|
||||
mode: "apply",
|
||||
port: 0,
|
||||
host: "",
|
||||
relayCfg: { url: "http://relay.test", apiKey: "k", clientId: "c" },
|
||||
};
|
||||
state = { db: {} as State["db"], cfg, index: null, autosync: null as unknown as State["autosync"] } as unknown as State;
|
||||
});
|
||||
|
||||
afterEach(async () => {
|
||||
await rm(dir, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
async function writeNote(body = "The gunslinger drew his revolver.\n"): Promise<void> {
|
||||
// contentHash baseline deliberately WRONG (64 zeros) so bodyHash !== baseline
|
||||
// and the "unchanged" skip does NOT fire — the push path is exercised.
|
||||
await writeFile(join(state.cfg.refinedDir, REL), seededNote(body, "0".repeat(64)), "utf8");
|
||||
}
|
||||
|
||||
/** Wait until the controller holds the lock for UUID (used to deterministically
|
||||
* pin a concurrent op's start to the window where the lock is held). */
|
||||
async function waitUntilHeld(controller: AutoSyncController, uuid: string): Promise<void> {
|
||||
while (!controller.lock.isHeld(uuid)) await new Promise<void>((r) => setImmediate(r));
|
||||
}
|
||||
|
||||
describe("AutoSyncController lock integration (E0.1)", () => {
|
||||
it("a normal save acquires the push lock, calls pushNote once, baselines, and releases", async () => {
|
||||
await writeNote();
|
||||
const controller = new AutoSyncController(state);
|
||||
await (controller as any).process(REL);
|
||||
|
||||
expect(vi.mocked(pushNote)).toHaveBeenCalledTimes(1);
|
||||
expect(controller.lock.isHeld(UUID)).toBe(false); // released after push
|
||||
|
||||
// Baseline wrote foundry.contentHash = current body hash (idempotency).
|
||||
const md = await readFile(join(state.cfg.refinedDir, REL), "utf8");
|
||||
const { fm, body } = splitFrontmatter(md);
|
||||
const fb = readFoundryBlock(fm);
|
||||
expect(fb?.contentHash).toBe(contentHash(body));
|
||||
expect(controller.events.some((e) => e.status === "pushed")).toBe(true);
|
||||
});
|
||||
|
||||
it("cross-direction: a pre-held F→O ('pull') lock blocks the O→F push — no PUT (fail-safe)", async () => {
|
||||
await writeNote();
|
||||
const controller = new AutoSyncController(state);
|
||||
// Simulate an in-flight F→O pull holding the uuid (E2's future path).
|
||||
expect(controller.lock.acquire(UUID, "pull")).toEqual({ acquired: true });
|
||||
|
||||
await (controller as any).process(REL);
|
||||
|
||||
expect(vi.mocked(pushNote)).not.toHaveBeenCalled();
|
||||
expect(controller.lock.isHeld(UUID)).toBe(true); // still held by pull
|
||||
expect(controller.events.some((e) => e.message.includes("lock busy"))).toBe(true);
|
||||
controller.lock.release(UUID, "pull");
|
||||
});
|
||||
|
||||
it("two concurrent process() calls for the same uuid push exactly once (lock dedup)", async () => {
|
||||
await writeNote();
|
||||
const controller = new AutoSyncController(state);
|
||||
|
||||
// Gate the FIRST pushNote call so p1 holds the lock long enough for p2 to
|
||||
// arrive and hit the lock-busy skip — deterministic dedup.
|
||||
const gate = deferred<void>();
|
||||
let calls = 0;
|
||||
vi.mocked(pushNote).mockImplementation(async () => {
|
||||
calls++;
|
||||
if (calls === 1) await gate.promise;
|
||||
return { dryRun: false, ccUuid: UUID, diff: {}, imageNote: "" };
|
||||
});
|
||||
|
||||
const p1 = (controller as any).process(REL) as Promise<void>;
|
||||
await waitUntilHeld(controller, UUID); // p1 acquired, now blocked in pushNote
|
||||
const p2 = (controller as any).process(REL) as Promise<void>;
|
||||
await p2; // p2 reads, withLock(skip) → held → skip (no push)
|
||||
// p1 has invoked pushNote once (it is parked on the gate); p2 did NOT
|
||||
// invoke it — so exactly one call so far, no duplicate.
|
||||
expect(vi.mocked(pushNote)).toHaveBeenCalledTimes(1);
|
||||
|
||||
gate.resolve(void 0);
|
||||
await p1; // p1 completes push + baseline
|
||||
expect(vi.mocked(pushNote)).toHaveBeenCalledTimes(1); // still exactly one PUT
|
||||
});
|
||||
|
||||
it("flag-off (SYNC_LOCK_ENABLED=false) uses the legacy per-relPath inflight guard, byte-identical behavior", async () => {
|
||||
await writeNote();
|
||||
const controller = new AutoSyncController(state);
|
||||
(controller as any).syncLockEnabled = false; // simulate the flag-off path
|
||||
const inflight = (controller as any).inflight as Set<string>;
|
||||
|
||||
await (controller as any).process(REL);
|
||||
|
||||
expect(vi.mocked(pushNote)).toHaveBeenCalledTimes(1);
|
||||
expect(inflight.has(REL)).toBe(false); // cleaned up in finally
|
||||
expect(controller.lock.isHeld(UUID)).toBe(false); // lock not consulted at all
|
||||
});
|
||||
|
||||
it("an unlinked note (no foundry.cc_uuid) is skipped before any push, and the relPath fallback key is cached", async () => {
|
||||
// No foundry block → unlinked. process should skip ("not linked") and NOT push.
|
||||
await writeFile(join(state.cfg.refinedDir, REL), "---\ntype: npc\n---\nBody with no foundry block.\n", "utf8");
|
||||
const controller = new AutoSyncController(state);
|
||||
await (controller as any).process(REL);
|
||||
|
||||
expect(vi.mocked(pushNote)).not.toHaveBeenCalled();
|
||||
expect(controller.events.some((e) => e.message.includes("not linked"))).toBe(true);
|
||||
// The relPath fallback key was cached (so the debounce pre-check can use it).
|
||||
expect((controller as any).uuidCache.get(REL)).toBe(`relPath:${REL}`);
|
||||
});
|
||||
});
|
||||
254
tests/synclock.test.ts
Normal file
254
tests/synclock.test.ts
Normal file
@@ -0,0 +1,254 @@
|
||||
import { describe, it, expect } from "vitest";
|
||||
import { SyncLock, relPathLockKey, LockAcquireTimeout } from "../src/synclock.js";
|
||||
|
||||
// Helper: a deferred promise the test controls, so a held `withLock` body can
|
||||
// be kept alive until an assertion outside it has run.
|
||||
interface Deferred<T = void> {
|
||||
promise: Promise<T>;
|
||||
resolve: (v: T) => void;
|
||||
reject: (e: unknown) => void;
|
||||
}
|
||||
function deferred<T = void>(): Deferred<T> {
|
||||
let resolve!: (v: T) => void;
|
||||
let reject!: (e: unknown) => void;
|
||||
const promise = new Promise<T>((res, rej) => { resolve = res; reject = rej; });
|
||||
return { promise, resolve, reject };
|
||||
}
|
||||
|
||||
describe("SyncLock acquire/release/isHeld/heldOps (E0.1)", () => {
|
||||
it("acquires when free and reports held", () => {
|
||||
const lock = new SyncLock();
|
||||
expect(lock.acquire("u1", "push")).toEqual({ acquired: true });
|
||||
expect(lock.isHeld("u1")).toBe(true);
|
||||
expect(lock.heldOps()).toEqual({ u1: "push" });
|
||||
lock.release("u1", "push");
|
||||
expect(lock.isHeld("u1")).toBe(false);
|
||||
expect(lock.heldOps()).toEqual({});
|
||||
});
|
||||
|
||||
it("acquire on a held uuid returns false with the held op", () => {
|
||||
const lock = new SyncLock();
|
||||
lock.acquire("u1", "push");
|
||||
expect(lock.acquire("u1", "pull")).toEqual({ acquired: false, heldOp: "push" });
|
||||
});
|
||||
|
||||
it("release of an un-held uuid is a no-op (does not throw)", () => {
|
||||
const lock = new SyncLock();
|
||||
expect(() => lock.release("never", "push")).not.toThrow();
|
||||
// wrong op on a held uuid is also a no-op
|
||||
lock.acquire("u1", "push");
|
||||
expect(() => lock.release("u1", "pull")).not.toThrow();
|
||||
expect(lock.isHeld("u1")).toBe(true);
|
||||
});
|
||||
});
|
||||
|
||||
describe("SyncLock cross-direction exclusion (E0.1, FR-3.1/FR-5.5)", () => {
|
||||
it("an O→F withLock running blocks an F→O acquire on the same uuid", async () => {
|
||||
const lock = new SyncLock();
|
||||
const gate = deferred<void>();
|
||||
let pushStarted = false;
|
||||
const pushDone = lock.withLock("u1", "push", async () => {
|
||||
pushStarted = true;
|
||||
await gate.promise;
|
||||
});
|
||||
// Wait until the push body is actually running.
|
||||
await new Promise<void>((r) => setImmediate(r));
|
||||
expect(pushStarted).toBe(true);
|
||||
|
||||
// While the push holds the uuid, an F→O acquire must be refused.
|
||||
expect(lock.acquire("u1", "pull")).toEqual({ acquired: false, heldOp: "push" });
|
||||
|
||||
gate.resolve(void 0);
|
||||
await pushDone;
|
||||
// After release, the pull can acquire.
|
||||
expect(lock.acquire("u1", "pull")).toEqual({ acquired: true });
|
||||
});
|
||||
});
|
||||
|
||||
describe("SyncLock burst serializes same-uuid ops (E0.1, NFR-3)", () => {
|
||||
it("N concurrent queue-policy withLock(uuid,'push') calls execute strictly one at a time", async () => {
|
||||
const lock = new SyncLock();
|
||||
const N = 10;
|
||||
let inFlight = 0;
|
||||
let maxInFlight = 0;
|
||||
const completed: number[] = [];
|
||||
|
||||
const tasks = Array.from({ length: N }, (_, i) =>
|
||||
lock.withLock("u1", "push", async () => {
|
||||
inFlight++;
|
||||
maxInFlight = Math.max(maxInFlight, inFlight);
|
||||
// Yield a few microtasks so other queued tasks would visibly overlap if allowed.
|
||||
await new Promise<void>((r) => setImmediate(r));
|
||||
await new Promise<void>((r) => setImmediate(r));
|
||||
completed.push(i);
|
||||
inFlight--;
|
||||
return i;
|
||||
}, { policy: "queue", maxWaitMs: 1000 }),
|
||||
);
|
||||
|
||||
const results = await Promise.all(tasks);
|
||||
expect(maxInFlight).toBe(1); // strictly one at a time
|
||||
expect(results).toHaveLength(N);
|
||||
expect(results.every((r) => r !== undefined)).toBe(true); // none skipped/timed out
|
||||
expect(completed).toHaveLength(N);
|
||||
});
|
||||
|
||||
it("skip-policy drops redundant same-uuid ops instead of queuing", async () => {
|
||||
const lock = new SyncLock();
|
||||
const gate = deferred<void>();
|
||||
let ran = 0;
|
||||
const first = lock.withLock("u1", "push", async () => { ran++; await gate.promise; }, { policy: "skip" });
|
||||
await new Promise<void>((r) => setImmediate(r));
|
||||
// Second skip-policy op while the first holds: dropped (undefined), does not run.
|
||||
const second = await lock.withLock("u1", "push", async () => { ran++; }, { policy: "skip" });
|
||||
expect(second).toBeUndefined();
|
||||
gate.resolve(void 0);
|
||||
await first;
|
||||
expect(ran).toBe(1);
|
||||
});
|
||||
});
|
||||
|
||||
describe("SyncLock release-on-throw (E0.1)", () => {
|
||||
it("a rejecting fn releases the lock so the next acquire succeeds", async () => {
|
||||
const lock = new SyncLock();
|
||||
await expect(lock.withLock("u1", "push", async () => { throw new Error("boom"); })).rejects.toThrow("boom");
|
||||
expect(lock.isHeld("u1")).toBe(false);
|
||||
expect(lock.acquire("u1", "push")).toEqual({ acquired: true });
|
||||
});
|
||||
|
||||
it("a queued waiter still acquires after the holder rejects (slot is released)", async () => {
|
||||
const lock = new SyncLock();
|
||||
let waiterRan = false;
|
||||
const holder = lock.withLock("u1", "push", async () => { throw new Error("holder fails"); });
|
||||
const waiter = lock.withLock("u1", "pull", async () => { waiterRan = true; }, { policy: "queue", maxWaitMs: 1000 });
|
||||
await expect(holder).rejects.toThrow("holder fails");
|
||||
const result = await waiter;
|
||||
expect(result).toBeUndefined(); // fn returned void → resolved to undefined
|
||||
expect(waiterRan).toBe(true);
|
||||
});
|
||||
});
|
||||
|
||||
describe("SyncLock different-uuid concurrency (E0.1 — no global lock)", () => {
|
||||
it("two different uuids proceed concurrently", async () => {
|
||||
const lock = new SyncLock();
|
||||
let maxInFlight = 0;
|
||||
let inFlight = 0;
|
||||
const both = Promise.all([
|
||||
lock.withLock("u1", "push", async () => {
|
||||
inFlight++; maxInFlight = Math.max(maxInFlight, inFlight);
|
||||
await new Promise<void>((r) => setImmediate(r));
|
||||
inFlight--;
|
||||
}),
|
||||
lock.withLock("u2", "push", async () => {
|
||||
inFlight++; maxInFlight = Math.max(maxInFlight, inFlight);
|
||||
await new Promise<void>((r) => setImmediate(r));
|
||||
inFlight--;
|
||||
}),
|
||||
]);
|
||||
await both;
|
||||
expect(maxInFlight).toBe(2);
|
||||
});
|
||||
});
|
||||
|
||||
describe("SyncLock reentrant-NO (E0.1)", () => {
|
||||
it("a second acquire of the same uuid from inside a held withLock returns false", async () => {
|
||||
const lock = new SyncLock();
|
||||
let innerAcquire: { acquired: boolean; heldOp?: string } | null = null;
|
||||
await lock.withLock("u1", "push", async () => {
|
||||
innerAcquire = lock.acquire("u1", "push"); // re-entrant attempt
|
||||
});
|
||||
expect(innerAcquire).toEqual({ acquired: false, heldOp: "push" });
|
||||
});
|
||||
|
||||
it("a skip-policy nested withLock for the same uuid returns undefined (no deadlock)", async () => {
|
||||
const lock = new SyncLock();
|
||||
let nestedRan = false;
|
||||
let nestedResult: unknown = "untouched";
|
||||
await lock.withLock("u1", "push", async () => {
|
||||
nestedResult = await lock.withLock("u1", "push", async () => { nestedRan = true; }, { policy: "skip" });
|
||||
});
|
||||
expect(nestedRan).toBe(false);
|
||||
expect(nestedResult).toBeUndefined();
|
||||
// Outer lock was still released.
|
||||
expect(lock.isHeld("u1")).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
describe("SyncLock queue timeout (E0.1)", () => {
|
||||
it("a queue-policy op THROWS LockAcquireTimeout after maxWaitMs if it cannot acquire (not a silent drop)", async () => {
|
||||
const lock = new SyncLock();
|
||||
const gate = deferred<void>();
|
||||
const holder = lock.withLock("u1", "push", async () => { await gate.promise; });
|
||||
await new Promise<void>((r) => setImmediate(r));
|
||||
await expect(
|
||||
lock.withLock("u1", "pull", async () => "ran", { policy: "queue", maxWaitMs: 30 }),
|
||||
).rejects.toBeInstanceOf(LockAcquireTimeout);
|
||||
gate.resolve(void 0);
|
||||
await holder;
|
||||
});
|
||||
|
||||
it("a nested queue-policy withLock for the same uuid throws after maxWait (no indefinite stall)", async () => {
|
||||
const lock = new SyncLock();
|
||||
// Outer holds; an inner queue-policy withLock for the same uuid cannot ever
|
||||
// acquire (the outer won't release until the inner returns) → throws after
|
||||
// maxWait instead of hanging forever.
|
||||
await expect(
|
||||
lock.withLock("u1", "push", async () => {
|
||||
await lock.withLock("u1", "push", async () => "inner", { policy: "queue", maxWaitMs: 20 });
|
||||
}),
|
||||
).rejects.toBeInstanceOf(LockAcquireTimeout);
|
||||
expect(lock.isHeld("u1")).toBe(false); // outer released in its finally despite inner throw
|
||||
});
|
||||
});
|
||||
|
||||
describe("SyncLock fairness (E0.1)", () => {
|
||||
it("a fresh public acquire DEFERS when waiters are queued (does not jump the queue)", async () => {
|
||||
const lock = new SyncLock();
|
||||
expect(lock.acquire("u1", "push")).toEqual({ acquired: true });
|
||||
|
||||
// Two queued waiters register while held.
|
||||
const gate = deferred<void>();
|
||||
const q1 = lock.withLock("u1", "pull", async () => { await gate.promise; return "q1"; }, { policy: "queue", maxWaitMs: 2000 });
|
||||
const q2 = lock.withLock("u1", "baseline", async () => "q2", { policy: "queue", maxWaitMs: 2000 });
|
||||
await new Promise<void>((r) => setImmediate(r)); // let both register in `waiters`
|
||||
|
||||
// Synchronously release, then synchronously acquire — between release and
|
||||
// the woken waiter's microtask, the lock is FREE with q2 still queued, so a
|
||||
// fresh public acquire must defer (not grab).
|
||||
lock.release("u1", "push"); // wakeOne shifts q1 (sync); waiters=[q2]; held deleted
|
||||
const fresh = lock.acquire("u1", "push");
|
||||
expect(fresh).toEqual({ acquired: false, deferred: true });
|
||||
|
||||
// Let q1 wake, grab, and run (it blocks on gate). q1 now holds; q2 queued.
|
||||
await new Promise<void>((r) => setImmediate(r));
|
||||
expect(lock.isHeld("u1")).toBe(true);
|
||||
gate.resolve(void 0);
|
||||
expect(await q1).toBe("q1"); // q1 releases → wakeOne shifts q2
|
||||
expect(await q2).toBe("q2"); // q2 grabs (FIFO after q1)
|
||||
expect(lock.isHeld("u1")).toBe(false);
|
||||
});
|
||||
|
||||
it("a queued waiter is served even if a fresh skip acquire is present (retry-loop, no silent give-up)", async () => {
|
||||
const lock = new SyncLock();
|
||||
const gate = deferred<void>();
|
||||
const holder = lock.withLock("u1", "push", async () => { await gate.promise; });
|
||||
const queued = lock.withLock("u1", "pull", async () => "served", { policy: "queue", maxWaitMs: 2000 });
|
||||
await new Promise<void>((r) => setImmediate(r));
|
||||
// A fresh skip op while held → drops (undefined), does not block the queue.
|
||||
const skip = await lock.withLock("u1", "push", async () => "skip-ran", { policy: "skip" });
|
||||
expect(skip).toBeUndefined();
|
||||
gate.resolve(void 0);
|
||||
expect(await queued).toBe("served");
|
||||
await holder;
|
||||
});
|
||||
});
|
||||
|
||||
describe("relPathLockKey fallback (E0.1)", () => {
|
||||
it("namespaces unlinked relPaths as a distinct pseudo-uuid", () => {
|
||||
expect(relPathLockKey("npcs/Roland.md")).toBe("relPath:npcs/Roland.md");
|
||||
expect(relPathLockKey("npcs/Roland.md")).not.toBe(relPathLockKey("npcs/Susan.md"));
|
||||
// The fallback is a distinct key from a real uuid — the bidirectional
|
||||
// claim holds for linked notes; the fallback only covers the unlinked tail.
|
||||
expect(relPathLockKey("npcs/Roland.md")).not.toBe("JournalEntry.abc1");
|
||||
});
|
||||
});
|
||||
13
wiki/components/_index.md
Normal file
13
wiki/components/_index.md
Normal file
@@ -0,0 +1,13 @@
|
||||
---
|
||||
type: meta
|
||||
title: "Components Index"
|
||||
updated: 2026-06-22
|
||||
---
|
||||
|
||||
# Components
|
||||
|
||||
Reusable pieces shared across modules. See `_templates/component.md`.
|
||||
|
||||
- [[content-hash]] — body contentHash idempotency guard
|
||||
- [[foundry-block]] — the `foundry:` frontmatter block + read/baseline helpers
|
||||
- [[name-uuid-resolver]] — name↔uuid map for link resolution
|
||||
35
wiki/components/content-hash.md
Normal file
35
wiki/components/content-hash.md
Normal file
@@ -0,0 +1,35 @@
|
||||
---
|
||||
type: component
|
||||
path: "src/normalize.ts"
|
||||
status: active
|
||||
purpose: "Stable body hash used as the sync baseline / idempotency guard."
|
||||
depends_on: []
|
||||
used_by: [[batch]], [[server]], [[autosync]], [[push]]
|
||||
tags: [component]
|
||||
created: 2026-06-22
|
||||
updated: 2026-06-22
|
||||
---
|
||||
|
||||
# content-hash
|
||||
|
||||
`contentHash(body)` in `src/normalize.ts`. Hashes a note's BODY (frontmatter excluded) to a
|
||||
stable string, stored in the note's `foundry.contentHash` and the cc file's `cc_sync_hash`.
|
||||
|
||||
## How it's used
|
||||
- **Drift detection** ([[index-recommend]]): `refinedChanged` = `foundry.contentHash` !=
|
||||
`contentHash(body)` (vault newer than last baseline). `ccChanged` = `cc_sync_hash` !=
|
||||
cc body hash.
|
||||
- **Idempotency guard** ([[autosync]]): a save only pushes if `contentHash(body) !=
|
||||
foundry.contentHash`. After a push, `baselineNote` sets `foundry.contentHash =
|
||||
contentHash(body)` → subsequent no-op saves skip.
|
||||
- **push-all**: only rows with `recommendation === "sync-cc"` (i.e. body hash drifted from
|
||||
baseline) are pushed; each is baselined after.
|
||||
|
||||
## Why it matters
|
||||
`pushNote` always PUTs. The hash is the ONLY thing preventing redundant pushes and the
|
||||
auto-sync feedback loop. It's the project's core sync primitive.
|
||||
|
||||
## Notes / gotchas
|
||||
- It hashes the BODY only, so a frontmatter-only edit (e.g. the post-push baseline rewrite)
|
||||
does NOT change the body hash → that's exactly why the baseline write is a no-op for the
|
||||
watcher.
|
||||
36
wiki/components/foundry-block.md
Normal file
36
wiki/components/foundry-block.md
Normal file
@@ -0,0 +1,36 @@
|
||||
---
|
||||
type: component
|
||||
path: "src/frontmatter.ts"
|
||||
status: active
|
||||
purpose: "Parse / read / rewrite the `foundry:` identity block in a refined note's frontmatter."
|
||||
depends_on: []
|
||||
used_by: [[batch]], [[push]], [[server]], [[autosync]]
|
||||
tags: [component]
|
||||
created: 2026-06-22
|
||||
updated: 2026-06-22
|
||||
---
|
||||
|
||||
# foundry-block
|
||||
|
||||
The `foundry:` nested mapping in a refined note's YAML frontmatter ties the note to its
|
||||
Foundry journal entry. Fields: `cc_uuid` (e.g. `JournalEntry.<id>`), `cc_type`, `folder_path`,
|
||||
`contentHash`, `syncedAt`.
|
||||
|
||||
## Helpers (src/frontmatter.ts)
|
||||
- `splitFrontmatter(md)` → `{fm, body}`.
|
||||
- `readFoundryBlock(fm)` → the `foundry:` mapping as `Record<string,string> | undefined`
|
||||
(so `fb.cc_uuid`, `fb.contentHash` are available).
|
||||
- `buildBlock(entry, body, stamp)` → the block content for seeding (used by `seedRow`/`handleLink`).
|
||||
- `seedBlockContent(md, block)` → inject/refresh the block in a note.
|
||||
|
||||
## In server.ts
|
||||
- `baselineFoundryBlock(md, newHash, newSyncedAt)` — text surgery that rewrites ONLY the
|
||||
`contentHash` + `syncedAt` lines inside the `foundry:` block, leaving `cc_uuid`/`cc_type`/
|
||||
`folder_path` and all curation untouched. Mirrors `scripts/resync.ts`.
|
||||
- `baselineNote(state, relPath, absPath)` — reads the note, computes `contentHash(body)`,
|
||||
calls `baselineFoundryBlock`, writes via the mirror-aware `targetPath`. Returns true if it
|
||||
wrote. Used by push-all and [[autosync]].
|
||||
|
||||
## Notes / gotchas
|
||||
- Baseline text-surgery is line-based: it scans for `foundry:` then the indented ` ` lines
|
||||
beneath it. A note without a `foundry:` block is returned unchanged (no baseline).
|
||||
28
wiki/components/name-uuid-resolver.md
Normal file
28
wiki/components/name-uuid-resolver.md
Normal file
@@ -0,0 +1,28 @@
|
||||
---
|
||||
type: component
|
||||
path: "src/resolver.ts"
|
||||
status: active
|
||||
purpose: "name↔uuid map for resolving wikilinks/mentions in notes to Foundry journal UUIDs during push."
|
||||
depends_on: [[relay-client]]
|
||||
used_by: [[push]], [[server]], [[cli]]
|
||||
tags: [component]
|
||||
created: 2026-06-22
|
||||
updated: 2026-06-22
|
||||
---
|
||||
|
||||
# name-uuid-resolver
|
||||
|
||||
`src/resolver.ts` — the `NameResolver` interface + `MapNameResolver` impl, plus
|
||||
`nameUuidIndexFromEntries`, `saveNameUuidIndex`, `loadNameUuidIndex`.
|
||||
|
||||
## How it's built / cached
|
||||
- `refresh` ([[refresh-flow]]) calls `relay.searchJournalEntries()` (minified) →
|
||||
`nameUuidIndexFromEntries([{name,uuid}…])` → saves `<out>/name-uuid.json`.
|
||||
- `pushNote` resolves links via: a preloaded `resolver` arg → else `loadNameUuidIndex(<out>/
|
||||
name-uuid.json)` → else build via relay `/search` (and cache it).
|
||||
|
||||
## Notes / gotchas
|
||||
- The map is keyed by entry NAME. Curated notes whose filename differs from the Foundry entry
|
||||
name (e.g. "Angro Harn" vs "Angro Harn - Journal") still resolve because the cc payload
|
||||
carries explicit UUID references, and the index's uuid-fallback matching handles filenames.
|
||||
- `name-uuid.json` is gitignored output (lives under `--out`).
|
||||
32
wiki/decisions/001-external-foundry.md
Normal file
32
wiki/decisions/001-external-foundry.md
Normal file
@@ -0,0 +1,32 @@
|
||||
---
|
||||
type: decision
|
||||
status: active
|
||||
date: 2026-06-20
|
||||
context: "The shareable compose bundle must let a cloner get running against their own Foundry."
|
||||
decision: "Foundry is external — the cloner supplies FOUNDRY_URL. No Foundry service in the compose."
|
||||
consequences: "Compose bundles only the relay (+ the host-run dashboard). The relay must be reachable FROM the Foundry host (the rest-api module connects OUT to it)."
|
||||
tags: [decision, adr]
|
||||
created: 2026-06-22
|
||||
updated: 2026-06-22
|
||||
---
|
||||
|
||||
# ADR 001 — External Foundry
|
||||
|
||||
## Context
|
||||
The live stack on the original host runs Foundry in its own compose (`~/docker/foundryvtt`,
|
||||
Traefik-fronted). For a shareable bundle, baking Foundry in is impractical (license, world
|
||||
data, secrets). The cloner already has Foundry somewhere.
|
||||
|
||||
## Decision
|
||||
Foundry stays external. The cloner supplies `FOUNDRY_URL` (+ world, user, password). The
|
||||
compose provides only the [[threehats-relay | relay]] and the (host-run) dashboard.
|
||||
|
||||
## Consequences
|
||||
- One fewer service to bundle/license.
|
||||
- The relay must be reachable from the Foundry host (the rest-api module connects OUT over
|
||||
WebSocket). If Foundry is remote, expose the relay port (port-forward / tailnet / public domain).
|
||||
- The `/setup` skill collects `FOUNDRY_URL`/`FOUNDRY_WORLD`/`RELAY_USER`/`RELAY_PASSWORD` from
|
||||
the user and points their rest-api module at the relay.
|
||||
|
||||
## Related
|
||||
- [[002-tool-host-run]], [[003-relay-no-journal-db-no-docker-stop]]
|
||||
30
wiki/decisions/002-tool-host-run.md
Normal file
30
wiki/decisions/002-tool-host-run.md
Normal file
@@ -0,0 +1,30 @@
|
||||
---
|
||||
type: decision
|
||||
status: active
|
||||
date: 2026-06-20
|
||||
context: "The sync tool reads the Foundry journal LevelDB and shells out to `docker` for refresh --full-index."
|
||||
decision: "The sync tool runs on the HOST, not in a container. The compose never includes it."
|
||||
consequences: "The tool needs host FS access (LevelDB) + the docker socket. The dashboard (:7788) is host-run via ./sync.sh ui."
|
||||
tags: [decision, adr]
|
||||
created: 2026-06-22
|
||||
updated: 2026-06-22
|
||||
---
|
||||
|
||||
# ADR 002 — Tool stays host-run
|
||||
|
||||
## Context
|
||||
The tool opens the Foundry journal LevelDB (read-only) for the dashboard's full index, and
|
||||
`refresh --full-index` stops/starts a local Foundry container via `docker`. Both need host
|
||||
access. Containerizing it would require bind-mounting the LevelDB + docker socket — fragile.
|
||||
|
||||
## Decision
|
||||
The sync tool (dashboard, CLI) runs on the host. The compose bundles only the relay. The
|
||||
dashboard binds `0.0.0.0:7788` (reachable on the tailnet) via `./sync.sh ui`.
|
||||
|
||||
## Consequences
|
||||
- Cloner needs Node on the host (the tool runs via `npx tsx`).
|
||||
- `.env` carries host-specific paths (`VAULT`, `JOURNAL`, `FOUNDRY_DATA_DIR`) — gitignored.
|
||||
- The `/setup` skill starts the dashboard on the host, not via compose.
|
||||
|
||||
## Related
|
||||
- [[001-external-foundry]], [[003-relay-no-journal-db-no-docker-stop]]
|
||||
35
wiki/decisions/003-relay-no-journal-db-no-docker-stop.md
Normal file
35
wiki/decisions/003-relay-no-journal-db-no-docker-stop.md
Normal file
@@ -0,0 +1,35 @@
|
||||
---
|
||||
type: decision
|
||||
status: active
|
||||
date: 2026-06-20
|
||||
context: "Live push/refresh must not stop Foundry or fight the LevelDB write lock."
|
||||
decision: "Live ops (push, push-all, refresh) go through the relay. The LevelDB snapshot is only for the full dashboard index (CLI, docker-stop)."
|
||||
consequences: "Zero-downtime live sync. The index's pull ops reflect the SNAPSHOT, not live Foundry — a known limitation."
|
||||
tags: [decision, adr]
|
||||
created: 2026-06-22
|
||||
updated: 2026-06-22
|
||||
---
|
||||
|
||||
# ADR 003 — Relay for live ops; LevelDB only for full index
|
||||
|
||||
## Context
|
||||
Foundry holds the journal LevelDB open with a write lock. Reading it live requires stopping
|
||||
Foundry (the docker-stop path). That's downtime — unacceptable for routine push/refresh.
|
||||
|
||||
## Decision
|
||||
- **Push / push-all / refresh** use the [[relay-client | relay]] (`/get`, `/update`, `/search`).
|
||||
Foundry keeps running; no LevelDB open, no Docker stop. `cp -r` of the live journal dir is
|
||||
used for the dev dashboard's snapshot (works while locked — only opening it is blocked).
|
||||
- The **full LevelDB read** (full entry docs, not minified) is CLI-only: `refresh --full-index`
|
||||
stops/starts a local Foundry container. The dashboard's `indexAll` uses the snapshot copy.
|
||||
|
||||
## Consequences
|
||||
- Live sync is zero-downtime.
|
||||
- The dashboard index's `entry` is the SNAPSHOT (`db.byId`), so pull ops (`rePullRow`) reflect
|
||||
the snapshot, not live Foundry edits. Live edits need a fresh snapshot or a future live-pull
|
||||
path. See [[index-recommend]].
|
||||
- `/search` is minified (no content hash) → can't detect live content changes cheaply
|
||||
([[005-autosync-o-to-f-only]]).
|
||||
|
||||
## Related
|
||||
- [[002-tool-host-run]], [[push-flow]], [[refresh-flow]]
|
||||
37
wiki/decisions/004-dropped-browser-obsidian.md
Normal file
37
wiki/decisions/004-dropped-browser-obsidian.md
Normal file
@@ -0,0 +1,37 @@
|
||||
---
|
||||
type: decision
|
||||
status: active
|
||||
date: 2026-06-21
|
||||
context: "The containerized browser-Obsidian (noVNC) + auto-installed plugin were clunky and had restricted-mode friction."
|
||||
decision: "Removed the obsidian + obsidian-init compose services and the plugin/. Edit in your own Obsidian desktop app; the dashboard is the only UI."
|
||||
consequences: "Compose bundles only the relay. The foundry-sync Obsidian plugin no longer exists. Dashboard push/pull/sync + auto-sync replace it."
|
||||
tags: [decision, adr]
|
||||
created: 2026-06-22
|
||||
updated: 2026-06-22
|
||||
---
|
||||
|
||||
# ADR 004 — Dropped browser Obsidian + plugin
|
||||
|
||||
## Context
|
||||
Phase 1 of the shareable bundle added `ghcr.io/sytone/obsidian-remote` (browser Obsidian via
|
||||
noVNC) bind-mounted to the vault, plus a `foundry-sync` Obsidian plugin auto-installed by an
|
||||
`obsidian-init` one-shot. In practice: the noVNC desktop was heavy/laggy, plugin loading hit
|
||||
Obsidian's restricted-mode trust prompt (uncertain whether a preseed would be honored), and the
|
||||
user already edits a real desktop-Obsidian vault on this host (`Land of Mardonar`).
|
||||
|
||||
## Decision
|
||||
Removed entirely (2026-06-21):
|
||||
- `obsidian` + `obsidian-init` services deleted from `docker-compose.yml`.
|
||||
- `plugin/` directory and `scripts/obsidian-init.js` deleted.
|
||||
- `.env` / `.env.example` obsidian vars (`OBSIDIAN_*`, `DASHBOARD_URL`, `PUID/PGID/TZ`) removed.
|
||||
- README + `/setup` skill rewritten: relay + dashboard only; edit notes in your own Obsidian
|
||||
desktop app pointed at `$VAULT`.
|
||||
|
||||
## Consequences
|
||||
- Compose now bundles ONLY the relay (`docker compose config` → `relay` + `relay-gpu` profile).
|
||||
- The dashboard (:7788) is the sole UI — consistent with the user's hard rule: "if it's not in
|
||||
the dashboard UI, it doesn't exist." Push/pull/sync + auto-sync cover what the plugin did.
|
||||
- The user edits the vault in their own Obsidian app; the dashboard reads/writes the same files.
|
||||
|
||||
## Related
|
||||
- [[005-autosync-o-to-f-only]] — the dashboard feature that replaced the plugin's push button.
|
||||
40
wiki/decisions/005-autosync-o-to-f-only.md
Normal file
40
wiki/decisions/005-autosync-o-to-f-only.md
Normal file
@@ -0,0 +1,40 @@
|
||||
---
|
||||
type: decision
|
||||
status: active
|
||||
date: 2026-06-21
|
||||
context: "User asked for an auto-sync button so 'any file syncs from either direction instantly.'"
|
||||
decision: "Auto-sync is Obsidian→Foundry instant only (inotify + contentHash guard). Foundry→Obsidian stays manual (Sync / Re-pull)."
|
||||
consequences: "The common pain (manual push after editing a note) is solved. Foundry→Obsidian remains a dashboard button click — the relay can't push change events."
|
||||
tags: [decision, adr]
|
||||
created: 2026-06-22
|
||||
updated: 2026-06-22
|
||||
---
|
||||
|
||||
# ADR 005 — Auto-sync is Obsidian→Foundry only
|
||||
|
||||
## Context
|
||||
The user wanted hands-off sync both ways. Two facts shaped feasibility:
|
||||
1. The relay has **no change-push / events stream**. `/search` is minified (`{uuid,id,name,img}`)
|
||||
with **no content hash** — so a live Foundry content edit can't be detected from `/search`.
|
||||
2. Detecting a live Foundry edit needs a `/get` per matched entry per cycle and a diff. For a
|
||||
~200-entry world, a fast poll (~15s) is ~800 `/get` calls/min through the headless Foundry —
|
||||
heavy and fragile. The index's drift detection compares LOCAL baselines + a stale snapshot,
|
||||
not live Foundry ([[index-recommend]]).
|
||||
|
||||
So "instantly from either direction" is impossible: Obsidian→Foundry can be instant (inotify on
|
||||
the host vault); Foundry→Obsidian cannot.
|
||||
|
||||
## Decision
|
||||
Auto-sync ([[autosync]]) is **Obsidian→Foundry instant only**. Foundry→Obsidian stays manual
|
||||
via the dashboard's Sync / Re-pull buttons. The user was offered three scopes (both + 60s poll,
|
||||
O→F only, both + 15s poll) and chose O→F only.
|
||||
|
||||
## Consequences
|
||||
- Solves the actual pain: saving a linked, seeded note pushes it to live Foundry within ~1s.
|
||||
- Foundry→Obsidian requires a manual button click. A future live-pull path (relay `/get` per
|
||||
matched entry + `entryToObsidian` on a poll) is feasible but not built — cost scales with
|
||||
matched-entry count.
|
||||
- The guard + `baselineNote` are what keep it loop-free ([[autosync-watch-loop]]).
|
||||
|
||||
## Related
|
||||
- [[autosync]], [[autosync-watch-loop]], [[003-relay-no-journal-db-no-docker-stop]]
|
||||
15
wiki/decisions/_index.md
Normal file
15
wiki/decisions/_index.md
Normal file
@@ -0,0 +1,15 @@
|
||||
---
|
||||
type: meta
|
||||
title: "Decisions Index (ADRs)"
|
||||
updated: 2026-06-22
|
||||
---
|
||||
|
||||
# Decisions (ADRs)
|
||||
|
||||
Architecture Decision Records. Newest context at the bottom of the file; highest number = most recent. See `_templates/decision.md`.
|
||||
|
||||
- [[001-external-foundry]] — external Foundry (cloner supplies FOUNDRY_URL)
|
||||
- [[002-tool-host-run]] — sync tool runs on the host, not in a container
|
||||
- [[003-relay-no-journal-db-no-docker-stop]] — live ops via relay; LevelDB only for full index
|
||||
- [[004-dropped-browser-obsidian]] — removed containerized Obsidian + plugin
|
||||
- [[005-autosync-o-to-f-only]] — auto-sync is Obsidian→Foundry instant only
|
||||
14
wiki/dependencies/_index.md
Normal file
14
wiki/dependencies/_index.md
Normal file
@@ -0,0 +1,14 @@
|
||||
---
|
||||
type: meta
|
||||
title: "Dependencies Index"
|
||||
updated: 2026-06-22
|
||||
---
|
||||
|
||||
# Dependencies
|
||||
|
||||
External services, libraries, and modules. See `_templates/dependency.md`.
|
||||
|
||||
- [[threehats-relay]] — foundryvtt-rest-api-relay (Go + headless Chrome)
|
||||
- [[foundryvtt-rest-api-module]] — Foundry module that connects OUT to the relay
|
||||
- [[classic-level]] — journal LevelDB reader (classic-level npm)
|
||||
- [[linkedom]] — HTML parsing for Obsidian→Foundry conversion
|
||||
34
wiki/dependencies/classic-level.md
Normal file
34
wiki/dependencies/classic-level.md
Normal file
@@ -0,0 +1,34 @@
|
||||
---
|
||||
type: dependency
|
||||
name: "classic-level"
|
||||
version: "^3.0.0"
|
||||
kind: library
|
||||
status: active
|
||||
purpose: "Read the Foundry journal LevelDB (read-only) for the dashboard's full index."
|
||||
risk: low
|
||||
tags: [dependency]
|
||||
created: 2026-06-22
|
||||
updated: 2026-06-22
|
||||
---
|
||||
|
||||
# classic-level
|
||||
|
||||
## What it is
|
||||
The `classic-level` npm package — a Node binding for LevelDB. Used by `src/db.ts`
|
||||
(`JournalDb.open`, `all()`, `byId()`).
|
||||
|
||||
## Why we depend on it
|
||||
The dashboard's full index (`indexAll`) needs full journal entry docs (not the minified
|
||||
`/search` list). Those live in Foundry's journal LevelDB.
|
||||
|
||||
## How we use it
|
||||
- `JournalDb` opens the LevelDB read-only (a `cp -r` snapshot in dev, since Foundry holds the
|
||||
live DB with a write lock — `cp` works while locked, opening doesn't).
|
||||
- `db.byId(ccId)` returns the snapshot entry used by [[batch]] row builders + `handleEntries`/`handleLink`.
|
||||
|
||||
## Risk / lock-in
|
||||
- Low: stable, narrow use (read-only). Only the full-index path needs the live DB (docker-stop,
|
||||
CLI-only — [[003-relay-no-journal-db-no-docker-stop]]).
|
||||
|
||||
## Related
|
||||
- [[batch]], [[index-recommend]], [[003-relay-no-journal-db-no-docker-stop]]
|
||||
36
wiki/dependencies/foundryvtt-rest-api-module.md
Normal file
36
wiki/dependencies/foundryvtt-rest-api-module.md
Normal file
@@ -0,0 +1,36 @@
|
||||
---
|
||||
type: dependency
|
||||
name: "Foundry rest-api module"
|
||||
version: ""
|
||||
kind: module
|
||||
status: active
|
||||
purpose: "The Foundry-side module that exposes the relay's endpoints by connecting OUT to it."
|
||||
risk: medium
|
||||
tags: [dependency]
|
||||
created: 2026-06-22
|
||||
updated: 2026-06-22
|
||||
---
|
||||
|
||||
# Foundry rest-api module
|
||||
|
||||
## What it is
|
||||
A Foundry VTT module (`github.com/ThreeHats/foundryvtt-rest-api-relay` ecosystem) installed in
|
||||
the user's Foundry world. It connects OUT to the relay over WebSocket and services the
|
||||
relay's forwarded requests (CRUD on journal entries, search).
|
||||
|
||||
## Why we depend on it
|
||||
Without it, the relay has nothing to forward to — `activeClients` is 0 and every
|
||||
`/get`/`/update`/`/search` 404s ("No connected Foundry clients found" / "Invalid client ID").
|
||||
|
||||
## How we use it
|
||||
- The `/setup` skill tells the user: install the module in Foundry, enable it, and set its
|
||||
relay URL to `ws(s)://<host-Foundry-can-reach>:3010`. The relay must be reachable FROM the
|
||||
Foundry host ([[001-external-foundry]]).
|
||||
- With >1 connected client, requests need a `clientId` query param ([[relay-client]]).
|
||||
|
||||
## Risk / lock-in
|
||||
- Medium: Foundry-version compatibility; module defines the response envelopes.
|
||||
- Must be reconnected after Foundry/server restarts.
|
||||
|
||||
## Related
|
||||
- [[threehats-relay]], [[relay-client]], [[001-external-foundry]]
|
||||
32
wiki/dependencies/linkedom.md
Normal file
32
wiki/dependencies/linkedom.md
Normal file
@@ -0,0 +1,32 @@
|
||||
---
|
||||
type: dependency
|
||||
name: "linkedom"
|
||||
version: "^0.18.12"
|
||||
kind: library
|
||||
status: active
|
||||
purpose: "Parse HTML in the Obsidian→Foundry conversion (markdown→HTML→Campaign Codex JSON)."
|
||||
risk: low
|
||||
tags: [dependency]
|
||||
created: 2026-06-22
|
||||
updated: 2026-06-22
|
||||
---
|
||||
|
||||
# linkedom
|
||||
|
||||
## What it is
|
||||
A DOM-like HTML parser (`linkedom` npm). Used in the Obsidian→Foundry conversion path
|
||||
(`src/toFoundry.ts` / `obsidianToFoundryJsonLive`).
|
||||
|
||||
## Why we depend on it
|
||||
The note body is markdown → HTML → the Campaign Codex `flags.campaign-codex.data` structure.
|
||||
linkedom parses that HTML without a browser.
|
||||
|
||||
## How we use it
|
||||
- `obsidianToFoundryJsonLive` (called by `buildPushPayload` in [[push]]) builds the full
|
||||
JournalEntry + the minimal `/update` diff.
|
||||
|
||||
## Risk / lock-in
|
||||
- Low: pure parsing, no network. Only the conversion path depends on it.
|
||||
|
||||
## Related
|
||||
- [[push]], [[push-flow]]
|
||||
38
wiki/dependencies/threehats-relay.md
Normal file
38
wiki/dependencies/threehats-relay.md
Normal file
@@ -0,0 +1,38 @@
|
||||
---
|
||||
type: dependency
|
||||
name: "ThreeHats foundryvtt-rest-api-relay"
|
||||
version: latest
|
||||
kind: service
|
||||
status: active
|
||||
purpose: "REST/WebSocket bridge to a running Foundry world for live push/refresh without downtime."
|
||||
risk: medium
|
||||
tags: [dependency]
|
||||
created: 2026-06-22
|
||||
updated: 2026-06-22
|
||||
---
|
||||
|
||||
# ThreeHats foundryvtt-rest-api-relay
|
||||
|
||||
## What it is
|
||||
A Go relay (`github.com/ThreeHats/foundryvtt-rest-api-relay`) that runs a headless Chrome
|
||||
session logged into Foundry. Foundry's rest-api module connects OUT to it over WebSocket; API
|
||||
clients call the relay's HTTP endpoints (`/get`, `/update`, `/create`, `/search`, `/session`,
|
||||
`/session-handshake`, `/start-session`), which it forwards to the module.
|
||||
|
||||
## Why we depend on it
|
||||
Live push/refresh without stopping Foundry or touching the LevelDB write lock ([[003-relay-no-journal-db-no-docker-stop]]).
|
||||
|
||||
## How we use it
|
||||
- Bundled in `docker-compose.yml` as the `relay` service (port 3010, sqlite, `ALLOW_HEADLESS`,
|
||||
`shm_size 1g`; `relay-gpu` profile for `/dev/dri` passthrough).
|
||||
- API keys created via the relay's web signup (no env/CLI) → `RELAY_API_KEY`.
|
||||
- Headless session launched by `scripts/start-relay-session.js` (handshake → RSA-OAEP encrypt
|
||||
password → start-session). On the original host, `relay-keepalive.sh` keeps it alive.
|
||||
- [[relay-client]] wraps the endpoints we use.
|
||||
|
||||
## Risk / lock-in
|
||||
- Medium: external project; envelope shapes are module-defined and per-endpoint inconsistent.
|
||||
- No change-push/events — `/search` is minified with no content hash ([[005-autosync-o-to-f-only]]).
|
||||
|
||||
## Related
|
||||
- [[relay-client]], [[foundryvtt-rest-api-module]], [[refresh-flow]], [[push-flow]]
|
||||
14
wiki/flows/_index.md
Normal file
14
wiki/flows/_index.md
Normal file
@@ -0,0 +1,14 @@
|
||||
---
|
||||
type: meta
|
||||
title: "Flows Index"
|
||||
updated: 2026-06-22
|
||||
---
|
||||
|
||||
# Flows
|
||||
|
||||
Data flows / request paths. See `_templates/flow.md`.
|
||||
|
||||
- [[push-flow]] — Obsidian → live Foundry (relay /get → diff → /update)
|
||||
- [[refresh-flow]] — name↔uuid map via relay /search (zero downtime)
|
||||
- [[autosync-watch-loop]] — fs.watch → debounce → guard → push → baseline
|
||||
- [[index-recommend]] — indexAll + recommend() drift detection
|
||||
41
wiki/flows/autosync-watch-loop.md
Normal file
41
wiki/flows/autosync-watch-loop.md
Normal file
@@ -0,0 +1,41 @@
|
||||
---
|
||||
type: flow
|
||||
status: active
|
||||
purpose: "On a vault save, push the changed note to live Foundry instantly (Obsidian→Foundry only)."
|
||||
actors: [[autosync]], [[push]], [[foundry-block]], [[content-hash]], [[relay-client]]
|
||||
steps: [fs.watch event, filter .md, debounce, guard, pushNote, baselineNote, log event]
|
||||
tags: [flow]
|
||||
created: 2026-06-22
|
||||
updated: 2026-06-22
|
||||
---
|
||||
|
||||
# autosync-watch-loop
|
||||
|
||||
Driven by [[autosync]] `AutoSyncController`, toggled from the dashboard's "Auto-sync" button.
|
||||
|
||||
## Steps
|
||||
1. `fs.watch(refinedDir, {recursive:true})` (per-dir fallback) fires on a `.md` change.
|
||||
2. Filter: must end `.md`, not be a `.bak`, not under `.obsidian/`.
|
||||
3. Debounce 700ms per file (coalesce rapid saves).
|
||||
4. `process(relPath)`:
|
||||
- read via `resolveRefined` (mirror-aware);
|
||||
- **guard**: skip unless `foundry.cc_uuid` (linked) AND `foundry.contentHash` (seeded) AND
|
||||
`contentHash(body) !== foundry.contentHash` (actually changed);
|
||||
- else `pushNote({dryRun:false})` ([[push-flow]]) → `baselineNote`.
|
||||
5. Log to the 100-event ring buffer (`pushed` / `skipped` / `error`); UI polls `GET /api/autosync`.
|
||||
|
||||
## Why no loop
|
||||
After a push, `baselineNote` sets `foundry.contentHash = contentHash(body)`. The watcher then
|
||||
fires on that baseline write too, but the guard reads `body hash == contentHash` → `skipped:
|
||||
unchanged`. So the baseline write is a no-op for the watcher. No feedback loop, no double-push.
|
||||
|
||||
## Edge cases / failure modes
|
||||
- Unlinked note (no `cc_uuid`) → `skipped: not linked` (seed/link first).
|
||||
- Unseeded note (no `contentHash` baseline) → `skipped: not seeded`.
|
||||
- Relay down / no headless session → `pushNote` → `relay.getEntry` 404 → `error` logged (no
|
||||
crash, no Foundry write); the note is NOT baselined, so it retries on the next real save.
|
||||
- New note created from scratch → no `foundry:` block → skipped (seed/link it manually first).
|
||||
|
||||
## Related
|
||||
- [[005-autosync-o-to-f-only]] — why Foundry→Obsidian isn't in this loop.
|
||||
- [[push-flow]] — the per-note push this loop invokes.
|
||||
41
wiki/flows/index-recommend.md
Normal file
41
wiki/flows/index-recommend.md
Normal file
@@ -0,0 +1,41 @@
|
||||
---
|
||||
type: flow
|
||||
status: active
|
||||
purpose: "Build the file↔entry index and decide a per-row recommendation from local baselines."
|
||||
actors: [[batch]], [[content-hash]], [[foundry-block]], db.ts
|
||||
steps: [walk refined+cc, read meta, basename match + uuid fallback, look up snapshot entry, recommend]
|
||||
tags: [flow]
|
||||
created: 2026-06-22
|
||||
updated: 2026-06-22
|
||||
---
|
||||
|
||||
# index-recommend
|
||||
|
||||
`indexAll` + `recommend` in [[batch]]. Rebuilt on every `GET /api/index` (hash-only, cheap) and
|
||||
after mutating ops. Powers the dashboard's recommendation badges + bulk actions.
|
||||
|
||||
## Steps
|
||||
1. Walk refined + cc dirs (with dev-mode `<out>` overlays).
|
||||
2. Read each file's meta: refined → `foundry.contentHash` (storedHash), body hash, mtime;
|
||||
cc → `cc_id`, `cc_sync_hash`, body hash, mtime.
|
||||
3. Match by **basename** (canonical); **uuid fallback** via `foundry.cc_uuid` for curated
|
||||
notes whose filename differs.
|
||||
4. `entry = db.byId(ccId)` — from the LevelDB **snapshot**.
|
||||
5. `recommend(...)`: `seed` | `sync-cc` (vault newer) | `repull` (cc newer) | `conflict` |
|
||||
`in-sync` | `import` | `review`.
|
||||
|
||||
## Drift is LOCAL-baseline-based, not live
|
||||
- `refinedChanged` = `foundry.contentHash` != `contentHash(body)` — vault newer than the LAST
|
||||
BASELINE (set by a prior push/sync). Not "newer than live Foundry."
|
||||
- `ccChanged` = `cc_sync_hash` != cc body hash — cc file newer than its baseline.
|
||||
- `entry` is the snapshot, so pull ops (`rePullRow`) reflect the snapshot, not live Foundry.
|
||||
|
||||
## Implications
|
||||
- The index is a fast, local, offline drift signal — great for "what have I edited since I
|
||||
last pushed?" It does NOT know about live Foundry edits someone else made.
|
||||
- Live Foundry edits are only surfaced via the relay push/refresh path. Detecting a live
|
||||
content change requires `/get` per entry (no hash in `/search`) — see [[005-autosync-o-to-f-only]].
|
||||
|
||||
## Related
|
||||
- [[content-hash]], [[foundry-block]] — the baselines this compares.
|
||||
- [[push-flow]] / [[refresh-flow]] — the live paths that complement this offline index.
|
||||
39
wiki/flows/push-flow.md
Normal file
39
wiki/flows/push-flow.md
Normal file
@@ -0,0 +1,39 @@
|
||||
---
|
||||
type: flow
|
||||
status: active
|
||||
purpose: "Push one refined note's content into LIVE Foundry without stopping Foundry."
|
||||
actors: [[push]], [[relay-client]], [[foundry-block]], [[name-uuid-resolver]], foundry/assets.ts
|
||||
steps: [read note, process body images, read foundry.cc_uuid, resolve name↔uuid, relay /get live entry, build minimal diff, backup live entry, relay /update, baseline note]
|
||||
tags: [flow]
|
||||
created: 2026-06-22
|
||||
updated: 2026-06-22
|
||||
---
|
||||
|
||||
# push-flow
|
||||
|
||||
Obsidian → live Foundry. Driven by [[push]] `pushNote`, invoked from the dashboard's push /
|
||||
push-all buttons and from [[autosync]].
|
||||
|
||||
## Steps
|
||||
1. Read the refined note; pre-process body `![[image]]` embeds → upload to Foundry's uploads
|
||||
dir, rewrite to `` (drop non-local). Frontmatter untouched.
|
||||
2. `readNoteFoundryMeta` → `foundry.cc_uuid` (required — else "run seed first") + portrait.
|
||||
3. Resolve the name↔uuid map (preloaded → `<out>/name-uuid.json` → build via `/search`).
|
||||
4. `relay.getEntry(cc_uuid)` — the LIVE entry (keeps ownership/folder/pages/existing image).
|
||||
5. Upload portrait if present (needs `FOUNDRY_DATA_DIR`/`WORLD`).
|
||||
6. `buildPushPayload` → minimal diff `{ name, "flags.campaign-codex": cc }` (dot-path merge
|
||||
preserves sibling flags; no `_id`/`pages`/`ownership`).
|
||||
7. dryRun → return diff. Apply → back up live entry to `<out>/bak/<name>.<stamp>.json`, then
|
||||
`relay.updateEntry(uuid, diff)`.
|
||||
8. (caller) `baselineNote` → set `foundry.contentHash = contentHash(body)` so a re-run skips.
|
||||
|
||||
## Edge cases / failure modes
|
||||
- No `foundry.cc_uuid` → throws (seed/link first).
|
||||
- No connected Foundry client → relay `404`/`400` ("Invalid client ID" / lists clients). Start
|
||||
the headless session.
|
||||
- Relay timeout `408`/`504` (~10s WS round-trip).
|
||||
- Image upload skipped when `FOUNDRY_DATA_DIR`/`WORLD` unset (existing image kept).
|
||||
|
||||
## Related
|
||||
- [[autosync-watch-loop]] — same flow, triggered by a vault save.
|
||||
- [[refresh-flow]] — builds the name↔uuid map this depends on.
|
||||
37
wiki/flows/refresh-flow.md
Normal file
37
wiki/flows/refresh-flow.md
Normal file
@@ -0,0 +1,37 @@
|
||||
---
|
||||
type: flow
|
||||
status: active
|
||||
purpose: "Rebuild the cached name↔uuid map live via relay /search — zero Foundry downtime."
|
||||
actors: [[relay-client]], [[name-uuid-resolver]], [[server]], [[cli]]
|
||||
steps: [relay /search minified, nameUuidIndexFromEntries, save name-uuid.json]
|
||||
tags: [flow]
|
||||
created: 2026-06-22
|
||||
updated: 2026-06-22
|
||||
---
|
||||
|
||||
# refresh-flow
|
||||
|
||||
`handleRefresh` (dashboard `POST /api/refresh`) and `cmd refresh` (CLI). Builds `name-uuid.json`
|
||||
so pushes can resolve note mentions to Foundry UUIDs.
|
||||
|
||||
## Steps
|
||||
1. `relay.searchJournalEntries()` → `GET /search?filter=documentType:JournalEntry&minified=true`
|
||||
→ `results: [{uuid,id,name,img}]`.
|
||||
2. `nameUuidIndexFromEntries(results.map(r => ({name, uuid})))` → `{nameToUuid, uuidToName}`.
|
||||
3. `saveNameUuidIndex(idx, <out>/name-uuid.json)`.
|
||||
4. Returns `{pairs, path}`.
|
||||
|
||||
## Why /search (not the LevelDB)
|
||||
`/search` is minified and live — it lists every journal entry with zero Foundry downtime
|
||||
(no Docker stop, no LevelDB lock). The heavy docker-stop full LevelDB read is CLI-only
|
||||
(`refresh --full-index`) and only needed for the dashboard's full `indexAll` (which needs full
|
||||
entry docs, not minified). See [[003-relay-no-journal-db-no-docker-stop]].
|
||||
|
||||
## Edge cases / failure modes
|
||||
- 0 connected clients → `404`; >1 → `400` (pass `clientId`).
|
||||
- Minified results have NO content hash — so refresh detects new/renamed entries but NOT
|
||||
content changes. That asymmetry is why Foundry→Obsidian isn't auto ([[005-autosync-o-to-f-only]]).
|
||||
|
||||
## Related
|
||||
- [[push-flow]] — consumes the map.
|
||||
- [[index-recommend]] — the index uses the LevelDB snapshot, not this map.
|
||||
17
wiki/modules/_index.md
Normal file
17
wiki/modules/_index.md
Normal file
@@ -0,0 +1,17 @@
|
||||
---
|
||||
type: meta
|
||||
title: "Modules Index"
|
||||
updated: 2026-06-22
|
||||
---
|
||||
|
||||
# Modules
|
||||
|
||||
One note per major module / package. See `_templates/module.md`.
|
||||
|
||||
- [[server]] — HTTP dashboard + JSON API
|
||||
- [[autosync]] — vault watcher → live push
|
||||
- [[push]] — single-note live push
|
||||
- [[relay-client]] — ThreeHats relay client
|
||||
- [[batch]] — index + row builders
|
||||
- [[cli]] — commands
|
||||
- [[config]] — env-driven config
|
||||
50
wiki/modules/autosync.md
Normal file
50
wiki/modules/autosync.md
Normal file
@@ -0,0 +1,50 @@
|
||||
---
|
||||
type: module
|
||||
path: "src/server.ts (AutoSyncController)"
|
||||
status: active
|
||||
language: typescript
|
||||
purpose: "Watch the refined vault and push each saved linked+seeded note into live Foundry instantly (Obsidian→Foundry only)."
|
||||
maintainer: Kaysser Kayyali
|
||||
last_updated: 2026-06-22
|
||||
linked_issues: []
|
||||
depends_on: [[push]], [[relay-client]], [[content-hash]], [[foundry-block]]
|
||||
used_by: [[server]]
|
||||
tags: [module]
|
||||
created: 2026-06-22
|
||||
updated: 2026-06-22
|
||||
---
|
||||
|
||||
# autosync
|
||||
|
||||
`AutoSyncController` in `src/server.ts`. Toggled via the dashboard's "Auto-sync" button
|
||||
(`POST /api/autosync {enabled}`); status + activity log at `GET /api/autosync` (the UI polls
|
||||
every 2s while on). See [[005-autosync-o-to-f-only]] for why it's one-directional.
|
||||
|
||||
## How it works
|
||||
1. `fs.watch(refinedDir, {recursive:true})` (per-dir fallback for older Node) → `onChange`.
|
||||
2. Filter to `*.md`, skip `.obsidian/` and `.bak`. Debounce 700ms per file.
|
||||
3. `process(relPath)`:
|
||||
- read the note via `resolveRefined` (mirror-aware);
|
||||
- **guard**: skip unless `foundry.cc_uuid` exists (linked), `foundry.contentHash` exists
|
||||
(seeded), and `contentHash(body) !== foundry.contentHash` (actually changed);
|
||||
- `pushNote` (live, `dryRun:false`) → `baselineNote` (rewrite `foundry.contentHash` to the
|
||||
new body hash).
|
||||
4. Bounded concurrency (3), per-file in-flight mutex, 100-event ring buffer (`events`).
|
||||
|
||||
## The guard is the loop/idempotency key
|
||||
`pushNote` always PUTs. The guard skips no-op saves (body hash == baseline). After a push,
|
||||
`baselineNote` sets the baseline to the new body hash, so the watcher's own baseline write
|
||||
(also a `.md` change) reads as unchanged → skipped. No feedback loop, no redundant push.
|
||||
|
||||
## Mode behavior
|
||||
- **dev**: baseline lands in the `--out` mirror (consistent with dev semantics); the mirror
|
||||
then wins `resolveRefined` until the real vault is edited again.
|
||||
- **apply**: baseline lands in the real vault (with `.bak`).
|
||||
- The push itself goes live via the relay regardless of mode.
|
||||
|
||||
## Notes / gotchas
|
||||
- Needs the relay + a live headless session (`activeClients > 0`), or `pushNote` → `relay.getEntry`
|
||||
404s ("Invalid client ID" / no connected clients) and logs an `error` (no crash, no Foundry write).
|
||||
- Verified 2026-06-22: no-op save → `skipped: unchanged`; scratch note with fake uuid + body
|
||||
change → guard passed → push attempted → relay 404 → `error` logged. Real-push end-to-end
|
||||
pending the dev headless session coming back up.
|
||||
56
wiki/modules/batch.md
Normal file
56
wiki/modules/batch.md
Normal file
@@ -0,0 +1,56 @@
|
||||
---
|
||||
type: module
|
||||
path: "src/batch.ts"
|
||||
status: active
|
||||
language: typescript
|
||||
purpose: "Index both corpora against the journal DB; produce match rows + recommendations; build per-row outputs."
|
||||
maintainer: Kaysser Kayyali
|
||||
last_updated: 2026-06-22
|
||||
linked_issues: []
|
||||
depends_on: [[content-hash]], [[foundry-block]], db.ts, toObsidian.ts
|
||||
used_by: [[server]], [[cli]]
|
||||
tags: [module]
|
||||
created: 2026-06-22
|
||||
updated: 2026-06-22
|
||||
---
|
||||
|
||||
# batch
|
||||
|
||||
`src/batch.ts` — `indexAll`, `recommend`, and the row builders (`seedRow`, `syncRow`,
|
||||
`rePullRow`, `importRow`). See [[index-recommend]].
|
||||
|
||||
## indexAll(db, ccDir, refinedDir, refinedOverlay?, ccOverlay?)
|
||||
Walks refined + cc dirs (with dev-mode overlays), reads each file's meta, matches by
|
||||
**basename** (canonical — the exporter names cc files by entry name), with **uuid fallback**
|
||||
for curated notes whose filename differs but are already linked by `foundry.cc_uuid`. Produces:
|
||||
`matched`, `ccOnly`, `refinedOnly`, `counts`, `byRecommendation`. In dev mode the server
|
||||
passes the `<out>/refined` + `<out>/cc` mirrors as overlays so dev writes are reflected.
|
||||
|
||||
## recommend(params) → Recommendation
|
||||
```
|
||||
refined-only → review
|
||||
cc-only (entry?) → import : review
|
||||
matched-unlinked → review
|
||||
matched+linked, !seeded → seed
|
||||
!hasCc | !ccSynced → sync-cc
|
||||
refinedChanged && ccChanged → conflict
|
||||
refinedChanged → sync-cc (vault newer → push)
|
||||
ccChanged → repull (cc/foundry newer → pull)
|
||||
else → in-sync
|
||||
```
|
||||
`refinedChanged` = note's `foundry.contentHash` != note body hash (vault newer than last
|
||||
baseline). `ccChanged` = cc file's `cc_sync_hash` != cc body hash. **Both compare against
|
||||
LOCAL baselines, not live Foundry.**
|
||||
|
||||
## Row builders
|
||||
- `seedRow` — inject/refresh the `foundry:` block from the journal entry (curation untouched).
|
||||
- `syncRow` — regenerate cc.md from the refined note AND refresh the refined `foundry:` block;
|
||||
baselines both sides.
|
||||
- `rePullRow` — regenerate the refined body from Foundry (`entryToObsidian(row.entry, …)`),
|
||||
preserving curated type/aliases/status tags. Uses the snapshot `row.entry`.
|
||||
- `importRow` — cc-only entry → new un-curated refined note under `refined/imported/<folder>/`.
|
||||
|
||||
## Notes / gotchas
|
||||
- `row.entry` comes from `db.byId(ccId)` — the LevelDB **snapshot**, not live Foundry. So
|
||||
pull ops reflect the snapshot; live Foundry edits need a fresh snapshot (docker-stop full
|
||||
index) or a future live-pull path. See [[003-relay-no-journal-db-no-docker-stop]].
|
||||
32
wiki/modules/cli.md
Normal file
32
wiki/modules/cli.md
Normal file
@@ -0,0 +1,32 @@
|
||||
---
|
||||
type: module
|
||||
path: "src/cli.ts"
|
||||
status: active
|
||||
language: typescript
|
||||
purpose: "Entry point (tsx). Commands: ui (dashboard), refresh, push. Env-driven via [[config]]."
|
||||
maintainer: Kaysser Kayyali
|
||||
last_updated: 2026-06-22
|
||||
linked_issues: []
|
||||
depends_on: [[server]], [[batch]], [[relay-client]], [[push]], [[config]]
|
||||
used_by: []
|
||||
tags: [module]
|
||||
created: 2026-06-22
|
||||
updated: 2026-06-22
|
||||
---
|
||||
|
||||
# cli
|
||||
|
||||
`src/cli.ts` — run via `npx tsx src/cli.ts <cmd>` or `./sync.sh <cmd>` (which sources `.env`
|
||||
and injects `--journal`).
|
||||
|
||||
## Commands (to document fully)
|
||||
- `ui` → `startServer` ([[server]]). `--vault`, `--cc` (optional — auto-builds stubs),
|
||||
`--out`, `--host` (default `0.0.0.0`), `--apply` (write real dirs; default dev/dry-run).
|
||||
Relay enabled when `RELAY_API_KEY` is set. Binds `0.0.0.0:7788`.
|
||||
- `refresh` → `name-uuid.json` via relay `/search` ([[refresh-flow]]). `--full-index` does
|
||||
the heavy docker-stop LevelDB read (CLI-only by design).
|
||||
- `push` → [[push]] one note live.
|
||||
|
||||
## Notes
|
||||
- `sync.sh` loads `.env` (`set -a; . ./.env`) and injects `--journal` from `$JOURNAL`.
|
||||
- This page is a stub — expand with exact flags + the `cmdUi`/`cmdRefresh`/`cmdPush` bodies.
|
||||
29
wiki/modules/config.md
Normal file
29
wiki/modules/config.md
Normal file
@@ -0,0 +1,29 @@
|
||||
---
|
||||
type: module
|
||||
path: "src/config.ts"
|
||||
status: active
|
||||
language: typescript
|
||||
purpose: "Env-driven config loaders for the relay + Foundry host control."
|
||||
maintainer: Kaysser Kayyali
|
||||
last_updated: 2026-06-22
|
||||
linked_issues: []
|
||||
depends_on: []
|
||||
used_by: [[server]], [[cli]], [[relay-client]]
|
||||
tags: [module]
|
||||
created: 2026-06-22
|
||||
updated: 2026-06-22
|
||||
---
|
||||
|
||||
# config
|
||||
|
||||
`src/config.ts` — `loadRelayConfig` / `loadFoundryConfig` (and the `RelayConfig` /
|
||||
`FoundryHostConfig` types). Reads `RELAY_URL`, `RELAY_API_KEY`, `RELAY_CLIENT_ID` for the
|
||||
relay; `FOUNDRY_CONTAINER`, `FOUNDRY_DATA_DIR`, world for host control. See `.env.example`
|
||||
for the full var list.
|
||||
|
||||
## Notes
|
||||
- The relay is enabled in the dashboard iff `RELAY_API_KEY` is set.
|
||||
- `FOUNDRY_CONTAINER` / `FOUNDRY_DATA_DIR` are only used by `refresh --full-index` (stop/start
|
||||
a local Foundry container + read its live LevelDB). Live push and plain refresh go through
|
||||
the relay and never touch them. See [[003-relay-no-journal-db-no-docker-stop]].
|
||||
- This page is a stub — expand with the exact env→field mapping.
|
||||
48
wiki/modules/push.md
Normal file
48
wiki/modules/push.md
Normal file
@@ -0,0 +1,48 @@
|
||||
---
|
||||
type: module
|
||||
path: "src/push.ts"
|
||||
status: active
|
||||
language: typescript
|
||||
purpose: "Push one refined note into LIVE Foundry via the relay (Foundry keeps running)."
|
||||
maintainer: Kaysser Kayyali
|
||||
last_updated: 2026-06-22
|
||||
linked_issues: []
|
||||
depends_on: [[relay-client]], [[name-uuid-resolver]], [[foundry-block]], foundry/assets.ts, toFoundry.ts
|
||||
used_by: [[server]], [[autosync]], [[cli]]
|
||||
tags: [module]
|
||||
created: 2026-06-22
|
||||
updated: 2026-06-22
|
||||
---
|
||||
|
||||
# push
|
||||
|
||||
`src/push.ts` — `pushNote(deps)` and the pure `buildPushPayload(md, noteName, liveEntry, resolver, imageOverride?)`.
|
||||
|
||||
## pushNote flow (see [[push-flow]])
|
||||
1. Read the note; pre-process body `![[image]]` embeds (upload co-located files to Foundry's
|
||||
uploads dir, rewrite to ``, drop non-local). Frontmatter untouched.
|
||||
2. `readNoteFoundryMeta` → `foundry.cc_uuid` (throws if absent — "run seed first") + portrait.
|
||||
3. Resolve the name↔uuid map: preloaded `resolver`, else `<out>/name-uuid.json`, else build via
|
||||
relay `/search`.
|
||||
4. `relay.getEntry(id)` — fetch the LIVE entry (preserves ownership/folder/pages/existing image).
|
||||
5. Portrait upload if the note has one (and `foundryDataDir`/`world` set).
|
||||
6. `buildPushPayload` → `obsidianToFoundryJsonLive` (full entry with `name` + `flags.campaign-codex`
|
||||
overridden, links resolved) → minimal diff `{ name, "flags.campaign-codex": cc }` (dot-path merge
|
||||
preserves sibling flags; never echoes `_id`/`pages`/`ownership`).
|
||||
7. `dryRun` → return diff. Apply → back up the live entry to `<out>/bak/<name>.<stamp>.json`
|
||||
(reversible), then `relay.updateEntry(id, diff)`.
|
||||
|
||||
## Critical: no internal idempotency
|
||||
`pushNote` ALWAYS PUTs (unless `dryRun`). It does NOT compare the note body to anything. The
|
||||
caller is responsible for gating: the dashboard's push-all and [[autosync]] call `baselineNote`
|
||||
afterward to set `foundry.contentHash = contentHash(body)` so a re-run skips unchanged notes.
|
||||
Without that, repeated pushes re-send the same diff.
|
||||
|
||||
## Depends on
|
||||
- [[relay-client]], [[name-uuid-resolver]], [[foundry-block]] (`splitFrontmatter`, `readFoundryBlock`),
|
||||
`src/foundry/assets.ts` (portrait + body-image upload), `src/toFoundry.ts` (`obsidianToFoundryJsonLive`).
|
||||
|
||||
## Notes / gotchas
|
||||
- Image upload needs `FOUNDRY_DATA_DIR` + `FOUNDRY_WORLD`; without them, images are skipped and
|
||||
existing ones kept. See [[foundry-uploads-convention]] (in the user's memory, not this wiki).
|
||||
- `skipImageUpload` is used for fast batch dry-run previews that shouldn't touch the Foundry data dir.
|
||||
43
wiki/modules/relay-client.md
Normal file
43
wiki/modules/relay-client.md
Normal file
@@ -0,0 +1,43 @@
|
||||
---
|
||||
type: module
|
||||
path: "src/relay/client.ts"
|
||||
status: active
|
||||
language: typescript
|
||||
purpose: "HTTP client over the ThreeHats foundryvtt-rest-api-relay (x-api-key auth, clientId query param)."
|
||||
maintainer: Kaysser Kayyali
|
||||
last_updated: 2026-06-22
|
||||
linked_issues: []
|
||||
depends_on: [[threehats-relay]], [[config]]
|
||||
used_by: [[push]], [[server]], [[cli]]
|
||||
tags: [module]
|
||||
created: 2026-06-22
|
||||
updated: 2026-06-22
|
||||
---
|
||||
|
||||
# relay-client
|
||||
|
||||
`src/relay/client.ts` — `RelayClient(cfg)` wraps the relay's pass-through REST API. See
|
||||
`docs/relay-api.md` for envelope shapes.
|
||||
|
||||
## Methods
|
||||
| Method | Relay call | Returns |
|
||||
|---|---|---|
|
||||
| `getEntry(uuid)` | `GET /get?uuid=` | `data: <JournalEntry>` |
|
||||
| `updateEntry(uuid, diff)` | `PUT /update?uuid=` body `{data:<diff>}` | `entity: [<doc>]` |
|
||||
| `createEntry(entityType, data)` | `POST /create` | `{uuid, data}` |
|
||||
| `searchJournalEntries()` | `GET /search?filter=documentType:JournalEntry&minified=true` | `results: [{uuid,id,name,img}]` |
|
||||
|
||||
## Mechanics
|
||||
- Base URL: no `/api` prefix. Auth: `x-api-key` header on every request.
|
||||
- **World selection**: `clientId` query param. Omit only when exactly one Foundry client is
|
||||
connected to the key (relay auto-resolves); 0 → `404`, >1 → `400` listing clients.
|
||||
- The relay forwards to Foundry's rest-api module over WebSocket and returns the module's
|
||||
response verbatim. Envelopes are per-endpoint (`data` / `entity` / `results`).
|
||||
- WS round-trip timeout ~10s → `408`/`504`.
|
||||
|
||||
## Notes / gotchas
|
||||
- For `update`, send a minimal diff (dot-path `flags.campaign-codex`), never the full doc —
|
||||
echoing `_id`/`pages`/`ownership` would clobber the live entry.
|
||||
- `/search` is **minified** (`{uuid,id,name,img}`) — no content hash. So you cannot detect a
|
||||
Foundry content change from `/search` alone; you'd need `/get` per entry. This is why
|
||||
Foundry→Obsidian auto-sync wasn't built ([[005-autosync-o-to-f-only]]).
|
||||
59
wiki/modules/server.md
Normal file
59
wiki/modules/server.md
Normal file
@@ -0,0 +1,59 @@
|
||||
---
|
||||
type: module
|
||||
path: "src/server.ts"
|
||||
status: active
|
||||
language: typescript
|
||||
purpose: "Local review dashboard + JSON API over the batch engine; binds 0.0.0.0:7788."
|
||||
maintainer: Kaysser Kayyali
|
||||
last_updated: 2026-06-22
|
||||
linked_issues: []
|
||||
depends_on: [[batch]], [[push]], [[relay-client]], [[content-hash]], [[foundry-block]], [[name-uuid-resolver]]
|
||||
used_by: [[cli]]
|
||||
tags: [module]
|
||||
created: 2026-06-22
|
||||
updated: 2026-06-22
|
||||
---
|
||||
|
||||
# server
|
||||
|
||||
`src/server.ts` — `startServer(cfg)` builds the index once, serves `dashboard.html` at `/`
|
||||
and a JSON API under `/api/*`. The mode (`dev` | `apply`) is fixed at startup; the UI cannot
|
||||
escalate to apply unless the server was started with `--apply`.
|
||||
|
||||
## Mode + mirror-aware reads
|
||||
- **dev**: writes land under `--out/<bucket>/<relPath>` (a mirror). `resolveRefined` /
|
||||
`resolveCc` read the mirror only if it exists AND is at least as new (mtime) as the real
|
||||
file — otherwise the real vault (edited in Obsidian) wins. This makes dev a true preview
|
||||
that never masks live vault edits.
|
||||
- **apply**: `targetPath` writes the real refined/cc dir; `writeWithBackup` copies a
|
||||
`.bak-<stamp>` first (reversible).
|
||||
- dry-run never writes; it collects `preview` entries.
|
||||
|
||||
## Routes
|
||||
| Method | Path | Handler |
|
||||
|---|---|---|
|
||||
| GET | `/` | `dashboard.html` |
|
||||
| GET | `/api/index` | rebuild index, return `IndexResult` (rebuilt every request — hash-only, cheap) |
|
||||
| GET | `/api/status` | `{mode, refinedDir, ccDir, outDir}` |
|
||||
| GET | `/api/file?name=` | per-row detail + seed/sync/re-pull previews |
|
||||
| GET | `/api/entries` | Foundry journal entries (name+uuid+type) from the snapshot, for the Link picker |
|
||||
| POST | `/api/action` | `runAction` — seed/sync/repull/import + All variants |
|
||||
| POST | `/api/push` | [[push]] one note (live) |
|
||||
| POST | `/api/push-all` | push every vault-newer matched note (live), baseline each |
|
||||
| POST | `/api/link` | inject the `foundry:` identity block (link a refined-only note by uuid) |
|
||||
| POST | `/api/refresh` | rebuild `name-uuid.json` via relay `/search` |
|
||||
| GET/POST | `/api/autosync` | [[autosync]] status / toggle |
|
||||
|
||||
## State
|
||||
`{ db, cfg, index, autosync }`. `index` is rebuilt on every `/api/index` and after mutating
|
||||
ops (push-all, link) so recommendations reflect current files.
|
||||
|
||||
## Depends on
|
||||
- [[batch]] (`indexAll`, row builders), [[push]] (`pushNote`), [[relay-client]], [[content-hash]],
|
||||
[[foundry-block]] (`readFoundryBlock`, `splitFrontmatter`), [[name-uuid-resolver]].
|
||||
|
||||
## Notes / gotchas
|
||||
- The index's drift detection compares against LOCAL baselines + the LevelDB snapshot — not
|
||||
live Foundry. Live edits are only seen via the relay push/refresh path. See [[index-recommend]].
|
||||
- `pushNote` has no internal idempotency; the dashboard's push-all (and [[autosync]]) baseline
|
||||
the note's `foundry.contentHash` afterward so re-runs skip unchanged notes.
|
||||
Reference in New Issue
Block a user