feat(E4.1): persistent sync-state.json + boot reconcile (Slice 1 foundation)

The first Slice 1 story. A durable aggregate (on/off, mode, parity, activity, last-sync) that survives restarts — the single status source E2/E3/E4 read from. E4.1 only delivers load/save + the boot reconcile; the dashboard migration to read sync-state.json is E4.2-E4.6 (gated behind features.syncStatus, defined here, default off). - src/sync-state.ts: loadSyncState/saveSyncState (atomic: tmp+rename, no truncation on crash), the shape {syncStateSchemaVersion, mode, autoSyncOn, lastSyncAt, parity, watchedDir, activity, updatedAt, conflict:null} (conflict reserved for E3 — E4 forces it null), appendActivity (newest-first, trimmed to 200), defaultSyncState (fresh install: autoSyncOn=false, mode=PREP). Schema mismatch (≠ SYNC_STATE_SCHEMA_VERSION from E0.3) → back up to sync-state.json.bak-<stamp> + fresh defaults + an error event. (Reconciliation: E0.3 froze SYNC_STATE_SCHEMA_VERSION = "sync-state/v1" — E4.1's AC "= 1" is superseded; E4.1 persists autoSyncOn, superseding E1b.5's "no persistence until E3" — a fresh install still defaults OFF; a user who toggled ON is restored ON.) - src/server.ts: features.syncStatus flag (env OFS_SYNC_STATUS=1, default off) on ServerConfig/State. startServer loads sync-state.json + reconciles AutoSyncController.enabled with autoSyncOn — if true, setEnabled(true); on throw (dev mode apply-gate / no relay) → autoSyncOn=false + persist + an "auto-sync could not resume" error event. State.syncState holds the aggregate. - tests/e4-1-syncstate.test.ts: 10 tests — load creates defaults; atomic save (no .tmp left); schema-mismatch backup + fresh + error event; restart survival (autoSyncOn + 17 activity preserved); activity trimmed to 200; conflict forced null; boot reconcile (autoSyncOn=true + apply + relay → enabled restored; no relay → flipped off + error; dev mode → flipped off; fresh → stays off). tsc clean; 222 passing project-wide (18 pre-existing fixture-missing unchanged). Co-Authored-By: Claude <noreply@anthropic.com>
2026-06-23 01:42:13 +00:00
parent 4ae4876695
commit fa4d36dbe4
3 changed files with 295 additions and 1 deletions
--- a/src/server.ts
+++ b/src/server.ts
@@ -24,6 +24,7 @@ import { contentHash } from "./normalize.js";
 import { SyncLock, relPathLockKey } from "./synclock.js";
 import { ccHash } from "./cchash.js";
 import { FLAGS_SCHEMA_VERSION } from "./schema-version.js";
+import { loadSyncState, appendActivity, type SyncState } from "./sync-state.js";
 import { backupStamp } from "./write.js";
 import type { RelayConfig, FoundryHostConfig } from "./config.js";
 import type { Mode, JournalEntry } from "./types.js";
@@ -241,10 +242,12 @@ export interface ServerConfig {
  outDir: string;
  mode: Mode; // dev | apply — fixed at startup
  port: number;
-  host: string; // bind address (0.0.0.0 by default to expose on the tailnet; 127.0.0.1 to restrict to localhost)
+  host: string; // bind address (127.0.0.1 by default; --host 0.0.0.0 to expose, needs a token when ENABLE_AUTH_MIDDLEWARE=on)
  // Live push path (optional). When relayCfg is set, /api/push and /api/refresh work.
  relayCfg?: RelayConfig;
  foundryCfg?: FoundryHostConfig;
+  // E4.1: feature flags read once at boot. syncStatus gates E4.2-E4.6 (default off).
+  features?: { syncStatus: boolean };
 }

 export interface ActionResult {
@@ -262,6 +265,8 @@ export interface State {
  cfg: ServerConfig;
  index: IndexResult | null;
  autosync: AutoSyncController;
+  // E4.1: the persisted sync-state aggregate (loaded at boot, saved on mutation).
+  syncState?: import("./sync-state.js").SyncState;
 }

 function send(res: ServerResponse, code: number, body: unknown): void {
@@ -1450,6 +1455,26 @@ export async function startServer(cfg: ServerConfig): Promise<{ server: Server;
    await state.autosync.migrateFlagsSchemaVersion();
  }

+  // E4.1: features.syncStatus flag (gates E4.2-E4.6; default off). Read once at boot.
+  if (!state.cfg.features) state.cfg.features = { syncStatus: process.env.OFS_SYNC_STATUS === "1" };
+
+  // E4.1: load the persisted sync-state.json (or create defaults). Reconcile the
+  // controller's `enabled` with the persisted `autoSyncOn` — a user who toggled
+  // auto-sync ON is restored ON across restarts (their choice; a fresh install
+  // defaults OFF). If start() throws (dev mode / no relay), flip autoSyncOn off +
+  // persist + log an error event.
+  const { state: syncState, freshened } = await loadSyncState(cfg.outDir, cfg.refinedDir);
+  state.syncState = syncState;
+  if (syncState.autoSyncOn) {
+    try {
+      await state.autosync.setEnabled(true);
+    } catch (e) {
+      syncState.autoSyncOn = false;
+      await appendActivity(cfg.outDir, syncState, { time: new Date().toISOString(), kind: "error", name: "(auto-sync)", status: "error", message: `auto-sync could not resume — ${(e as Error).message}` }).catch(() => {});
+    }
+  }
+  void freshened;
+
  // E7.1: the route table. Every endpoint is declared here with its auth/CSRF
  // requirement + handler; the dispatch loop consults the table before calling
  // the handler. Read routes are requireAuth:false; mutation (POST) routes are
--- a/src/sync-state.ts
+++ b/src/sync-state.ts
@@ -0,0 +1,133 @@
+// E4.1 — persistent sync-state.json (the single status source E2/E3/E4 read from).
+//
+// The AutoSyncController keeps an in-memory view (the dashboard's fast poll); this
+// module is the DURABLE aggregate that survives restarts: on/off, mode, parity,
+// activity (last 200), last-sync. Atomic writes (tmp + rename) so a crash mid-
+// write never leaves a truncated file. Schema-versioned: a mismatch backs up the
+// old file + writes fresh defaults + an error event.
+//
+// E4.1 does NOT change the existing /api/autosync or the dashboard panel — those
+// migrate to read sync-state.json in E4.2-E4.6 (gated behind features.syncStatus,
+// defined here). E4.1 only delivers load/save + the boot reconcile.
+//
+// Reconciliations: (1) E0.3 froze SYNC_STATE_SCHEMA_VERSION = "sync-state/v1"
+// (string); E4.1's AC said "= 1" (number) — E0.3 wins, so the file's
+// syncStateSchemaVersion is the E0.3 string. (2) E4.1 persists autoSyncOn across
+// restarts, superseding E1b.5's "no persistence until E3" — Slice 1 introduces the
+// persistence layer. A fresh install (no sync-state.json) still defaults
+// autoSyncOn=false; a user who explicitly toggled ON is restored ON (their choice).
+
+import { readFile, writeFile, rename, mkdir, stat } from "node:fs/promises";
+import { join, dirname } from "node:path";
+import { FLAGS_SCHEMA_VERSION, SYNC_STATE_SCHEMA_VERSION } from "./schema-version.js";
+import { backupStamp } from "./write.js";
+
+export interface SyncStateActivityEvent {
+  time: string;
+  kind: string; // push | pull | baseline | skip | error | mode | status-note
+  name: string;
+  status: string;
+  message: string;
+}
+
+export interface SyncStateParity {
+  status: string; // in-parity | O-pending | F-pending | conflict | unsynced-linked
+  oPending: number;
+  fPending: number;
+  conflict: number;
+  unsyncedLinked: number;
+  lastPollAt: string | null;
+}
+
+export interface SyncState {
+  syncStateSchemaVersion: string;
+  mode: string; // PREP | RUN-THE-MATCH
+  autoSyncOn: boolean;
+  lastSyncAt: string | null;
+  parity: SyncStateParity;
+  watchedDir: string;
+  activity: SyncStateActivityEvent[];
+  updatedAt: string;
+  conflict: null; // reserved for E3 — E4 MUST NOT populate this
+}
+
+/** Default state for a fresh install (or after a schema-mismatch reset). */
+export function defaultSyncState(watchedDir: string): SyncState {
+  return {
+    syncStateSchemaVersion: SYNC_STATE_SCHEMA_VERSION,
+    mode: "PREP",
+    autoSyncOn: false, // fresh install — OFF (opt-in)
+    lastSyncAt: null,
+    parity: { status: "in-parity", oPending: 0, fPending: 0, conflict: 0, unsyncedLinked: 0, lastPollAt: null },
+    watchedDir,
+    activity: [],
+    updatedAt: new Date().toISOString(),
+    conflict: null,
+  };
+}
+
+/** Trim activity to the last 200 (newest first) on each append. */
+export const MAX_ACTIVITY = 200;
+
+/** E4.1: load sync-state.json from <outDir>/sync-state.json. If absent, create
+ *  defaults. If the schema version mismatches, back up the old file to
+ *  sync-state.json.bak-<stamp> and write fresh defaults (with an error event).
+ *  Returns { state, freshened } where `freshened` is true if a reset happened. */
+export async function loadSyncState(outDir: string, watchedDir: string): Promise<{ state: SyncState; freshened: boolean }> {
+  const path = join(outDir, "sync-state.json");
+  let raw: string | null = null;
+  try { raw = await readFile(path, "utf8"); } catch { /* absent */ }
+  if (!raw) {
+    const state = defaultSyncState(watchedDir);
+    await saveSyncState(outDir, state);
+    return { state, freshened: false };
+  }
+  let parsed: Partial<SyncState>;
+  try { parsed = JSON.parse(raw) as Partial<SyncState>; }
+  catch { parsed = {}; }
+  if (parsed.syncStateSchemaVersion !== SYNC_STATE_SCHEMA_VERSION) {
+    // Schema mismatch — back up the old file + write fresh defaults + an error event.
+    try { await rename(path, `${path}.bak-${backupStamp()}`); } catch { /* best-effort */ }
+    const state = defaultSyncState(watchedDir);
+    state.activity.unshift({ time: new Date().toISOString(), kind: "error", name: "(state)", status: "error", message: `sync-state.json schema reset: ${parsed.syncStateSchemaVersion ?? "(absent)"} → ${SYNC_STATE_SCHEMA_VERSION}` });
+    await saveSyncState(outDir, state);
+    return { state, freshened: true };
+  }
+  // Merge with defaults so missing fields don't break readers (forward-compat).
+  const state: SyncState = {
+    ...defaultSyncState(watchedDir),
+    ...parsed,
+    parity: { ...defaultSyncState(watchedDir).parity, ...(parsed.parity ?? {}) },
+    activity: Array.isArray(parsed.activity) ? parsed.activity.slice(0, MAX_ACTIVITY) : [],
+    conflict: null, // E3 owns this; force null in E4
+  };
+  return { state, freshened: false };
+}
+
+/** E4.1: atomically write sync-state.json (tmp + rename). Never throws out of a
+ *  state-mutation path — callers catch + log (the flag-off-failure-path rule). */
+export async function saveSyncState(outDir: string, state: SyncState): Promise<void> {
+  const path = join(outDir, "sync-state.json");
+  const tmp = `${path}.tmp`;
+  await mkdir(dirname(path), { recursive: true });
+  state.updatedAt = new Date().toISOString();
+  await writeFile(tmp, JSON.stringify(state, null, 2), "utf8");
+  await rename(tmp, path);
+}
+
+/** E4.1: append an activity event + save (atomic). The activity array is newest-
+ *  first, trimmed to MAX_ACTIVITY. Best-effort (never throws). */
+export async function appendActivity(outDir: string, state: SyncState, event: SyncStateActivityEvent): Promise<void> {
+  state.activity.unshift(event);
+  if (state.activity.length > MAX_ACTIVITY) state.activity.length = MAX_ACTIVITY;
+  await saveSyncState(outDir, state).catch(() => { /* best-effort */ });
+}
+
+/** Read the sync-state.json mtime (for tests / diagnostics), or null if absent. */
+export async function syncStateMtime(outDir: string): Promise<number | null> {
+  try { return (await stat(join(outDir, "sync-state.json"))).mtimeMs; } catch { return null; }
+}
+
+// Re-export so E1b's flagsSchemaVersion migration + E4's syncStateSchemaVersion
+// are both reachable from one import (distinct names, distinct owners).
+export { FLAGS_SCHEMA_VERSION };
--- a/tests/e4-1-syncstate.test.ts
+++ b/tests/e4-1-syncstate.test.ts
@@ -0,0 +1,136 @@
+// E4.1 — persistent sync-state.json (load/save/atomic/schema-mismatch/restart/reconcile).
+
+import { describe, it, expect, beforeEach, afterEach } from "vitest";
+import { ClassicLevel } from "classic-level";
+import { mkdtemp, mkdir, rm, readFile, readdir } from "node:fs/promises";
+import { existsSync } from "node:fs";
+import { join } from "node:path";
+import { tmpdir } from "node:os";
+import type { Server } from "node:http";
+
+import { loadSyncState, saveSyncState, appendActivity, defaultSyncState, MAX_ACTIVITY, type SyncState } from "../src/sync-state.js";
+import { SYNC_STATE_SCHEMA_VERSION } from "../src/schema-version.js";
+import { startServer } from "../src/server.js";
+
+let dir: string;
+beforeEach(async () => { dir = await mkdtemp(join(tmpdir(), "e4-1-")); await mkdir(join(dir, "refined"), { recursive: true }); });
+afterEach(async () => { await rm(dir, { recursive: true, force: true }); });
+
+describe("E4.1 SyncState load/save", () => {
+  it("load creates defaults if absent (autoSyncOn=false, mode=PREP) + writes the file", async () => {
+    const { state, freshened } = await loadSyncState(dir, "/vault/refined");
+    expect(freshened).toBe(false);
+    expect(state.autoSyncOn).toBe(false);
+    expect(state.mode).toBe("PREP");
+    expect(state.syncStateSchemaVersion).toBe(SYNC_STATE_SCHEMA_VERSION);
+    expect(state.activity).toEqual([]);
+    expect(state.conflict).toBeNull();
+    expect(existsSync(join(dir, "sync-state.json"))).toBe(true);
+  });
+
+  it("save is atomic (tmp+rename — no .tmp left after)", async () => {
+    const { state } = await loadSyncState(dir, "/vault/refined");
+    state.autoSyncOn = true;
+    await saveSyncState(dir, state);
+    const files = await readdir(dir);
+    expect(files).toContain("sync-state.json");
+    expect(files.some((f) => f.endsWith(".tmp"))).toBe(false);
+    const reloaded = JSON.parse(await readFile(join(dir, "sync-state.json"), "utf8")) as SyncState;
+    expect(reloaded.autoSyncOn).toBe(true);
+  });
+
+  it("schema mismatch → backs up the old file + writes fresh defaults + an error event", async () => {
+    // Write a stale-version file.
+    const stale = { ...defaultSyncState("/vault/refined"), syncStateSchemaVersion: "sync-state/v0" };
+    await saveSyncState(dir, stale);
+    const { state, freshened } = await loadSyncState(dir, "/vault/refined");
+    expect(freshened).toBe(true);
+    expect(state.syncStateSchemaVersion).toBe(SYNC_STATE_SCHEMA_VERSION);
+    expect(state.autoSyncOn).toBe(false); // fresh defaults
+    expect(state.activity.length).toBe(1);
+    expect(state.activity[0].kind).toBe("error");
+    expect(state.activity[0].message).toMatch(/schema reset/);
+    // The old file was backed up (.bak-<stamp>).
+    const files = await readdir(dir);
+    expect(files.some((f) => f.startsWith("sync-state.json.bak-"))).toBe(true);
+  });
+
+  it("restart survival: autoSyncOn + activity preserved across reload", async () => {
+    const { state } = await loadSyncState(dir, "/vault/refined");
+    state.autoSyncOn = true;
+    state.lastSyncAt = "2026-06-23T01:00:00.000Z";
+    for (let i = 0; i < 17; i++) state.activity.push({ time: `2026-06-23T0:${i}:00.000Z`, kind: "push", name: `N${i}`, status: "pushed", message: `m${i}` });
+    await saveSyncState(dir, state);
+    // Reload — the persisted state is restored.
+    const { state: reloaded } = await loadSyncState(dir, "/vault/refined");
+    expect(reloaded.autoSyncOn).toBe(true);
+    expect(reloaded.lastSyncAt).toBe("2026-06-23T01:00:00.000Z");
+    expect(reloaded.activity.length).toBe(17);
+  });
+
+  it("activity is trimmed to MAX_ACTIVITY on append", async () => {
+    const { state } = await loadSyncState(dir, "/vault/refined");
+    for (let i = 0; i < MAX_ACTIVITY + 50; i++) {
+      await appendActivity(dir, state, { time: `t${i}`, kind: "push", name: `N${i}`, status: "pushed", message: `m${i}` });
+    }
+    expect(state.activity.length).toBe(MAX_ACTIVITY);
+    const reloaded = JSON.parse(await readFile(join(dir, "sync-state.json"), "utf8")) as SyncState;
+    expect(reloaded.activity.length).toBe(MAX_ACTIVITY);
+  });
+
+  it("conflict field is forced to null by E4 (reserved for E3)", async () => {
+    // Even if a file has a non-null conflict, E4 forces it null on load.
+    const { state } = await loadSyncState(dir, "/vault/refined");
+    (state as unknown as { conflict: unknown }).conflict = { foo: "bar" };
+    await saveSyncState(dir, state);
+    const { state: reloaded } = await loadSyncState(dir, "/vault/refined");
+    expect(reloaded.conflict).toBeNull();
+  });
+});
+
+describe("E4.1 boot reconcile (startServer restores autoSyncOn)", () => {
+  async function bootWithState(state: Partial<SyncState>, opts: { relay?: boolean; mode?: "dev" | "apply" } = {}): Promise<{ server: Server; enabledAfter: boolean; stateAfter: SyncState }> {
+    // Pre-write sync-state.json in outDir.
+    const outDir = join(dir, "out");
+    await mkdir(outDir, { recursive: true });
+    const full = { ...defaultSyncState(join(dir, "refined")), ...state } as SyncState;
+    await saveSyncState(outDir, full);
+    // Create an empty journal LevelDB.
+    const jdb = new ClassicLevel<string, string>(join(dir, "journal"));
+    await jdb.open(); await jdb.close();
+    const { server, state: srvState } = await startServer({
+      journal: join(dir, "journal"), refinedDir: join(dir, "refined"), ccDir: join(dir, "cc"),
+      outDir, mode: opts.mode ?? "apply", port: 0, host: "127.0.0.1",
+      relayCfg: opts.relay ? { url: "http://relay.test", apiKey: "k", clientId: "c" } : undefined,
+    });
+    const enabledAfter = srvState.autosync.enabled;
+    const stateAfter = JSON.parse(await readFile(join(outDir, "sync-state.json"), "utf8")) as SyncState;
+    // Clean up the watcher (auto-sync may be ON).
+    srvState.autosync.stop();
+    await new Promise<void>((r) => server.close(() => r()));
+    return { server, enabledAfter, stateAfter };
+  }
+
+  it("autoSyncOn=true + apply mode + relay → controller.enabled restored to true", async () => {
+    const { enabledAfter } = await bootWithState({ autoSyncOn: true }, { relay: true, mode: "apply" });
+    expect(enabledAfter).toBe(true);
+  });
+
+  it("autoSyncOn=true + no relay → autoSyncOn flipped to false + error event", async () => {
+    const { enabledAfter, stateAfter } = await bootWithState({ autoSyncOn: true }, { relay: false, mode: "apply" });
+    expect(enabledAfter).toBe(false); // start() threw (no relay) → flipped off
+    expect(stateAfter.autoSyncOn).toBe(false);
+    expect(stateAfter.activity.some((e) => e.message.includes("auto-sync could not resume"))).toBe(true);
+  });
+
+  it("autoSyncOn=true + dev mode (no apply) → autoSyncOn flipped to false (apply-mode gate)", async () => {
+    const { enabledAfter, stateAfter } = await bootWithState({ autoSyncOn: true }, { relay: true, mode: "dev" });
+    expect(enabledAfter).toBe(false); // start() threw (dev mode) → flipped off
+    expect(stateAfter.autoSyncOn).toBe(false);
+  });
+
+  it("autoSyncOn=false (fresh install) → stays off", async () => {
+    const { enabledAfter } = await bootWithState({ autoSyncOn: false }, { relay: true, mode: "apply" });
+    expect(enabledAfter).toBe(false);
+  });
+});