From 0846dacdd93baa071fba59ca3cd97ca4b87ff4d3 Mon Sep 17 00:00:00 2001 From: hermes-agent Date: Fri, 26 Jun 2026 20:23:22 +0000 Subject: [PATCH] =?UTF-8?q?docs(18):=20substrate=20merge=20plan=20?= =?UTF-8?q?=E2=80=94=206-phase=20execution=20roadmap=20for=20the=20lore-en?= =?UTF-8?q?gine=20+=20GraphMCP=20merge?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Phase 0 inventory (gate) → Phase 1 substrate merge → Phase 2 ontology + time + planes → Phase 3 consistency engine → Phase 4 structured + dialogue ingestion → Phase 5 bot integration → Phase 6 connector template + first new source. Each phase produces a PR with verify.sh per [[Verify Gate]]. Cost estimate: ~11.5h wall clock, ~$9.20 LLM at minimax-m3 rates. Comfortable within Ollama $100/mo + Gemini $20/day caps. Open questions (deferred to the phase that needs them): bot language (Python decided 2026-06-26, mardonar-npcs uses Pydantic v2), first new source to template (TBD before phase 6), plane taxonomy for non-Mardonari settings (TBD for v1.3). Companion to [[2026-06-26 Lore Engine GraphMCP Merge]]. --- docs/18-substrate-merge-plan.md | 231 ++++++++++++++++++++++++++++++++ 1 file changed, 231 insertions(+) create mode 100644 docs/18-substrate-merge-plan.md diff --git a/docs/18-substrate-merge-plan.md b/docs/18-substrate-merge-plan.md new file mode 100644 index 0000000..5281ef6 --- /dev/null +++ b/docs/18-substrate-merge-plan.md @@ -0,0 +1,231 @@ +# 18 — Substrate Merge Plan + +The execution plan that turns the lore-engine design into a working +merged runtime that absorbs the [[GraphMCP Example]] substrate and +serves the [[Mardonar Specs]] bot. Companion to [[2026-06-26 Lore +Engine GraphMCP Merge]]. + +## What we're building + +A single MCP runtime that: +- Hosts the 14-node lore ontology + time model + v1.2 planes on top of + GraphMCP's existing Person/Location/Faction/Event/Encounter graph +- Preserves all 7 GraphMCP ingestion workers (Go, Redis Streams) +- Adds 2 new workers (structured-ingestor, dialogue-processor) +- Exposes a unified MCP surface: 8 inherited GraphMCP tools + 12 lore-engine + POC plugins + 4 new v1.2 plane tools + the consistency generalizations +- Lets `mardonar-npcs` (Discord bot) call `query_as_npc` + `log_encounter` + and publish NPC dialogue to `raw.dialogue` + +## Repo layout after the merge + +``` +kaykayyali/lore-engine/ # design docs (unchanged — current main=7d81a76) + docs/ + 00-overview.md ... 17-planes.md + 18-substrate-merge-plan.md # THIS FILE + 19-mcp-server-contract.md # the contract for the merged MCP surface + 20-verification.md # the verify-gate for the merged stack + +kaykayyali/lore-engine-poc/ # runtime — gains workers + structured-ingest + dialogue-processor + docker-compose.yml # neo4j + postgres + minio + redis + gateway + 7 workers + 2 new + plugins/ + world.py, lineage.py, trade.py, images.py, embeddings.py, consistency.py # existing 6 + nsc/ # NEW: NPC scoping plugin (query_as_npc, log_encounter, witness_who) + planes/ # NEW: v1.2 plane tools + workers/ # NEW: Go services (port from GraphMCP-Example) + discord-connector/ # copied, env-adapted + discord-filter/ + lore-watcher/ + ingestion-worker/ + entity-extractor/ + entity-extractor-2/ + lore-extractor/ + lore-extractor-2/ + encounter-processor/ + encounter-processor-2/ + structured-ingestor/ # NEW + dialogue-processor/ # NEW + gateway/server.py # gains the new plugins + neo4j/init.cypher # extended with ontology + time + planes + postgres/init.sql # extended with witness + dialogue tables + verify-merge.sh # NEW: exercises every plugin + every inherited tool + tests/ # gained E2E for the merge + +kaykayyali/mardonar-specs/ # UNCHANGED — content only + *.yaml # 9 encounter specs + +kaykayyali/mardonar-npcs/ # NEW + src/ + spec/loader.py # Pydantic v2 EncounterSpecSchema + bot/main.py # Discord client + bot/dm.py # LLM narrative driver + specs/ # build-time injected from mardonar-specs + Dockerfile # has SPECS_GIT_URL build arg + docker-compose.yml # bot + MCP client config + docs/spec-authoring-guide.md # the canonical guide (moved from README reference) + tests/ + test_spec.py + test_bot_encounter.py +``` + +## Phased execution + +The merge runs as 6 phases. Each phase is one kanban wave with the +shape `[dev-implementation-cards] → [tester-validation] → [integration-gate]`. + +### Phase 0 — Schema + worker inventory (gate) +**Owner:** dev + tester. **Outputs:** one PR with a `docs/merge/00-inventory.md` +listing every worker, every tool, every stream, every Cypher query, every +LLM call in GraphMCP-Example today — so the merge knows what must be +preserved. + +**Why first:** without the inventory, "preserved as-is" is a vibe, not a +contract. The orchestrator auto-rejects any merge card that references +something not in the inventory. + +### Phase 1 — Substrate merge +**Owner:** dev. **Outputs:** lore-engine-poc gains Redis, 7 GraphMCP workers +ported in (with env var adaptations: NEO4J_URL, REDIS_URL, the 3 lore-engine-poc +container names), and all 8 GraphMCP tools re-implemented as a single +`nsc` (NPC scoping) Python plugin. The gateway gains a `tools/list` that +shows the merged 24+ tools. + +**Verification:** `verify-merge.sh` exercises every tool. Neo4j Browser +shows the merged graph. The gateway at `:8765/mcp` advertises the full +surface. `docker compose ps` shows all 11 services healthy. + +### Phase 2 — Ontology + time + planes +**Owner:** dev. **Outputs:** the 14-node ontology + time-bounded relations ++ v1.2 Setting/Plane nodes land on top of GraphMCP's existing nodes via +the lore-engine's existing Cypher schema (`neo4j/init.cypher` is extended). +Migration script: collapse the 2 Roland nodes into 1 with two LOCATED_IN +edges, deprecate the v1.1 `world_id` strings. + +**Verification:** existing 12-test `test.sh` still green + 4 new tests for +the plane model. Neo4j Browser shows Setting/Plane nodes + the plane +relations. `entity_context(Roland)` returns one node with 2 planes. + +### Phase 3 — Consistency engine +**Owner:** dev. **Outputs:** generalize `get_contradictions` → `find_contradictions`, +add `find_anachronisms` + `find_ontology_violations` + `find_orphans` + +`find_plane_violations`. The existing `consistency.py` plugin in lore-engine-poc +gets these — they're already designed in `04-consistency.md`. + +**Verification:** every planted contradiction in the seed data surfaces. +A new adversarial test injects 5 contradictions and verifies all 5 are +caught. + +### Phase 4 — Structured + dialogue ingestion +**Owner:** dev. **Outputs:** 2 new Go workers (structured-ingestor, +dialogue-processor) + the `POST /ingest/structured` + `POST /ingest/dialogue` +HTTP endpoints on `ingestion-worker`. The lore-engine-poc gateway gains +`add_timeline_yaml`, `add_family_tree_yaml`, `add_dialogue` MCP tools +(forwarders to the HTTP endpoints, since the workers are Go). + +**Verification:** `curl -F file=@timeline.yaml -F source_type=timeline +http://localhost:8080/ingest/structured` → 202 + raw.structured stream +entry. structured-ingestor consumes it within 1s. Cypher shows the new +Date/Event/Edge nodes. `find_anachronisms` doesn't flag them. + +### Phase 5 — Bot integration +**Owner:** dev + tester. **Outputs:** `kaykayyali/mardonar-npcs` repo +created with Pydantic v2 EncounterSpecSchema + Discord client + DM +narrative driver. `SPECS_GIT_URL=mardonar-specs` baked into the Dockerfile. +The bot calls the merged MCP server for `query_as_npc` + `log_encounter` +and publishes NPC dialogue to `raw.dialogue`. + +**Verification:** bot loads `the-clock-maker.yaml`, starts an encounter in +Discord, narrates the opening scene, the player interacts, the bot calls +`log_encounter` synchronously, the second `query_as_npc` returns the new +encounter in the witness graph. Live test against the hp-grey-public +deployment. + +### Phase 6 — Connector template + first new source +**Owner:** dev. **Outputs:** `workers/connector-template/` Go service +that becomes the copy-paste starting point for any new `*-connector`. +README explains the 5 producer shapes (connector / watcher / structured / +LLM-extractor / event). Two worked examples: `slack-connector` and +`pdf-watcher` (both real, both wire up to existing extractors). + +**Verification:** `cp -r connector-template slack-connector && sed -i +'s/TEMPLATE/SLACK/' ...` produces a working service that polls a Slack +workspace and writes to `raw.messages`. New workers appear in +`docker compose ps` healthy. + +## Dependencies between phases + +``` +P0 (inventory) + └── P1 (substrate merge) + └── P2 (ontology + time + planes) + └── P3 (consistency engine) + └── P4 (structured + dialogue ingestion) + └── P5 (bot integration) + └── P6 (connector template + first new source) +``` + +P0 → P1 → P2 → P3 → P4 → P5 → P6 is a strict linear chain. P6 can fan out +into multiple `connector-*` workers once the template ships. + +Each phase produces a PR. Each PR ships with `verify.sh` per the +[[Verify Gate]] pattern. Merging into `main` auto-promotes the next +phase's task in kanban. + +## Risk register (excerpt; full version in merge ADR research file) + +| Risk | Mitigation | Phase | +|---|---|---| +| Lore-engine-poc plugins break when GraphMCP worker writes hit Neo4j | Phase 1 includes verify-merge.sh that exercises every plugin against merged stack | P1 | +| WITNESSED-edge semantics drift when plane model lands | Spell out: WITNESSED is Person↔Encounter, NOT Person↔Plane; orthogonal | P2 | +| Two-LLM arbitration (`-2` replicas) writes conflicting nodes | Add `source_lv` property check in `find_contradictions` | P3 | +| `world_id` → plane migration corrupts existing Mardonar data | One-shot Cypher with rollback, run against v1.2 seed; 2 Roland nodes collapse to 1 with two LOCATED_IN | P2 | +| Bot `log_encounter` writes fail during active DM | Sync write is the contract; failure → bot retries; encounter graph is source of truth | P5 | + +## Open questions (deferred to the phase that needs them) + +- **P5:** confirm Python for mardonar-npcs (decision made 2026-06-26; + Pydantic v2 replaces Zod) +- **P6:** pick first real new source to template (Slack / RSS / Foundry + CSV export / other) +- **P2:** plane taxonomy for non-Mardonari settings (Eberron already + has data; Darksun / Forgotten Realms / homebrew TBD) + +## Cost + time estimate + +Per-phase rough budget on the existing worker fleet (dev/tester, +minimax-m3): + +| Phase | Worker turns | Wall clock | LLM cost (est.) | +|---|---|---|---| +| P0 inventory | 60 | 30 min | ~$0.40 | +| P1 substrate merge | 240 | 2 h | ~$1.60 | +| P2 ontology + planes | 240 | 2 h | ~$1.60 | +| P3 consistency engine | 180 | 1.5 h | ~$1.20 | +| P4 structured + dialogue | 240 | 2 h | ~$1.60 | +| P5 bot integration | 240 | 2 h | ~$1.60 | +| P6 connector template | 180 | 1.5 h | ~$1.20 | +| **Total** | **1380 turns** | **~11.5 h** | **~$9.20** | + +Runs cleanly within the Ollama $100/mo + Gemini $20/day caps. Use +minimax-m3 throughout (already the active model). Sub-agent routing per +the [[LLM Cost Caps]] pattern. + +## How to start + +1. **Architecture review gate** — Kay reviews this plan + the merge ADR + research file. (You are here.) +2. **Phase 0 dispatch** — one kanban task to dev: produce the inventory + doc. Worker is gated on approval, not on review of later phases. +3. **Phases 1-6** — kanban board `lore-engine-merge` (new), one wave per + phase. Board watchdog cron at 1m cadence during active phases, 15m + when dormant. + +## See also + +- [[Lore Engine]] — the merged runtime home +- [[GraphMCP Example]] — the substrate being merged in +- [[Mardonar Specs]] — the encounter corpus +- [[2026-06-26 Lore Engine GraphMCP Merge]] — the ADR +- [[Patterns/Verify Gate]] — every shipped phase needs verify.sh +- [[Patterns/Orchestrator Default Resolves Policy Choices]] — + the orchestrator resolves worker design defaults, doesn't escalate +- [[Patterns/Three PR 21 Process Rules]] — serial actions, read stderr + verbatim, don't trust worker local-green when CI is red \ No newline at end of file