lore-engine-poc/planning-artifacts/architecture.md

Architecture — Lore Engine × GraphMCP Substrate Merge

Template: BMAD architecture at _bmad-output/planning-artifacts/architecture.md (and mirrored at meta/architecture.md). This file MUST live at planning-artifacts/architecture.md exactly — the orchestrator's spec-refiner hardcodes this path.

Date: 2026-06-26 Companion to: meta/prd.md, meta/epics.md Companion to design: Lore Engine design docs (00-17, especially 06-ingestion, 08-architecture, 17-planes) Companion to ADR: 2026-06-26 Lore Engine GraphMCP Merge + research file

Substrate catalog (Phase 0, the merge gate): docs/merge/00-inventory.md. Every GraphMCP worker, MCP tool, and Redis stream the merged runtime must preserve, replace, or deprecate is enumerated there. Anything not in the inventory is out of scope for this architecture until it lands.

---

1. System context

┌────────────────────────────────────────────────────────────────────────┐
│                      USER: Kay (DM/operator)                            │
│  - authors YAML specs in mardonar-specs                                │
│  - inspects Neo4j Browser at :7474                                     │
│  - runs the bot, plays encounters, queries lore                        │
└──────────────┬─────────────────────────────────────────────────────────┘
               │
               ▼
┌────────────────────────────────────────────────────────────────────────┐
│                   BUILD-TIME: SPECS INJECT                              │
│  Dockerfile SPECS_GIT_URL=mardonar-specs ─────▶ mardonar-bot image     │
│  (the bot image is self-contained, no runtime fetch)                   │
└────────────────────────────────────────────────────────────────────────┘

RUNTIME:
┌──────────────┐         ┌─────────────────────────────────────────────────┐
│ mardonar-bot │ ──JSON─▶│  lore-gateway :8765  (Python FastAPI plugin      │
│ (Discord)    │ ──RPC──▶│   server, MCP HTTP+SSE)                          │
│              │         │   plugins: world, lineage, trade, images,        │
│              │         │   embeddings, consistency, nsc (NEW), planes     │
│              │         │   (NEW)                                          │
│              │         └─────┬───────────────────────────────────────────┘
└──────────────┘               │
                               │ reads/writes Neo4j
                               ▼
┌────────────────────────────────────────────────────────────────────────┐
│                       lore-neo4j  :7474 / :7687                        │
│  Graph stores: Plane/Setting/Person/Faction/Encounter/LoreFragment/  │
│  Era/Lineage/Calendar/Culture/Deity/MagicSystem/Spell/Language/...    │
│  Edges: WITNESSED, FEATURES, OCCURRED_AT, EXISTS_IN, LOCATED_IN,      │
│  REFLECTS, LAYER_OF, ADJACENT_TO, ACCESSIBLE_VIA, ALLIED_WITH, ...    │
└────────────────────────────────────────────────────────────────────────┘
       ▲                                            ▲
       │ writes                                     │ reads (embeddings)
       │                                            │
┌──────┴─────────────────────────────────────────────────────┐ ┌─────────┴──────┐
│         INGESTION LAYER (Go workers, Redis Streams)        │ │  Polyglot      │
│                                                            │ │  storage       │
│  ┌─────────────────────────────────────────────────────┐ │ │                │
│  │ Redis Streams (4 inherited + 2 new):                 │ │ │ ┌────────────┐ │
│  │   raw.discord    → discord-filter → raw.messages     │ │ │ │ lore-      │ │
│  │   raw.messages   → entity-extractor, entity-x2       │ │ │ │ postgres   │ │
│  │   raw.lore       → lore-extractor, lore-extractor-2  │ │ │ │ (pgvector) │ │
│  │   raw.encounters → encounter-processor, encounter-2   │ │ │ └────────────┘ │
│  │   raw.structured (NEW) → structured-ingestor (NEW)   │ │ │ ┌────────────┐ │
│  │   raw.dialogue   (NEW) → dialogue-processor (NEW)    │ │ │ │ lore-minio │ │
│  └─────────────────────────────────────────────────────┘ │ │ │ (images)   │ │
│                                                            │ │ └────────────┘ │
│  Producers (live external feeds + HTTP endpoints):        │ │                │
│    - discord-connector   (Discord API)                    │ │                │
│    - discord-filter      (raw.discord → raw.messages)     │ │                │
│    - lore-watcher        (./lore-data/ filesystem)       │ │                │
│    - ingestion-worker    (HTTP /ingest/lore, /structured, │ │                │
│                            /dialogue, /encounter)         │ │                │
│  [FUTURE: slack-connector, pdf-watcher, etc. — copy       │ │                │
│   connector-template/]                                    │ │                │
└────────────────────────────────────────────────────────────┘

2. Component diagram (post-merge lore-engine-poc/)

kaykayyali/lore-engine-poc/
├── docker-compose.yml              # neo4j + postgres + minio + redis + gateway + 9 workers
├── gateway/
│   ├── server.py                   # FastAPI + MCP JSON-RPC, plugin autoreload
│   └── plugins/
│       ├── world.py                # entity_context, was_true_at, state_at
│       ├── lineage.py              # ancestors_of, descendants_of, lineage_of
│       ├── trade.py                # log_trade, trades_by_buyer, market_price
│       ├── images.py               # register_image, recall_images, search_images_by_caption
│       ├── embeddings.py           # embed_images, search_images_semantic
│       ├── consistency.py          # find_contradictions, find_anachronisms, find_ontology_violations, find_orphans
│       ├── nsc/                    # NEW (NPC Scoping — wraps GraphMCP's 4 NPC tools)
│       │   ├── __init__.py         # exports query_as_npc, log_encounter, list_encounters, search_encounters, get_encounter, get_unresolved
│       │   └── client.py           # httpx client to GraphMCP's Go MCP server (or local if Go MCP is ported)
│       └── planes/                 # NEW (v1.2 plane model)
│           ├── __init__.py         # exports list_planes, entity_planes, entity_planes_at_time, find_plane_violations
│           └── cyper.py            # the Cypher for EXISTS_IN / LOCATED_IN / REFLECTS / LAYER_OF / etc.
├── workers/                        # NEW Go services
│   ├── discord-connector/
│   ├── discord-filter/
│   ├── lore-watcher/
│   ├── ingestion-worker/
│   ├── entity-extractor/  + entity-extractor-2/
│   ├── lore-extractor/    + lore-extractor-2/
│   ├── encounter-processor/ + encounter-processor-2/
│   ├── structured-ingestor/        # NEW
│   ├── dialogue-processor/         # NEW
│   └── connector-template/         # NEW (canonical starter for any new producer)
├── neo4j/init.cypher               # extended with Plane/Setting ontology + plane edges
├── postgres/init.sql               # extended with witness + dialogue tables
├── verify-merge.sh                 # NEW (exercises every plugin + every inherited tool)
└── tests/                          # gained E2E for the merge

3. State shape

3.1 Graph nodes (the merged ontology)

// INHERITED FROM GRAPHMCP
(:Person     {id, name, tier})             // tier: commoner | merchant | noble | spy | scholar
(:Location   {id, name, kind})
(:Faction    {id, name})
(:Event      {id, name, in_fiction_date})
(:Item       {id, name})
(:Creature   {id, name})
(:Encounter  {id, title, summary, timestamp})
(:Chunk      {text, embedding})             // 768-dim embedgemma vectors
(:LoreChunk  {text, embedding})
(:LoreFragment {claim, source_doc_id})
(:Message    {content, author, timestamp, msgID})
(:Contradiction {subject, kind, severity, sources})

// NEW FROM LORE-ENGINE
(:Era        {id, name, start_year, end_year})
(:Calendar   {id, name, era_id, year_offset})
(:Lineage    {id, name, founding_ancestor_id})
(:Culture    {id, name, language_id, homeland_id})
(:Deity      {id, name, domain, alignment})
(:MagicSystem {id, name, description})
(:Spell      {id, name, system_id, level, school})
(:Language   {id, name, script})
(:Title      {id, name, holder_id})
(:Artifact   {id, name, owner_id, current_location_id})
(:Region     {id, name, parent_region_id, kind})
(:Setting    {id, name, kind})              // v1.2 (was: world_id string)
(:Plane      {id, name, kind, setting_id})  // v1.2 — material | reflection | transit | outer | demiplane | etc.
(:Dialogue   {text, speaker_id, in_fiction_date, location_id, plane_id})  // NEW — from dialogue-processor

// INHERITED FROM GRAPHMCP, ENHANCED WITH LORE-ENGINE METADATA
(:LoreSource {id, kind, source_type, confidence})   // source_type: prose | timeline | family_tree | gazetteer | bestiary | magic_system | culture
(:Entity     {id, name, lore_verified})             // generic entity for v1.1 extensibility

3.2 Graph edges

// INHERITED
(Person)-[:WITNESSED {since}]->(Encounter)
(Encounter)-[:FEATURES]->(Entity)
(Encounter)-[:OCCURRED_AT]->(Location)
(Encounter)-[:OCCURRED_DURING]->(Era)
(Person)-[:MEMBER_OF {valid_from, valid_until}]->(Faction)
(Person)-[:PARENT_OF]->(Person)
(Person)-[:SPOUSE_OF]->(Person)
(Person)-[:LOCATED_IN {valid_from, valid_until, plane_id}]->(Location)
(Location)-[:PART_OF]->(Location)
(Person)-[:POSSESSES]->(Item)
(Person)-[:PARTICIPATED_IN]->(Event)
(Faction)-[:ALLIED_WITH | :ENEMY_OF]->(Faction)
(Faction)-[:CONCERNS]->(Region)
(Event)-[:FOUNDED_BY]->(Faction)
(LoreFragment)-[:CONTRADICTS]->(LoreFragment)
(LoreChunk)-[:FEATURES]->(Entity)
(Encounter)-[:ABOUT]->(Entity)

// NEW FROM LORE-ENGINE
(Entity)-[:EXISTS_IN]->(Plane)               // timeless type-assertion
(Entity)-[:LOCATED_IN {valid_from, valid_until}]->(Plane)   // time-bounded, redundant w/ Location LOCATED_IN
(Plane)-[:REFLECTS]->(Plane)                 // e.g., Shadowfell REFLECTS Material
(Plane)-[:LAYER_OF]->(Plane)                  // transitive plane composition
(Plane)-[:ADJACENT_TO]->(Plane)              // reachable without transit
(Plane)-[:ACCESSIBLE_VIA]->(Portal | Spell)  // traversal mechanism
(Person)-[:WORSHIPS]->(Deity)
(Person)-[:SPEAKS]->(Language)
(Person)-[:BELONGS_TO]->(Culture)
(Person)-[:MEMBER_OF]->(Lineage)
(Faction)-[:PRACTICED_AT {valid_from, valid_until}]->(Location)
(Setting)-[:CONTAINS]->(Plane)                // setting has planes (each setting has its own Material/Shadowfell/etc.)

3.3 Postgres tables (operational data)

-- INHERITED
trade_log       (id, buyer_id, seller_id, item_id, qty, price_gp, location_id, occurred_at)
image_manifest  (id, entity_id, caption, storage_key, content_type, byte_size, uploaded_at)
image_embedding (id, image_id, embedding vector(768))

-- NEW
witness_attestation (id, person_id, encounter_id, attested_at, source_lv)  -- tracks the -2 replica arbitration
dialogue_log       (id, speaker_id, text, in_fiction_date, location_id, plane_id, encounter_id, received_at)

3.4 Redis Streams

raw.discord     -- chat messages from Discord (raw, unfiltered)
raw.messages    -- canonical messages (after discord-filter dedup/relevance gate)
raw.lore        -- lore documents from filesystem watcher + HTTP ingest
raw.encounters  -- encounter events from Discord (raw, unfiltered) + HTTP log_encounter
raw.structured  -- NEW: YAML structured facts (timeline, family_tree, gazetteer, etc.)
raw.dialogue    -- NEW: in-character NPC dialogue from mardonar-bot

4. Component boundaries (what each piece owns)

Component	Owns	Reads	Writes
mardonar-bot	Discord session lifecycle; LLM narration; name draws; in-world error messages	query_as_npc(name, question) from MCP; encounter spec from ./specs/	log_encounter(...) to MCP (sync); POST /ingest/dialogue (async)
lore-gateway	MCP JSON-RPC dispatch; plugin autoreload; tools/list; SSE sessions	Neo4j (bolt); Postgres (psycopg); MinIO (S3)	Neo4j (Cypher); Postgres (SQL); MinIO (S3 PUT)
workers/ (Go)*	Stream consumption; LLM extraction; dual-LLM arbitration; Cypher writes	Redis Streams; Neo4j	Redis Streams (XADD); Neo4j (Cypher)
ingestion-worker	HTTP chunking + embedding; LLM-callable embedgemma	Redis Streams (XREADGROUP)	Neo4j (Chunk/LoreChunk)
structured-ingestor (NEW)	YAML parsing + deterministic Cypher; no LLM	Redis raw.structured	Neo4j (typed MERGE)
dialogue-processor (NEW)	Parses Dialogue payloads; writes Dialogue nodes	Redis raw.dialogue	Neo4j (Dialogue + LOCATED_IN)
connector-template (NEW)	Canonical Go starter for any new producer	Environment (env-driven)	Redis Streams (XADD)

5. Data flow — a Mardonar encounter (end-to-end)

1. Kay commits `the-clock-maker.yaml` to mardonar-specs
2. Rebuild mardonar-bot image: docker build --build-arg SPECS_GIT_URL=mardonar-specs .
3. Bot loads spec at /app/specs/the-clock-maker.yaml; EncounterSpecSchema validates it
4. Player triggers `/encounter start the-clock-maker` in Discord
5. Bot calls mcp:query_as_npc(name="clockmaker", question="what's my opening line?")
   → MCP gateway → nsc plugin → Neo4j Cypher:
       MATCH (p:Person {name: $name})-[:WITNESSED]->(e:Encounter)-[:FEATURES]->(entity)
       RETURN e, entity ORDER BY e.timestamp DESC LIMIT 5
   → returns: previous encounters the clockmaker has witnessed
6. Bot narrates opening scene (LLM-generated, persona-grounded)
7. Player interacts → bot loops: query_as_npc → LLM narrate
8. Each NPC line → POST /ingest/dialogue → ingestion-worker → XADD raw.dialogue
9. Scene resolves → bot calls mcp:log_encounter(title, participants, summary, location)
   → MCP gateway → nsc plugin → Neo4j Cypher:
       MERGE (enc:Encounter {id: $id})
       WITH enc
       UNWIND $participants AS p_name
       MATCH (p:Person {name: p_name})
       MERGE (p)-[:WITNESSED]->(enc)
       RETURN enc, count(*) as witnesses
   → returns: encounter + witness count
10. Next session, same NPC, same query_as_npc → returns the new encounter

6. Data flow — world-builder ingests a structured fact

1. World-builder writes timeline.yaml for "Battle of Black Spire"
2. World-builder → POST /ingest/structured -F file=@timeline.yaml -F source_type=timeline
3. ingestion-worker validates the multipart; XADD raw.structured with YAML body + source_type tag
4. structured-ingestor (Go) consumes the entry:
   a. Parses YAML (no LLM)
   b. Validates against per-type schema (timeline.yaml vs family_tree.yaml vs gazetteer.yaml etc.)
   c. Materializes Cypher:
       MERGE (era:Era {id: $era_id}) SET era.name = $name, era.start_year = $start
       MERGE (event:Event {id: $event_slug})
         SET event.label = $label, event.in_fiction_date = $date, event.year = $year
       MERGE (event)-[:OCCURRED_DURING]->(era)
       MERGE (event)-[:OCCURRED_AT]->(:Location {id: $location_id})
       WITH event
       UNWIND $participants AS p_id
       MATCH (p:Person {id: p_id})
       MERGE (p)-[:PARTICIPATED_IN {valid_from: $date, valid_until: $date}]->(event)
       RETURN event
5. consistency-engine runs: find_anachronisms, find_plane_violations (no scheduled batch — runs on-demand)
6. World-builder queries mcp:was_true_at(relation=PARTICIPATED_IN, subject="aldric_raventhorne", object="battle_of_black_spire", at_time="3rd_age.year_340")
   → MCP gateway → world plugin → Neo4j Cypher → returns: was_true: true, sources: ["chronicles-vyr.md"]

7. Auth + access model

• No public ingress. The merged MCP runtime is on hp-grey-public.tailcb2b60.ts.net:8765 via Tailscale Serve. Tailnet-only.
• MCP server has no auth layer. It's a private tailnet service. If we ever expose it externally, we add bearer auth (the lore-engine-poc gateway already has Authorization: Bearer plumbing from an earlier spike).
• Bot has DISCORD_TOKEN auth to Discord. Bot-to-MCP traffic is on the trusted tailnet.
• Gitea API token for the bot's git push workflow (if we ever make specs push hot-reload) — currently not implemented.

8. Deploy plan

Step	Action
1	Phase N's PR merged to lore-engine-poc main
2	docker compose -f lore-engine-poc/docker-compose.yml up -d --build
3	Verify with verify-merge.sh (one per phase)
4	Inspect Neo4j Browser at :7474 for the new graph shape
5	Curl :8765/mcp tools/list for the new tool surface
6	Update wiki Projects/Lore Engine.md with the new state
7	Update ADR with the merge SHA

Per-phase rebuild + verify is the cycle. The orchestrator handles the PR-merge-and-build on a 1m cron tick; the dev worker commits + opens the PR; the operator (me) merges after verify-gate is green.

9. Backwards compatibility

• GraphMCP-Example tools: the 8 inherited MCP tools keep their input/output contracts verbatim. get_contradictions becomes an alias for find_contradictions (the new generalized version) for one minor version, then get_contradictions is removed.
• lore-engine-poc plugins: the existing 12 plugins extend, not break, their surface. New params are optional; old params keep their meaning.
• lore-engine-poc data: the v1.2 plane migration is a one-shot Cypher script that adds Setting/Plane nodes, writes EXISTS_IN/LOCATED_IN edges, and collapses the 2 Roland nodes into 1. The script is idempotent — running it twice does nothing the second time.
• world_id string property: deprecated but readable. New writes don't set it. Old reads still work for one minor version, then the property is dropped.
• MULTIVERSE_COUNTERPART_OF relation: removed in the migration. The 2 Roland nodes are collapsed, so the relation has nothing to point to.

10. Open questions for Kay

1. Phase 6 first new source: Slack, RSS feed, Foundry CSV export, GitHub repo watcher, raw paste scraper, or something else? The canonical connector-template/ is built around whichever you pick.
2. Plane taxonomy for non-Mardonari settings (v1.3): Eberron already has data in the lore-engine-poc seed. Darksun, Forgotten Realms, homebrew settings — do we model them as full Settings or as ad-hoc Plane collections under a default unknown setting?
3. Bot's text-to-speech (audio NPC dialogue) is explicitly out-of-scope for v1. If you want to ship v1 with audio, we add it as a v1.1 epic.
4. Multi-party concurrent sessions (multiple Discord sessions running the same bot at once, with shared encounter state) is explicitly out-of-scope for v1. v1.5 if you want it.

11. Verification

Per Verify Gate, every shipped phase ships with verify-merge.sh + docs/VERIFICATION.md. The final rolled-up verify-gate at end-of-epic:

#!/usr/bin/env bash
# verify-merge.sh — exercises every plugin + every inherited tool
set -euo pipefail

cd /root/lore-engine-poc

# 1. All 11 services healthy
docker compose ps --services | xargs -I{} docker compose ps {} | grep -E "Up|healthy" || exit 1

# 2. Plugin tool surface complete (24 tools)
TOOL_COUNT=$(curl -s -X POST http://localhost:8765/mcp \
  -H 'Content-Type: application/json' \
  -d '{"jsonrpc":"2.0","id":1,"method":"tools/list"}' \
  | python3 -c "import json,sys; print(len(json.load(sys.stdin)['result']['tools']))")
[ "$TOOL_COUNT" -ge 24 ] || { echo "FAIL: only $TOOL_COUNT tools"; exit 1; }

# 3. Neo4j has merged ontology
NEO4J=$(docker exec lore-neo4j cypher-shell -u neo4j -p lore-d...n -d neo4j)
echo "$NEO4J" "MATCH (n) WHERE n:Plane OR n:Setting RETURN count(n)" | grep -E "^[0-9]+$" || exit 1

# 4. Lore-engine-poc tests green
bash test.sh

# 5. LLM consumer E2E green
bash examples/run_questions.sh

# 6. Bot integration E2E (Phase 5 onwards)
bash /root/mardonar-npcs/tests/test_bot_encounter.sh

echo "PASSED"

12. See also

• Lore Engine design docs 00-17
• GraphMCP Example — the substrate being merged
• Mardonar Specs — the encounter corpus
• 2026-06-26 Lore Engine GraphMCP Merge — the ADR
• 2026-06-26 Lore Engine GraphMCP Merge Research — research file
• Patterns/Verify Gate — VERIFICATION.md + verify.sh per shipped card
• Patterns/Orchestrator Default Resolves Policy Choices — orchestrator resolves worker design defaults
• Patterns/Three PR 21 Process Rules — serial actions, read stderr verbatim