# 08 — Architecture The Lore Engine is a **domain layer** on top of [Cognee](https://github.com/topoteretes/cognee), a self-hosted, MIT-licensed knowledge-graph framework. Cognee provides the storage abstraction (**Neo4j** + vector index — pinned by ADR 0008, overriding Cognee's Kuzu default), the LLM-based extraction pipeline, the embedding pipeline, and the agent-native `remember/recall/forget` API. The Lore Engine adds a typed high-fantasy ontology, a temporal model, a consistency engine, NPC knowledge scoping, and a TypeTemplate polymorphic extension system on top. Cognee is the **plumbing**; the Lore Engine is the **domain layer** that the world-builder cares about. ## System diagram ``` ┌─────────────────────────────────────┐ │ World-Builder Authoring │ │ markdown · YAML · dialogue JSON │ └──────────────┬──────────────────────┘ │ ┌────────────────────────────┼────────────────────────────┐ │ │ │ ▼ ▼ ▼ ┌───────────────┐ ┌────────────────────┐ ┌────────────────────┐ │ prose path │ │ structured path │ │ dialogue path │ │ (LLM extract) │ │ (YAML parse) │ │ (HTTP POST) │ └───────┬───────┘ └─────────┬──────────┘ └─────────┬──────────┘ │ │ │ ▼ ▼ ▼ cognee.add() cognee.add() + Lore cognee.add() cognee.cognify() Engine parser + NPC scope │ │ │ └───────────────────────────┼──────────────────────────────┘ │ ▼ ┌─────────────────────────┐ │ Cognee Storage │ │ ───────────────────── │ │ Neo4j (graph) │ │ + Vector store │ │ + Postgres (metadata) │ │ ───────────────────── │ │ Cognee manages: │ │ DataPoint, Chunk, │ │ + Lore Engine types: │ │ Person, Faction, │ │ Location, Era, │ │ Lineage, MagicSystem,│ │ Item, Setting, Plane,│ │ DomainEntity, ... │ └────────────┬────────────┘ │ │ Cypher / cognee.recall() │ ┌────────────▼────────────┐ │ Lore Engine extension │ │ (in-process with │ │ Cognee MCP server) │ │ ───────────────────── │ │ 8 inherited tools │ │ 37 new tools │ │ 10 consistency tools │ └────────────┬────────────┘ │ JSON-RPC over MCP │ ┌────────────▼────────────┐ │ LLM Client │ │ (Claude, gpt-4, etc.) │ │ with system prompt │ │ from reasoning harness│ └─────────────────────────┘ Background jobs (Cognee data-pipelines): ┌─────────────────────────┐ ┌─────────────────────────┐ │ consistency-pipeline │ │ template-watcher │ │ (cron, 03:00 daily) │ │ (file watcher on │ │ runs all rules │ │ ./templates/, │ │ materializes nodes │ │ hot-reload tools) │ └─────────────────────────┘ └─────────────────────────┘ ``` The architecture is **one process, in-process extensions**: Cognee runs its MCP server; the Lore Engine registers as a Cognee data-model extension (typed labels, constraints, indexes) plus a tool extension (the 45 MCP tools). The world-builder never interacts with Cognee directly — the Lore Engine is the surface. The two background jobs are Cognee data-pipelines (no separate Go services): the consistency pipeline runs the 4 rule categories on a schedule and on demand; the template-watcher watches `./templates/` and re-registers the TypeTemplate-driven tools on change. ## Data flow: a question ``` 1. User → LLM Client "Did House Vyr rule Valdorn in 340 TA?" 2. LLM Client → LLM (with system prompt + active context) LLM picks tool: was_true_at(RULED, "House Vyr", "Valdorn", "3rd_age.year_340") 3. LLM Client → Cognee MCP Server (JSON-RPC POST /mcp) {"method": "tools/call", "params": {"name": "was_true_at", "arguments": {...}}} 4. Cognee MCP Server → Lore Engine tool handler → Cognee storage The handler composes a Cypher query and executes it through Cognee's graph adapter (which may be Neo4j, Kuzu, or another Cognee-supported backend): MATCH (f:Faction {name: "House Vyr"})-[r:RULED]->(l:Location {name: "Valdorn"}) WHERE time_in_window("3rd_age.year_340", r.valid_from, r.valid_until) RETURN r, r.valid_from, r.valid_until, sources 5. Neo4j → MCP Server [{valid_from: "3rd_age.year_312", valid_until: "3rd_age.year_360", sources: [...]}] 6. MCP Server → LLM Client (JSON-RPC response) {"was_true": true, "valid_from": "...", "valid_until": "...", "sources": [...]} 7. LLM Client → LLM Adds the result to its context. 8. LLM → User "Yes — House Vyr ruled Valdorn from 312 TA to 360 TA, which covers 340 TA. Sources: chronicles-vyr.md." ``` End-to-end latency target: **<500ms** for a single-tool call, **<2s** for a 3-tool chain. ## Data flow: a plane question (v1.2) The plane model (per `17-planes.md`) is the new substrate for multi-setting, multi-plane worlds. A question that traverses planes looks like this: ``` 1. User → LLM Client "Can Asmodeus reach the Material Plane from the Nine Hells? And what planes is the Roland of Mardonari connected to in 430 TA?" 2. LLM Client → LLM (with system prompt + active context) LLM picks tools: list_accessible_targets(plane='nine_hells'), entity_planes_at_time(entity='roland_raventhorne', at='3rd_age.year_430') 3. LLM Client → Cognee MCP Server (JSON-RPC POST /mcp) Two tool calls, possibly chained. 4. Cognee MCP Server → Lore Engine tool handler → Cognee storage Cypher for accessibility: MATCH (start:Plane {id: 'mardonari.nine_hells'})-[:ACCESSIBLE_VIA|ADJACENT_TO*1..2]->(target:Plane) RETURN DISTINCT target.id, target.kind Cypher for time-bounded location: MATCH (p:Person {id: 'roland_raventhorne'})-[r:LOCATED_IN]->(plane:Plane) WHERE time_in_window('3rd_age.year_430', r.valid_from, r.valid_until) RETURN plane.id, plane.kind, plane.name 5. Neo4j → MCP Server Combined response: { reachable_planes: [...], rolands_planes_at_430: [...] } 6. MCP Server → LLM Client (JSON-RPC response) 7. LLM Client → LLM Adds both results to its context. 8. LLM → User "Yes — Plane Shift (a 7th-level spell) connects the Nine Hells to the Material. And in 430 TA, Roland was in the Mardonari Material Plane (pottery workshop) and had recently returned from a 2-year stint in Voldramir (a Mardonari demiplane)." ``` Plane relations (`REFLECTS`, `LAYER_OF`, `ADJACENT_TO`, `ACCESSIBLE_VIA`) make these questions a single Cypher traversal instead of a multi-step string-matching exercise. ## Data flow: structured ingestion ``` 1. World-builder writes timeline.yaml, family_tree.yaml, gazetteer.yaml. 2. World-builder → POST /ingest/structured (multipart) Files attached, source_type per file. 3. The Lore Engine extension (running in Cognee) routes the upload: - Structured YAML files (timeline, family_tree, gazetteer, bestiary, magic_system, culture) go through the per-type parser and are merged as Cypher. - Prose .md files are forwarded to `cognee.add()` + `cognee.cognify()` for LLM-based extraction. 4. Cognee persists the data; the Lore Engine's typed ontology is what Cognee stores (the labels and constraints live in the Lore Engine data-model extension, registered with Cognee at startup). 5. The consistency pipeline (a Cognee data-pipeline) gets a write event. - Runs the relevant anachronism + contradiction checks on the new edges. - Materializes any new :Contradiction / :Anachronism / :Orphan nodes. 6. Cognee state is now consistent. The next MCP tool call sees the new data. ``` ## Services layout (Lore Engine on Cognee) ``` lore-engine-extension/ [NEW] the Lore Engine on Cognee ├── lore_engine/ Cognee data-model extension package │ ├── __init__.py registers labels, constraints, indexes │ ├── ontology/ typed labels (Person, Faction, ...) │ ├── time_model/ time_in_window, era-tree helpers │ ├── consistency/ 4 rule categories + 10 starter rules │ └── templates/ TypeTemplate validator + registry │ ├── tools/ the 45 MCP tools, in-process with Cognee │ ├── lookup.py Group 1 — disambiguation │ ├── entity_context.py Group 1 │ ├── was_true_at.py Group 2 — time-aware │ ├── state_at.py Group 2 │ ├── list_lineage.py Group 3 │ ├── event_chain.py Group 4 │ ├── lore_about.py Group 5 │ ├── consistency_tools.py Group 6 │ ├── generation_tools.py Group 7 │ └── worldbuilder_tools.py Group 8 │ ├── pipelines/ Cognee data-pipelines │ ├── consistency_pipeline.py nightly + on-demand rule run │ └── template_watcher.py watches ./templates/, hot-reload │ ├── parsers/ structured YAML ingest │ ├── timeline.py │ ├── family_tree.py │ ├── gazetteer.py │ ├── bestiary.py │ ├── magic_system.py │ └── culture.py │ └── schema/ init.cognify / init.cypher ├── init.cypher constraints, indexes └── udfs/ time_in_window, time_windows_overlap ``` The Lore Engine extension is **one Python package** that registers with Cognee at startup. Cognee provides the MCP server, the storage abstraction, the embedding pipeline, and the agent API; the Lore Engine provides the domain types, the time model, the consistency rules, and the 45 tools. There are no separate Go workers; the entire backend is in-process with Cognee. ## Schema bootstrap (new Cypher) The full schema lives in `schema/init.cypher` (to be generated during the build phase). These constraints and indexes are **the Lore Engine data-model extension** that Cognee applies on top of its own schema at startup: ```cypher // New label constraints CREATE CONSTRAINT era_slug IF NOT EXISTS FOR (e:Era) REQUIRE e.slug IS UNIQUE; CREATE CONSTRAINT calendar_name IF NOT EXISTS FOR (c:Calendar) REQUIRE c.name IS UNIQUE; CREATE CONSTRAINT date_slug IF NOT EXISTS FOR (d:Date) REQUIRE d.slug IS UNIQUE; CREATE CONSTRAINT lineage_name IF NOT EXISTS FOR (l:Lineage) REQUIRE l.name IS UNIQUE; CREATE CONSTRAINT culture_name IF NOT EXISTS FOR (c:Culture) REQUIRE c.name IS UNIQUE; CREATE CONSTRAINT magic_system_name IF NOT EXISTS FOR (m:MagicSystem) REQUIRE m.name IS UNIQUE; CREATE CONSTRAINT language_name IF NOT EXISTS FOR (l:Language) REQUIRE l.name IS UNIQUE; CREATE CONSTRAINT deity_name IF NOT EXISTS FOR (d:Deity) REQUIRE d.name IS UNIQUE; CREATE CONSTRAINT spell_name IF NOT EXISTS FOR (s:Spell) REQUIRE s.name IS UNIQUE; CREATE CONSTRAINT title_name IF NOT EXISTS FOR (t:Title) REQUIRE (t.name, t.domain) IS UNIQUE; CREATE CONSTRAINT region_name IF NOT EXISTS FOR (r:Region) REQUIRE r.name IS UNIQUE; CREATE CONSTRAINT material_name IF NOT EXISTS FOR (m:Material) REQUIRE m.name IS UNIQUE; // New violation label constraints CREATE CONSTRAINT anachronism_id IF NOT EXISTS FOR (a:Anachronism) REQUIRE a.id IS UNIQUE; CREATE CONSTRAINT ontology_violation_id IF NOT EXISTS FOR (o:OntologyViolation) REQUIRE o.id IS UNIQUE; CREATE CONSTRAINT orphan_id IF NOT EXISTS FOR (o:Orphan) REQUIRE o.id IS UNIQUE; CREATE CONSTRAINT ontology_rule_id IF NOT EXISTS FOR (r:OntologyRule) REQUIRE r.id IS UNIQUE; CREATE CONSTRAINT consistency_run_id IF NOT EXISTS FOR (c:ConsistencyRun) REQUIRE c.id IS UNIQUE; // Composite index for the most common query: "what was X like at T?" CREATE INDEX relation_time_window IF NOT EXISTS FOR ()-[r:RULED|CONTROLS|LOCATED_IN|MEMBER_OF|PARTICIPATED_IN|ALLIED_WITH|ENEMY_OF|POSSESSES|SPOUSE_OF|PARENT_OF|WORSHIPS|PRACTICES|SPEAKS|BELONGS_TO|CLAIMS_TITLE]-() ON (r.valid_from, r.valid_until); // Index for era-tree traversal CREATE INDEX era_parent IF NOT EXISTS FOR (e:Era) ON (e.parent_era); CREATE INDEX era_slug_idx IF NOT EXISTS FOR (e:Era) ON (e.slug); // Index for violation queries CREATE INDEX anachronism_flagged IF NOT EXISTS FOR (a:Anachronism) ON (a.flagged); CREATE INDEX ontology_violation_flagged IF NOT EXISTS FOR (o:OntologyViolation) ON (o.flagged); CREATE INDEX orphan_flagged IF NOT EXISTS FOR (o:Orphan) ON (o.flagged); ``` ## User-defined functions (UDFs) Two critical UDFs. They live in `schema/udfs/`: ### `time_in_window(t, valid_from, valid_until) → bool` The heart of the time model. See `02-time-model.md` for the full specification. ```cypher // pseudocode, not runnable Cypher RETURN time_in_window('3rd_age.year_345', '3rd_age.year_340', '3rd_age.year_352'); // → true RETURN time_in_window('3rd_age.year_360', '3rd_age.year_340', '3rd_age.year_352'); // → false ``` Implementation notes: - Resolves `current` against the `:Now` config node. - Walks the era tree for parent-era membership. - Treats `null` as open-ended. - Implementation: a **Neo4j user-defined function (Java)** — the graph backend is Neo4j (ADR 0008), so the time model ships as a Java UDF queryable inline in Cypher, not as Python application-layer code. The POC's pure-Python `time_in_window` port is the reference implementation and test oracle for the UDF. ### `time_windows_overlap(from_a, until_a, from_b, until_b) → bool` For contradiction detection. Two windows overlap unless one ends before the other starts. ```cypher RETURN time_windows_overlap('3rd_age.year_340', '3rd_age.year_360', '3rd_age.year_345', '3rd_age.year_370'); // → true ``` The `time_in_window` and `time_windows_overlap` UDFs are the only ones the engine needs. Everything else is regular Cypher. ## Hosting & deployment The Lore Engine on Cognee is **one process** that runs the Cognee MCP server, the storage adapter (Neo4j), and the Lore Engine extension. The hosting story is: - **Cognee** — runs in a Docker container, exposes the MCP server on `:8000` (or whatever port the deployment chooses), manages the storage adapter. The Lore Engine extension is a Python package installed into the Cognee image. - **Storage backend** — Cognee supports Neo4j 5.x (most common in production), Kuzu (embedded, single-binary), and others via the `cognee-neo4j` / `cognee-kuzu` adapters. The Lore Engine does not need to know which one is in use. - **Postgres (v1.1)** — Cognee uses Postgres for metadata (sessions, tasks, the `setting` table). Single instance, managed by Cognee. - **LLM provider** — Cognee is LLM-provider-agnostic; it calls any OpenAI-compatible endpoint. The world-builder points it at a local model or a hosted one. The deployment story is: build a Cognee image with the Lore Engine extension installed, point the storage adapter at Neo4j, set the LLM endpoint env vars, run. **No new infrastructure on top of Cognee**; the Lore Engine is code, not a new service. ## Resource budget (Lore Engine delta on Cognee) | Component | Memory | Notes | |---|---|---| | Cognee MCP server (base) | ~300MB | Cognee is a single Python process; this is its baseline. | | Lore Engine extension (in-process) | ~+80MB | The 45 tools, the typed ontology, the consistency rules. No separate process. | | Neo4j (additional indexes for typed labels) | ~+200MB | 19 v1 + 2 v1.2 + 6 v1.1 + 5 consistency labels. | | Postgres (v1.1) | ~+50MB | Single instance, the `setting`/`lore_event`/`retcon` tables. | | Consistency pipeline (peak) | ~+150MB | Short-lived during nightly run. | | **Total delta over bare Cognee** | **~480MB** | Cheap. | The Lore Engine is **cheaper** to run than the prior GraphMCP-Example stack because the 5+ Go services are gone; everything is in-process with Cognee. ## What is intentionally not in this architecture - **No real-time updates.** The consistency engine runs on a schedule and on demand. Streaming consistency (per-write checks) is for a future v2. - **No LLM in the read path.** Tool calls are pure Cypher. Only `summarize_chain` and `narrate_arc` call an LLM, and only on the generation side. - **No external world-data sources.** Wikipedia imports, fantasy-name generators, etc. are explicitly out. The engine reasons about *the world as defined by its sources*, not the world at large. - **No user authentication.** The MCP server is internal. The reasoning harness runs in a trusted context. - **No graph versioning.** Edits overwrite. If you need history, that's a v2 feature (and a real cost in storage and Cypher complexity). The architecture is a *deliberate floor*. The features we don't have are features we can add when we have evidence they're needed, not features we can remove once added.