lore-engine/docs/08-architecture.md

# 08 — Architecture

The Lore Engine is a **domain layer** on top of [Cognee](https://github.com/topoteretes/cognee), a self-hosted, MIT-licensed knowledge-graph framework. Cognee provides the storage abstraction (**Neo4j** + vector index — pinned by ADR 0008, overriding Cognee's Kuzu default), the LLM-based extraction pipeline, the embedding pipeline, and the agent-native `remember/recall/forget` API. The Lore Engine adds a typed high-fantasy ontology, a temporal model, a consistency engine, NPC knowledge scoping, and a TypeTemplate polymorphic extension system on top.

Cognee is the **plumbing**; the Lore Engine is the **domain layer** that the world-builder cares about.

## System diagram

```
                            ┌─────────────────────────────────────┐
                            │         World-Builder Authoring      │
                            │   markdown · YAML · dialogue JSON    │
                            └──────────────┬──────────────────────┘
                                           │
              ┌────────────────────────────┼────────────────────────────┐
              │                            │                            │
              ▼                            ▼                            ▼
      ┌───────────────┐         ┌────────────────────┐         ┌────────────────────┐
      │ prose path    │         │ structured path    │         │ dialogue path      │
      │ (LLM extract) │         │ (YAML parse)       │         │ (HTTP POST)        │
      └───────┬───────┘         └─────────┬──────────┘         └─────────┬──────────┘
              │                           │                              │
              ▼                           ▼                              ▼
       cognee.add()              cognee.add() + Lore              cognee.add()
       cognee.cognify()          Engine parser                   + NPC scope
              │                           │                              │
              └───────────────────────────┼──────────────────────────────┘
                                           │
                                           ▼
                              ┌─────────────────────────┐
                              │     Cognee Storage      │
                              │  ─────────────────────  │
                              │  Neo4j (graph)          │
                              │  + Vector store         │
                              │  + Postgres (metadata)  │
                              │  ─────────────────────  │
                              │  Cognee manages:        │
                              │  DataPoint, Chunk,      │
                              │  + Lore Engine types:   │
                              │    Person, Faction,     │
                              │    Location, Era,       │
                              │    Lineage, MagicSystem,│
                              │    Item, Setting, Plane,│
                              │    DomainEntity, ...    │
                              └────────────┬────────────┘
                                           │
                                           │  Cypher / cognee.recall()
                                           │
                              ┌────────────▼────────────┐
                              │   Lore Engine extension │
                              │   (in-process with      │
                              │    Cognee MCP server)   │
                              │  ─────────────────────  │
                              │  8 inherited tools      │
                              │  37 new tools           │
                              │  10 consistency tools   │
                              └────────────┬────────────┘
                                           │  JSON-RPC over MCP
                                           │
                              ┌────────────▼────────────┐
                              │   LLM Client            │
                              │   (Claude, gpt-4, etc.) │
                              │   with system prompt    │
                              │   from reasoning harness│
                              └─────────────────────────┘

   Background jobs (Cognee data-pipelines):
   ┌─────────────────────────┐         ┌─────────────────────────┐
   │  consistency-pipeline   │         │  template-watcher        │
   │  (cron, 03:00 daily)    │         │  (file watcher on       │
   │  runs all rules         │         │   ./templates/,         │
   │  materializes nodes     │         │   hot-reload tools)     │
   └─────────────────────────┘         └─────────────────────────┘
```

The architecture is **one process, in-process extensions**: Cognee runs its MCP server; the Lore Engine registers as a Cognee data-model extension (typed labels, constraints, indexes) plus a tool extension (the 45 MCP tools). The world-builder never interacts with Cognee directly — the Lore Engine is the surface.

The two background jobs are Cognee data-pipelines (no separate Go services): the consistency pipeline runs the 4 rule categories on a schedule and on demand; the template-watcher watches `./templates/` and re-registers the TypeTemplate-driven tools on change.

## Data flow: a question

```
1. User → LLM Client
   "Did House Vyr rule Valdorn in 340 TA?"

2. LLM Client → LLM (with system prompt + active context)
   LLM picks tool: was_true_at(RULED, "House Vyr", "Valdorn", "3rd_age.year_340")

3. LLM Client → Cognee MCP Server (JSON-RPC POST /mcp)
   {"method": "tools/call", "params": {"name": "was_true_at", "arguments": {...}}}

4. Cognee MCP Server → Lore Engine tool handler → Cognee storage
   The handler composes a Cypher query and executes it through Cognee's graph adapter (which may be Neo4j, Kuzu, or another Cognee-supported backend):
   MATCH (f:Faction {name: "House Vyr"})-[r:RULED]->(l:Location {name: "Valdorn"})
   WHERE time_in_window("3rd_age.year_340", r.valid_from, r.valid_until)
   RETURN r, r.valid_from, r.valid_until, sources

5. Neo4j → MCP Server
   [{valid_from: "3rd_age.year_312", valid_until: "3rd_age.year_360", sources: [...]}]

6. MCP Server → LLM Client (JSON-RPC response)
   {"was_true": true, "valid_from": "...", "valid_until": "...", "sources": [...]}

7. LLM Client → LLM
   Adds the result to its context.

8. LLM → User
   "Yes — House Vyr ruled Valdorn from 312 TA to 360 TA, which covers 340 TA.
    Sources: chronicles-vyr.md."
```

End-to-end latency target: **<500ms** for a single-tool call, **<2s** for a 3-tool chain.

## Data flow: a plane question (v1.2)

The plane model (per `17-planes.md`) is the new substrate for multi-setting, multi-plane worlds. A question that traverses planes looks like this:

```
1. User → LLM Client
   "Can Asmodeus reach the Material Plane from the Nine Hells? And what planes
    is the Roland of Mardonari connected to in 430 TA?"

2. LLM Client → LLM (with system prompt + active context)
   LLM picks tools: list_accessible_targets(plane='nine_hells'),
                     entity_planes_at_time(entity='roland_raventhorne', at='3rd_age.year_430')

3. LLM Client → Cognee MCP Server (JSON-RPC POST /mcp)
   Two tool calls, possibly chained.

4. Cognee MCP Server → Lore Engine tool handler → Cognee storage
   Cypher for accessibility:
     MATCH (start:Plane {id: 'mardonari.nine_hells'})-[:ACCESSIBLE_VIA|ADJACENT_TO*1..2]->(target:Plane)
     RETURN DISTINCT target.id, target.kind
   Cypher for time-bounded location:
     MATCH (p:Person {id: 'roland_raventhorne'})-[r:LOCATED_IN]->(plane:Plane)
     WHERE time_in_window('3rd_age.year_430', r.valid_from, r.valid_until)
     RETURN plane.id, plane.kind, plane.name

5. Neo4j → MCP Server
   Combined response: { reachable_planes: [...], rolands_planes_at_430: [...] }

6. MCP Server → LLM Client (JSON-RPC response)

7. LLM Client → LLM
   Adds both results to its context.

8. LLM → User
   "Yes — Plane Shift (a 7th-level spell) connects the Nine Hells to the Material.
    And in 430 TA, Roland was in the Mardonari Material Plane (pottery workshop) and
    had recently returned from a 2-year stint in Voldramir (a Mardonari demiplane)."
```

Plane relations (`REFLECTS`, `LAYER_OF`, `ADJACENT_TO`, `ACCESSIBLE_VIA`) make these questions a single Cypher traversal instead of a multi-step string-matching exercise.

## Data flow: structured ingestion

```
1. World-builder writes timeline.yaml, family_tree.yaml, gazetteer.yaml.

2. World-builder → POST /ingest/structured (multipart)
   Files attached, source_type per file.

3. The Lore Engine extension (running in Cognee) routes the upload:
   - Structured YAML files (timeline, family_tree, gazetteer, bestiary, magic_system, culture)
     go through the per-type parser and are merged as Cypher.
   - Prose .md files are forwarded to `cognee.add()` + `cognee.cognify()`
     for LLM-based extraction.

4. Cognee persists the data; the Lore Engine's typed ontology is what
   Cognee stores (the labels and constraints live in the Lore Engine
   data-model extension, registered with Cognee at startup).

5. The consistency pipeline (a Cognee data-pipeline) gets a write event.
   - Runs the relevant anachronism + contradiction checks on the new edges.
   - Materializes any new :Contradiction / :Anachronism / :Orphan nodes.

6. Cognee state is now consistent.
   The next MCP tool call sees the new data.
```

## Services layout (Lore Engine on Cognee)

```
lore-engine-extension/                [NEW]   the Lore Engine on Cognee
├── lore_engine/                      Cognee data-model extension package
│   ├── __init__.py                   registers labels, constraints, indexes
│   ├── ontology/                     typed labels (Person, Faction, ...)
│   ├── time_model/                   time_in_window, era-tree helpers
│   ├── consistency/                  4 rule categories + 10 starter rules
│   └── templates/                    TypeTemplate validator + registry
│
├── tools/                            the 45 MCP tools, in-process with Cognee
│   ├── lookup.py                     Group 1 — disambiguation
│   ├── entity_context.py             Group 1
│   ├── was_true_at.py                Group 2 — time-aware
│   ├── state_at.py                   Group 2
│   ├── list_lineage.py               Group 3
│   ├── event_chain.py                Group 4
│   ├── lore_about.py                 Group 5
│   ├── consistency_tools.py          Group 6
│   ├── generation_tools.py           Group 7
│   └── worldbuilder_tools.py         Group 8
│
├── pipelines/                        Cognee data-pipelines
│   ├── consistency_pipeline.py       nightly + on-demand rule run
│   └── template_watcher.py           watches ./templates/, hot-reload
│
├── parsers/                          structured YAML ingest
│   ├── timeline.py
│   ├── family_tree.py
│   ├── gazetteer.py
│   ├── bestiary.py
│   ├── magic_system.py
│   └── culture.py
│
└── schema/                           init.cognify / init.cypher
    ├── init.cypher                   constraints, indexes
    └── udfs/                         time_in_window, time_windows_overlap
```

The Lore Engine extension is **one Python package** that registers with Cognee at startup. Cognee provides the MCP server, the storage abstraction, the embedding pipeline, and the agent API; the Lore Engine provides the domain types, the time model, the consistency rules, and the 45 tools. There are no separate Go workers; the entire backend is in-process with Cognee.

## Schema bootstrap (new Cypher)

The full schema lives in `schema/init.cypher` (to be generated during the build phase). These constraints and indexes are **the Lore Engine data-model extension** that Cognee applies on top of its own schema at startup:

```cypher
// New label constraints
CREATE CONSTRAINT era_slug IF NOT EXISTS FOR (e:Era) REQUIRE e.slug IS UNIQUE;
CREATE CONSTRAINT calendar_name IF NOT EXISTS FOR (c:Calendar) REQUIRE c.name IS UNIQUE;
CREATE CONSTRAINT date_slug IF NOT EXISTS FOR (d:Date) REQUIRE d.slug IS UNIQUE;
CREATE CONSTRAINT lineage_name IF NOT EXISTS FOR (l:Lineage) REQUIRE l.name IS UNIQUE;
CREATE CONSTRAINT culture_name IF NOT EXISTS FOR (c:Culture) REQUIRE c.name IS UNIQUE;
CREATE CONSTRAINT magic_system_name IF NOT EXISTS FOR (m:MagicSystem) REQUIRE m.name IS UNIQUE;
CREATE CONSTRAINT language_name IF NOT EXISTS FOR (l:Language) REQUIRE l.name IS UNIQUE;
CREATE CONSTRAINT deity_name IF NOT EXISTS FOR (d:Deity) REQUIRE d.name IS UNIQUE;
CREATE CONSTRAINT spell_name IF NOT EXISTS FOR (s:Spell) REQUIRE s.name IS UNIQUE;
CREATE CONSTRAINT title_name IF NOT EXISTS FOR (t:Title) REQUIRE (t.name, t.domain) IS UNIQUE;
CREATE CONSTRAINT region_name IF NOT EXISTS FOR (r:Region) REQUIRE r.name IS UNIQUE;
CREATE CONSTRAINT material_name IF NOT EXISTS FOR (m:Material) REQUIRE m.name IS UNIQUE;

// New violation label constraints
CREATE CONSTRAINT anachronism_id IF NOT EXISTS FOR (a:Anachronism) REQUIRE a.id IS UNIQUE;
CREATE CONSTRAINT ontology_violation_id IF NOT EXISTS FOR (o:OntologyViolation) REQUIRE o.id IS UNIQUE;
CREATE CONSTRAINT orphan_id IF NOT EXISTS FOR (o:Orphan) REQUIRE o.id IS UNIQUE;
CREATE CONSTRAINT ontology_rule_id IF NOT EXISTS FOR (r:OntologyRule) REQUIRE r.id IS UNIQUE;
CREATE CONSTRAINT consistency_run_id IF NOT EXISTS FOR (c:ConsistencyRun) REQUIRE c.id IS UNIQUE;

// Composite index for the most common query: "what was X like at T?"
CREATE INDEX relation_time_window IF NOT EXISTS
FOR ()-[r:RULED|CONTROLS|LOCATED_IN|MEMBER_OF|PARTICIPATED_IN|ALLIED_WITH|ENEMY_OF|POSSESSES|SPOUSE_OF|PARENT_OF|WORSHIPS|PRACTICES|SPEAKS|BELONGS_TO|CLAIMS_TITLE]-()
ON (r.valid_from, r.valid_until);

// Index for era-tree traversal
CREATE INDEX era_parent IF NOT EXISTS FOR (e:Era) ON (e.parent_era);
CREATE INDEX era_slug_idx IF NOT EXISTS FOR (e:Era) ON (e.slug);

// Index for violation queries
CREATE INDEX anachronism_flagged IF NOT EXISTS FOR (a:Anachronism) ON (a.flagged);
CREATE INDEX ontology_violation_flagged IF NOT EXISTS FOR (o:OntologyViolation) ON (o.flagged);
CREATE INDEX orphan_flagged IF NOT EXISTS FOR (o:Orphan) ON (o.flagged);
```

## User-defined functions (UDFs)

Two critical UDFs. They live in `schema/udfs/`:

### `time_in_window(t, valid_from, valid_until) → bool`

The heart of the time model. See `02-time-model.md` for the full specification.

```cypher
// pseudocode, not runnable Cypher
RETURN time_in_window('3rd_age.year_345', '3rd_age.year_340', '3rd_age.year_352');
// → true

RETURN time_in_window('3rd_age.year_360', '3rd_age.year_340', '3rd_age.year_352');
// → false
```

Implementation notes:
- Resolves `current` against the `:Now` config node.
- Walks the era tree for parent-era membership.
- Treats `null` as open-ended.
- Implementation: a **Neo4j user-defined function (Java)** — the graph backend is Neo4j (ADR 0008), so the time model ships as a Java UDF queryable inline in Cypher, not as Python application-layer code. The POC's pure-Python `time_in_window` port is the reference implementation and test oracle for the UDF.

### `time_windows_overlap(from_a, until_a, from_b, until_b) → bool`

For contradiction detection. Two windows overlap unless one ends before the other starts.

```cypher
RETURN time_windows_overlap('3rd_age.year_340', '3rd_age.year_360',
                            '3rd_age.year_345', '3rd_age.year_370');
// → true
```

The `time_in_window` and `time_windows_overlap` UDFs are the only ones the engine needs. Everything else is regular Cypher.

## Hosting & deployment

The Lore Engine on Cognee is **one process** that runs the Cognee MCP server, the storage adapter (Neo4j), and the Lore Engine extension. The hosting story is:

- **Cognee** — runs in a Docker container, exposes the MCP server on `:8000` (or whatever port the deployment chooses), manages the storage adapter. The Lore Engine extension is a Python package installed into the Cognee image.
- **Storage backend** — Cognee supports Neo4j 5.x (most common in production), Kuzu (embedded, single-binary), and others via the `cognee-neo4j` / `cognee-kuzu` adapters. The Lore Engine does not need to know which one is in use.
- **Postgres (v1.1)** — Cognee uses Postgres for metadata (sessions, tasks, the `setting` table). Single instance, managed by Cognee.
- **LLM provider** — Cognee is LLM-provider-agnostic; it calls any OpenAI-compatible endpoint. The world-builder points it at a local model or a hosted one.

The deployment story is: build a Cognee image with the Lore Engine extension installed, point the storage adapter at Neo4j, set the LLM endpoint env vars, run. **No new infrastructure on top of Cognee**; the Lore Engine is code, not a new service.

## Resource budget (Lore Engine delta on Cognee)

| Component | Memory | Notes |
|---|---|---|
| Cognee MCP server (base) | ~300MB | Cognee is a single Python process; this is its baseline. |
| Lore Engine extension (in-process) | ~+80MB | The 45 tools, the typed ontology, the consistency rules. No separate process. |
| Neo4j (additional indexes for typed labels) | ~+200MB | 19 v1 + 2 v1.2 + 6 v1.1 + 5 consistency labels. |
| Postgres (v1.1) | ~+50MB | Single instance, the `setting`/`lore_event`/`retcon` tables. |
| Consistency pipeline (peak) | ~+150MB | Short-lived during nightly run. |
| **Total delta over bare Cognee** | **~480MB** | Cheap. |

The Lore Engine is **cheaper** to run than the prior GraphMCP-Example stack because the 5+ Go services are gone; everything is in-process with Cognee.

## What is intentionally not in this architecture

- **No real-time updates.** The consistency engine runs on a schedule and on demand. Streaming consistency (per-write checks) is for a future v2.
- **No LLM in the read path.** Tool calls are pure Cypher. Only `summarize_chain` and `narrate_arc` call an LLM, and only on the generation side.
- **No external world-data sources.** Wikipedia imports, fantasy-name generators, etc. are explicitly out. The engine reasons about *the world as defined by its sources*, not the world at large.
- **No user authentication.** The MCP server is internal. The reasoning harness runs in a trusted context.
- **No graph versioning.** Edits overwrite. If you need history, that's a v2 feature (and a real cost in storage and Cypher complexity).

The architecture is a *deliberate floor*. The features we don't have are features we can add when we have evidence they're needed, not features we can remove once added.