lore-engine/docs/00-overview.md

# 00 — Overview

## What this is

A Lore Engine is a **reasoning substrate** for a high-fantasy world. It is not a CRM, not a wiki, and not a search engine. Its single product is this: *when an LLM asks a question about the world, it gets a precise, time-bound, contradiction-checked answer from the canonical source — and the answer can be traced back to the document it came from.*

The Lore Engine is built on top of [Cognee](https://github.com/topoteretes/cognee) — a self-hosted, MIT-licensed knowledge-graph framework with ~17.8k stars and an Apache 2.0 paper. Cognee provides the storage abstraction (**Neo4j** — pinned by ADR 0008, overriding Cognee's Kuzu default), the extraction pipeline (LLM-based entity/relation extraction), the embedding pipeline, and the agent-native `remember/recall/forget` API. The Lore Engine adds a typed high-fantasy ontology, a temporal model, a consistency engine, NPC knowledge scoping, and a TypeTemplate polymorphic extension system on top. Cognee is the substrate; the Lore Engine is the domain layer.

## Design goals, ranked

1. **Historical accuracy** — the LLM must not be able to confuse "Aldric's father" with "Aldric's son," or claim that a character did X at a time when X had not yet been invented, or state a fact that contradicts a lore document.
2. **Macro↔micro association** — micro details (this dagger, this scar, this debt) must be reachable from macro context (this house, this age, this war) without manual joins.
3. **Logical thinking** — the LLM must be able to chain facts: if `A ruled X` and `B succeeded A` then `B rules X`, and the engine should support that inference path, not just return strings.
4. **Modular tools** — each MCP tool does one job well; higher-level patterns are compositions, not mega-tools.
5. **Source attribution** — every claim returned to the LLM is tagged with the document it came from. The LLM can then say *"according to the chronicles of House Vyr..."* or know to discount a single low-confidence source.
6. **Failure surfacing** — the engine reports what it *doesn't* know as loudly as what it does. Missing lineage, orphaned locations, temporal gaps — all surfaced via dedicated tools, never silently dropped.

## What we inherit from Cognee

| Component | Status | What we keep |
|---|---|---|
| Storage abstraction (Neo4j 5.x + vector index — ADR 0008) | Production | Substrate. We extend the schema, not replace it. Cognee stores the data; we add the typed labels and constraints. |
| Extraction pipeline (LLM-based entity/relation extraction) | Production | Prose path. Cognee handles the "give me a chunk of markdown, get me triples" step; we customize the prompt to emit Lore Engine typed entities. |
| Embedding pipeline (vector embeddings of chunks + entities) | Production | Semantic search back-end. Cognee manages the embedding store; we query it through the `lore_about` and `cite` tools. |
| Agent-native `remember/recall/forget` API | Production | Storage-agnostic interface. The Lore Engine wraps `recall` with typed ontology + time model. |
| `DataPoint` / `Entity` schema | Production | Base node types. The Lore Engine adds `Person`, `Faction`, `Location`, etc. as typed extensions. |
| Session / task registry | Production | The Lore Engine ships its **own** MCP server exposing the 45 typed tools; it calls Cognee's Python API (`remember`/`recall`/`add_data_points`) in-process. Cognee's `cognee-mcp` (a fixed 14-tool surface) is a separate reference server — we don't register our tools into it. |

## What we add

- **Deeper ontology** — Era, Calendar, Lineage, Culture, Deity, MagicSystem, Spell, Language, Title, Item (covers Artifact), Region, `Relation` (reified-edge node, promoted to v1 per ADR 0009), plus 2 v1.2 nodes (Plane, Setting) and 5 consistency nodes. Roughly 36 node labels total: 7 base (Cognee `DataPoint` originals), the v1 core entities + relations (incl. `Relation`), 2 v1.2 plane labels, 5 v1.1 polymorphic labels, and 5 consistency labels. The 7 base types are Lore Engine originals built on Cognee's `DataPoint` — not inherited as-is (Cognee ships no `Contradiction`/`Message`/etc. nodes of its own). The v1.1 docs (`11-extensibility.md`) add the polymorphic `DomainEntity`, `TypeTemplate`, `NPC`, `PC`, and `Human` labels (`Relation` graduated to v1).
- **Time as a first-class concept** — temporal validity windows on relations, era filters, "was X true at time T?" via dedicated tools. The LLM no longer has to guess which version of Aldric it is talking to.
- **Structured ingestion** — not just `.md` files. We ingest `timeline.yaml`, `family-tree.yaml`, `gazetteer.yaml`, `bestiary.yaml` as first-class sources. Free prose stays, but is no longer the only path.
- **A consistency engine** — Cypher-based rules that flag anachronisms (Aldric can't be at the Battle of Black Spire if it happened 200 years before his birth), missing lineage (a noble with no recorded parents), and ontological violations (a region claimed to be inside two non-overlapping kingdoms).
- **A new generation tool** — `narrate_arc` and `summarize_chain` let the LLM produce in-character responses grounded in the graph instead of freehand hallucinations.
- **A reasoning harness doc** — a written contract for the LLM on *how* to use these tools, including the 5 most important question types and the exact tool sequence for each.

## What we explicitly do NOT do

- We do not host narrative prose. The Lore Engine is the *ground truth* layer; the LLM writes the prose.
- We do not try to model everything. Magic systems, prophecy, divine intervention, planes, and the metaphysical weirdness of high fantasy all get *named* in the ontology so they can be referenced, but the engine does not try to *compute* them.
- We do not implement in-world physics. The engine does not adjudicate whether a fireball would hit a dragon. It tells you what the world's books say about fireballs and dragons.
- We do not invent details the LLM might want. If a fact isn't in the graph, the tool returns `null` or `not_found` and the LLM is told to say so.

## How a question flows

```
User: "Did House Vyr hold the Crimson Throne during the Second Age?"
            │
            ▼
LLM picks tool: was_true_at("RULED", "House Vyr", "Crimson Throne", "2nd_age")
            │
            ▼
MCP server (Cognee-hosted) → Lore Engine extension → Neo4j Cypher
  MATCH (f:Faction {name: "House Vyr"})-[r:RULED]->(t:Throne {name: "Crimson Throne"})
  WHERE time_in_window('2nd_age.year_120', r.valid_from, r.valid_until)
  RETURN f, t, r, sources
            │
            ▼
LLM gets: { "was_true": true, "valid_from": "2nd_age.year_120", "valid_until": "2nd_age.year_340", "sources": ["chronicles-vyr.md"], "confidence": 0.92 }
            │
            ▼
LLM responds to user in narrative voice, citing the source if asked.
```

That is the contract. Every question in the engine follows that shape: **typed input → Cypher → typed, source-attributed output.**

## Naming conventions

- **Node labels:** PascalCase, singular. `Person`, `Faction`, `Era`, `Lineage`.
- **Edge types:** UPPER_SNAKE_CASE, verb-form, directional. `RULED`, `PARENT_OF`, `OCCURRED_DURING`, `LOCATED_IN`.
- **Properties:** camelCase. `lore_verified`, `temporal_hint`, `source_doc_id`.
- **MCP tool names:** snake_case, verb-noun. `query_faction_at_time`, `list_lineage`, `flag_anachronism`.
- **Time references:** always `{era}.{year}` or `{era}` alone. E.g. `2nd_age`, `3rd_age.year_340`. See `02-time-model.md`.

## How to read this design

If you only have 10 minutes: read this doc, then `01-ontology.md`, then the tool catalog headers in `05-mcp-tools.md`.

If you have an hour: read it in order. Each doc assumes the previous.

If you want to challenge it: jump to `10-critique.md` first. I tried to break it.

## The full doc set

**Core design (00–17):** the engine itself.

**Operations & integration (18–22):** how to run it.

- `18-eval-policy.md` — thresholds + cadence for the 50-question harness
- `19-retcon-policy.md` — how retcons are declared, preserved, surfaced
- `20-multi-setting-policy.md` — cross-setting queries, `settings.yaml`
- `21-quickstart.md` — 1-page "first 5 minutes"
- `22-cognee-boundary.md` — substrate/domain contract

**Specialized registries:**

- `prompts/` — the prompt registry (system-prompt mirror + extraction-prompt stub)
- `models/` — the model registry (Minimax-M3 notes, cross-vendor sanity-check models)
- `cognee-integration.md` — the recipe for overriding Cognee's extraction prompt and routing through LiteLLM

**ADRs (`adr/`):** hard-to-reverse decisions.

- `0001-aggregate-confidence-floor.md` — aggregate = min of per-source (extraction × source)
- `0002-disputed-edges-stay-separate.md` — conflicting bounds produce two edges
- `0003-lineage-vs-faction.md` — Lineage is blood, Faction is association
- `0004-region-vs-plane.md` — Region is geographic, Plane is mode of existence
- `0005-primary-llm-minimax-m3.md` — primary LLM is Minimax-M3
- `0006-cognee-version-pin.md` — Cognee pinned at 1.1.2; harness is the upgrade gate
- `0007-graph-model-ontology-contract.md` — ontology enforced via `graph_model=`, not RDF/OWL
- `0008-graph-backend-neo4j.md` — Neo4j (not Cognee's Kuzu default); time model ships as Java UDF
- `0009-reified-relation-edges.md` — time-bounded/confident edges are `:Relation` nodes (Cognee `graph_model` can't carry edge properties)

**Build plan (`plan/`):** the 10 slice specs and an index.