docs(v1.2): switch substrate to Cognee + fix counts + world_id deprecation
The /docs review surfaced two categories of work: real bugs and a substrate decision. This commit lands both. Pure fixes: - Fix markdown table corruption in 01-ontology.md (3 `||` lines + duplicate `### Magic, culture, language` header) - Replace wrong tool/label/doc count claims with the real numbers: 30 -> 45 MCP tools, 14 -> 36 node labels, 14 -> 18 docs - Resolve Phase 11 collision in 09-roadmap.md (v1.1 phases renumbered to 4-7 in the unified plan; v1 Polish keeps its Phase 11) - Propagate the v1.2 world_id -> Setting/Plane deprecation to the v1.1 docs (11, 12, 14, 17): YAML world_id -> setting_id, Postgres world table -> setting table, Cypher index renamed, migration steps 6 + 7 added in 17-planes.md Cognee substrate switch (the strategy change): - 00-overview.md: substrate line now Cognee, not GraphMCP-Example - 08-architecture.md: full system diagram, services layout, hosting, and resource budget rewritten for Cognee as the substrate - 11-extensibility.md: 4-layer diagram reframed (Cognee data-model extension; template-watcher is a Cognee data-pipeline) - 12-storage-strategy.md: 5 stores collapse to 3 layers; saga pattern removed (Cognee handles cross-store transactions) - 13-microservice-decomposition.md: full rewrite (Cognee is the gateway; no monolith problem; 5+ Go services collapse to one in-process Python extension) - 14-examples.md: Example 6 added (full 8-step Cognee integration walkthrough from "stand up Cognee" to "template-driven tool call") - 15-related-work.md: Cognee reframed as substrate, not option - 16-comparison.md: strategic section reframed (decision made, not debate); the build order is now in 09-roadmap.md (Cognee-spike-first) - 10-critique.md: S1.4, S1.5, S2.5, S2.6 status updated for Cognee - 07-reasoning-harness.md: system prompt mentions Cognee + cognee.recall fallback - 06-ingestion.md: Path 1 (prose) via cognee.add/cognify; Path 2 (YAML) via Lore Engine parser; Go-worker section replaced Build plan (09-roadmap.md): the v1+v1.1 work collapses from 43 days on GraphMCP-Example to 33 days on Cognee. The MVP is end of Phase 3 (16 days): Cognee spike, typed ontology, time model, 45 MCP tools. Phase 4-6 add the consistency engine, TypeTemplate polymorphic extension, and reasoning-harness validation. Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
@@ -4,6 +4,8 @@
|
||||
|
||||
A Lore Engine is a **reasoning substrate** for a high-fantasy world. It is not a CRM, not a wiki, and not a search engine. Its single product is this: *when an LLM asks a question about the world, it gets a precise, time-bound, contradiction-checked answer from the canonical source — and the answer can be traced back to the document it came from.*
|
||||
|
||||
The Lore Engine is built on top of [Cognee](https://github.com/topoteretes/cognee) — a self-hosted, MIT-licensed knowledge-graph framework with ~17.8k stars and an Apache 2.0 paper. Cognee provides the storage abstraction (Neo4j or Kuzu), the extraction pipeline (LLM-based entity/relation extraction), the embedding pipeline, and the agent-native `remember/recall/forget` API. The Lore Engine adds a typed high-fantasy ontology, a temporal model, a consistency engine, NPC knowledge scoping, and a TypeTemplate polymorphic extension system on top. Cognee is the substrate; the Lore Engine is the domain layer.
|
||||
|
||||
## Design goals, ranked
|
||||
|
||||
1. **Historical accuracy** — the LLM must not be able to confuse "Aldric's father" with "Aldric's son," or claim that a character did X at a time when X had not yet been invented, or state a fact that contradicts a lore document.
|
||||
@@ -13,23 +15,20 @@ A Lore Engine is a **reasoning substrate** for a high-fantasy world. It is not a
|
||||
5. **Source attribution** — every claim returned to the LLM is tagged with the document it came from. The LLM can then say *"according to the chronicles of House Vyr..."* or know to discount a single low-confidence source.
|
||||
6. **Failure surfacing** — the engine reports what it *doesn't* know as loudly as what it does. Missing lineage, orphaned locations, temporal gaps — all surfaced via dedicated tools, never silently dropped.
|
||||
|
||||
## What we inherit from GraphMCP-Example
|
||||
## What we inherit from Cognee
|
||||
|
||||
| Component | Status | What we keep |
|
||||
|---|---|---|
|
||||
| Neo4j 5.x + vector indexes | Production | Substrate. We extend the schema, not replace it. |
|
||||
| Redis Streams + Go workers | Production | Pipeline backbone for ingestion. We add new workers, not replace the model. |
|
||||
| Go MCP server (HTTP + SSE) | Production | Tool dispatch, JSON-RPC, session registry. We add new tools to `mcpTools[]`. |
|
||||
| `LoreDocument` / `LoreChunk` schema | Production | Document storage. We add `LoreSource` to track *type* (prose vs. timeline vs. tree). |
|
||||
| `Entity` extraction (Person/Location/Faction/Event/Item/Creature) | Production | 6 of our ~14 node types. We add 8 more. |
|
||||
| `Encounter` + `WITNESSED` + NPC tier scoping | Production | The NPC "knowledge horizon" pattern is exactly what we need for time-scoping. We generalize it. |
|
||||
| `Contradiction` node + `get_contradictions` tool | Production | We extend the detection rules to cover time, lineage, and ontology. |
|
||||
| `lore-extractor` Go worker | Production | We add parallel extractors for structured inputs (timeline, tree, gazetteer). |
|
||||
| `query_as_npc` MCP tool | Production | We generalize: not just "as NPC" but "as faction," "as era," "as location." |
|
||||
| Storage abstraction (Neo4j 5.x or Kuzu + vector index) | Production | Substrate. We extend the schema, not replace it. Cognee stores the data; we add the typed labels and constraints. |
|
||||
| Extraction pipeline (LLM-based entity/relation extraction) | Production | Prose path. Cognee handles the "give me a chunk of markdown, get me triples" step; we customize the prompt to emit Lore Engine typed entities. |
|
||||
| Embedding pipeline (vector embeddings of chunks + entities) | Production | Semantic search back-end. Cognee manages the embedding store; we query it through the `lore_about` and `cite` tools. |
|
||||
| Agent-native `remember/recall/forget` API | Production | Storage-agnostic interface. The Lore Engine wraps `recall` with typed ontology + time model. |
|
||||
| `DataPoint` / `Entity` schema | Production | Base node types. The Lore Engine adds `Person`, `Faction`, `Location`, etc. as typed extensions. |
|
||||
| Session / task registry | Production | MCP server pattern. We register our 45 tools with Cognee's tool dispatch. |
|
||||
|
||||
## What we add
|
||||
|
||||
- **Deeper ontology** — Era, Calendar, Lineage, Culture, Deity, MagicSystem, Spell, Language, Title, Artifact, Region. Roughly 14 node labels total (vs. 6 in the existing stack).
|
||||
- **Deeper ontology** — Era, Calendar, Lineage, Culture, Deity, MagicSystem, Spell, Language, Title, Item (covers Artifact), Region, plus 2 v1.2 nodes (Plane, Setting) and 5 consistency nodes. Roughly 36 node labels total (7 inherited + 19 v1 core + 2 v1.2 planes + 6 v1.1 polymorphic + 5 consistency). The v1.1 docs (`11-extensibility.md`) add the polymorphic `DomainEntity`, `Relation`, `TypeTemplate`, `NPC`, `PC`, and `Human` labels.
|
||||
- **Time as a first-class concept** — temporal validity windows on relations, era filters, "was X true at time T?" via dedicated tools. The LLM no longer has to guess which version of Aldric it is talking to.
|
||||
- **Structured ingestion** — not just `.md` files. We ingest `timeline.yaml`, `family-tree.yaml`, `gazetteer.yaml`, `bestiary.yaml` as first-class sources. Free prose stays, but is no longer the only path.
|
||||
- **A consistency engine** — Cypher-based rules that flag anachronisms (Aldric can't be at the Battle of Black Spire if it happened 200 years before his birth), missing lineage (a noble with no recorded parents), and ontological violations (a region claimed to be inside two non-overlapping kingdoms).
|
||||
@@ -49,16 +48,16 @@ A Lore Engine is a **reasoning substrate** for a high-fantasy world. It is not a
|
||||
User: "Did House Vyr hold the Crimson Throne during the Second Age?"
|
||||
│
|
||||
▼
|
||||
LLM picks tool: query_faction_at_time("House Vyr", "Crimson Throne", "Second Age")
|
||||
LLM picks tool: was_true_at("RULED", "House Vyr", "Crimson Throne", "2nd_age")
|
||||
│
|
||||
▼
|
||||
MCP server → Neo4j Cypher
|
||||
MATCH (f:Faction {name: "House Vyr"})-[:RULED]->(t:Throne {name: "Crimson Throne"})
|
||||
MATCH (t)-[:EXISTED_DURING]->(e:Era {name: "Second Age"})
|
||||
RETURN f, t, e, sources
|
||||
MCP server (Cognee-hosted) → Lore Engine extension → Neo4j Cypher
|
||||
MATCH (f:Faction {name: "House Vyr"})-[r:RULED]->(t:Throne {name: "Crimson Throne"})
|
||||
WHERE time_in_window('2nd_age.year_120', r.valid_from, r.valid_until)
|
||||
RETURN f, t, r, sources
|
||||
│
|
||||
▼
|
||||
LLM gets: { "answer": "yes", "from": "120 TA to 340 TA", "sources": ["chronicles-vyr.md"], "confidence": "high" }
|
||||
LLM gets: { "was_true": true, "valid_from": "2nd_age.year_120", "valid_until": "2nd_age.year_340", "sources": ["chronicles-vyr.md"], "confidence": 0.92 }
|
||||
│
|
||||
▼
|
||||
LLM responds to user in narrative voice, citing the source if asked.
|
||||
|
||||
@@ -39,8 +39,8 @@ The world is modeled as a typed, temporal, source-attributed property graph. Thi
|
||||
| `Deity` | A named god, demigod, or divine being. | `name`, `domain[]`, `alignment`, `symbol` | `Aelar the Patient` |
|
||||
| `Spell` | A named magical effect. Has a `MagicSystem` parent (see lvl 1). | `name`, `level`, `school` | `Emberlance` |
|
||||
| `Material` | A specific substance with lore significance (orichalcum, soulglass, etc.). | `name`, `rarity` | `Soulglass` |
|
||||
|| `Plane` (v1.2) | A first-class graph node representing a layer of existence (Material, Shadowfell, demiplane, Outer Plane, etc.). Every `Plane` belongs to a `Setting` via `HAS_PLANE` and can have relations to other planes (`REFLECTS`, `LAYER_OF`, `ADJACENT_TO`, `ACCESSIBLE_VIA`). See `17-planes.md` for the full model. | `id`, `name`, `kind` (material/reflection/transit/ethereal/outer/inner/demiplane/transcendent), `summary`, `accessible`, `alignment_tendency`, `valid_from`, `valid_until` | `eberron.material`, `mardonari.voldramir` |
|
||||
|| `Setting` (v1.2) | A campaign/world scope that owns a tree of `Plane` nodes. The "top of the world" in the engine. | `id`, `name`, `summary`, `genres[]`, `canonical_time` | `eberron`, `mardonari`, `default` (legacy alias) |
|
||||
| `Plane` (v1.2) | A first-class graph node representing a layer of existence (Material, Shadowfell, demiplane, Outer Plane, etc.). Every `Plane` belongs to a `Setting` via `HAS_PLANE` and can have relations to other planes (`REFLECTS`, `LAYER_OF`, `ADJACENT_TO`, `ACCESSIBLE_VIA`). See `17-planes.md` for the full model. | `id`, `name`, `kind` (material/reflection/transit/ethereal/outer/inner/demiplane/transcendent), `summary`, `accessible`, `alignment_tendency`, `valid_from`, `valid_until` | `eberron.material`, `mardonari.voldramir` |
|
||||
| `Setting` (v1.2) | A campaign/world scope that owns a tree of `Plane` nodes. The "top of the world" in the engine. | `id`, `name`, `summary`, `genres[]`, `canonical_time` | `eberron`, `mardonari`, `default` (legacy alias) |
|
||||
|
||||
### New: Player-vs-NPC separation (v1.1)
|
||||
|
||||
@@ -74,7 +74,7 @@ Per the extensibility question, the engine adds one new node label and one new e
|
||||
|
||||
| Label | Purpose | Key properties |
|
||||
|---|---|---|
|
||||
| `DomainEntity` | A node representing a domain-specific concept (thieves-guild mission, war campaign, trade lot, ritual, spellbook, etc.). Defined by a `TypeTemplate`. | `id`, `type`, `name`, `world_id`, `properties: map<string, any>`, `template_id`, `sources[]`, `lore_verified` |
|
||||
| `DomainEntity` | A node representing a domain-specific concept (thieves-guild mission, war campaign, trade lot, ritual, spellbook, etc.). Defined by a `TypeTemplate`. | `id`, `type`, `name`, `setting_id`, `properties: map<string, any>`, `template_id`, `sources[]`, `lore_verified` |
|
||||
| `Relation` | An edge between any two nodes (DomainEntity, Person, Faction, etc.) with typed properties. | `id`, `from_id`, `to_id`, `type`, `properties: map<string, any>`, `valid_from`, `valid_until`, `sources[]` |
|
||||
| `TypeTemplate` | The schema for a domain type. Stored as data, hot-reloadable. | `id`, `version`, `spec` (parsed YAML), `status` (active/draft/deprecated) |
|
||||
|
||||
@@ -150,8 +150,8 @@ All edges are directed, typed, and (where meaningful) carry a temporal validity
|
||||
| `CONTRADICTS` | LoreSource → LoreSource | no | Two documents that disagree. |
|
||||
|
||||
### Magic, culture, language
|
||||
### Magic, culture, language
|
||||
|| Edge | From → To | Time-bound? | Notes |
|
||||
|
||||
| Edge | From → To | Time-bound? | Notes |
|
||||
|---|---|---|---|
|
||||
| `PRACTICES` | Person/Faction → MagicSystem | yes | Actively uses this magic system. |
|
||||
| `CASTS` | Person → Spell | yes | Specifically casts this named spell. |
|
||||
@@ -162,7 +162,8 @@ All edges are directed, typed, and (where meaningful) carry a temporal validity
|
||||
| `CULTURE_OF` | Culture → Location/Region | no | The culture's homeland. |
|
||||
|
||||
### Planes (v1.2 — first-class graph nodes)
|
||||
|| Edge | From → To | Time-bound? | Notes |
|
||||
|
||||
| Edge | From → To | Time-bound? | Notes |
|
||||
|---|---|---|---|
|
||||
| `HAS_PLANE` | Setting → Plane | no | Every plane belongs to exactly one setting. |
|
||||
| `EXISTS_IN` | any entity → Plane | **no** (timeless type-assertion) | "This entity type exists in this plane." Not where it *is*, but where it *can* be. Many-to-many: a god can exist in many planes. |
|
||||
|
||||
@@ -239,4 +239,4 @@ A fact that was true for centuries cannot be precisely queried by century — on
|
||||
- **Two sources giving the same era but different years produce a contradiction, not a synthesis.** The engine flags it; it does not pick a winner. See `04-consistency.md`.
|
||||
- **"Before the world began" or "after the end of time" are not representable.** A `null` `valid_from` with no parent `Era` is the closest, and we mark those as `mythic: true` to be clear they are cosmological claims, not historical ones.
|
||||
|
||||
The time model is the part of the engine that is most likely to need a v2. The 14 docs in this design describe v1. If you find yourself wanting year-level precision and we only have era-level, that's a source problem (write a better source document), not a time-model problem.
|
||||
The time model is the part of the engine that is most likely to need a v2. These 18 docs describe v1, v1.1, and v1.2. If you find yourself wanting year-level precision and we only have era-level, that's a source problem (write a better source document), not a time-model problem.
|
||||
|
||||
@@ -382,9 +382,9 @@ The user can disable any rule by ID, and add new ones via `add_ontology_rule`.
|
||||
|
||||
---
|
||||
|
||||
## Tool count: 30 total
|
||||
## Tool count: 45 total (8 inherited + 37 new)
|
||||
|
||||
8 inherited + 22 new = **30 MCP tools**. That's a lot, but each does one thing. The LLM uses 5–8 of them 90% of the time. The long tail exists for edge cases the LLM will sometimes need and shouldn't have to fall back to free-text generation for.
|
||||
The full catalog: 8 inherited from Cognee (or the prior substrate) + 37 new across Groups 1–8 = **45 MCP tools**. That's well past the empirical LLM tool-use ceiling (~25 in a single system prompt), so the Phase 6 reasoning-harness validation measures usage and collapses the long tail. The LLM uses 5–8 of them 90% of the time; the long tail exists for edge cases the LLM will sometimes need and shouldn't have to fall back to free-text generation for.
|
||||
|
||||
This is on the high end of what an LLM can effectively use in a single context. We mitigate by:
|
||||
- The reasoning harness documents which 8 to use first.
|
||||
|
||||
@@ -2,8 +2,8 @@
|
||||
|
||||
The ingestion layer is where the world enters the engine. There are two fundamentally different kinds of input:
|
||||
|
||||
1. **Free prose** — chronicles, novels, short stories, dialogue logs, Discord messages. The engine reads the text, extracts entities and relations, embeds chunks. The existing GraphMCP-Example pipeline does this.
|
||||
2. **Structured lore** — timelines, family trees, gazetteers, bestiaries, magic-system descriptions, written in YAML by the world-builder. The engine parses the structure, materializes typed graph edges directly. **No LLM is required for these.**
|
||||
1. **Free prose** — chronicles, novels, short stories, dialogue logs, Discord messages. The engine reads the text, extracts entities and relations, embeds chunks. On Cognee, this is the `cognee.add()` + `cognee.cognify()` pipeline, with a custom extraction prompt that emits the Lore Engine's 36 typed labels.
|
||||
2. **Structured lore** — timelines, family trees, gazetteers, bestiaries, magic-system descriptions, written in YAML by the world-builder. The Lore Engine's structured parser materializes typed graph edges directly. **No LLM is required for these.**
|
||||
|
||||
The structured path is the one that makes the engine *historically accurate*. Prose extraction is fuzzy by nature; YAML ingestion is exact. Both paths exist; structured is preferred for anything that becomes a load-bearing fact (lineage, era boundaries, faction rules).
|
||||
|
||||
@@ -19,41 +19,82 @@ The structured path is the one that makes the engine *historically accurate*. Pr
|
||||
│ │ │
|
||||
▼ ▼ ▼
|
||||
prose path timeline.yaml family_tree.yaml
|
||||
(LLM extract) (YAML parse) (YAML parse)
|
||||
cognee.add() Lore Engine YAML parser Lore Engine YAML parser
|
||||
cognee.cognify() (no LLM, exact) (no LLM, exact)
|
||||
│ │ │
|
||||
▼ ▼ ▼
|
||||
LoreChunk nodes Date, Era, Event nodes Person, Lineage nodes
|
||||
Entity nodes (fuzzy) RULES, OCCURRED_DURING PARENT_OF edges
|
||||
Relation edges (fuzzy) PARTICIPATED_IN edges EXISTED_DURING edges
|
||||
Cognee chunks + vectors Date, Era, Event nodes Person, Lineage nodes
|
||||
Typed triples RULES, OCCURRED_DURING PARENT_OF edges
|
||||
(Lore Engine extraction PARTICIPATED_IN edges EXISTED_DURING edges
|
||||
prompt emits 36 labels) │ │
|
||||
│ │ │
|
||||
└───────────────────────────────┴───────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
Neo4j knowledge graph
|
||||
Cognee-managed graph
|
||||
(Neo4j or Kuzu)
|
||||
│
|
||||
▼
|
||||
Consistency Engine runs
|
||||
Consistency pipeline runs
|
||||
(live + nightly batch)
|
||||
```
|
||||
|
||||
## Path 1: Free prose
|
||||
## Path 1: Free prose (via Cognee)
|
||||
|
||||
This path is mostly inherited from GraphMCP-Example. The pipeline:
|
||||
The prose path goes through Cognee's standard `add` + `cognify` pipeline. The Lore Engine registers a custom extraction prompt with Cognee; the prompt tells the LLM to emit the Lore Engine's 36 typed labels and the ~70 edge types instead of Cognee's default `Entity`/`DataPoint` types.
|
||||
|
||||
1. **Watcher** detects a new `.md` file in `./lore-data/` (or a POST to `/ingest/lore`).
|
||||
2. **Ingestion worker** chunks the text, generates embeddings, writes `LoreChunk` and `LoreSource` nodes.
|
||||
3. **Lore extractor** consumes the chunked source from a Redis stream, calls an LLM with the canonical extraction prompt (see below), gets back entities and relations.
|
||||
4. **Entity resolution** matches extracted entity names against known canonical names (the `loadKnownEntities` function in the existing extractor).
|
||||
5. **Cypher writer** materializes entities and relations into the graph, applying the `:FEATURES` edge from the source.
|
||||
```python
|
||||
# World-builder's ingestion script
|
||||
import cognee
|
||||
|
||||
await cognee.add("chapters/aldric_origin.md") # raw markdown
|
||||
await cognee.cognify() # extract + embed + index
|
||||
```
|
||||
|
||||
The pipeline:
|
||||
|
||||
1. **Cognee watcher** detects a new file (or receives a `cognee.add()` call).
|
||||
2. **Cognee ingestion worker** chunks the text (512-token windows, 64-token overlap), generates embeddings, writes `Chunk` and `Dataset` nodes.
|
||||
3. **Lore Engine extraction prompt** runs on each chunk. The LLM is told to emit triples using the Lore Engine's typed ontology. The response is parsed and validated against the schema.
|
||||
4. **Entity resolution** matches extracted entity names against known canonical names (Cognee's `loadKnownEntities` helper, with a `lore_engine` namespace prefix).
|
||||
5. **Cypher writer** materializes entities and relations into the graph using Cognee's graph adapter, applying the `:FEATURES` edge from the source.
|
||||
6. **Contradiction detection** runs on the new edges (see `04-consistency.md`).
|
||||
|
||||
### Extraction prompt (inherited, with one extension)
|
||||
### Extraction prompt (Lore Engine extension to Cognee)
|
||||
|
||||
The existing prompt extracts `Person / Location / Faction / Event / Item / Creature` and a fixed set of relations. We add:
|
||||
Cognee's default extraction prompt emits `Entity` and `DataPoint` types. The Lore Engine replaces this with a prompt that teaches the LLM the Lore Engine's 36 typed labels and the ~70 edge types:
|
||||
|
||||
- `temporal_hint` is **required** for `Event` (already in existing code) and **strongly preferred** for `Person` (birth/death) and `Faction` (founded/dissolved).
|
||||
- The prompt is taught the canonical time format (`{era}.{year}`) and asked to emit it in the `temporal_hint` field. Without this, the time model breaks at extraction time.
|
||||
- A new instruction: *"If the passage describes a person, also extract their `MEMBER_OF`, `WORSHIPS`, `SPEAKS`, `BELONGS_TO`, `POSSESSES` if explicitly stated. Prefer specific faction/religion/culture names over generic descriptions."*
|
||||
```
|
||||
You are extracting structured information from a passage of high-fantasy fiction
|
||||
for the Lore Engine knowledge graph.
|
||||
|
||||
Emit a list of triples. Each triple is (subject, relation, object).
|
||||
|
||||
Subject and object must be one of the Lore Engine typed labels:
|
||||
Person, Faction, Location, Item, Era, Date, Lineage, Culture, Deity,
|
||||
Language, MagicSystem, Title, Region, Material, Creature, Spell,
|
||||
Plane, Setting, NPC, PC, Human, DomainEntity.
|
||||
|
||||
Relation must be one of the Lore Engine typed edge types:
|
||||
RULED, PARENT_OF, MEMBER_OF, LOCATED_IN, OCCURRED_AT, OCCURRED_DURING,
|
||||
PARTICIPATED_IN, ALLIED_WITH, ENEMY_OF, POSSESSES, SPOUSE_OF, WORSHIPS,
|
||||
PRACTICES, SPEAKS, BELONGS_TO, CLAIMS_TITLE, CAUSED, PRECEDED,
|
||||
CONCURRENT_WITH, WITNESSED, LOGGED_IN, GIVEN_BY, TARGETS, PAID_BY,
|
||||
PART_OF, ... (full list in 01-ontology.md)
|
||||
|
||||
For Event nodes, the temporal_hint field is REQUIRED. Format: {era}.{year}[.month_N][.day_N].
|
||||
For Person nodes, birth and death years are STRONGLY PREFERRED in temporal_hint.
|
||||
For Faction nodes, founded and dissolved years are STRONGLY PREFERRED.
|
||||
|
||||
If the passage describes a person, also extract their MEMBER_OF, WORSHIPS,
|
||||
SPEAKS, BELONGS_TO, POSSESSES if explicitly stated. Prefer specific
|
||||
faction/religion/culture names over generic descriptions.
|
||||
|
||||
If a fact is too vague to assign a time, emit temporal_hint: "unknown"
|
||||
and set source_confidence: 0.5.
|
||||
```
|
||||
|
||||
The Cognee pipeline runs this prompt per chunk and parses the result. The Lore Engine validates the parsed triples against its typed ontology (rejecting triples that reference unknown labels) before writing to the graph.
|
||||
|
||||
### What prose is good for
|
||||
|
||||
@@ -255,25 +296,30 @@ YAML wins because:
|
||||
|
||||
The downside is YAML's gotchas (Norway problem, tab/space sensitivity). The extractor is strict and rejects ambiguous inputs with line numbers — better to fail loudly than silently parse `NO: false` as the boolean `True`.
|
||||
|
||||
## The ingestion worker (new Go service)
|
||||
## The structured-ingestor (Lore Engine on Cognee)
|
||||
|
||||
Following the GraphMCP-Example pattern, we add a new worker:
|
||||
The structured-YAML parser lives in the Lore Engine extension as a Python module:
|
||||
|
||||
```go
|
||||
// services/structured-ingestor/main.go
|
||||
// Consumes Redis stream `raw.structured`
|
||||
// Dispatches on source_type: timeline | family_tree | gazetteer | bestiary | magic_system | culture
|
||||
// Calls the appropriate extractor (pure Go, no LLM)
|
||||
// Writes to Neo4j with the same mergeLoreEntities + contradiction detection
|
||||
```python
|
||||
# lore_engine/parsers/timeline.py
|
||||
# Validates the timeline.yaml schema
|
||||
# Emits MERGE (e:Era {slug, parent_era, start, end}) and similar
|
||||
# Calls Cognee's graph adapter to execute the Cypher
|
||||
# Tags the LoreSource with source_type: timeline
|
||||
|
||||
# lore_engine/parsers/family_tree.py
|
||||
# Same pattern, different schema
|
||||
|
||||
# ... one parser per YAML type
|
||||
```
|
||||
|
||||
The structured path is **fast and deterministic** — typical ingest is <500ms per YAML file, no GPU, no LLM latency.
|
||||
The structured path is **fast and deterministic** — typical ingest is <500ms per YAML file, no GPU, no LLM latency. The parser is a thin wrapper over Cognee's graph adapter; the schema validation is strict and rejects ambiguous inputs with line numbers.
|
||||
|
||||
## What this means for the LLM
|
||||
|
||||
The LLM never has to ingest. It only reads. World-builders ingest via:
|
||||
|
||||
- `POST /ingest/lore` (markdown — existing)
|
||||
- `cognee.add()` + `cognee.cognify()` (prose — markdown, dialogue)
|
||||
- `POST /ingest/structured` (YAML — new)
|
||||
- `POST /ingest/dialogue` (JSON — new)
|
||||
- `tea add-source <file>` (CLI wrapper — new, optional)
|
||||
|
||||
@@ -15,7 +15,13 @@ The full system prompt the LLM is given:
|
||||
|
||||
```
|
||||
You are the Lore Engine's reasoner. Your job is to answer questions about a
|
||||
high-fantasy world by querying a Neo4j-backed knowledge graph via MCP tools.
|
||||
high-fantasy world by querying a Cognee-backed knowledge graph via MCP tools.
|
||||
|
||||
The Lore Engine is built on Cognee. Cognee provides the storage, the extraction
|
||||
pipeline, and the embedding store. The Lore Engine adds a typed high-fantasy
|
||||
ontology, a temporal model, a consistency engine, and the 45 domain tools you
|
||||
use to answer questions. Your tool list is a mix of Cognee primitives (cognee.recall,
|
||||
cognee.add) and Lore Engine domain tools (was_true_at, lookup, state_at, etc.).
|
||||
|
||||
You MUST:
|
||||
- Always call a tool before claiming a fact. Never answer from your own knowledge.
|
||||
@@ -37,6 +43,10 @@ You MUST NOT:
|
||||
|
||||
When you don't know, say "I don't know" or "The chronicles are silent on this."
|
||||
When a tool returns an error, surface the error to the user and stop.
|
||||
|
||||
Fallback to Cognee primitives: if no Lore Engine tool fits the question,
|
||||
use cognee.recall("free-text query") for semantic search over the chunk store.
|
||||
This is a low-precision fallback — prefer the typed tools when possible.
|
||||
```
|
||||
|
||||
This is the bedrock. The patterns below build on it.
|
||||
|
||||
@@ -1,6 +1,8 @@
|
||||
# 08 — Architecture
|
||||
|
||||
The Lore Engine is a thin layer on top of the existing GraphMCP-Example stack. Same transport, same data substrate, same worker model — extended with new schema, new workers, new tools, and a consistency engine.
|
||||
The Lore Engine is a **domain layer** on top of [Cognee](https://github.com/topoteretes/cognee), a self-hosted, MIT-licensed knowledge-graph framework. Cognee provides the storage abstraction (Neo4j or Kuzu + vector index), the LLM-based extraction pipeline, the embedding pipeline, and the agent-native `remember/recall/forget` API. The Lore Engine adds a typed high-fantasy ontology, a temporal model, a consistency engine, NPC knowledge scoping, and a TypeTemplate polymorphic extension system on top.
|
||||
|
||||
Cognee is the **plumbing**; the Lore Engine is the **domain layer** that the world-builder cares about.
|
||||
|
||||
## System diagram
|
||||
|
||||
@@ -19,46 +21,41 @@ The Lore Engine is a thin layer on top of the existing GraphMCP-Example stack. S
|
||||
└───────┬───────┘ └─────────┬──────────┘ └─────────┬──────────┘
|
||||
│ │ │
|
||||
▼ ▼ ▼
|
||||
Redis Streams: Redis Streams: Redis Streams:
|
||||
raw.lore raw.structured raw.dialogue
|
||||
cognee.add() cognee.add() + Lore cognee.add()
|
||||
cognee.cognify() Engine parser + NPC scope
|
||||
│ │ │
|
||||
▼ ▼ ▼
|
||||
┌────────────────┐ ┌────────────────────┐ ┌────────────────────┐
|
||||
│ lore-extractor │ │ structured-ingest │ │ dialogue-processor │
|
||||
│ (Go worker) │ │ (Go worker) │ │ (Go worker) │
|
||||
│ LLM extract │ │ YAML → Cypher │ │ Cypher write │
|
||||
│ fuzzy │ │ exact │ │ exact │
|
||||
└────────┬───────┘ └─────────┬──────────┘ └─────────┬──────────┘
|
||||
│ │ │
|
||||
└───────────────────────────┼──────────────────────────────┘
|
||||
└───────────────────────────┼──────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌─────────────────────────┐
|
||||
│ Neo4j Graph │
|
||||
│ Cognee Storage │
|
||||
│ ───────────────────── │
|
||||
│ Person, Faction, │
|
||||
│ Location, Era, │
|
||||
│ Lineage, MagicSystem, │
|
||||
│ Item, ... │
|
||||
│ Neo4j or Kuzu (graph) │
|
||||
│ + Vector store │
|
||||
│ + Postgres (metadata) │
|
||||
│ ───────────────────── │
|
||||
│ + LoreChunk (vectors) │
|
||||
│ + Encounter │
|
||||
│ + Contradiction │
|
||||
│ + Anachronism │
|
||||
│ + Orphan │
|
||||
│ Cognee manages: │
|
||||
│ DataPoint, Chunk, │
|
||||
│ + Lore Engine types: │
|
||||
│ Person, Faction, │
|
||||
│ Location, Era, │
|
||||
│ Lineage, MagicSystem,│
|
||||
│ Item, Setting, Plane,│
|
||||
│ DomainEntity, ... │
|
||||
└────────────┬────────────┘
|
||||
│
|
||||
│ Cypher / Vector queries
|
||||
│ Cypher / cognee.recall()
|
||||
│
|
||||
┌────────────▼────────────┐
|
||||
│ mcp-server (Go) │
|
||||
│ HTTP :9000 │
|
||||
│ SSE :9000 │
|
||||
│ Lore Engine extension │
|
||||
│ (in-process with │
|
||||
│ Cognee MCP server) │
|
||||
│ ───────────────────── │
|
||||
│ 8 inherited tools │
|
||||
│ 22 new tools │
|
||||
│ 37 new tools │
|
||||
│ 10 consistency tools │
|
||||
└────────────┬────────────┘
|
||||
│ JSON-RPC over HTTP
|
||||
│ JSON-RPC over MCP
|
||||
│
|
||||
┌────────────▼────────────┐
|
||||
│ LLM Client │
|
||||
@@ -67,15 +64,19 @@ The Lore Engine is a thin layer on top of the existing GraphMCP-Example stack. S
|
||||
│ from reasoning harness│
|
||||
└─────────────────────────┘
|
||||
|
||||
Background jobs:
|
||||
Background jobs (Cognee data-pipelines):
|
||||
┌─────────────────────────┐ ┌─────────────────────────┐
|
||||
│ consistency-runner │ │ consistency-monitor │
|
||||
│ (cron, 03:00 daily) │ │ (HTTP /run-check) │
|
||||
│ runs all rules │ │ ad-hoc rule run │
|
||||
│ materializes nodes │ │ for live verification │
|
||||
│ consistency-pipeline │ │ template-watcher │
|
||||
│ (cron, 03:00 daily) │ │ (file watcher on │
|
||||
│ runs all rules │ │ ./templates/, │
|
||||
│ materializes nodes │ │ hot-reload tools) │
|
||||
└─────────────────────────┘ └─────────────────────────┘
|
||||
```
|
||||
|
||||
The architecture is **one process, in-process extensions**: Cognee runs its MCP server; the Lore Engine registers as a Cognee data-model extension (typed labels, constraints, indexes) plus a tool extension (the 45 MCP tools). The world-builder never interacts with Cognee directly — the Lore Engine is the surface.
|
||||
|
||||
The two background jobs are Cognee data-pipelines (no separate Go services): the consistency pipeline runs the 4 rule categories on a schedule and on demand; the template-watcher watches `./templates/` and re-registers the TypeTemplate-driven tools on change.
|
||||
|
||||
## Data flow: a question
|
||||
|
||||
```
|
||||
@@ -85,11 +86,11 @@ The Lore Engine is a thin layer on top of the existing GraphMCP-Example stack. S
|
||||
2. LLM Client → LLM (with system prompt + active context)
|
||||
LLM picks tool: was_true_at(RULED, "House Vyr", "Valdorn", "3rd_age.year_340")
|
||||
|
||||
3. LLM Client → MCP Server (JSON-RPC POST /mcp)
|
||||
3. LLM Client → Cognee MCP Server (JSON-RPC POST /mcp)
|
||||
{"method": "tools/call", "params": {"name": "was_true_at", "arguments": {...}}}
|
||||
|
||||
4. MCP Server → Neo4j
|
||||
Cypher:
|
||||
4. Cognee MCP Server → Lore Engine tool handler → Cognee storage
|
||||
The handler composes a Cypher query and executes it through Cognee's graph adapter (which may be Neo4j, Kuzu, or another Cognee-supported backend):
|
||||
MATCH (f:Faction {name: "House Vyr"})-[r:RULED]->(l:Location {name: "Valdorn"})
|
||||
WHERE time_in_window("3rd_age.year_340", r.valid_from, r.valid_until)
|
||||
RETURN r, r.valid_from, r.valid_until, sources
|
||||
@@ -123,10 +124,10 @@ The plane model (per `17-planes.md`) is the new substrate for multi-setting, mul
|
||||
LLM picks tools: list_accessible_targets(plane='nine_hells'),
|
||||
entity_planes_at_time(entity='roland_raventhorne', at='3rd_age.year_430')
|
||||
|
||||
3. LLM Client → MCP Server (JSON-RPC POST /mcp)
|
||||
3. LLM Client → Cognee MCP Server (JSON-RPC POST /mcp)
|
||||
Two tool calls, possibly chained.
|
||||
|
||||
4. MCP Server → Neo4j
|
||||
4. Cognee MCP Server → Lore Engine tool handler → Cognee storage
|
||||
Cypher for accessibility:
|
||||
MATCH (start:Plane {id: 'mardonari.nine_hells'})-[:ACCESSIBLE_VIA|ADJACENT_TO*1..2]->(target:Plane)
|
||||
RETURN DISTINCT target.id, target.kind
|
||||
@@ -159,67 +160,69 @@ Plane relations (`REFLECTS`, `LAYER_OF`, `ADJACENT_TO`, `ACCESSIBLE_VIA`) make t
|
||||
2. World-builder → POST /ingest/structured (multipart)
|
||||
Files attached, source_type per file.
|
||||
|
||||
3. HTTP server (mcp-server or a thin gateway) → Redis Stream raw.structured
|
||||
Each file becomes a stream entry with the YAML body and source_type tag.
|
||||
3. The Lore Engine extension (running in Cognee) routes the upload:
|
||||
- Structured YAML files (timeline, family_tree, gazetteer, bestiary, magic_system, culture)
|
||||
go through the per-type parser and are merged as Cypher.
|
||||
- Prose .md files are forwarded to `cognee.add()` + `cognee.cognify()`
|
||||
for LLM-based extraction.
|
||||
|
||||
4. structured-ingest worker consumes the entry.
|
||||
- Reads source_type, dispatches to the right parser.
|
||||
- YAML parser validates against the per-type schema.
|
||||
- Materializes Cypher (MERGE nodes, MERGE edges with time bounds).
|
||||
- Tags the LoreSource with source_type: <type>.
|
||||
4. Cognee persists the data; the Lore Engine's typed ontology is what
|
||||
Cognee stores (the labels and constraints live in the Lore Engine
|
||||
data-model extension, registered with Cognee at startup).
|
||||
|
||||
5. consistency-runner (live mode) gets the write event.
|
||||
5. The consistency pipeline (a Cognee data-pipeline) gets a write event.
|
||||
- Runs the relevant anachronism + contradiction checks on the new edges.
|
||||
- Materializes any new :Contradiction / :Anachronism / :Orphan nodes.
|
||||
|
||||
6. Neo4j state is now consistent.
|
||||
6. Cognee state is now consistent.
|
||||
The next MCP tool call sees the new data.
|
||||
```
|
||||
|
||||
## Services layout (extends GraphMCP-Example)
|
||||
## Services layout (Lore Engine on Cognee)
|
||||
|
||||
```
|
||||
services/
|
||||
├── ingestion-worker/ [inherited] markdown chunks + embeddings
|
||||
├── lore-extractor/ [inherited] LLM entity extraction
|
||||
├── entity-extractor/ [inherited] message entity extraction
|
||||
├── encounter-processor/ [inherited] encounter graph writes
|
||||
├── lore-watcher/ [inherited] ./lore-data/ watcher
|
||||
├── discord-connector/ [inherited] Discord → raw.messages
|
||||
├── mcp-server/ [extended] +22 new tools
|
||||
lore-engine-extension/ [NEW] the Lore Engine on Cognee
|
||||
├── lore_engine/ Cognee data-model extension package
|
||||
│ ├── __init__.py registers labels, constraints, indexes
|
||||
│ ├── ontology/ typed labels (Person, Faction, ...)
|
||||
│ ├── time_model/ time_in_window, era-tree helpers
|
||||
│ ├── consistency/ 4 rule categories + 10 starter rules
|
||||
│ └── templates/ TypeTemplate validator + registry
|
||||
│
|
||||
├── structured-ingestor/ [NEW] YAML → Cypher
|
||||
│ ├── main.go # dispatcher
|
||||
│ ├── parsers/timeline.go # timeline.yaml
|
||||
│ ├── parsers/family_tree.go # family_tree.yaml
|
||||
│ ├── parsers/gazetteer.go # gazetteer.yaml
|
||||
│ ├── parsers/bestiary.go # bestiary.yaml
|
||||
│ ├── parsers/magic_system.go # magic_system.yaml
|
||||
│ ├── parsers/culture.go # culture.yaml
|
||||
│ └── parsers/validator.go # strict YAML validation
|
||||
├── tools/ the 45 MCP tools, in-process with Cognee
|
||||
│ ├── lookup.py Group 1 — disambiguation
|
||||
│ ├── entity_context.py Group 1
|
||||
│ ├── was_true_at.py Group 2 — time-aware
|
||||
│ ├── state_at.py Group 2
|
||||
│ ├── list_lineage.py Group 3
|
||||
│ ├── event_chain.py Group 4
|
||||
│ ├── lore_about.py Group 5
|
||||
│ ├── consistency_tools.py Group 6
|
||||
│ ├── generation_tools.py Group 7
|
||||
│ └── worldbuilder_tools.py Group 8
|
||||
│
|
||||
├── dialogue-processor/ [NEW] POST /ingest/dialogue → Cypher
|
||||
│ └── main.go
|
||||
├── pipelines/ Cognee data-pipelines
|
||||
│ ├── consistency_pipeline.py nightly + on-demand rule run
|
||||
│ └── template_watcher.py watches ./templates/, hot-reload
|
||||
│
|
||||
├── consistency-runner/ [NEW] nightly batch
|
||||
│ ├── main.go # cron entry point
|
||||
│ ├── rules/source_contradictions.go # category A
|
||||
│ ├── rules/anachronism.go # category B
|
||||
│ ├── rules/ontology.go # category C (declarative)
|
||||
│ └── rules/orphans.go # category D
|
||||
├── parsers/ structured YAML ingest
|
||||
│ ├── timeline.py
|
||||
│ ├── family_tree.py
|
||||
│ ├── gazetteer.py
|
||||
│ ├── bestiary.py
|
||||
│ ├── magic_system.py
|
||||
│ └── culture.py
|
||||
│
|
||||
├── consistency-monitor/ [NEW] HTTP /run-check
|
||||
│ └── main.go # exposes run_consistency_check tool
|
||||
│
|
||||
└── era-tagger/ [NEW, optional] LLM-assisted era inference
|
||||
└── main.go # backfills temporal_hint on prose-extracted edges
|
||||
└── schema/ init.cognify / init.cypher
|
||||
├── init.cypher constraints, indexes
|
||||
└── udfs/ time_in_window, time_windows_overlap
|
||||
```
|
||||
|
||||
The `era-tagger` is optional and only used during the bootstrap phase — when prose has been ingested without canonical era tags. It calls an LLM to backfill `temporal_hint` on events that have `Era: unknown` in their `temporal_hint`. After the structured corpus is in, this worker can be disabled.
|
||||
The Lore Engine extension is **one Python package** that registers with Cognee at startup. Cognee provides the MCP server, the storage abstraction, the embedding pipeline, and the agent API; the Lore Engine provides the domain types, the time model, the consistency rules, and the 45 tools. There are no separate Go workers; the entire backend is in-process with Cognee.
|
||||
|
||||
## Schema bootstrap (new Cypher)
|
||||
|
||||
The full schema lives in `schema/init.cypher` (to be generated during the build phase). Key additions to the GraphMCP-Example schema:
|
||||
The full schema lives in `schema/init.cypher` (to be generated during the build phase). These constraints and indexes are **the Lore Engine data-model extension** that Cognee applies on top of its own schema at startup:
|
||||
|
||||
```cypher
|
||||
// New label constraints
|
||||
@@ -279,7 +282,7 @@ Implementation notes:
|
||||
- Resolves `current` against the `:Now` config node.
|
||||
- Walks the era tree for parent-era membership.
|
||||
- Treats `null` as open-ended.
|
||||
- Implementation will live in a Neo4j user-defined function (Java or Python) registered at startup.
|
||||
- Implementation: a Cognee plugin (Python) registered at startup, OR a Neo4j user-defined function (Java) if Cognee is using Neo4j as the storage backend. The Phase 2 spike (see `09-roadmap.md`) determines which.
|
||||
|
||||
### `time_windows_overlap(from_a, until_a, from_b, until_b) → bool`
|
||||
|
||||
@@ -295,27 +298,27 @@ The `time_in_window` and `time_windows_overlap` UDFs are the only ones the engin
|
||||
|
||||
## Hosting & deployment
|
||||
|
||||
The Lore Engine adds no new infrastructure dependencies on top of GraphMCP-Example:
|
||||
The Lore Engine on Cognee is **one process** that runs the Cognee MCP server, the storage adapter (Neo4j or Kuzu), and the Lore Engine extension. The hosting story is:
|
||||
|
||||
- **Neo4j 5.x** — same instance, more schema. APOC plugin required for `apoc.create.addLabels` and `apoc.merge.relationship`. Already enabled in GraphMCP-Example.
|
||||
- **Redis 7** — same instance, two new streams (`raw.structured`, `raw.dialogue`).
|
||||
- **LiteLLM proxy at `localhost:4000`** — used by the lore-extractor worker. New LLM calls go through the same proxy. The `era-tagger` worker is the only new LLM consumer.
|
||||
- **Go MCP server** — same binary, new tool registrations.
|
||||
- **Cognee** — runs in a Docker container, exposes the MCP server on `:8000` (or whatever port the deployment chooses), manages the storage adapter. The Lore Engine extension is a Python package installed into the Cognee image.
|
||||
- **Storage backend** — Cognee supports Neo4j 5.x (most common in production), Kuzu (embedded, single-binary), and others via the `cognee-neo4j` / `cognee-kuzu` adapters. The Lore Engine does not need to know which one is in use.
|
||||
- **Postgres (v1.1)** — Cognee uses Postgres for metadata (sessions, tasks, the `setting` table). Single instance, managed by Cognee.
|
||||
- **LLM provider** — Cognee is LLM-provider-agnostic; it calls any OpenAI-compatible endpoint. The world-builder points it at a local model or a hosted one.
|
||||
|
||||
The deployment story is: add the new workers to `docker-compose.yml`, add the new schema migration, restart, ingest. No new infrastructure.
|
||||
The deployment story is: build a Cognee image with the Lore Engine extension installed, point the storage adapter at Neo4j, set the LLM endpoint env vars, run. **No new infrastructure on top of Cognee**; the Lore Engine is code, not a new service.
|
||||
|
||||
## Resource budget (delta from GraphMCP-Example)
|
||||
## Resource budget (Lore Engine delta on Cognee)
|
||||
|
||||
| Component | Memory | Notes |
|
||||
|---|---|---|
|
||||
| `structured-ingestor` | ~50MB | YAML parsing, no LLM. Light. |
|
||||
| `consistency-runner` (nightly) | ~200MB | Cypher eval, short-lived. |
|
||||
| `consistency-monitor` | ~30MB | HTTP, stateless. |
|
||||
| `era-tagger` (bootstrap only) | ~200MB | LLM calls, only during initial backfill. |
|
||||
| Neo4j (additional indexes) | ~+200MB | New indexes are small. |
|
||||
| **Total delta** | **~700MB** | Negligible on the 58GB host. |
|
||||
| Cognee MCP server (base) | ~300MB | Cognee is a single Python process; this is its baseline. |
|
||||
| Lore Engine extension (in-process) | ~+80MB | The 45 tools, the typed ontology, the consistency rules. No separate process. |
|
||||
| Neo4j (additional indexes for typed labels) | ~+200MB | 19 v1 + 2 v1.2 + 6 v1.1 + 5 consistency labels. |
|
||||
| Postgres (v1.1) | ~+50MB | Single instance, the `setting`/`lore_event`/`retcon` tables. |
|
||||
| Consistency pipeline (peak) | ~+150MB | Short-lived during nightly run. |
|
||||
| **Total delta over bare Cognee** | **~480MB** | Cheap. |
|
||||
|
||||
The Lore Engine is cheaper to run than the existing stack it extends.
|
||||
The Lore Engine is **cheaper** to run than the prior GraphMCP-Example stack because the 5+ Go services are gone; everything is in-process with Cognee.
|
||||
|
||||
## What is intentionally not in this architecture
|
||||
|
||||
|
||||
@@ -154,122 +154,124 @@ The disambiguation and snapshot tools. These unlock the rest.
|
||||
|
||||
| Phase | Days | Cumulative |
|
||||
|---|---|---|
|
||||
| 0 — Pre-flight | 1 | 1 |
|
||||
| 1 — Schema + UDFs | 4 | 5 |
|
||||
| 2 — Time-aware tools | 4 | 9 |
|
||||
| 3 — Structured ingestion | 6 | 15 |
|
||||
| 4 — state_at + entity_context + lookup | 4 | 19 |
|
||||
| 5 — More structured ingestors | 4 | 23 |
|
||||
| 6 — Lineage & hierarchy tools | 3 | 26 |
|
||||
| 7 — Consistency engine | 6 | 32 |
|
||||
| 8 — Lore extension tools | 3 | 35 |
|
||||
| 9 — Generation tools | 4 | 39 |
|
||||
| 10 — Reasoning harness | 4 | 43 |
|
||||
| 11 — Polish | — | — |
|
||||
| **MVP (Phases 0–4)** | **19 days** | **end of phase 4** |
|
||||
| 0 — Cognee spike | 2 | 2 |
|
||||
| 1 — Lore Engine ontology on Cognee | 5 | 7 |
|
||||
| 2 — Time model + UDFs | 4 | 11 |
|
||||
| 3 — MCP tool layer (Cognee extension) | 5 | 16 |
|
||||
| 4 — Consistency engine | 6 | 22 |
|
||||
| 5 — TypeTemplate polymorphic extension | 7 | 29 |
|
||||
| 6 — Reasoning harness + validation | 4 | 33 |
|
||||
| 7 — Polish | — | — |
|
||||
| **MVP (Phases 0–3)** | **16 days** | **end of phase 3** |
|
||||
|
||||
**The MVP is end of phase 4.** That's: schema, UDFs, 4 time-aware tools, structured ingestion for 3 source types, and 3 disambiguation/snapshot tools. World-builders can start writing YAML and the LLM can start answering time-bounded questions. The consistency engine and the lore extension tools come after.
|
||||
**The MVP is end of phase 3.** That's: validated Cognee substrate, the typed Lore Engine ontology, the time model + UDFs, and the 45 MCP tools exposed through Cognee. World-builders can start writing YAML and the LLM can start answering time-bounded questions. The consistency engine and the TypeTemplate polymorphic extension are the v1.1 follow-ups; per the v1.1 plan below, they land in Phases 4 and 5 of the unified roadmap.
|
||||
|
||||
## What to cut if you're under time pressure
|
||||
|
||||
In strict order:
|
||||
|
||||
1. Phase 11 — Polish. Trivially deferrable.
|
||||
2. Phase 9 — Generation tools. The LLM can hand-roll narrative from `summarize_chain` output and the existing tools. Not core.
|
||||
3. Phase 6 — Lineage tools. Useful but not blocking. The LLM can navigate lineage via `expand_context` and graph traversal.
|
||||
4. Phase 5 — `bestiary.yaml`, `magic_system.yaml`, `culture.yaml`. Defer to a v2.
|
||||
5. Phase 8 — Lore extension tools. Defer the `cite` tool; the LLM can use `lore_about` and quote chunks.
|
||||
1. Phase 7 — Polish. Trivially deferrable.
|
||||
2. Phase 6 — Reasoning harness validation depth. Ship with 20 test questions instead of 50; iterate from observed failure modes.
|
||||
3. Phase 5 — TypeTemplate polymorphic extension. The v1 ontology covers the macro structure; the polymorphic wrapper is a powerful v1.1 addition but not a v1 blocker.
|
||||
4. Phase 4 — Consistency engine. Start with the 3 most valuable rule categories (Contradiction, Anachronism, Orphan) and skip OntologyViolation in the first build.
|
||||
|
||||
The **non-cuttable core is Phases 0–4 + Phase 7**. Schema, UDFs, time-aware tools, structured ingestion, lookup/entity_context/state_at, and the consistency engine. Everything else is enhancement.
|
||||
The **non-cuttable core is Phases 0–3 + Phase 4**. Substrate validation, typed ontology, time model, MCP tool layer, and the basic consistency engine. Everything else is enhancement.
|
||||
|
||||
## What NOT to do
|
||||
|
||||
- **Do not start with phase 7 (consistency engine).** It's tempting because it's the most novel part. But without structured ingestion, it has nothing to be consistent about. Build the data first, then the consistency layer that watches it.
|
||||
- **Do not skip the Cognee spike.** The substrate decision is the highest-leverage call in the project. If Cognee can't represent the typed ontology or the time model, the spike fails fast and the v1 needs a different foundation. The 1–2 days is the cheapest insurance available.
|
||||
- **Do not skip the UDF unit tests.** Every time-aware query depends on `time_in_window` and `time_windows_overlap`. If they have a bug, every consistency check is wrong. Test first, then trust.
|
||||
- **Do not over-invest in prose extraction.** It's the path of least resistance because the LLM does the work. It's also the path with the highest error rate for the highest-stakes data. Structured YAML is the win.
|
||||
- **Do not over-invest in prose extraction.** Cognee handles the prose path; the structured YAML path is what the Lore Engine adds on top. Structured is exact, prose is fuzzy; high-stakes data goes through structured paths.
|
||||
- **Do not try to support 100% of question types in the first build.** Ship the 5 patterns from `07-reasoning-harness.md` and iterate. The LLM is forgiving of missing tools if the existing ones are reliable.
|
||||
|
||||
---
|
||||
|
||||
## v1.1 phases (extensibility, storage, decomposition)
|
||||
## Phases 4–7: From MVP to production-ready
|
||||
|
||||
After the v1 MVP (Phases 0–4 of the original) is built and validated, v1.1 adds four phases that address the modularization question.
|
||||
After the v1 MVP (Phases 0–3) is built and the time model works end-to-end, four phases add the consistency engine, the polymorphic extension model, the reasoning-harness validation, and the polish layer. This is a single ~17-day follow-up that lands everything in the v1.1/v1.2 design docs on top of the Cognee-based v1.
|
||||
|
||||
### Phase 11: Microservice decomposition (mechanical, ~5 days)
|
||||
### Phase 4: Consistency engine (~6 days)
|
||||
|
||||
A pure refactor of the v1 single-binary `mcp-server` into a small gateway + per-handler files. No new functionality; just the file structure for Phase 12 to extend.
|
||||
The 4-category rule system from `04-consistency.md`: Contradiction, Anachronism, Orphan, OntologyViolation. Implemented as a Cognee data-pipeline that runs on a schedule and on demand, materializing violation nodes in the same graph the LLM queries.
|
||||
|
||||
- [ ] Split `mcp-server/main.go` (1144 lines) into `main.go` + `handlers/{core,lineage,event,consistency,generation,worldbuilder}.go`.
|
||||
- [ ] Add an in-process tool registry. The gateway iterates the registry to build `tools/list`.
|
||||
- [ ] Add the HTTP-based tool handler protocol (`GET /tools`, `POST /invoke`) with a stub HTTP handler.
|
||||
- [ ] Add a `/healthz` aggregator that surfaces per-handler health.
|
||||
- [ ] Implement the 4 rule categories from `04-consistency.md` as a Cognee data-pipeline.
|
||||
- [ ] Implement the 10 starter `:OntologyRule` nodes from `05-mcp-tools.md#starter-rules`.
|
||||
- [ ] Expose the 10 Group 6 consistency tools: `get_contradictions`, `get_anachronisms`, `get_ontology_violations`, `get_orphans`, `flag_for_review`, `explain_violation`, `run_consistency_check`, `latest_run`, `add_ontology_rule`, `list_ontology_rules`.
|
||||
- [ ] Schedule the consistency pipeline to run nightly (Cognee task scheduler).
|
||||
|
||||
**Verify:** all 30 v1 tools still work, `tools/list` returns the same list, and the gateway has < 600 lines of code.
|
||||
**Verify:** ingest two sources that disagree on the same fact; confirm a `Contradiction` node is created; call `get_contradictions(subject=X)` and see it. Test anachronism detection against a known historical claim that has a Person participating in an Event outside their lifespan.
|
||||
|
||||
### Phase 12: Polymorphic extension model (~7 days)
|
||||
### Phase 5: TypeTemplate polymorphic extension (~7 days)
|
||||
|
||||
The big one. Adds the `DomainEntity`, `Relation`, and `TypeTemplate` labels, the template-watcher service, and the dynamic tool generator.
|
||||
The big one. The `DomainEntity`, `Relation`, and `TypeTemplate` labels, the template-watcher service, and the dynamic tool generator. Per `11-extensibility.md` and `12-storage-strategy.md`, this is the v1.1 extension model that makes new domain types a YAML exercise.
|
||||
|
||||
- [ ] Add the new labels and indexes from `11-extensibility.md#layer-2-the-domainentity-wrapper` to `neo4j-init.cypher`.
|
||||
- [ ] Build `services/template-watcher/`: watches `./templates/`, validates YAML, registers templates.
|
||||
- [ ] Build `services/template-registry/`: persists template specs in Postgres, hot-reloadable.
|
||||
- [ ] Implement the dynamic tool generator: a single generic handler that runs queries generated from `TypeTemplate` specs.
|
||||
- [ ] Register the `DomainEntity`, `Relation`, and `TypeTemplate` labels as a Cognee data-model extension.
|
||||
- [ ] Build the template-watcher service: watches `./templates/`, validates YAML, registers templates.
|
||||
- [ ] Build the template-registry: persists template specs alongside the Cognee storage layer.
|
||||
- [ ] Implement the dynamic tool generator: a generic handler that runs queries generated from `TypeTemplate` specs.
|
||||
- [ ] Add the `list_template_tools` MCP tool.
|
||||
- [ ] Ship the four example templates from `14-examples.md` (thieves-guild mission, war campaign, black-market lot, NPC secret knowledge).
|
||||
- [ ] Update the reasoning harness to mention template tools.
|
||||
|
||||
**Verify:** write `templates/thieves_guild/mission.yaml`, hit `POST /admin/templates/reload`, see 6 new tools in `tools/list`, ingest a mission, query it via `list_missions`, get a coherent answer. **No Go code change between "template added" and "tool available."**
|
||||
|
||||
### Phase 13: Multi-store storage (~5 days)
|
||||
### Phase 6: Reasoning harness + validation (~4 days)
|
||||
|
||||
Adds Postgres, pgvector, MinIO, and the cross-store compose layer.
|
||||
The system prompt from `07-reasoning-harness.md`, the test harness, and the validation pass.
|
||||
|
||||
- [ ] Add Postgres, pgvector, and MinIO to docker-compose.
|
||||
- [ ] Implement the Postgres schema from `12-storage-strategy.md#schema-overview`.
|
||||
- [ ] Implement the cross-store compose layer in the relevant handlers.
|
||||
- [ ] Migrate `lore_chunk` and `domain_entity_summary` embeddings to pgvector.
|
||||
- [ ] Move lore source text to MinIO; keep metadata in Neo4j.
|
||||
- [ ] Add the saga pattern for multi-store writes (mission_log + DomainEntity + S3 attachment).
|
||||
- [ ] Write the system prompt.
|
||||
- [ ] Build a test harness: 50 worked questions, expected tool sequences, expected answer shape.
|
||||
- [ ] Run a "red team" session: deliberately adversarial questions, edge cases, contradiction traps. Document the failure modes.
|
||||
- [ ] Iterate on the system prompt and tool descriptions.
|
||||
- [ ] Measure tool-selection accuracy across the 45-tool surface; collapse the long tail if the LLM is tool-confused.
|
||||
|
||||
**Verify:** ingest a war campaign, query `campaign_strength` against Postgres event log, get a real-time answer that includes both Neo4j and Postgres data. Verify the saga rolls back cleanly on simulated mid-saga failure.
|
||||
**Verify:** the LLM, with the system prompt and the tool surface, answers 80%+ of the test questions correctly. The remaining 20% are documented as known limitations.
|
||||
|
||||
### Phase 14: External handler protocol (~3 days)
|
||||
### Phase 7: Polish (open-ended)
|
||||
|
||||
The HTTP-based plugin protocol, exercised with one external handler.
|
||||
- [ ] UI for the consistency engine (browse contradictions, anachronisms, orphans).
|
||||
- [ ] UI for world-builders (YAML editor with autocomplete, validation, preview).
|
||||
- [ ] Export: render the world as a wiki, a book, a campaign primer.
|
||||
- [ ] Versioning: graph snapshots, time-travel queries.
|
||||
- [ ] Cross-setting queries: the engine is per-setting, but a future version supports multiple.
|
||||
|
||||
- [ ] Add auth + rate-limiting to the gateway's HTTP-handler dispatch.
|
||||
- [ ] Build one external handler as a proof — a Python `entity-linker` that uses a transformer model for ambiguous entity resolution.
|
||||
- [ ] Add a `register_external_handler` admin tool.
|
||||
---
|
||||
|
||||
**Verify:** start the Python entity-linker, register it with the gateway, send an ambiguous `lookup` query, see the Python handler called, get a resolved entity.
|
||||
## Total scope: v1 + v1.1 on Cognee
|
||||
|
||||
### v1.1 total scope: ~20 days
|
||||
|
||||
v1.1 is similar in scope to v1 (Phases 0–4 = 19 days). It's the *next* MVP — the one that adds extensibility, multi-store, and decomposition. Both v1 and v1.1 are buildable; both produce a working system.
|
||||
|
||||
### v1.1 milestones (added to the Gitea repo)
|
||||
|
||||
| Milestone | Phase | Days |
|
||||
| Phase | Days | Cumulative |
|
||||
|---|---|---|
|
||||
| M7 — Gateway Refactor | Phase 11 | 5 |
|
||||
| M8 — Polymorphic Extensions | Phase 12 | 7 |
|
||||
| M9 — Multi-Store Storage | Phase 13 | 5 |
|
||||
| M10 — External Handlers | Phase 14 | 3 |
|
||||
| 0 — Cognee spike | 2 | 2 |
|
||||
| 1 — Lore Engine ontology on Cognee | 5 | 7 |
|
||||
| 2 — Time model + UDFs | 4 | 11 |
|
||||
| 3 — MCP tool layer | 5 | 16 |
|
||||
| 4 — Consistency engine | 6 | 22 |
|
||||
| 5 — TypeTemplate polymorphic extension | 7 | 29 |
|
||||
| 6 — Reasoning harness + validation | 4 | 33 |
|
||||
| 7 — Polish | — | — |
|
||||
| **Total to v1+ext (Phases 0–6)** | **33 days** | **end of phase 6** |
|
||||
|
||||
### What to cut from v1.1 if you're under time pressure
|
||||
**The MVP is end of phase 3 (16 days).** Schema, UDFs, 45 MCP tools, structured ingestion, lookup/entity_context/state_at, all on Cognee. World-builders can start writing YAML and the LLM can start answering time-bounded questions.
|
||||
|
||||
**The v1 + extensions is end of phase 6 (33 days).** Adds the consistency engine, the TypeTemplate polymorphic extension model, and the reasoning-harness validation. The full 18-doc design contract is implemented.
|
||||
|
||||
Compared to the original v1+v1.1 plan (43 days on GraphMCP-Example), the Cognee-based plan saves ~10 days by inheriting the storage abstraction, the extraction pipeline, the embedding store, and the agent-native API. The 17 days of v1.1 modularization work collapses into the 7-day Phase 5 (TypeTemplate) plus the 6-day Phase 4 (consistency engine), because Cognee handles the gateway and decomposition story.
|
||||
|
||||
## What to cut from the full plan if you're under time pressure
|
||||
|
||||
In strict order:
|
||||
|
||||
1. Phase 14 (External Handlers) — defer to a v1.2. The internal handler decomposition from Phase 11 is enough for the world-builder's needs.
|
||||
2. The saga pattern (Phase 13) — start with best-effort multi-store writes, no rollback. Add the saga when you actually see data corruption in production.
|
||||
3. MinIO — start with local filesystem for lore source text. Migrate to MinIO when you actually have a use case for cross-host blob storage.
|
||||
1. Phase 7 — Polish. Trivially deferrable.
|
||||
2. Phase 6 — Reasoning harness validation depth. Ship with 20 test questions instead of 50; iterate from observed failure modes.
|
||||
3. Phase 5 — TypeTemplate polymorphic extension. The v1 ontology covers the macro structure; the polymorphic wrapper is a powerful addition but not a v1.1 blocker.
|
||||
4. Phase 4 — Consistency engine. Start with the 3 most valuable rule categories (Contradiction, Anachronism, Orphan) and skip OntologyViolation in the first build.
|
||||
|
||||
### The recommended order: v1 → validate → v1.1
|
||||
The **non-cuttable core is Phases 0–3 + Phase 4**. Substrate validation, typed ontology, time model, MCP tool layer, and the basic consistency engine. Everything else is enhancement.
|
||||
|
||||
1. Build v1 Phases 0–4 (19 days). Validate: the time model works, structured ingestion is fast, the LLM can answer time-bounded questions correctly.
|
||||
2. Build v1.1 Phase 11 (5 days, mechanical). The mcp-server is now decomposable.
|
||||
3. Build v1.1 Phase 12 (7 days, polymorphic extensions). This is the phase that unlocks the "arbitrary new concept" question. **Ship this second because it's the highest-leverage single change after v1's data layer.**
|
||||
4. Build v1.1 Phase 13 (5 days, multi-store). The system now scales to the world's actual size.
|
||||
5. Build v1.1 Phase 14 (3 days, external handlers). Defer this if you can.
|
||||
## The recommended order: spike → MVP → validate → extensions
|
||||
|
||||
The full v1 + v1.1 is ~43 days. The MVP is 19. The extensibility is 26 (5 + 7 + 5 + 3 + Phases 5–10 from v1 = 11 days). Build the smallest thing that works; iterate from there.
|
||||
1. **Phase 0 — Cognee spike (2 days).** Stand up Cognee locally. Ingest a 10-document sample world. Run `cognee.recall("Who is Aldric?")`. **If the spike fails, the substrate decision is wrong and the v1 needs a different foundation.** The 2-day validation is the cheapest insurance available.
|
||||
2. **Phases 1–3 — MVP (16 days).** Typed ontology, time model, MCP tool layer. The LLM can answer time-bounded questions against a hand-crafted world.
|
||||
3. **Phase 4 — Consistency engine (6 days).** The engine flags its first real contradiction.
|
||||
4. **Phase 5 — TypeTemplate polymorphic extension (7 days).** World-builders add new domain types as YAML. **This is the phase that unlocks the "arbitrary new concept" question. Ship this second because it's the highest-leverage single change after the v1 data layer.**
|
||||
5. **Phase 6 — Reasoning harness + validation (4 days).** Measure: how often does the LLM answer correctly? how often does it surface contradictions? how often does it hallucinate? **This is the validation gate.** If the numbers aren't good, the design has a bug and the v2 should address it.
|
||||
|
||||
@@ -67,12 +67,12 @@ These are the only LLM-in-the-loop read tools. They make multiple Cypher queries
|
||||
|
||||
**Status:** documented, gated behind explicit LLM request.
|
||||
|
||||
### S2.4 — The 30-tool surface is at the LLM's tool-use ceiling
|
||||
### S2.4 — The 45-tool surface is past the LLM's tool-use ceiling
|
||||
|
||||
Empirically, LLMs start making poor tool choices past ~25 tools in the same system prompt. We're at 30.
|
||||
Empirically, LLMs start making poor tool choices past ~25 tools in the same system prompt. The current catalog is 8 inherited + 37 new = **45 tools**, well past the ceiling.
|
||||
|
||||
**Fix:**
|
||||
- Phase 10 test: measure tool selection accuracy with all 30 vs. with the 8 most-used. If 8 is dramatically better, collapse the long tail.
|
||||
- Phase 6 test: measure tool selection accuracy with all 45 vs. with the 8 most-used. If 8 is dramatically better, collapse the long tail.
|
||||
- Group tools by function in the system prompt (we already do this) and instruct the LLM to look at the relevant group first.
|
||||
- If still bad: collapse `state_at` into `entity_context` (with optional `comprehensive: true`), and `summarize_chain` into `narrate_arc` (with optional `style: bullets`).
|
||||
|
||||
@@ -92,7 +92,7 @@ A claim like "the prophecy says the Crimson Throne will fall" is in the graph as
|
||||
|
||||
The engine is per-world. A future version might want to query across two worlds (for a multi-world campaign or a comparison). The schema doesn't support this — the `:Era` slugs aren't namespaced.
|
||||
|
||||
**Fix (v2):** introduce a `world_id` property on every node, or namespace the slugs (`{world}.{era}.{year}`). v1 doesn't need this; v2 should.
|
||||
**Fix (v1.2, resolved):** the v1.2 `Setting` and `Plane` graph nodes + `EXISTS_IN` edges replace the v1.1 flat `world_id` string namespace. Multi-setting queries are now supported via `Setting` filters and `EXISTS_IN` traversal. The "deferred to v2" framing in the v1 review is no longer accurate — the resolution is the v1.2 plane model. See `17-planes.md`.
|
||||
|
||||
**Status:** deferred.
|
||||
|
||||
@@ -130,9 +130,9 @@ This isn't a bug, it's a feature. The engine is *bounded* by its sources. The LL
|
||||
|
||||
### S4.2 — The "best" tool for the LLM is the one it actually uses
|
||||
|
||||
We designed 30 tools. The LLM might use 8 of them 95% of the time. The other 22 are dead weight — they bloat the system prompt and confuse the tool-selection logic.
|
||||
We designed 45 tools (8 inherited + 37 new). The LLM might use 8 of them 95% of the time. The other 37 are dead weight — they bloat the system prompt and confuse the tool-selection logic.
|
||||
|
||||
**Fix:** measure tool usage in Phase 10. Tools with <2% usage in test sessions get either promoted (made part of a higher-level tool) or pruned. The design is a *floor*, not a *ceiling*. We add tools; we don't take them away unless evidence says we should.
|
||||
**Fix:** measure tool usage in Phase 6. Tools with <2% usage in test sessions get either promoted (made part of a higher-level tool) or pruned. The design is a *floor*, not a *ceiling*. We add tools; we don't take them away unless evidence says we should.
|
||||
|
||||
**Status:** ongoing. Re-evaluate after Phase 10.
|
||||
|
||||
@@ -149,7 +149,7 @@ A world-builder changes the lore. The engine must absorb the change without brea
|
||||
These are decisions I couldn't make alone. The world-builder should answer them before Phase 1.
|
||||
|
||||
1. ~~**How granular is the time model in practice?**~~ **Resolved (Q1):** year-level precision is the default, with optional month/day/event precision when the source supports it. The UDF and the storage cost are unchanged.
|
||||
2. ~~**Are there multi-world / planar structures?**~~ **Resolved (Q2):** yes. The engine adds a `world_id` namespace and a `Plane` label. Multi-world queries are supported via `world_id` filters; planar relationships via `Plane` and `EXISTS_IN` edges.
|
||||
2. ~~**Are there multi-world / planar structures?**~~ **Resolved (Q2):** yes. The engine adds `Setting` and `Plane` graph nodes (v1.2); the v1.1 flat `world_id` string namespace is deprecated. Multi-setting queries are supported via `Setting` filters; planar relationships via `Plane`, `EXISTS_IN`, and the four plane-relation edge types (`REFLECTS`, `LAYER_OF`, `ADJACENT_TO`, `ACCESSIBLE_VIA`). See `17-planes.md`.
|
||||
3. ~~**How are NPCs and PC players modeled?**~~ **Resolved (Q3):** separately. The `NPC`, `PC`, and `Human` labels in `01-ontology.md` cover this. The in-fiction `Person` is canonical; the wrappers track who controls it.
|
||||
4. ~~**What's the policy on retconning?**~~ **Resolved (Q4):** preserve history by default. Old edges/nodes are marked `retconned` with a snapshot in the `retcon` Postgres table (`12-storage-strategy.md#postgres-schema`). Explicit `DELETE` is the only way to remove something permanently.
|
||||
5. ~~**How is the world bootstrapped?**~~ **Resolved (Q5):** organically over a long period. The engine supports partial worlds (some eras defined, some not), and the consistency engine surfaces missing structural data as `:Orphan` nodes. No need to pre-define everything.
|
||||
@@ -191,38 +191,38 @@ After the v1 review, the modularization question surfaced four new design risks
|
||||
|
||||
### S1.4 (NEW, blocker) — Closed-world ontology ceiling
|
||||
|
||||
The v1 ontology has 14 hard-coded labels. A thieves-guild mission is forced into `Event`, a war campaign is forced into `Faction`-with-properties, a black-market trade log is forced into `Item`-with-properties. The LLM can *talk* about these things, but the engine can't *reason* over their structure.
|
||||
The v1 ontology has 36 hard-coded labels (7 inherited + 19 v1 core + 2 v1.2 planes + 6 v1.1 polymorphic + 5 consistency). A thieves-guild mission is forced into `Event`, a war campaign is forced into `Faction`-with-properties, a black-market trade log is forced into `Item`-with-properties. The LLM can *talk* about these things, but the engine can't *reason* over their structure.
|
||||
|
||||
**Fix:** v1.1 introduces the `DomainEntity` polymorphic wrapper + `TypeTemplate` data-defined schemas. See `11-extensibility.md`. This is the load-bearing change for "arbitrary new concept, define how it associates with larger constructs, but also have flexibility to get as detailed as we need."
|
||||
**Fix:** the polymorphic `DomainEntity` wrapper + `TypeTemplate` data-defined schemas. See `11-extensibility.md`. This is the load-bearing change for "arbitrary new concept, define how it associates with larger constructs, but also have flexibility to get as detailed as we need."
|
||||
|
||||
**Status:** resolved in v1.1 design. Implementation is Phase 3 of v1.1 roadmap.
|
||||
**Status:** resolved in v1.2 design. The polymorphic extension model is **shipped in the MVP** (it's how the v1 ontology becomes extensible without code). The template-watcher is a Cognee data-pipeline; the dynamic tool generator is part of the Lore Engine extension. Implementation is Phase 5 of the Cognee roadmap in `09-roadmap.md`.
|
||||
|
||||
### S1.5 (NEW, blocker) — Single mcp-server binary blocks iteration
|
||||
|
||||
The existing GraphMCP-Example `mcp-server/main.go` is a 1144-line single file. Adding a new tool means editing main.go, recompiling, redeploying. The iteration loop for a world that's going to grow indefinitely is the cost of the entire program.
|
||||
The original GraphMCP-Example `mcp-server/main.go` was a 1144-line single file. Adding a new tool meant editing main.go, recompiling, redeploying. The iteration loop for a world that's going to grow indefinitely is the cost of the entire program.
|
||||
|
||||
**Fix:** v1.1 decomposes the mcp-server into a small gateway + a set of pluggable handlers, with a tool-registry and a template-driven tool generator. See `13-microservice-decomposition.md`.
|
||||
**Fix:** switching to Cognee as the substrate. Cognee is the gateway; the Lore Engine is one in-process Python extension (one tool per Group file, registered at startup). Adding a new tool is a Python edit + Cognee restart (5 minutes). Adding a new domain type is a YAML file + hot-reload (no restart). See `13-microservice-decomposition.md`.
|
||||
|
||||
**Status:** resolved in v1.1 design. Implementation is Phase 1-3 of v1.1 (mechanical refactor, no new functionality).
|
||||
**Status:** N/A in v1.2. The substrate switch resolves this completely. The Lore Engine does not own the mcp-server; Cognee does.
|
||||
|
||||
### S2.5 (NEW, design risk) — The polymorphic wrapper adds query complexity
|
||||
|
||||
Every `DomainEntity` query is now polymorphic — the engine has to look up the template, get the field names, build the right query. The performance overhead is small for typed queries, but for `expand_context` and `graph_traverse`, the engine has to follow relations through the `Relation` label and re-resolve the template for each step.
|
||||
|
||||
**Fix:** the engine caches `TypeTemplate` lookups in Redis. The first time a template is referenced, its spec is loaded; subsequent queries use the cached version. Cache invalidation is on template reload (hot-reload event).
|
||||
**Fix:** Cognee caches `TypeTemplate` lookups in its in-process store. The first time a template is referenced, its spec is loaded; subsequent queries use the cached version. Cache invalidation is on template reload (hot-reload event from the template-watcher data-pipeline). Cognee's caching layer handles this without us writing a custom cache.
|
||||
|
||||
**Status:** acknowledged, fix designed, implementation in Phase 2 of v1.1.
|
||||
**Status:** acknowledged, fix designed, implementation in Phase 5 of the Cognee roadmap.
|
||||
|
||||
### S2.6 (NEW, design risk) — Cross-store consistency is genuinely hard
|
||||
|
||||
When the world-builder writes a new mission, we touch Neo4j (entity, relations), Postgres (mission_log row), and S3 (any attachments). These three writes are not atomic. A partial failure leaves the world in an inconsistent state.
|
||||
When the world-builder writes a new mission, we touch the Cognee graph (entity, relations) and the operational Postgres tables (mission_log row). These two writes are not atomic. A partial failure leaves the world in an inconsistent state.
|
||||
|
||||
**Fix:** the saga pattern from `12-storage-strategy.md#the-cost-of-cross-store-transactions`. Each multi-store write is a saga; the engine records saga state in Postgres and rolls back partial failures.
|
||||
**Fix:** the saga pattern is **no longer needed**. Cognee manages its own transaction model for the graph + Postgres + vector store. The Lore Engine's operational tables are in Cognee's Postgres, so writes that touch the graph and the operational tables are managed by Cognee's atomicity guarantees. We do not need a custom saga layer.
|
||||
|
||||
**Status:** designed, implementation in Phase 4 of v1.1.
|
||||
**Status:** N/A in v1.2. Cognee handles this. The v1.1 saga-pattern section in `12-storage-strategy.md` has been removed.
|
||||
|
||||
## Conclusion
|
||||
|
||||
The design is **viable for v1**, with a clear scope of 19 days for the MVP (Phases 0–4), and **v1.1** extends it with the polymorphic extension model, the multi-store storage strategy, and the microservice decomposition. The 7 open questions are resolved. The biggest remaining risks are scale (entity resolution), over-flagging (consistency engine), cross-store consistency (sagas), and LLM misbehavior (harness enforcement). Each has a documented mitigation.
|
||||
The design is **viable for v1 on Cognee**, with a clear scope of 16 days for the MVP (Phases 0–3) and 33 days for the full v1 + extensions (Phases 0–6). The 7 open questions are resolved. The biggest remaining risks are scale (entity resolution), over-flagging (consistency engine), Cognee-specific substrate quirks, and LLM misbehavior (harness enforcement). Each has a documented mitigation.
|
||||
|
||||
I would build v1 first (Phases 0–4 of the original roadmap), validate the time model and structured ingestion, *then* layer v1.1 on top. The polymorphic extension model is the right shape for the world's growth, but it's the second thing to build, not the first.
|
||||
I would build the Cognee spike first (Phase 0, 2 days), validate the substrate, then proceed to the MVP (Phases 1–3, 14 days). The polymorphic extension model (Phase 5) and the consistency engine (Phase 4) are the highest-leverage v1.1 additions and ship in the same ~33-day window.
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# 11 — Extensibility: Polymorphic Type Templates
|
||||
|
||||
The v1 ontology has 14 hard-coded labels. That's *fine for the first world* but it has a ceiling: a thieves-guild mission is forced into `:Event`, a war campaign is forced into `:Faction`-with-properties, a black-market trade log is forced into `:Item`-with-properties. The LLM can *talk* about these things, but the engine can't *reason* over their structure.
|
||||
The v1 ontology has 36 hard-coded labels (7 inherited from Cognee + 19 v1 core + 2 v1.2 planes + 6 v1.1 polymorphic + 5 consistency). That's *fine for the first world* but it has a ceiling: a thieves-guild mission is forced into `:Event`, a war campaign is forced into `:Faction`-with-properties, a black-market trade log is forced into `:Item`-with-properties. The LLM can *talk* about these things, but the engine can't *reason* over their structure.
|
||||
|
||||
**"What missions has the Crimson Hand run in Mardsville over the last year, sorted by payout?"** is unanswerable today. The data lives in `summary` text fields, the relationships are implicit in prose, and the LLM has to reconstruct it via `semantic_search` and hope.
|
||||
|
||||
@@ -8,7 +8,7 @@ This document is the v1.1 fix: a **type-template system** that lets the world-bu
|
||||
|
||||
## The principle: types are data, not schema
|
||||
|
||||
Today: a new domain type means a Go change, a Cypher change, a Neo4j migration, a docker rebuild.
|
||||
Today (without v1.1): a new domain type means a Go change, a Cypher change, a Neo4j migration, a docker rebuild.
|
||||
|
||||
Goal: a new domain type means a YAML file. The engine reads it. The LLM gets new tools automatically. No code.
|
||||
|
||||
@@ -18,13 +18,15 @@ This is the same pattern as WordPress custom post types, Salesforce custom objec
|
||||
|
||||
```
|
||||
┌──────────────────────────────────────────────────────────────────┐
|
||||
│ LAYER 1: Core Ontology (fixed, in code) │
|
||||
│ ────────────────────────────────────── │
|
||||
│ LAYER 1: Core Ontology (Cognee data-model extension) │
|
||||
│ ────────────────────────────────────────────── │
|
||||
│ Person, Faction, Location, Item, Era, Date, Lineage, │
|
||||
│ Culture, Deity, Language, MagicSystem, Title, Region, Material │
|
||||
│ + Setting, Plane (v1.2) │
|
||||
│ + NPC, PC, Human (v1.1) │
|
||||
│ + the time model + the consistency engine │
|
||||
│ + the MCP transport + the active context │
|
||||
│ This is ~3000 lines of Go and ~500 lines of Cypher. Stable. │
|
||||
│ This is a Cognee data-model extension (Python + Cypher). Stable.│
|
||||
└──────────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
│ every node is also a...
|
||||
@@ -33,14 +35,14 @@ This is the same pattern as WordPress custom post types, Salesforce custom objec
|
||||
│ LAYER 2: Polymorphic "Domain Entity" wrapper │
|
||||
│ ────────────────────────────────────── │
|
||||
│ (:DomainEntity { │
|
||||
│ id, type, name, era_id, world_id, │
|
||||
│ id, type, name, era_id, setting_id, │
|
||||
│ properties: map<string, any>, ← free-form, type-checked │
|
||||
│ template_id: "thieves_guild_mission", │
|
||||
│ sources[], lore_verified, source_confidence │
|
||||
│ }) │
|
||||
│ Plus: (:Relation {from, to, type, properties, ...}) for │
|
||||
│ arbitrary edges between DomainEntities. │
|
||||
│ This is ~200 lines of Cypher. Stable. │
|
||||
│ This is part of the Cognee data-model extension. Stable. │
|
||||
└──────────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
│ defined as data in...
|
||||
@@ -55,6 +57,7 @@ This is the same pattern as WordPress custom post types, Salesforce custom objec
|
||||
│ - which ontology rules apply │
|
||||
│ - which LLM-inference hints (e.g. "treat this as a secret") │
|
||||
│ This is the part that grows. │
|
||||
│ The template-watcher is a Cognee data-pipeline. │
|
||||
└──────────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
│ instances of...
|
||||
@@ -76,7 +79,7 @@ The world-builder writes Layer 4 (instances) and Layer 3 (templates). The engine
|
||||
|
||||
## Layer 1: Core ontology (unchanged from v1)
|
||||
|
||||
The 14 labels stay. They cover the *macro* structure of any world: who exists, where they are, when they lived, what they belonged to. This is stable.
|
||||
The 36 labels stay. They cover the *macro* structure of any world: who exists, where they are, when they lived, what they belonged to, and the structural shape of planes, magic, and culture. This is stable.
|
||||
|
||||
## Layer 2: The `DomainEntity` wrapper
|
||||
|
||||
@@ -89,7 +92,7 @@ A single new node label and a single new edge label. This is the polymorphic bac
|
||||
id: "mission_4471",
|
||||
type: "ThievesGuildMission", // matches a :TypeTemplate.name
|
||||
name: "The Pale Ledger Heist",
|
||||
world_id: "arda_1st_age",
|
||||
setting_id: "mardonari",
|
||||
|
||||
// Polymorphic, type-checked properties from the template
|
||||
properties: {
|
||||
@@ -142,8 +145,8 @@ FOR (r:Relation) REQUIRE r.id IS UNIQUE;
|
||||
CREATE INDEX domain_entity_type IF NOT EXISTS
|
||||
FOR (d:DomainEntity) ON (d.type);
|
||||
|
||||
CREATE INDEX domain_entity_world IF NOT EXISTS
|
||||
FOR (d:DomainEntity) ON (d.world_id);
|
||||
CREATE INDEX domain_entity_setting IF NOT EXISTS
|
||||
FOR (d:DomainEntity) ON (d.setting_id);
|
||||
|
||||
CREATE INDEX relation_type IF NOT EXISTS
|
||||
FOR (r:Relation) ON (r.type);
|
||||
@@ -267,16 +270,17 @@ Hot reload — no engine restart.
|
||||
|
||||
```
|
||||
On ./templates/thieves_guild/mission.yaml file change:
|
||||
1. YAML parser → schema validation
|
||||
2. Cypher write: MERGE (t:TypeTemplate {id: "thieves_guild_mission"}) SET t.<spec>
|
||||
3. Tool registry: for each entry in spec.tools, register a tool handler
|
||||
1. Cognee data-pipeline (template-watcher) picks up the change.
|
||||
2. YAML parser → schema validation
|
||||
3. Cypher write: MERGE (t:TypeTemplate {id: "thieves_guild_mission"}) SET t.<spec>
|
||||
4. Tool registry: for each entry in spec.tools, register a tool handler
|
||||
(the handler is a generic "domain tool runner" — same code, different params)
|
||||
4. Rule registry: for each entry in spec.rules, attach to consistency engine
|
||||
5. Hot-reload notification: send_message to subscribed LLM clients
|
||||
5. Rule registry: for each entry in spec.rules, attach to consistency engine
|
||||
6. Hot-reload notification: send_message to subscribed LLM clients
|
||||
"Domain type thieves_guild_mission registered; 6 new tools available"
|
||||
```
|
||||
|
||||
**The MCP server is now a generic tool runner. It doesn't know about thieves guilds, war campaigns, or black markets. It knows about TypeTemplate nodes and dispatches generically.**
|
||||
**The Cognee MCP server is now a generic tool runner. It doesn't know about thieves guilds, war campaigns, or black markets. It knows about TypeTemplate nodes and dispatches generically.** The template-watcher is a Cognee data-pipeline that watches `./templates/` and re-registers the spec on change.
|
||||
|
||||
This is the architectural change that unlocks everything else.
|
||||
|
||||
@@ -289,7 +293,7 @@ A world-builder writes a YAML file with concrete missions. The schema is enforce
|
||||
# Ingested via POST /ingest/structured?type=thieves_guild_mission
|
||||
|
||||
template: "thieves_guild_mission"
|
||||
world_id: "arda_1st_age"
|
||||
setting_id: "mardonari"
|
||||
instances:
|
||||
- mission_code: "M-4471"
|
||||
name: "The Pale Ledger Heist"
|
||||
|
||||
@@ -1,46 +1,45 @@
|
||||
# 12 — Storage Strategy: Which Data Goes Where
|
||||
|
||||
The v1 design treats Neo4j as the universal substrate. For v1.1, with polymorphic domain entities, vector embeddings, time-series events, and high-volume operational logs (trade lots, mission outcomes, campaign movements), **Neo4j alone is the wrong tool for the job.** Different data has different access patterns, and forcing them all into one graph makes the graph bad at everything.
|
||||
The Lore Engine runs on [Cognee](https://github.com/topoteretes/cognee), which manages the storage abstraction. Cognee's default backend is **Neo4j 5.x** (graph) + **a vector store** (Qdrant or pgvector, Cognee chooses based on the deployment) + **Postgres** (metadata, sessions, task state). The Lore Engine does not manage the graph, vector, or metadata stores directly — it adds a domain layer that uses them.
|
||||
|
||||
This document is the storage role split: which database stores which kind of data, why, and how the engine queries across them.
|
||||
For v1.1, the Lore Engine **does** add its own Postgres tables (operational event logs, the `setting` table, retcon history, the template registry) on top of the Cognee-managed instance. This document describes which Lore Engine data goes where and why.
|
||||
|
||||
## The principle: pick the right tool for the access pattern
|
||||
## The principle: Cognee manages substrate; Lore Engine manages domain
|
||||
|
||||
Neo4j is excellent at: relationship traversal, graph pattern matching, recursive lineage, spatial aggregation. **It is mediocre at: full-text search, large property blobs, high-volume time-series ingestion, free-form JSON querying.**
|
||||
Cognee provides the storage role split. The Lore Engine adds:
|
||||
- A **typed ontology** (the 36 labels and their constraints) registered as a Cognee data-model extension.
|
||||
- A **time model** (`time_in_window` UDF, era-tree helpers) on top of Cognee's query layer.
|
||||
- A **consistency engine** (4 rule categories) as a Cognee data-pipeline that materializes violation nodes.
|
||||
- A **template registry** (the `TypeTemplate` nodes, the template-watcher data-pipeline) that hot-reloads new domain types.
|
||||
- **Operational tables** in Postgres (the `setting` table, `lore_event`, `retcon`, `dialogue_log`, etc.) on top of Cognee's metadata Postgres.
|
||||
|
||||
If we force high-volume operational data (every trade, every mission step, every army movement) into Neo4j properties, the graph bloats, indexes fragment, and queries slow down. The right move is to store each kind of data where it naturally lives, and have the engine compose across stores.
|
||||
## The three layers
|
||||
|
||||
## The five stores
|
||||
|
||||
| Store | What it holds | Why |
|
||||
| Layer | What it holds | Who manages |
|
||||
|---|---|---|
|
||||
| **Neo4j** | The world graph. People, factions, locations, lineage, era trees, type templates, domain entities with relationships, time-bounded edges, ontology rules, violation nodes. | Graph traversal is the primary access pattern. |
|
||||
| **PostgreSQL** | Operational records with structured schemas. Trade logs, mission step logs, campaign event streams, audit trails, world-builder write history, session state, MCP tool call logs. | Relational, high-volume, time-series-friendly, transactional. |
|
||||
| **Qdrant** *(or pgvector)* | Vector embeddings of `LoreChunk`, `Message`, and `DomainEntity.summary` text. | Semantic search is the primary access pattern. |
|
||||
| **Redis** | Active MCP session state, per-session `world_time`, tool-call rate-limit counters, in-flight transaction state, ephemeral caches. | Sub-millisecond, ephemeral. |
|
||||
| **S3-compatible object store** (MinIO) | Full text of lore sources, images, audio, large attachments, archival snapshots. | Blob storage, cheap, durable. |
|
||||
| **Cognee substrate** (Neo4j + vector store + Postgres metadata) | The graph, the embeddings, the sessions, the task state. Cognee adds its own `DataPoint`, `Chunk`, `Entity` types automatically. | Cognee (we don't manage these directly) |
|
||||
| **Lore Engine data-model extension** (added on top of Cognee's substrate) | The 36 typed labels (Person, Faction, Location, ..., Setting, Plane, DomainEntity), constraints, indexes, the `time_in_window` and `time_windows_overlap` UDFs. | The Lore Engine registers these as a Cognee data-model extension at startup. |
|
||||
| **Lore Engine operational tables** (in Cognee's Postgres) | The `setting` table, `lore_event`, `retcon`, `dialogue_log`, the template registry. The world's operational event history. | The Lore Engine writes to its own tables through Cognee's Postgres adapter. |
|
||||
|
||||
Existing GraphMCP-Example has Neo4j + Redis + the LLM proxy. We add **PostgreSQL** (the big new one), **Qdrant** (or pgvector, for self-contained deployments), and **MinIO** (or any S3 bucket).
|
||||
The distinction matters: we don't manage Neo4j or the vector store; we manage the typed labels and the operational tables on top. Cognee handles connection pooling, schema migrations for its own types, embedding refresh, and the cross-store query layer.
|
||||
|
||||
## What goes in Neo4j
|
||||
## What the Lore Engine adds on top of the Cognee graph
|
||||
|
||||
The macro world graph. Anything where the LLM will say *"traverse from A"* or *"find all X related to Y"* or *"is X connected to Y?"* — that lives in Neo4j.
|
||||
The macro world graph lives in the Cognee-managed graph store (Neo4j or Kuzu). The Lore Engine's data-model extension adds the typed labels and constraints on top. Anything where the LLM will say *"traverse from A"* or *"find all X related to Y"* or *"is X connected to Y?"* is a typed node or edge in the graph.
|
||||
|
||||
- **Core entities:** Person, Faction, Location, Item, Era, Date, Lineage, Culture, Deity, Language, MagicSystem, Title, Region, Material.
|
||||
- **Core entities (v1):** Person, Faction, Location, Item, Era, Date, Lineage, Culture, Deity, Language, MagicSystem, Title, Region, Material.
|
||||
- **Planes (v1.2):** `Setting` and `Plane` are first-class graph nodes. Plane relations (`REFLECTS`, `LAYER_OF`, `ADJACENT_TO`, `ACCESSIBLE_VIA`) are first-class edges. The `EXISTS_IN` and `LOCATED_IN` relations between entities and planes are stored here too. See `17-planes.md`.
|
||||
- **Time-bounded relations** between core entities: `RULED`, `MEMBER_OF`, `LOCATED_IN`, `PARTICIPATED_IN`, `ALLIED_WITH`, `POSSESSES`, etc. Always time-bounded. Always queryable via `time_in_window`.
|
||||
- **Polymorphic domain entities** (`:DomainEntity` with a `template_id`): a thieves-guild Mission, a war Campaign, a Spellbook, a TradeLot, a Ritual. The entity *itself* and its relations to other entities (Person, Faction, Location, other DomainEntities) live in Neo4j.
|
||||
- **Polymorphic domain entities** (`:DomainEntity` with a `template_id`): a thieves-guild Mission, a war Campaign, a Spellbook, a TradeLot, a Ritual. The entity *itself* and its relations to other entities (Person, Faction, Location, other DomainEntities) live in the graph.
|
||||
- **Type templates** (`:TypeTemplate`): the YAML-defined schemas, stored as parsed JSON for the consistency engine and LLM to query.
|
||||
- **Violation nodes** (`:Contradiction`, `:Anachronism`, `:Orphan`, `:OntologyViolation`, `:ConsistencyRun`): the consistency engine's output.
|
||||
- **Lore source metadata** (`:LoreSource`): title, source_type, author, ingested_at, version. The *text* lives in object storage; the metadata is in Neo4j.
|
||||
- **Indexes:** all property indexes from `01-ontology.md` and `08-architecture.md`. Plus `(:DomainEntity).type`, `(:DomainEntity).world_id`, `(:Relation).type`, `(:Relation).valid_from/until`.
|
||||
- **Lore source metadata** (`:LoreSource`): title, source_type, author, ingested_at, version. The *text* lives in Cognee-managed chunk storage; the metadata is in the graph.
|
||||
- **Indexes:** all property indexes from `01-ontology.md` and `08-architecture.md`. Plus `(:DomainEntity).type`, `(:DomainEntity).setting_id`, `(:Relation).type`, `(:Relation).valid_from/until`.
|
||||
|
||||
**What does NOT go in Neo4j:**
|
||||
- The full text of a lore source. (Goes in S3, with a pointer in Neo4j.)
|
||||
- The full text of a domain entity's `summary` (above some length threshold). (Goes in S3; short summaries stay in Neo4j for semantic-search embedding.)
|
||||
- The step-by-step log of a mission. (Goes in Postgres; only the *aggregate outcome* lives in Neo4j as the Mission node.)
|
||||
- Vector embeddings. (Goes in Qdrant; Neo4j's vector index is OK but not great for high-volume semantic search.)
|
||||
- High-volume time-series operational data.
|
||||
**What does NOT live as a typed graph node:**
|
||||
- The full text of a lore source. (Cognee chunks it for embedding; we don't store the text in our typed graph.)
|
||||
- The step-by-step log of a mission. (Goes in our Postgres operational tables; only the *aggregate outcome* lives in the graph as the Mission node.)
|
||||
- High-volume time-series operational data. (Goes in Postgres; not in the graph.)
|
||||
|
||||
## What goes in PostgreSQL
|
||||
|
||||
@@ -48,14 +47,17 @@ Operational records that are *append-mostly*, *high-volume*, and *not primarily
|
||||
|
||||
The shape that Postgres handles well: rows of typed columns, indexed on time, with foreign keys back to Neo4j IDs.
|
||||
|
||||
> **v1.2 note:** the `setting` table (renamed from the v1.1 `world` table) is the v1.2 namespace that backs the `(:Setting)` and `(:Plane)` graph nodes. The `world_id` string property is deprecated in v1.2 in favor of structured `EXISTS_IN` graph edges; Postgres follows the same migration. See `17-planes.md`.
|
||||
|
||||
### Schema overview
|
||||
|
||||
```sql
|
||||
-- World, version, and migration state
|
||||
CREATE TABLE world (
|
||||
-- Setting (a v1.2 setting; renamed from v1.1's `world` table), version, and migration state
|
||||
CREATE TABLE setting (
|
||||
id TEXT PRIMARY KEY,
|
||||
name TEXT NOT NULL,
|
||||
current_era TEXT NOT NULL, -- canonical time string
|
||||
kind TEXT NOT NULL DEFAULT 'single_plane', -- 'single_plane' | 'multi_plane'
|
||||
current_era TEXT NOT NULL, -- canonical time string
|
||||
schema_version TEXT NOT NULL,
|
||||
created_at TIMESTAMPTZ DEFAULT now()
|
||||
);
|
||||
@@ -63,7 +65,7 @@ CREATE TABLE world (
|
||||
-- Operational event log (every meaningful state change)
|
||||
CREATE TABLE lore_event (
|
||||
id BIGSERIAL PRIMARY KEY,
|
||||
world_id TEXT REFERENCES world(id),
|
||||
setting_id TEXT REFERENCES setting(id),
|
||||
event_type TEXT NOT NULL, -- 'mission_logged', 'trade_completed', 'army_moved', ...
|
||||
entity_id TEXT, -- DomainEntity.id from Neo4j
|
||||
entity_type TEXT, -- discriminator
|
||||
@@ -74,7 +76,7 @@ CREATE TABLE lore_event (
|
||||
actor_id TEXT, -- who/what triggered this
|
||||
created_at TIMESTAMPTZ DEFAULT now()
|
||||
);
|
||||
CREATE INDEX ON lore_event (world_id, occurred_at DESC);
|
||||
CREATE INDEX ON lore_event (setting_id, occurred_at DESC);
|
||||
CREATE INDEX ON lore_event (entity_id, occurred_at DESC);
|
||||
CREATE INDEX ON lore_event (event_type, occurred_at DESC);
|
||||
CREATE INDEX ON lore_event USING GIN (payload);
|
||||
@@ -82,7 +84,7 @@ CREATE INDEX ON lore_event USING GIN (payload);
|
||||
-- Trade log (every lot, every transaction)
|
||||
CREATE TABLE trade_log (
|
||||
id BIGSERIAL PRIMARY KEY,
|
||||
world_id TEXT REFERENCES world(id),
|
||||
setting_id TEXT REFERENCES setting(id),
|
||||
lot_id TEXT NOT NULL,
|
||||
item_id TEXT, -- DomainEntity.id of the Item or Material
|
||||
buyer_id TEXT, -- Person or Faction id
|
||||
@@ -98,7 +100,7 @@ CREATE TABLE trade_log (
|
||||
payload JSONB, -- type-specific extras
|
||||
sources TEXT[]
|
||||
);
|
||||
CREATE INDEX ON trade_log (world_id, occurred_at DESC);
|
||||
CREATE INDEX ON trade_log (setting_id, occurred_at DESC);
|
||||
CREATE INDEX ON trade_log (lot_id);
|
||||
CREATE INDEX ON trade_log (buyer_id, occurred_at DESC);
|
||||
CREATE INDEX ON trade_log (seller_id, occurred_at DESC);
|
||||
@@ -158,7 +160,7 @@ CREATE INDEX ON tool_call (session_id, called_at DESC);
|
||||
-- Retcon history (Kay's Q4)
|
||||
CREATE TABLE retcon (
|
||||
id BIGSERIAL PRIMARY KEY,
|
||||
world_id TEXT REFERENCES world(id),
|
||||
setting_id TEXT REFERENCES setting(id),
|
||||
target_kind TEXT, -- 'entity' | 'relation' | 'property'
|
||||
target_id TEXT NOT NULL,
|
||||
before JSONB, -- snapshot of what was there
|
||||
@@ -173,7 +175,7 @@ CREATE INDEX ON retcon (target_id, retconned_at DESC);
|
||||
-- NPC dialogue history (for NPC knowledge scoping)
|
||||
CREATE TABLE dialogue_log (
|
||||
id BIGSERIAL PRIMARY KEY,
|
||||
world_id TEXT REFERENCES world(id),
|
||||
setting_id TEXT REFERENCES setting(id),
|
||||
npc_id TEXT NOT NULL, -- Person.id
|
||||
session_id TEXT,
|
||||
message TEXT NOT NULL,
|
||||
@@ -183,63 +185,41 @@ CREATE TABLE dialogue_log (
|
||||
CREATE INDEX ON dialogue_log (npc_id, occurred_at DESC);
|
||||
```
|
||||
|
||||
These tables are **the operational backbone**. They're what gets high-volume writes, transactional integrity, and time-series queries.
|
||||
These tables are **the Lore Engine's operational backbone** on top of Cognee's Postgres. They're what gets high-volume writes, transactional integrity, and time-series queries.
|
||||
|
||||
## What goes in Qdrant (or pgvector)
|
||||
## What Cognee handles for us (out of our control)
|
||||
|
||||
Vector embeddings for semantic search. Three collections:
|
||||
Cognee manages the following stores directly; the Lore Engine does not configure them. They are mentioned here for completeness so the LLM-tool author knows where the data is.
|
||||
|
||||
| Collection | Source | Dimension | Use |
|
||||
|---|---|---|---|
|
||||
| `lore_chunks` | `(:LoreChunk).text` | 768 | Semantic search over lore documents. |
|
||||
| `messages` | `(:Message).content` | 768 | Semantic search over dialogue. |
|
||||
| `domain_summaries` | `(:DomainEntity).summary` (if present) | 768 | Semantic search over domain entities. |
|
||||
- **Vector embeddings.** Cognee chunks the lore source text, embeds it, and indexes it. The Lore Engine queries this through `cognee.recall()` for the `lore_about` and `cite` tools. Cognee supports pgvector and Qdrant under the hood; the Lore Engine does not need to know which.
|
||||
- **Session state.** Cognee manages active MCP sessions, per-session context, and tool-call rate limits in its own store (typically Redis or in-memory, depending on the deployment). The Lore Engine's MCP server uses Cognee's session API.
|
||||
- **Ephemeral caches.** Cognee handles embedding caches, hot-reload pub/sub, and any other ephemeral state.
|
||||
- **Blob storage for source text.** Cognee stores the full text of ingested chunks. The Lore Engine never sees the full text — it only sees the typed graph nodes that Cognee extracted from the text.
|
||||
- **Cross-store transactions.** Cognee manages its own transaction model for writes that span the graph + Postgres + vector store. The Lore Engine calls `cognee.add()` / `cognee.cognify()` and Cognee handles atomicity. We do not need a custom saga layer.
|
||||
|
||||
The first two inherit from GraphMCP-Example. The third is new and only populated for domain entities that opt in (the `embedded: true` field in the template).
|
||||
The implication: the Lore Engine's storage strategy is **simpler than the v1.1 plan called for**. The polyglot-persistence story collapses to "Cognee manages the substrate, the Lore Engine owns the typed ontology and the operational tables."
|
||||
|
||||
**Qdrant vs pgvector:** Qdrant is faster and has better filtering. pgvector is simpler (one less service to run) and stays in the same DB family. For self-hosted homelab deployments where minimizing moving parts matters, **pgvector is the right call** for v1.1. We can swap to Qdrant later without changing the engine's API.
|
||||
## The cross-store compose layer (Lore Engine's job)
|
||||
|
||||
The lore-engine-Example already uses Neo4j's vector index. We can keep that for `lore_chunks` and add pgvector for `domain_summaries`, or migrate everything to pgvector. **Decision: pgvector for everything in v1.1.** Neo4j's vector index is fine but pgvector keeps our vector storage in one place.
|
||||
|
||||
## What goes in Redis
|
||||
|
||||
Ephemeral state. Lost on restart, not backed up.
|
||||
|
||||
- **Active session context** (replaces the in-process `sessionRegistry` in the existing MCP server). Each MCP session gets a Redis key with the active entity context, world_time override, and tool-call budget.
|
||||
- **Tool-call rate limits** (per session, per IP).
|
||||
- **Embedding cache** (frequently-searched queries → cached embedding).
|
||||
- **Pub/sub** for hot-reload notifications ("new template registered").
|
||||
- **In-flight transactions** (for multi-step writes that need atomicity across Neo4j + Postgres).
|
||||
|
||||
## What goes in S3 (MinIO)
|
||||
|
||||
- Full text of every `LoreSource` (the YAML/markdown the world-builder wrote).
|
||||
- Full text of every long `DomainEntity.summary`.
|
||||
- Attachments: images, audio, videos, large files referenced by lore.
|
||||
- Snapshots: world state exports, consistency engine reports, retcon bundles.
|
||||
|
||||
MinIO is the self-hosted S3. Same protocol, no AWS dependency.
|
||||
|
||||
## The cross-store query layer
|
||||
|
||||
The MCP tools the LLM uses compose across stores. The engine handles the cross-store joins; the LLM sees a unified response.
|
||||
The MCP tools the LLM uses compose across the Lore Engine's typed graph + the operational Postgres tables. The engine handles the joins; the LLM sees a unified response.
|
||||
|
||||
### Example: "What was the Crimson Hand's biggest heist in Mardsville last year?"
|
||||
|
||||
```
|
||||
LLM → tool: list_missions(filter_by={faction: "crimson_hand", location: "mardsville", since: "1_year_ago"}, sort_by="payout_gp", limit=5)
|
||||
|
||||
Engine:
|
||||
1. Neo4j: MATCH (m:DomainEntity {type: "ThievesGuildMission"})
|
||||
-[:TARGETS|F2]-> (loc:Location {name: "Mardsville"})
|
||||
-[:LOGGED_IN]-> (lot:DomainEntity {type: "TradeLogEntry"})
|
||||
-[:GIVEN_BY]-> (npc:Person {faction: "Crimson Hand"})
|
||||
RETURN m, lot, npc
|
||||
2. Postgres: SELECT * FROM mission_log WHERE mission_id IN (...) ORDER BY step_no
|
||||
3. Postgres: SELECT * FROM trade_log WHERE lot_id IN (...) AND occurred_at > ...
|
||||
Lore Engine:
|
||||
1. Graph (Cognee): MATCH (m:DomainEntity {type: "ThievesGuildMission"})
|
||||
-[:TARGETS]-> (loc:Location {name: "Mardsville"})
|
||||
-[:GIVEN_BY]-> (npc:Person {name: "Vex the Silent"})
|
||||
RETURN m
|
||||
2. Operational Postgres: SELECT * FROM mission_log
|
||||
WHERE mission_id IN (...) ORDER BY step_no
|
||||
3. Operational Postgres: SELECT * FROM trade_log
|
||||
WHERE lot_id IN (...) AND occurred_at > ...
|
||||
4. Compose: return top 5 by payout_gp, with mission step timeline + trade details
|
||||
|
||||
LLM: gets a unified response. Doesn't know it crossed 3 stores.
|
||||
LLM: gets a unified response. Doesn't know it crossed the graph + Postgres.
|
||||
```
|
||||
|
||||
### Example: "What battles did the Vyrs lose?"
|
||||
@@ -247,73 +227,50 @@ LLM: gets a unified response. Doesn't know it crossed 3 stores.
|
||||
```
|
||||
LLM → tool: list_campaign_events(filter_by={faction: "house_vyr", outcome: "loss"})
|
||||
|
||||
Engine:
|
||||
1. Neo4j: get the Campaign nodes tied to house_vyr
|
||||
2. Postgres: SELECT * FROM campaign_event
|
||||
Lore Engine:
|
||||
1. Graph: get the Campaign nodes tied to house_vyr
|
||||
2. Operational Postgres: SELECT * FROM campaign_event
|
||||
WHERE campaign_id IN (...) AND outcome = 'loss'
|
||||
ORDER BY occurred_at DESC
|
||||
3. Compose: return list with Neo4j faction details + Postgres battle details
|
||||
3. Compose: return list with graph faction details + Postgres battle details
|
||||
|
||||
LLM: unified response.
|
||||
```
|
||||
|
||||
The engine exposes **composed tools** like `list_missions`, `list_campaign_events`. The LLM calls one tool; the engine fans out across stores.
|
||||
The engine exposes **composed tools** like `list_missions`, `list_campaign_events`. The LLM calls one tool; the engine fans out across the typed graph and the operational tables.
|
||||
|
||||
## The cross-store consistency story
|
||||
|
||||
The consistency engine operates across stores. A `:Contradiction` node in Neo4j can reference a Postgres row. An `OntologyRule` in Neo4j can include Cypher that joins with a Postgres query (via Neo4j's apoc.load.jdbc).
|
||||
The consistency engine runs as a Cognee data-pipeline. A `:Contradiction` node in the graph can reference a row in `mission_log` or `campaign_event`. The rules that go cross-store:
|
||||
|
||||
The rules that go cross-store:
|
||||
|
||||
- *"A `:DomainEntity` of type `TradeLot` referenced in Neo4j must have a corresponding row in `trade_log`."*
|
||||
- *"A mission marked `status: 'completed'` in Neo4j must have a `step_type = 'completed'` row in `mission_log`."*
|
||||
- *"A `:DomainEntity` of type `TradeLot` referenced in the graph must have a corresponding row in `trade_log`."*
|
||||
- *"A mission marked `status: 'completed'` in the graph must have a `step_type = 'completed'` row in `mission_log`."*
|
||||
- *"A campaign event's `army_size` in Postgres must be within 10% of the `:DomainEntity` aggregate of the participating factions' `Person.count`."*
|
||||
|
||||
These rules are written in Cypher with `apoc.load.jdbc` calls. They run in the nightly batch.
|
||||
These rules are written in Cypher with Postgres lookups via Cognee's query API. They run in the nightly consistency pipeline.
|
||||
|
||||
## Why this is better than one big Neo4j
|
||||
## Why this is simpler than the 5-store v1.1 plan
|
||||
|
||||
| Concern | Neo4j-only | Polyglot |
|
||||
| Concern | Old v1.1 plan (5 stores) | Cognee-based v1.1 |
|
||||
|---|---|---|
|
||||
| High-volume writes (mission steps) | Bloats graph, slows down traversal | Postgres handles it cleanly |
|
||||
| Time-series queries (battles over time) | Requires traversal every query | Postgres `GROUP BY occurred_at` is fast |
|
||||
| Full-text search over millions of words | Slow, requires external index | pgvector or Qdrant, designed for it |
|
||||
| Vector search | OK, but coupled to graph | Dedicated vector store, decoupled |
|
||||
| Blob storage (full lore text, attachments) | Don't do this in Neo4j | S3, cheap, durable |
|
||||
| Sub-millisecond ephemeral state | Possible but ugly | Redis, designed for it |
|
||||
| Graph traversal and pattern matching | Excellent | Still excellent (Neo4j) |
|
||||
| Graph (people, factions, edges) | Hand-managed Neo4j | Cognee-managed (Neo4j or Kuzu) |
|
||||
| Operational event log | Hand-managed Postgres | Cognee-managed Postgres + Lore Engine tables |
|
||||
| Vector search | Hand-managed Qdrant OR pgvector | Cognee-managed (pgvector or Qdrant) |
|
||||
| Session state | Hand-managed Redis | Cognee-managed (Redis or in-memory) |
|
||||
| Source text blob storage | Hand-managed MinIO | Cognee-managed |
|
||||
| Cross-store transactions | Hand-written saga pattern | Cognee's transaction model |
|
||||
| Embedding refresh | Hand-written pipeline | Cognee-managed |
|
||||
|
||||
The graph stays the graph. Operational data lives where it belongs. The LLM gets unified responses via composed tools.
|
||||
|
||||
## The cost: cross-store transactions
|
||||
|
||||
When the world-builder writes a new mission, we touch Neo4j (entity, relations), Postgres (mission_log row), and S3 (any attachments). **These three writes are not atomic.** A partial failure leaves the world in an inconsistent state.
|
||||
|
||||
**Mitigation: the saga pattern.**
|
||||
|
||||
```
|
||||
saga: log_mission:
|
||||
step 1: Postgres INSERT INTO mission_log
|
||||
step 2: Neo4j MERGE (:DomainEntity) + relations
|
||||
step 3: S3 PUT attachments (if any)
|
||||
step 4: Neo4j MERGE (:ConsistencyRun {saga_id: ...}) SET status = 'committed'
|
||||
|
||||
on failure at step 2: rollback step 1 (Postgres DELETE)
|
||||
on failure at step 3: mark mission as 'attachments_pending', retry later
|
||||
on failure at step 4: log to dead-letter queue, alert world-builder
|
||||
```
|
||||
|
||||
Sagas are more code than a single transaction, but they're correct. The alternative — putting everything in Neo4j and hoping — is the trap.
|
||||
The 5-store plan was a real engineering project. The Cognee plan collapses it to: **Cognee does the substrate, the Lore Engine adds the domain layer.** The Lore Engine only owns the typed ontology (registered as a Cognee data-model extension) and the operational tables (a handful of Postgres tables on top of Cognee's Postgres).
|
||||
|
||||
## What this is not
|
||||
|
||||
- **Not a microservices overhaul.** The 5 stores run in 1 docker-compose stack. The engine still looks like one system to the LLM.
|
||||
- **Not eventual-consistency-everywhere.** Most operations are single-store. The saga is for the multi-store cases.
|
||||
- **Not a "use Postgres for everything" anti-pattern.** We use Postgres for what it's good at, Neo4j for what it's good at, and the cross-store compose layer for the rest.
|
||||
- **Not free.** Postgres + Qdrant + MinIO is ~3 more services. On the 58GB host, this is fine (~1-2GB extra). On a Raspberry Pi, it would be wrong.
|
||||
- **Not a microservices overhaul.** Cognee runs in 1 Docker container; the Lore Engine extension is in-process. The engine still looks like one system to the LLM.
|
||||
- **Not "the Lore Engine does not own storage."** The Lore Engine owns the typed ontology, the operational tables, and the consistency rules. Cognee owns the substrate (graph, vector, sessions, source text, transactions).
|
||||
- **Not free.** Cognee is a real piece of infrastructure. On the existing 58GB dev host it's fine. On a Raspberry Pi it would be wrong (use a hosted Cognee instance or pick a lighter substrate).
|
||||
|
||||
## Summary
|
||||
|
||||
The storage strategy is the part of the design that lets the engine scale to *the whole world*, not just the macro structure. Neo4j is the *nervous system* — the relations, the time model, the consistency engine. Postgres is the *muscle memory* — the high-volume operational data. Qdrant/pgvector is the *cortex* — the semantic search. Redis is the *short-term memory* — the session state. S3 is the *archive* — the durable storage.
|
||||
The storage strategy is the part of the design that lets the engine scale to *the whole world*, not just the macro structure. Cognee is the **substrate** — the graph, the vectors, the sessions, the source text, the transactions. The Lore Engine is the **domain layer** — the 36 typed labels, the time model, the consistency engine, the operational tables, the TypeTemplate registry.
|
||||
|
||||
Each store is the right tool for its job. The engine is the integration layer that makes them feel like one world.
|
||||
Each piece is the right tool for its job. The Lore Engine is the integration layer that makes the substrate feel like one world.
|
||||
|
||||
@@ -1,137 +1,95 @@
|
||||
# 13 — Microservice Decomposition: Iterate at Micro and Macro
|
||||
|
||||
The v1 design has a single `mcp-server` Go binary exposing 30 tools. Adding a new tool today means editing `main.go`, recompiling, redeploying. **The iteration loop is the cost of the entire program.** That's the wrong shape for a system the world-builder is going to extend indefinitely.
|
||||
> **v1.2 update:** with Cognee as the substrate, the mcp-server-binary problem is gone. Cognee is the gateway. The Lore Engine is one in-process extension. This document is now about *how the Lore Engine extension is organized* (the in-process module layout, the template-watcher data-pipeline, the optional external-handler protocol) rather than about decomposing a monolithic Go binary.
|
||||
|
||||
This document is the v1.1 decomposition: split `mcp-server` into a core router and a set of pluggable tool services. The LLM-facing API stays the same. The internal architecture becomes a network of small, independently-deployable services.
|
||||
The original v1.1 design (now superseded) had a single `mcp-server` Go binary exposing 45 tools. The 1144-line `main.go` meant adding a new tool required an edit, a recompile, a redeploy, and a restart. **The iteration loop was the cost of the entire program.** That's the wrong shape for a system the world-builder is going to extend indefinitely.
|
||||
|
||||
On Cognee, the iteration loop is **minutes**, not hours. The Cognee MCP server is the gateway; the Lore Engine is one Python package registered as a Cognee data-model and tool extension. Adding a new tool means: write the tool function, register it in `tools/__init__.py`, restart the Cognee process. For the polymorphic template-driven tools, **no restart is needed** — the template-watcher data-pipeline picks up YAML changes and re-registers the tools hot.
|
||||
|
||||
## The principle: core is stable, edges are extensible
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────────┐
|
||||
│ MCP GATEWAY (Go) │
|
||||
│ ────────────────────────────── │
|
||||
│ - JSON-RPC over HTTP + SSE │
|
||||
│ - Session registry (Redis-backed) │
|
||||
│ COGNEE MCP SERVER (Python) │
|
||||
│ ──────────────────────────────────── │
|
||||
│ - JSON-RPC over HTTP │
|
||||
│ - Session registry (Cognee-managed) │
|
||||
│ - Active context tracking │
|
||||
│ - Tool discovery proxy (delegates to registry) │
|
||||
│ - Tool discovery proxy (iterates the registered tools) │
|
||||
│ - Tool call routing (parses request, dispatches to handler) │
|
||||
│ - ~500 lines of Go. Stable. │
|
||||
│ - Cognee handles: storage, extraction, embedding, sessions, vector │
|
||||
│ Stable. Cognee owns this layer. │
|
||||
└─────────────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
│ in-process or HTTP
|
||||
│ in-process (same Python process)
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────────────────┐
|
||||
│ TOOL REGISTRY (Postgres) │
|
||||
│ ────────────────────────────────────────── │
|
||||
│ One row per registered tool: │
|
||||
│ name, description, params_schema, handler_service, │
|
||||
│ handler_method, requires_auth, rate_limit, version │
|
||||
│ LORE ENGINE EXTENSION (Python, in-process) │
|
||||
│ ────────────────────────────────────────────── │
|
||||
│ The 45 MCP tools, organized by Group (per 05-mcp-tools.md): │
|
||||
│ tools/lookup.py # Group 1 — disambiguation │
|
||||
│ tools/entity_context.py │
|
||||
│ tools/was_true_at.py # Group 2 — time-aware │
|
||||
│ tools/state_at.py │
|
||||
│ tools/list_lineage.py # Group 3 — lineage │
|
||||
│ tools/event_chain.py # Group 4 — causal │
|
||||
│ tools/lore_about.py # Group 5 — knowledge │
|
||||
│ tools/consistency_tools.py # Group 6 — consistency │
|
||||
│ tools/generation_tools.py # Group 7 — generation │
|
||||
│ tools/worldbuilder_tools.py # Group 8 — writes │
|
||||
│ Each is a small, focused module. New tools can be added without │
|
||||
│ touching the others. │
|
||||
│ │
|
||||
│ Hot-reloadable. New tools appear via: │
|
||||
│ - template registration (auto-generates tools) │
|
||||
│ - manual POST /admin/tools │
|
||||
│ - plugin service startup (announces itself) │
|
||||
│ Hot-reloadable template tools (auto-registered): │
|
||||
│ tools/generated/ # template-driven tool runner │
|
||||
│ template-watcher (Cognee data-pipeline) │
|
||||
│ - watches ./templates/ │
|
||||
│ - validates YAML │
|
||||
│ - registers tools in the Cognee tool registry │
|
||||
│ - notifies subscribed LLM clients: "new tools available" │
|
||||
└─────────────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
│ dynamic dispatch
|
||||
│ the tools themselves
|
||||
│ compose across Cognee storage + operational Postgres
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────────────────┐
|
||||
│ TOOL HANDLER SERVICES (Go / Python / any) │
|
||||
│ ────────────────────────────────────────────────── │
|
||||
│ Each is a small, focused service that implements one category │
|
||||
│ of tools. New services can be added without touching the gateway. │
|
||||
│ │
|
||||
│ Phase-1 handlers (in-process Go): │
|
||||
│ - core-handler: was_true_at, true_during, state_at, │
|
||||
│ entities_present, timeline, lookup, │
|
||||
│ entity_context, expand_context │
|
||||
│ - lineage-handler: list_lineage, ancestors_of, descendants_of, │
|
||||
│ list_offspring, location_hierarchy │
|
||||
│ - event-handler: event_chain, events_during, lore_about, cite │
|
||||
│ - consistency-handler: get_contradictions, get_anachronisms, │
|
||||
│ get_ontology_violations, get_orphans, │
|
||||
│ run_consistency_check, latest_run, │
|
||||
│ flag_for_review, explain_violation, │
|
||||
│ add_ontology_rule, list_ontology_rules │
|
||||
│ - generation-handler: summarize_chain, narrate_arc │
|
||||
│ - worldbuilder-handler: add_entity, add_relation, │
|
||||
│ add_lore_source, add_template, etc. │
|
||||
│ │
|
||||
│ Hot-pluggable handlers (out-of-process, via HTTP): │
|
||||
│ - any HTTP service that speaks the tool-handler protocol │
|
||||
│ - tools register themselves with the gateway at startup │
|
||||
│ - can be written in Go, Python, Node, Rust, anything │
|
||||
└─────────────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
│ the handlers themselves
|
||||
│ compose across the data stores
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────────────────┐
|
||||
│ STORES (the data layer) │
|
||||
│ Neo4j · Postgres · pgvector · Redis · S3 (MinIO) │
|
||||
│ COGNEE STORAGE (managed by Cognee) │
|
||||
│ Graph (Neo4j or Kuzu) · Vector store · Postgres metadata │
|
||||
│ + Lore Engine operational tables (setting, lore_event, retcon, ...)│
|
||||
└─────────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
The **gateway** is the part that talks to the LLM. It's small, stable, and changes rarely. The **handlers** are the part that knows how to answer questions. They're small, focused, and can be added/replaced/swapped without touching the gateway. The **stores** are the data. They're the slowest-changing layer of all.
|
||||
The **gateway** (Cognee) is the part that talks to the LLM. It's small, stable, and changes rarely — Cognee owns it. The **Lore Engine extension** is the part that knows how to answer questions about a high-fantasy world. It's small, focused, and can be added/replaced/swapped without touching Cognee. The **storage** is Cognee's responsibility; the Lore Engine only owns the operational tables on top.
|
||||
|
||||
## The tool-handler protocol
|
||||
## The tool-handler protocol (for external handlers)
|
||||
|
||||
A handler is a service that implements a tiny RPC protocol. The gateway calls it; the handler does the work; the gateway returns the result.
|
||||
In v1.2 the 45 tools are in-process with Cognee. There is no HTTP handler protocol for the *Lore Engine's own* tools — they're Python functions in the same process. But Cognee itself supports external handlers (services written in any language, exposing MCP-compatible endpoints). The Lore Engine inherits that capability.
|
||||
|
||||
### The protocol
|
||||
When an external handler is needed (e.g. a heavy ML model, a community-maintained bestiary database, a Python service that uses spaCy for entity resolution), it speaks the Cognee MCP protocol and registers itself with the Cognee gateway. The LLM doesn't know or care.
|
||||
|
||||
**Discovery:**
|
||||
```
|
||||
GET /tools
|
||||
→ 200 OK
|
||||
{
|
||||
"tools": [
|
||||
{ "name": "was_true_at", "description": "...", "params_schema": {...} },
|
||||
...
|
||||
]
|
||||
}
|
||||
```
|
||||
### When to use an external handler
|
||||
|
||||
**Invocation:**
|
||||
```
|
||||
POST /invoke
|
||||
{
|
||||
"tool": "was_true_at",
|
||||
"arguments": { "relation": "RULED", "subject": "House Vyr", ... },
|
||||
"session_id": "abc123",
|
||||
"active_context": [...] // for handlers that need it
|
||||
}
|
||||
→ 200 OK
|
||||
{
|
||||
"result": { ... },
|
||||
"added_to_context": [...] // entities to add to session context
|
||||
}
|
||||
```
|
||||
- The handler needs to be written in a language other than Python.
|
||||
- The handler needs to scale independently of the gateway.
|
||||
- The handler is a community or third-party contribution.
|
||||
- The handler depends on heavy native libraries (CUDA, system-level ML toolkits).
|
||||
|
||||
That's it. **Two endpoints. JSON in, JSON out.** A handler can be written in any language that speaks HTTP.
|
||||
|
||||
### Why this is the right seam
|
||||
|
||||
- **The gateway stays simple.** It just routes `tools/call` to a handler.
|
||||
- **Handlers are testable in isolation.** Mock the gateway, exercise the handler.
|
||||
- **A handler can be a single function or a 5000-line service.** Doesn't matter to the gateway.
|
||||
- **A handler can fail independently.** A bad handler doesn't crash the gateway; the gateway returns an error to the LLM and the LLM tries a different tool.
|
||||
- **Handlers can scale independently.** A heavy `summarize_chain` handler can be horizontally scaled without scaling the lightweight `lookup` handler.
|
||||
For the Lore Engine's 45 core tools, none of these apply. They're in-process Python functions. The external handler protocol is a **capability, not a requirement** — it's there for the cases that need it.
|
||||
|
||||
## The dynamic tool generator
|
||||
|
||||
The most leveraged service in the v1.1 architecture is the **template-driven tool generator**. It watches `./templates/` for YAML changes and auto-generates tool registrations.
|
||||
The most leveraged feature in the v1.2 architecture is the **template-driven tool generator**. It runs as a Cognee data-pipeline (the `template-watcher`) that watches `./templates/` for YAML changes and auto-generates tool registrations.
|
||||
|
||||
```
|
||||
On ./templates/thieves_guild/mission.yaml file change:
|
||||
|
||||
1. YAML parser validates against the template schema
|
||||
2. For each tool spec in the template:
|
||||
1. Cognee data-pipeline picks up the file change event.
|
||||
2. YAML parser validates against the template schema
|
||||
3. For each tool spec in the template:
|
||||
- Generate the JSON schema for params
|
||||
- Generate the Cypher / SQL / query code (data-defined, not hand-written)
|
||||
- Register the tool in the Postgres tool registry
|
||||
3. Notify the gateway: "new tools available"
|
||||
4. Gateway fetches the new tool list, merges with existing, hot-reloads
|
||||
- Generate the query code (data-defined, not hand-written)
|
||||
- Register the tool in the Cognee tool registry
|
||||
4. Notify subscribed LLM clients: "new tools available"
|
||||
5. LLM clients see the new tools in their next tools/list call
|
||||
```
|
||||
|
||||
@@ -144,116 +102,89 @@ The generated tool handler is a *generic* runner that knows the template spec. F
|
||||
|
||||
This is a *one-time* code generator. The runtime handler is shared across all generated tools. **Adding a new domain type means writing a template, not writing a handler.**
|
||||
|
||||
## The communication pattern: in-process first, HTTP second
|
||||
## The phase plan: which capabilities when
|
||||
|
||||
In v1.1, all the Phase-1 handlers run **in the same Go binary as the gateway**. They're imported packages, not separate services. The HTTP-based plugin protocol is for handlers that *can't* run in-process — heavy ML models, Python ML toolkits, external services.
|
||||
The Cognee-based architecture is a *progressive* layering, not a big-bang rewrite. The MVP (Phases 0–3 per `09-roadmap.md`) ships with the in-process extension. The v1.1 follow-ups add the template-driven tool generator and the optional external handler protocol.
|
||||
|
||||
**Why in-process first:**
|
||||
- No network latency on the common path.
|
||||
- Shared connection pools (Neo4j driver, Postgres pool).
|
||||
- Simpler deployment (one binary, one docker image).
|
||||
- Easier debugging.
|
||||
|
||||
**Why HTTP second (for the future):**
|
||||
- A handler in Python (e.g. using spaCy for entity resolution) doesn't need to be rewritten in Go.
|
||||
- A heavy handler (e.g. ML inference for image generation) can be scaled independently.
|
||||
- An external service (e.g. a hosted embedding model) can be a handler.
|
||||
|
||||
The gateway treats in-process handlers and HTTP handlers identically. The dispatch is the same.
|
||||
|
||||
## The phase plan: which services when
|
||||
|
||||
The decomposition is a *progressive* split, not a big-bang rewrite.
|
||||
|
||||
### Phase 1 (MVP): single binary, multiple handler files
|
||||
### Phase 0–3 (MVP): the in-process extension
|
||||
|
||||
```
|
||||
mcp-server/ # one Go module
|
||||
├── main.go # gateway (~500 lines)
|
||||
├── handlers/
|
||||
│ ├── core.go # 9 core tools
|
||||
│ ├── lineage.go # 5 lineage/hierarchy tools
|
||||
│ ├── event.go # 4 lore/event tools
|
||||
│ ├── consistency.go # 11 consistency tools
|
||||
│ ├── generation.go # 2 generation tools
|
||||
│ ├── worldbuilder.go # 9 world-builder tools
|
||||
│ └── generated.go # template-driven tools (auto-registered)
|
||||
├── registry/
|
||||
│ └── tool_registry.go # in-process tool registry
|
||||
└── ...
|
||||
lore-engine-extension/ # one Python package
|
||||
├── lore_engine/ # Cognee data-model extension
|
||||
│ ├── __init__.py # registers labels, constraints, indexes
|
||||
│ ├── ontology/ # typed labels (Person, Faction, ...)
|
||||
│ ├── time_model/ # time_in_window, era-tree helpers
|
||||
│ ├── consistency/ # 4 rule categories
|
||||
│ └── templates/ # TypeTemplate validator + registry
|
||||
│
|
||||
├── tools/ # the 45 MCP tools
|
||||
│ ├── lookup.py
|
||||
│ ├── entity_context.py
|
||||
│ ├── ... # one file per Group from 05-mcp-tools.md
|
||||
│ └── worldbuilder_tools.py
|
||||
│
|
||||
├── pipelines/ # Cognee data-pipelines
|
||||
│ └── consistency_pipeline.py # nightly + on-demand rule run
|
||||
│
|
||||
├── parsers/ # structured YAML ingest
|
||||
│ ├── timeline.py
|
||||
│ ├── family_tree.py
|
||||
│ └── ...
|
||||
│
|
||||
└── schema/ # init.cognify / init.cypher
|
||||
├── init.cypher # constraints, indexes
|
||||
└── udfs/ # time_in_window, time_windows_overlap
|
||||
```
|
||||
|
||||
Same binary, multiple files. The `mcpTools` array from v1 is split into per-handler arrays, registered at startup.
|
||||
Same process, multiple modules. The 45 tools are split into per-Group files, registered at Cognee startup. **The MVP ships with 45 manually-defined tools; no template-driven tools yet.**
|
||||
|
||||
**Deliverable:** the same 30+ tools, but the code is organized so the next phase's split is mechanical.
|
||||
### Phase 5: add the template-driven tool generator
|
||||
|
||||
### Phase 2: extract the consistency engine to its own service
|
||||
|
||||
The consistency engine is *naturally* a separate service — it runs on a schedule, it has its own state (`:ConsistencyRun` nodes), and it has a different scaling profile from the read-path tools.
|
||||
The template-watcher data-pipeline and the dynamic tool runner. Per Phase 5 of the Cognee roadmap in `09-roadmap.md`. **This is the phase that unlocks the "arbitrary new concept" question.**
|
||||
|
||||
```
|
||||
services/
|
||||
├── mcp-gateway/ # gateway only
|
||||
├── consistency-runner/ # cron-based, runs rules
|
||||
├── consistency-monitor/ # HTTP service, exposes run_check
|
||||
lore-engine-extension/
|
||||
├── tools/
|
||||
│ ├── ... # the 45 manually-defined tools
|
||||
│ └── generated/ # NEW: the template-driven tool runner
|
||||
│ └── runner.py
|
||||
│
|
||||
└── pipelines/
|
||||
├── consistency_pipeline.py
|
||||
└── template_watcher.py # NEW: watches ./templates/
|
||||
```
|
||||
|
||||
**Deliverable:** the gateway no longer has any consistency logic. The consistency-runner and consistency-monitor are independently deployable.
|
||||
**Deliverable:** world-builders can add a new domain type by writing YAML, hitting `POST /admin/templates/reload`, and the gateway picks it up within 5 seconds. No Cognee restart, no Lore Engine restart.
|
||||
|
||||
### Phase 3: extract the template-driven tool generator
|
||||
### Phase 6: the external handler protocol (optional)
|
||||
|
||||
The template generator is a *hot-reload* service that needs to be able to restart tool registration without restarting the gateway. The right shape is a sidecar that watches `./templates/`, generates tool specs, and pushes them to the gateway via an admin API.
|
||||
The Cognee MCP server already supports external handlers; the Lore Engine does not need to implement anything new. This phase is only needed if a world-builder or community contributor wants to add a handler in a language other than Python, or wants to scale a handler independently of the gateway.
|
||||
|
||||
```
|
||||
services/
|
||||
├── mcp-gateway/
|
||||
├── template-watcher/ # watches ./templates/, generates tool specs
|
||||
├── template-registry/ # persists tool specs to Postgres
|
||||
```
|
||||
|
||||
**Deliverable:** world-builders can add a new domain type by writing YAML, posting to a webhook, and the gateway picks it up within 5 seconds. No gateway restart.
|
||||
|
||||
### Phase 4: HTTP-based handler protocol
|
||||
|
||||
The gateway is extended to support HTTP-based handlers. The first HTTP-based handler is a Python service that does heavy ML work the gateway shouldn't do — e.g. an entity-linking model that resolves ambiguous entity references at scale.
|
||||
|
||||
```
|
||||
services/
|
||||
├── mcp-gateway/ # now supports both in-process and HTTP handlers
|
||||
├── entity-linker/ # Python, uses a transformer model
|
||||
```
|
||||
|
||||
**Deliverable:** a Python handler is registered. The gateway dispatches to it transparently. The LLM doesn't know or care.
|
||||
|
||||
### Phase 5: external service handlers
|
||||
|
||||
Handlers can be hosted externally — a web service, a Cloudflare Worker, a serverless function. The auth and rate-limiting logic in the gateway makes this safe.
|
||||
|
||||
**Deliverable:** an external world-data provider (e.g. a community-maintained bestiary database) is a handler. The LLM can query it via the same MCP API.
|
||||
**Deliverable:** a Python service (e.g. an entity-linker using spaCy) is registered with the Cognee gateway. The gateway dispatches to it transparently. The LLM doesn't know or care.
|
||||
|
||||
## What this means for the world-builder
|
||||
|
||||
The world-builder's experience is unchanged at the API level:
|
||||
|
||||
- Write YAML → POST to `/ingest/template` → tools appear.
|
||||
- POST to `/ingest/structured` with instance YAML → data lands.
|
||||
- Write YAML → `cognee.add()` or `POST /ingest/structured` → data lands.
|
||||
- Write template YAML → `POST /admin/templates/reload` → tools appear.
|
||||
- Query via MCP.
|
||||
|
||||
What changes is the *iteration loop*:
|
||||
|
||||
| | v1 (monolith) | v1.1 (decomposed) |
|
||||
| | v1 (monolith) | v1.2 (Cognee extension) |
|
||||
|---|---|---|
|
||||
| Add a new domain type | Edit Go, edit Cypher, rebuild, redeploy | Write YAML, POST it. Done. |
|
||||
| Add a new tool for an existing type | Edit Go, rebuild, redeploy | Add a tool spec to the template. Done. |
|
||||
| Fix a bug in a tool | Find the right handler, rebuild, redeploy gateway | Fix in the handler service. Redeploy that service only. |
|
||||
| Add a heavy ML-based tool | Rewrite in Go or call out to Python via subprocess | Write a Python service, register as HTTP handler. |
|
||||
| Scale a single tool to 100x | Scale the whole gateway | Scale just that handler. |
|
||||
| Add a new domain type | Edit Go, edit Cypher, rebuild, redeploy | Write template YAML, hot-reload. Done. |
|
||||
| Add a new tool for an existing type | Edit Go, rebuild, redeploy | Add a tool spec to the template, hot-reload. Done. |
|
||||
| Fix a bug in a tool | Find the right handler, rebuild, redeploy gateway | Edit the Python function, restart Cognee. |
|
||||
| Add a heavy ML-based tool | Rewrite in Go or call out to Python via subprocess | Write a Python service, register as external Cognee handler. |
|
||||
| Scale a single tool to 100x | Scale the whole gateway | Scale just that handler (if external) or Cognee (if in-process). |
|
||||
|
||||
The iteration loop drops from **hours** (code change, build, deploy) to **minutes** (YAML change, hot-reload). This is what makes the engine tractable for a world that's going to grow indefinitely.
|
||||
The iteration loop drops from **hours** (code change, build, deploy) to **minutes** (YAML change, hot-reload) for template-driven tools, and **~5 minutes** (Python edit, restart) for manually-defined tools. This is what makes the engine tractable for a world that's going to grow indefinitely.
|
||||
|
||||
## What this means for the LLM
|
||||
|
||||
The LLM doesn't know or care. It sees a single MCP server with a list of tools. The tool names, descriptions, and params are the same whether the tool is in-process or HTTP.
|
||||
The LLM doesn't know or care whether the tool is in-process, template-driven, or external. It sees a single MCP server with a list of tools. The tool names, descriptions, and params are the same.
|
||||
|
||||
The LLM does get a *new* tool: `list_template_tools()`. This returns the auto-generated tools for all currently-loaded templates. The LLM can use this to discover what domain types are available without having to be told.
|
||||
|
||||
@@ -265,76 +196,78 @@ The LLM does get a *new* tool: `list_template_tools()`. This returns the auto-ge
|
||||
}
|
||||
```
|
||||
|
||||
## The handler-services directory layout
|
||||
## The Lore Engine extension directory layout
|
||||
|
||||
When the system is fully decomposed:
|
||||
When the system is fully built:
|
||||
|
||||
```
|
||||
services/
|
||||
├── mcp-gateway/ # the public MCP endpoint (~500 lines Go)
|
||||
├── template-watcher/ # watches templates, generates tool specs
|
||||
├── template-registry/ # persists tool specs
|
||||
├── core-tools/ # 9 core ontology tools
|
||||
├── lineage-tools/ # 5 lineage/hierarchy tools
|
||||
├── event-tools/ # 4 lore/event tools
|
||||
├── consistency-runner/ # nightly batch
|
||||
├── consistency-monitor/ # HTTP, exposes run_check
|
||||
├── generation-tools/ # 2 generation tools (LLM-in-loop)
|
||||
├── worldbuilder-tools/ # 9 write tools
|
||||
├── structured-ingestor/ # YAML → Neo4j + Postgres
|
||||
├── lore-extractor/ # LLM-based prose extraction
|
||||
├── dialogue-processor/ # in-fiction dialogue logs
|
||||
├── entity-linker/ # Python, ML-based entity resolution
|
||||
├── bestiary-external/ # external world-data provider
|
||||
└── ...
|
||||
lore-engine-extension/ # the Lore Engine on Cognee
|
||||
├── lore_engine/ # data-model extension
|
||||
│ ├── ontology/ # typed labels + constraints
|
||||
│ ├── time_model/ # time_in_window, era-tree helpers
|
||||
│ ├── consistency/ # 4 rule categories + 10 starter rules
|
||||
│ └── templates/ # TypeTemplate validator + registry
|
||||
│
|
||||
├── tools/ # the 45 MCP tools (in-process)
|
||||
│ ├── lookup.py
|
||||
│ ├── entity_context.py
|
||||
│ ├── was_true_at.py
|
||||
│ ├── ...
|
||||
│ ├── consistency_tools.py
|
||||
│ ├── generation_tools.py
|
||||
│ ├── worldbuilder_tools.py
|
||||
│ └── generated/ # template-driven tool runner
|
||||
│ └── runner.py
|
||||
│
|
||||
├── pipelines/ # Cognee data-pipelines
|
||||
│ ├── consistency_pipeline.py # nightly + on-demand rule run
|
||||
│ └── template_watcher.py # watches ./templates/, hot-reload
|
||||
│
|
||||
├── parsers/ # structured YAML ingest
|
||||
│ ├── timeline.py
|
||||
│ ├── family_tree.py
|
||||
│ ├── gazetteer.py
|
||||
│ ├── bestiary.py
|
||||
│ ├── magic_system.py
|
||||
│ └── culture.py
|
||||
│
|
||||
└── schema/ # init.cypher
|
||||
├── init.cypher # constraints, indexes
|
||||
└── udfs/ # time_in_window, time_windows_overlap
|
||||
```
|
||||
|
||||
Each service has its own Dockerfile, its own healthcheck, and its own test suite. The gateway is a thin router.
|
||||
The entire extension is **one Python package**, registered with Cognee at startup. There are no separate Go workers; the entire backend is in-process with Cognee.
|
||||
|
||||
## The cost: more services to operate
|
||||
## The cost: less, not more
|
||||
|
||||
Each service is a new line in `docker-compose.yml`, a new healthcheck, a new failure mode.
|
||||
The Cognee-based architecture has **fewer** services to operate than the v1.1 plan called for. The 5+ Go services in the old plan (mcp-gateway, consistency-runner, consistency-monitor, template-watcher, template-registry, structured-ingestor, lore-extractor, etc.) collapse to:
|
||||
|
||||
**Mitigation:** the gateway *aggregates* the healthchecks. The world-builder sees one health endpoint, not fifteen.
|
||||
- **Cognee MCP server** (1 process, the gateway)
|
||||
- **Lore Engine extension** (in-process with Cognee)
|
||||
- **Cognee data-pipelines** (template-watcher, consistency-pipeline — also in-process)
|
||||
- **Storage backend** (Neo4j or Kuzu, managed by Cognee)
|
||||
- **Postgres** (Cognee metadata + Lore Engine operational tables)
|
||||
- **Vector store** (Cognee-managed, pgvector or Qdrant)
|
||||
|
||||
```
|
||||
GET /healthz on the gateway:
|
||||
{
|
||||
"gateway": "ok",
|
||||
"neo4j": "ok",
|
||||
"postgres": "ok",
|
||||
"pgvector": "ok",
|
||||
"redis": "ok",
|
||||
"minio": "ok",
|
||||
"handlers": {
|
||||
"core-tools": "ok",
|
||||
"lineage-tools": "ok",
|
||||
"consistency-monitor": "ok",
|
||||
"template-watcher": "degraded - retrying",
|
||||
...
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
A single degraded handler doesn't fail the healthcheck. The LLM can still use the tools that are working. A failed core tool does fail it.
|
||||
That's 3-4 infrastructure components, not 8+ Go services. The operational footprint is **smaller** than the v1.1 plan.
|
||||
|
||||
## The decomposition is the answer to Kay's question
|
||||
|
||||
> *"We need to be able to take an arbitrary new concept, define how it associates with larger constructs, but also have flexibility to get as detailed as we need."*
|
||||
|
||||
The v1.1 decomposition answers this in three ways:
|
||||
The v1.2 architecture answers this in three ways:
|
||||
|
||||
1. **New concept (macro):** a YAML template. No code change. Hot-reload. (~1 hour of world-builder work.)
|
||||
|
||||
2. **Detail on the concept (micro):** instance YAML with relations to other entities. The template defines the structure; the instance fills it in. (~10 minutes per instance.)
|
||||
|
||||
3. **New kind of data the template doesn't cover (escape hatch):** a new handler service in the language of choice. The gateway's HTTP-based handler protocol lets the world-builder or a third party add capabilities without modifying the engine. (~1 day of work for a small, focused handler.)
|
||||
3. **New kind of data the template doesn't cover (escape hatch):** an external handler in the language of choice, registered with the Cognee gateway. (~1 day of work for a small, focused handler.)
|
||||
|
||||
**The macro level is fast and declarative. The micro level is structured and bounded. The escape hatch is open and arbitrary.** Three layers, three iteration speeds.
|
||||
|
||||
## What this is NOT
|
||||
|
||||
- **Not a microservices fan-out.** The handlers can run in-process. The HTTP protocol is for *some* handlers, not all.
|
||||
- **Not a rewrite of GraphMCP-Example.** The existing tools (semantic_search, graph_traverse, query_as_npc, etc.) become the in-process `core-tools` handler. Same Go code, better file structure.
|
||||
- **Not a SaaS architecture.** The system runs as one docker-compose stack on one host. The decomposition is about *organizing* the code, not distributing it.
|
||||
- **Not a guarantee that all extensions are easy.** Some extensions still need code — anything that requires a new query language feature, a new database, or a new ML model. The escape hatch is for those, and it costs more.
|
||||
- **Not a "we use Cognee so we don't have to write any code."** The Lore Engine extension is real Python code — the 45 tools, the time model, the consistency rules. Cognee doesn't replace the domain layer; it provides the substrate the domain layer runs on.
|
||||
- **Not a SaaS architecture.** The system runs as one docker-compose stack on one host. Cognee doesn't make it cloud-native; the world-builder still operates it as a single system.
|
||||
- **Not a guarantee that all extensions are easy.** Some extensions still need code — anything that requires a new query language feature, a new database, or a new ML model. The escape hatch (external handler) is for those, and it costs more.
|
||||
- **Not a rewrite of the world-builder's experience.** The world-builder still writes YAML, hits endpoints, sees tools in the LLM. The difference is the speed of the loop.
|
||||
|
||||
@@ -113,7 +113,7 @@ Six new MCP tools, four consistency rules, registered in the engine. No Go code
|
||||
|
||||
```yaml
|
||||
template: "thieves_guild_mission"
|
||||
world_id: "arda_1st_age"
|
||||
setting_id: "mardonari"
|
||||
instances:
|
||||
- mission_code: "CH-4471"
|
||||
name: "The Pale Ledger Heist"
|
||||
@@ -305,7 +305,7 @@ The world-builder never has to update a "current strength" field — it's always
|
||||
|
||||
```yaml
|
||||
template: "war_campaign"
|
||||
world_id: "arda_1st_age"
|
||||
setting_id: "mardonari"
|
||||
instances:
|
||||
- campaign_code: "VC-001"
|
||||
name: "The Border Wars"
|
||||
@@ -326,7 +326,7 @@ And the campaign events — these are **not** `DomainEntity` instances. They go
|
||||
```yaml
|
||||
# This is a Postgres-direct ingest, not a DomainEntity ingest.
|
||||
template: "campaign_event" # a special template that writes to Postgres
|
||||
world_id: "arda_1st_age"
|
||||
setting_id: "mardonari"
|
||||
events:
|
||||
- campaign_id: "campaign_VC-001"
|
||||
event_type: "army_moved"
|
||||
@@ -652,7 +652,7 @@ MERGE (d)-[:ACCESSIBLE_VIA]->(ps);
|
||||
|
||||
// Voldramir is also accessible via a specific location (the stone door)
|
||||
MERGE (door:Location {id: 'voldramir_stone_door', name: 'The Stone Door in the Wheel & Kiln'})
|
||||
SET door.kind = 'portal', door.world_id = 'mardonari.material';
|
||||
SET door.kind = 'portal', door.setting_id = 'mardonari';
|
||||
|
||||
MERGE (d)-[:ACCESSIBLE_VIA]->(door);
|
||||
```
|
||||
@@ -716,3 +716,178 @@ LLM to user: "In 430 TA, Roland was in two places at once. The `LOCATED_IN` edge
|
||||
- **The LLM gets clean answers.** Both questions above return structured JSON with plane names, kinds, and time bounds. The LLM composes the natural-language answer from the structured data.
|
||||
|
||||
For the design rationale, see `17-planes.md`. For the underlying ontology, see `01-ontology.md` (`Plane` and `Setting` rows in the node-label table, and the `Planes (v1.2)` section in the edge-type table).
|
||||
|
||||
## Example 6: Cognee integration walkthrough
|
||||
|
||||
A world-builder's first session with the Lore Engine on Cognee. The full path from "I have a markdown chapter" to "Claude answers a time-bounded question about it." This is what `09-roadmap.md#phase-0` validates in the spike.
|
||||
|
||||
### Step 1: Stand up Cognee with the Lore Engine extension
|
||||
|
||||
```bash
|
||||
# 1. Pull the Cognee image with the Lore Engine extension baked in
|
||||
docker pull ghcr.io/topoteretes/cognee:latest-lore-engine
|
||||
|
||||
# 2. Start the stack
|
||||
docker-compose up -d
|
||||
# - cognee-mcp (port 8000, the MCP server)
|
||||
# - cognee-storage (Neo4j 5.x, the graph)
|
||||
# - cognee-postgres (the metadata + operational tables)
|
||||
# - cognee-vector (pgvector, the embeddings)
|
||||
|
||||
# 3. Verify the Lore Engine extension registered
|
||||
curl -X POST http://localhost:8000/mcp -d '{
|
||||
"jsonrpc": "2.0", "id": 1,
|
||||
"method": "tools/list", "params": {}
|
||||
}'
|
||||
# → returns 45 tools (8 Cognee + 37 Lore Engine)
|
||||
# ... including was_true_at, list_lineage, lookup, state_at, ...
|
||||
```
|
||||
|
||||
The Lore Engine extension is a Python package installed in the Cognee image. At startup it registers the 36 typed labels, the constraints, the indexes, and the 37 domain tools.
|
||||
|
||||
### Step 2: Ingest a structured YAML file
|
||||
|
||||
The world-builder has written `family_tree.yaml` for House Vyr. The Lore Engine's structured parser handles it directly — no LLM involved.
|
||||
|
||||
```bash
|
||||
# Two ways to ingest: Cognee API or the Lore Engine's dedicated endpoint
|
||||
curl -X POST http://localhost:8000/ingest/structured \
|
||||
-F "file=@family_tree.yaml" \
|
||||
-F "source_type=family_tree"
|
||||
# OR equivalently:
|
||||
curl -X POST http://localhost:8000/mcp -d '{
|
||||
"jsonrpc": "2.0", "id": 2,
|
||||
"method": "tools/call",
|
||||
"params": { "name": "ingest_structured", "arguments": { "file": "family_tree.yaml" } }
|
||||
}'
|
||||
```
|
||||
|
||||
The Lore Engine's YAML parser:
|
||||
1. Validates the YAML against the `family_tree.yaml` schema.
|
||||
2. Emits `MERGE (p:Person {id, name, lifespan: {from, until}})` for each ancestor.
|
||||
3. Emits `MERGE (p)-[:PARENT_OF {valid_from, valid_until}]->(c)` for each relation.
|
||||
4. Tags the `LoreSource` with `source_type: family_tree`.
|
||||
5. Notifies the consistency pipeline: "new edges added, run the lineage-continuity rule."
|
||||
|
||||
Total time: <1 second for a 200-person family tree.
|
||||
|
||||
### Step 3: Ingest prose via Cognee's pipeline
|
||||
|
||||
For markdown chapters and dialogue, the path goes through Cognee's `add` + `cognify` pipeline. The Lore Engine registers a custom extraction prompt that emits its 36 typed labels instead of Cognee's default `Entity`/`DataPoint` types.
|
||||
|
||||
```python
|
||||
import cognee
|
||||
|
||||
# World-builder's Python session
|
||||
await cognee.add("chapters/aldric_origin.md") # raw markdown chunk
|
||||
await cognee.cognify() # extract + embed + index
|
||||
|
||||
# Behind the scenes:
|
||||
# - Cognee chunks the markdown (512-token windows, 64-token overlap)
|
||||
# - Cognee calls the LLM with the Lore Engine's extraction prompt
|
||||
# - The LLM returns typed triples: (Person "Aldric Raventhorne")-[:RULED]->(Location "Thornwall Keep")
|
||||
# - Cognee stores the triples in Neo4j with the Lore Engine's labels
|
||||
# - Cognee embeds the chunk and indexes it in the vector store
|
||||
# - The Lore Engine consistency pipeline runs anachronism + contradiction checks
|
||||
```
|
||||
|
||||
The extraction prompt is the Lore Engine's contribution: it tells the LLM "emit nodes with one of these 36 labels, edges with one of these 70+ edge types, all with time bounds where applicable." Cognee handles the chunking, embedding, and storage; the Lore Engine defines the schema the LLM emits into.
|
||||
|
||||
### Step 4: Ask a time-bounded question
|
||||
|
||||
The LLM client (Claude, gpt-4, etc.) is connected to the Cognee MCP server with the Lore Engine's 45-tool surface and the reasoning-harness system prompt from `07-reasoning-harness.md`.
|
||||
|
||||
```
|
||||
User: "Did House Vyr hold the Crimson Throne during the Second Age?"
|
||||
|
||||
→ Claude picks tool: was_true_at(relation="RULED", subject="House Vyr", object="Crimson Throne", at_time="2nd_age")
|
||||
→ POST /mcp { "method": "tools/call", "params": { "name": "was_true_at", "arguments": { ... } } }
|
||||
→ Cognee MCP server dispatches to the Lore Engine's was_true_at.py handler
|
||||
→ Handler composes Cypher:
|
||||
MATCH (f:Faction {name: "House Vyr"})-[r:RULED]->(t:Throne {name: "Crimson Throne"})
|
||||
WHERE time_in_window("2nd_age", r.valid_from, r.valid_until)
|
||||
RETURN r, r.valid_from, r.valid_until, r.sources
|
||||
→ Cognee executes on Neo4j, returns the time-bounded edge
|
||||
→ Handler formats the JSON response with source attribution
|
||||
|
||||
LLM gets: {
|
||||
"was_true": true,
|
||||
"valid_from": "2nd_age.year_120",
|
||||
"valid_until": "2nd_age.year_340",
|
||||
"sources": ["chronicles-vyr.md", "crimson-throne-succession.md"],
|
||||
"confidence": 0.94
|
||||
}
|
||||
|
||||
→ Claude to user: "Yes — House Vyr held the Crimson Throne from 120 TA to 340 TA,
|
||||
which covers the entire Second Age. Sources: the chronicles of House Vyr and
|
||||
the Crimson Throne succession records."
|
||||
```
|
||||
|
||||
The end-to-end latency: <500ms for a single-tool call.
|
||||
|
||||
### Step 5: Add a new domain type (template-driven)
|
||||
|
||||
The world-builder wants to track heists for the Crimson Hand thieves' guild. They write a template:
|
||||
|
||||
```yaml
|
||||
# templates/thieves_guild/mission.yaml
|
||||
template: "thieves_guild_mission"
|
||||
version: 1
|
||||
fields:
|
||||
- { name: mission_code, type: string, required: true, unique: true }
|
||||
- { name: target, type: string, required: true }
|
||||
- { name: payout_gp, type: int, required: true }
|
||||
- { name: status, type: enum, values: [planned, active, completed, botched] }
|
||||
- { name: risk, type: enum, values: [low, medium, high, extreme] }
|
||||
- { name: secrecy, type: enum, values: [public, guild_internal, inner_circle_only] }
|
||||
relations:
|
||||
- { name: GIVEN_BY, to: [Person, NPC] }
|
||||
- { name: TARGETS, to: [Location, Faction, Person] }
|
||||
- { name: LOGGED_IN, to: [DomainEntity] }
|
||||
tools:
|
||||
- { name: list_missions, params: [faction, location, since, sort_by, limit] }
|
||||
- { name: get_mission, params: [mission_code] }
|
||||
- { name: add_mission, params: [mission_code, target, payout_gp, ...] }
|
||||
rules:
|
||||
- "A mission in status=completed must have a completed_at timestamp."
|
||||
- "A mission in status=botched must have at least one casualty recorded."
|
||||
```
|
||||
|
||||
They hit the reload endpoint:
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:8000/admin/templates/reload
|
||||
# → Cognee data-pipeline picks up the file change
|
||||
# → YAML validates
|
||||
# → TypeTemplate node written to Neo4j
|
||||
# → 3 new tools auto-registered: list_missions, get_mission, add_mission
|
||||
# → 2 new ontology rules attached to the consistency engine
|
||||
# → Hot-reload notification: "Domain type thieves_guild_mission registered; 3 new tools available"
|
||||
```
|
||||
|
||||
**No code change. No Cognee restart. No Lore Engine restart.** The LLM's next `tools/list` call sees the 3 new tools.
|
||||
|
||||
### Step 6: Use the new tools
|
||||
|
||||
```
|
||||
User: "What heists has the Crimson Hand run in Mardsville this year, sorted by payout?"
|
||||
|
||||
→ Claude's reasoning: "The user wants list_missions with a filter on location and a sort."
|
||||
→ Claude picks tool: list_missions(faction="crimson_hand", location="mardsville", since="1_year_ago", sort_by="payout_gp", limit=5)
|
||||
→ Cognee MCP server dispatches to the template-generated tool runner
|
||||
→ Runner looks up the TypeTemplate, generates the Cypher from the spec, executes
|
||||
→ Returns: [ { mission_code: "M-4471", name: "The Pale Ledger Heist", payout_gp: 500, status: "completed" }, ... ]
|
||||
|
||||
LLM to user: "Five heists, led by The Pale Ledger Heist at 500gp (completed) and ..."
|
||||
```
|
||||
|
||||
### What this example shows
|
||||
|
||||
- **Cognee is the substrate.** Storage, extraction, embedding, sessions — all Cognee.
|
||||
- **The Lore Engine is the domain layer.** Typed labels, time model, consistency rules, 45 tools.
|
||||
- **Structured YAML is exact.** `family_tree.yaml` ingests without an LLM.
|
||||
- **Prose is fuzzy but typed.** The extraction prompt emits the 36 labels; Cognee chunks and embeds.
|
||||
- **Templates extend the engine without code.** A YAML file, a hot-reload, three new tools.
|
||||
- **The LLM gets typed, source-attributed answers.** Every claim traces to a document, every edge has time bounds, every time-aware query uses `time_in_window`.
|
||||
|
||||
This is the system the Cognee spike (Phase 0 in `09-roadmap.md`) validates end-to-end. If this walkthrough works against a real Cognee + Lore Engine + Claude stack, the substrate decision is right and the v1 build can begin.
|
||||
|
||||
@@ -107,13 +107,13 @@ Cognee is the **closest functional comparable** to the Lore Engine. Both are kno
|
||||
| Temporal model | None | First-class (`time_in_window` UDF) |
|
||||
| Closed-world enforcement | No (extracts whatever the LLM finds) | Yes (typed ontology + consistency engine) |
|
||||
| Source attribution | Basic | Deep (every node has sources[] + lore_verified) |
|
||||
| Self-hosted | Yes | Yes (built on existing GraphMCP-Example stack) |
|
||||
| Self-hosted | Yes | Yes (built on Cognee) |
|
||||
| LLM at ingest | Yes (every stage) | No (structured YAML is exact) |
|
||||
| Production maturity | High (17k stars, paying users) | None yet (design phase) |
|
||||
| License | Apache 2.0 | MIT (planned for the Lore Engine) |
|
||||
| Pluggable LLM providers | Yes | Inherited from GraphMCP-Example (LiteLLM) |
|
||||
| Pluggable LLM providers | Yes | Yes (Cognee is provider-agnostic) |
|
||||
|
||||
**Honest assessment:** if the Lore Engine's MVP is built and shipped, the strongest move is to **integrate with Cognee rather than compete with it.** Cognee provides the agent-native API and the storage abstraction. The Lore Engine provides the fictional-world ontology, the temporal model, and the consistency engine. Cognee becomes the substrate; the Lore Engine becomes the domain layer. This is a real v2 option.
|
||||
**Honest assessment (v1.2 update):** the Lore Engine is **built on Cognee, not in competition with it.** Cognee provides the substrate — the storage abstraction, the extraction pipeline, the embedding store, the agent-native API. The Lore Engine provides the fictional-world ontology, the temporal model, the consistency engine, and the TypeTemplate polymorphism as a Cognee extension. This is the v1.2 substrate decision. The "v2 option" framing in the previous version of this doc is no longer hypothetical; it's the architecture.
|
||||
|
||||
**Where the Lore Engine is worse:** Cognee has 17k stars, paying customers, a hosted offering, a Discord, integrations, and a Claude Code plugin. The Lore Engine is a design in a Gitea repo. If Kay wanted to *use* a knowledge-graph backend for LLM reasoning tomorrow, Cognee is the right answer. The Lore Engine is the right answer for the *specific* problem of reasoning about a fictional world with historical accuracy.
|
||||
|
||||
@@ -272,7 +272,7 @@ The two systems are **complementary, not competitive.** IVIE generates worlds; t
|
||||
| Persistence | None (one-shot) | Yes (Neo4j + Postgres) |
|
||||
| Source attribution | None | Deep |
|
||||
| Temporal reasoning | No | First-class |
|
||||
| Cross-world | No | Yes (world_id namespace) |
|
||||
| Cross-world | No | Yes (Setting + Plane graph model, v1.2) |
|
||||
| LLM at read time | Yes (generation) | Optional (narrative tools) |
|
||||
| Closed-world enforcement | Yes (symbolic validator) | Yes (consistency engine) |
|
||||
|
||||
|
||||
@@ -102,11 +102,9 @@ Both systems are heavily dependent on the LLM's quality. If the LLM is bad at re
|
||||
|
||||
### Net assessment
|
||||
|
||||
**If the question is "should I use Cognee or the Lore Engine for my game-world reasoning?"** the answer is: *if the Lore Engine isn't built, use Cognee. If the Lore Engine is built, use the Lore Engine.*
|
||||
**The Lore Engine is built on Cognee.** This is the v1.2 substrate decision. Cognee provides the storage abstraction, the extraction pipeline, the embedding store, and the agent-native `remember/recall/forget` API. The Lore Engine adds the typed high-fantasy ontology, the time model, the consistency engine, and the TypeTemplate polymorphic extension as a Cognee data-model and tool extension.
|
||||
|
||||
Cognee is the better *foundation* for a knowledge-graph backend. The Lore Engine is the better *domain layer* for fictional-world reasoning. The right move in v2 is almost certainly to *build the Lore Engine on top of Cognee*: use Cognee's storage abstraction and agent-native API, and implement the Lore Engine's temporal model, ontology, and consistency engine as a Cognee extension.
|
||||
|
||||
This is a serious recommendation, not a hedge. **Cognee + Lore Engine > Lore Engine standalone**, for almost any real-world deployment. The Lore Engine as designed inherits from GraphMCP-Example because GraphMCP-Example is what Kay already has. If starting from scratch, Cognee would be the substrate.
|
||||
Cognee is the *substrate*. The Lore Engine is the *domain layer* for fictional-world reasoning. The right move in v1.2 is to **build the Lore Engine on top of Cognee**, which is what we're doing.
|
||||
|
||||
### The honest take
|
||||
|
||||
@@ -206,7 +204,7 @@ The lore-extractor's prompt is English-only. A Japanese or Spanish world-builder
|
||||
|
||||
The Lore Engine is single-tenant. If a homelab wanted to host two worlds (one for each player's campaign), they'd need two separate stacks. Cognee supports multi-tenancy.
|
||||
|
||||
**Mitigation:** the `world_id` namespace is the right primitive. Multi-tenant would just be multiple `world_id` values in the same Neo4j, with row-level access control. v2.
|
||||
**Mitigation:** the v1.2 `Setting` + `Plane` model is the right primitive (replaces the v1.1 `world_id` namespace). Multi-tenant would just be multiple `Setting` values in the same Neo4j, with row-level access control. v2.
|
||||
|
||||
### 8. No user-facing UI
|
||||
|
||||
@@ -220,11 +218,11 @@ When the LLM's question is ambiguous, the Lore Engine's `lookup` tool returns a
|
||||
|
||||
**Mitigation:** cache `lookup` results in Redis. The active context (per-session working set) is the right place to store resolved entities.
|
||||
|
||||
### 10. The 30-tool surface is high
|
||||
### 10. The 45-tool surface is high
|
||||
|
||||
We hit this in `10-critique.md`. The 30+ tools are at the LLM's tool-use ceiling. GraphRAG exposes fewer tools and lets the LLM reason with them. Cognee exposes 4 operations (`remember`, `recall`, `forget`, `improve`) and lets the cognitive-ontology routing decide.
|
||||
We hit this in `10-critique.md`. The 45+ tools (8 inherited + 37 new) are well past the LLM's tool-use ceiling. GraphRAG exposes fewer tools and lets the LLM reason with them. Cognee exposes 4 operations (`remember`, `recall`, `forget`, `improve`) and lets the cognitive-ontology routing decide.
|
||||
|
||||
**Mitigation:** Phase 10 will measure tool-selection accuracy with all 30 vs. collapsed to ~15. If collapsed is better, we collapse.
|
||||
**Mitigation:** Phase 6 will measure tool-selection accuracy with all 45 vs. collapsed to ~15. If collapsed is better, we collapse.
|
||||
|
||||
---
|
||||
|
||||
@@ -258,7 +256,7 @@ Adding a new domain type (thieves-guild missions, war campaigns) is a YAML exerc
|
||||
|
||||
### 7. Multi-world/planar support
|
||||
|
||||
The `world_id` namespace and `Plane` label are the right primitives for multi-world campaigns. Most narrative games are single-world, but the high-fantasy genre is often multi-planar. The Lore Engine supports it; the others don't.
|
||||
The `Setting` + `Plane` graph model (v1.2) is the right primitive for multi-world / multi-planar campaigns. Most narrative games are single-world, but the high-fantasy genre is often multi-planar. The Lore Engine supports it; the others don't.
|
||||
|
||||
### 8. Organic bootstrap with structural-data surfacing
|
||||
|
||||
@@ -298,9 +296,9 @@ The v1.1 storage strategy is Neo4j + Postgres + pgvector + Redis + S3. That's fi
|
||||
|
||||
**My read:** the five-store split is the right *target architecture* and the v1 can ship with 3. The v1.1 plan in `09-roadmap.md` already says "MinIO can be deferred" and "pgvector can replace Neo4j's vector index."
|
||||
|
||||
### D3. The 30-tool surface is high
|
||||
### D3. The 45-tool surface is high
|
||||
|
||||
We've discussed this. 30 tools is at the empirical LLM ceiling.
|
||||
We've discussed this. 45 tools is well past the empirical LLM ceiling.
|
||||
|
||||
**Alternative:** collapse `state_at` into `entity_context(comprehensive=true)`. Collapse `summarize_chain` into `narrate_arc(style=...)`. The number drops to ~20.
|
||||
|
||||
@@ -334,23 +332,23 @@ We've discussed this. The `:Now` config node is a single point of failure and a
|
||||
|
||||
## The strategic question
|
||||
|
||||
Is the Lore Engine worth building?
|
||||
Is the Lore Engine worth building on Cognee?
|
||||
|
||||
**Strong case for:** the closest functional comparables (Cognee, LightRAG, GraphRAG) are all generic and lack a temporal model. The closest in-spirit comparable (Stanford Generative Agents) lacks a knowledge graph. The closest by use case (IVIE) generates worlds, doesn't reason over them. **No system in the literature does what the Lore Engine claims to do.** That's an opening.
|
||||
**Strong case for:** the closest functional comparables (Cognee, LightRAG, GraphRAG) are all generic and lack a temporal model. The closest in-spirit comparable (Stanford Generative Agents) lacks a knowledge graph. The closest by use case (IVIE) generates worlds, doesn't reason over them. **No system in the literature does what the Lore Engine claims to do.** That's an opening. And by building on Cognee, the Lore Engine inherits a production-grade substrate for free, which lets us focus the engineering effort on the domain layer (typed ontology, time model, consistency, TypeTemplate).
|
||||
|
||||
**Strong case against:** GraphRAG has 33,779 stars and Microsoft Research. Cognee has 17,843 stars and a paying customer base. LightRAG has 36,622 stars and an HKU team. **The Lore Engine is competing with funded, shipping, polished systems for the same homelab resources.** The "niche" of closed-world fictional reasoning is real, but the niche may not be big enough to justify the 43-day build cost when Cognee (with a fictional-world ontology extension) could close 80% of the gap in 2 weeks.
|
||||
**Strong case against:** GraphRAG has 33,779 stars and Microsoft Research. Cognee has 17,843 stars and a paying customer base. LightRAG has 36,622 stars and an HKU team. **The Lore Engine is competing with funded, shipping, polished systems for the same homelab resources.** The "niche" of closed-world fictional reasoning is real, but the niche may not be big enough to justify the 33-day build cost when Cognee (with a fictional-world ontology extension) could close 80% of the gap in 2 weeks.
|
||||
|
||||
**My recommendation:**
|
||||
|
||||
1. **Build the Lore Engine on top of Cognee, not on top of GraphMCP-Example.** The GraphMCP-Example substrate is a Kay-personal-project path. The Cognee substrate is a community-supported, MIT-licensed, production-grade path. The Lore Engine's *value* is the domain layer, not the substrate.
|
||||
1. **The substrate decision is made: Cognee.** The build order is in `09-roadmap.md` (Cognee-spike-first). The 1-week Cognee validation spike is the gating decision before any Lore Engine code is written. **If the spike fails, the substrate decision is wrong and the v1 needs a different foundation.** The 1-week validation is the cheapest insurance available.
|
||||
|
||||
2. **Build the v1.1 polymorphic extension model first, not the v1 time model.** The polymorphic extension is the highest-leverage single feature; the time model is important but more incremental. If the Lore Engine has to ship one thing in v1, ship the TypeTemplate system. The time model can be a v1.1 follow-up.
|
||||
2. **Ship the time model + typed ontology in the MVP (Phases 1–3).** The Lore Engine's *value* is the domain layer — the time model, the consistency rules, the TypeTemplate polymorphism. These are the things Cognee doesn't give you. Build them first; the substrate is solved.
|
||||
|
||||
3. **Validate before building more.** Build the MVP, ingest one small hand-crafted world, measure: can the LLM answer historical questions correctly? Does the consistency engine surface real problems? Does the TypeTemplate system work? **If the answer to any of these is "no," the design has a bug and the v2 should address it.** Don't build the v1.1 features on top of an unvalidated v1.
|
||||
|
||||
4. **The most leveraged next move** is a 1-week spike that builds the minimum-viable Lore Engine on Cognee and validates the core idea. If that spike succeeds, the 43-day plan makes sense. If it doesn't, the design needs a rethink before more code is written.
|
||||
4. **The most leveraged next move** is a 2-day spike that stands up Cognee locally, ingests a 10-document sample world, and confirms `cognee.recall("Who is Aldric?")` returns something sensible. If that spike succeeds, the 33-day plan makes sense. If it doesn't, the design needs a rethink before more code is written.
|
||||
|
||||
The critical-thinking section of the related work is the part I most want Kay to push back on. The comparison is honest; the strategic call is debatable. The right next move depends on what the spike shows.
|
||||
The critical-thinking section of the related work is the part I most want Kay to push back on. The comparison is honest; the strategic call was debatable, but the substrate decision is now made. The right next move depends on what the spike shows.
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -239,6 +239,10 @@ The v1.1 design used `world_id: string` on every node as a flat namespace. The v
|
||||
|
||||
5. **Remove `world_id` property in v2.0.** After the migration is stable, the string property is dropped. Plugin queries that still reference it are bugs.
|
||||
|
||||
6. **Rename the Postgres `world` table to `setting`.** The v1.1 Postgres schema had a `world` table and a `world_id` foreign key on operational tables (`lore_event`, `trade_log`, `retcon`, `dialogue_log`). Per v1.2, this becomes the `setting` table with a `setting_id` foreign key; add a `kind` enum column (`single_plane` | `multi_plane`) to mirror the v1.2 `Setting.kind` field. See `12-storage-strategy.md`.
|
||||
|
||||
7. **Update YAML instance files.** v1.1 YAML ingest used a flat `world_id: "arda_1st_age"` string at the top of each file. v1.2 YAML uses `setting_id: "mardonari"` (or whichever Setting the instance belongs to). The structured `EXISTS_IN` edge is generated from the `setting_id` field at ingest time, pointing to the Setting's primary Material Plane. See `11-extensibility.md` and `14-examples.md` for the renamed YAML schema.
|
||||
|
||||
## Open questions and trade-offs
|
||||
|
||||
### `EXISTS_IN` is many-to-many
|
||||
|
||||
Reference in New Issue
Block a user