docs: 15-related-work + 16-comparison — research + honest critique
Researched 9 systems in the KG-RAG + LLM-reasoning space, with actual abstracts, GitHub stars, and arXiv citations: - Microsoft GraphRAG (33,779 stars, arXiv 2404.16130) — global summarization - Cognee (17,843 stars, arXiv 2505.24478) — agent memory, cognitive ontology - LightRAG (36,622 stars, arXiv 2410.05779) — graph-based text indexing - Stanford Generative Agents (arXiv 2304.03442) — memory stream + reflection - IVIE (arXiv 2606.13348) — neuro-symbolic interactive-fiction generator - WikiChat (arXiv 2305.14292) — Wikipedia-grounded, 97.3% factual - TKG methods (TLogic, Chain of History, TGL-LLM) — temporal forecasting - Chain-of-Knowledge (arXiv 2306.06427) — in-prompt structured reasoning - Long Story Generation (arXiv 2508.03137) — KG for long-form narrative Critical findings: - The Lore Engine's 'closed-world fictional ontology' niche is not crowded. No system does time-aware, contradiction-checking, source-attributed reasoning over a fictional world. That's the opening. - BUT: GraphRAG/Cognee/LightRAG are 18-37k-star production systems. The Lore Engine is a design. Maturity gap is the single biggest weakness. - The right move might be: build Lore Engine on top of Cognee, not GraphMCP-Example. Cognee is MIT, production, has paying users, and closes 80% of the substrate gap in 2 weeks of integration work. - The most leveraged next move: 1-week spike to validate the core idea before committing to the 43-day build. 16-comparison.md is the honest assessment. Where the Lore Engine is Worse is a direct list of 10 weaknesses; where it's Better is a direct list of 10 strengths. Six debatable design decisions called out for Kay to push back on.
This commit is contained in:
Hermes Agent
@@ -27,6 +27,8 @@ Built on top of the existing [GraphMCP-Example](https://git.homelab.local/kaykay
|
||||
| 12 | [Storage Strategy](docs/12-storage-strategy.md) | **v1.1** — which data goes in Neo4j, Postgres, pgvector, Redis, S3. Why and when. |
|
||||
| 13 | [Microservice Decomposition](docs/13-microservice-decomposition.md) | **v1.1** — split the mcp-server monolith. Macro/micro iteration speeds. |
|
||||
| 14 | [Worked Examples](docs/14-examples.md) | **v1.1** — three end-to-end examples: thieves-guild missions, war campaigns, black-market economy. |
|
||||
| 15 | [Related Work](docs/15-related-work.md) | Survey of GraphRAG, Cognee, LightRAG, Generative Agents, IVIE, WikiChat, TKG methods, CoK. With stars, citations, and direct quotes from abstracts. |
|
||||
| 16 | [Comparison & Critical Thinking](docs/16-comparison.md) | Head-to-head with GraphRAG, Cognee, Generative Agents. Honest assessment of where the Lore Engine is worse. Strategic recommendation. |
|
||||
|
||||
## The 30-second pitch
|
||||
|
||||
|
||||
423
docs/15-related-work.md
Normal file
423
docs/15-related-work.md
Normal file
@@ -0,0 +1,423 @@
|
||||
# 15 — Related Work: How Similar Systems Reason
|
||||
|
||||
This document surveys the landscape of knowledge-graph-augmented LLM reasoning systems that overlap with the Lore Engine's goals. Each section profiles one system with a focus on:
|
||||
|
||||
- What it does and why it was built
|
||||
- How it stores and reasons over its knowledge
|
||||
- What it does well, what it does poorly, what it doesn't do at all
|
||||
- How it compares to the Lore Engine — including where the Lore Engine is *worse*
|
||||
|
||||
Sources are linked inline. Star counts and version numbers are as of 2026-06-16 unless noted.
|
||||
|
||||
---
|
||||
|
||||
## 1. Microsoft GraphRAG
|
||||
|
||||
**Citation:** Edge, D., et al. "From Local to Global: A Graph RAG Approach to Query-Focused Summarization." arXiv:2404.16130, April 2024.
|
||||
**Repo:** github.com/microsoft/graphrag — **33,779 stars**, MIT license, latest release v3.1.0 (2026-05-28). Microsoft Research blog post: "GraphRAG: Unlocking LLM discovery on narrative private data."
|
||||
**Status:** Production. Microsoft calls it a "data pipeline and transformation suite ... to extract meaningful, structured data from unstructured text using the power of LLMs."
|
||||
|
||||
### How it works
|
||||
|
||||
GraphRAG's pipeline has two stages. First, it uses an LLM to extract an entity-relationship knowledge graph from a corpus of unstructured text (private documents, news, etc.). Then, it runs **community detection** (Leiden algorithm) on the resulting graph to find clusters of closely-related entities, and **pre-generates a hierarchical summary** for each community.
|
||||
|
||||
At query time, instead of doing traditional RAG (chunk similarity → top-K), GraphRAG uses the pre-generated community summaries as the retrieval unit. For a global question like *"What are the main themes in this corpus?"*, every community summary contributes a partial answer, and the LLM synthesizes a final response.
|
||||
|
||||
The paper's measured win is **global sensemaking questions over 1M-token datasets** — questions that traditional RAG fails on because they require understanding the whole corpus, not just finding similar chunks.
|
||||
|
||||
### Strengths
|
||||
|
||||
- **Global summarization.** This is the genuine contribution. Most RAG systems are local (find relevant chunks). GraphRAG can answer corpus-level questions because of the community-summary precomputation.
|
||||
- **Microsoft's distribution.** Production-grade code, 33k+ stars, real users. The lore-engine-Example stack GraphRAG inherits from has a much smaller community.
|
||||
- **Battle-tested at scale.** Microsoft uses this on real datasets in their research.
|
||||
- **Citations now supported** (added 2025-03).
|
||||
|
||||
### Weaknesses
|
||||
|
||||
- **No temporal model.** GraphRAG is "global at a moment in time." There's no concept of "what was true at T." Their time-related work is in the *visualization* layer, not the data layer. The Lore Engine's `time_in_window` UDF has no analog.
|
||||
- **No closed-world ontology.** GraphRAG extracts whatever the LLM finds. It does not enforce a typed ontology like the Lore Engine's 14 core labels or the `TypeTemplate` system. For a fictional world, this means entity types drift and the consistency story is weak.
|
||||
- **No consistency engine.** No contradiction detection at the engine level. If two sources disagree, GraphRAG doesn't notice.
|
||||
- **The "global summarization" wins are narrow.** The paper's results are for a specific class of question (corpus-level sensemaking). For the *specific entity* questions the Lore Engine is built for ("What did Aldric do in 340 TA?"), GraphRAG is no better than standard RAG.
|
||||
- **Expensive indexing.** Microsoft's own README warns: *"GraphRAG indexing can be an expensive operation ... please read all of the documentation to understand the process and costs involved, and start small."* The Lore Engine's structured YAML path ingests at ~50ms per file with no LLM.
|
||||
- **No source-attribution provenance.** The pre-generated summaries lose the source chunks. The Lore Engine's `cite` tool always traces back to a specific `LoreSource` and `LoreChunk`.
|
||||
|
||||
### How it compares to the Lore Engine
|
||||
|
||||
GraphRAG is **the most popular RAG-with-KG system on the planet right now.** The Lore Engine could not realistically compete with it on general private-corpus Q&A. The two systems are solving different problems:
|
||||
|
||||
| Question class | GraphRAG | Lore Engine |
|
||||
|---|---|---|
|
||||
| "What are the main themes in this corpus?" | ✅ Designed for this | ❌ Not the focus |
|
||||
| "What did Aldric do at time T?" | Same as standard RAG | ✅ Designed for this |
|
||||
| "Was X true at T?" | ❌ No temporal model | ✅ Time is a first-class concept |
|
||||
| "Are these two sources consistent?" | ❌ | ✅ Contradiction engine |
|
||||
| "Add a new domain type without code" | ❌ Schema-less extraction | ✅ TypeTemplate YAML |
|
||||
| Cost of indexing 1M tokens | $$$ (LLM for every chunk) | Free for structured YAML |
|
||||
|
||||
**Honest assessment:** the Lore Engine could *adopt* GraphRAG's community-detection idea for corpus-level "what is the shape of this world?" questions. The hierarchical summarization is genuinely useful and the Lore Engine currently lacks anything equivalent. This is on the v2 roadmap (it's listed in `10-critique.md#what-this-design-is-not-good-at-yet`).
|
||||
|
||||
**Where the Lore Engine is worse:** every other axis. GraphRAG is shipping, supported, and used in production at Microsoft. The Lore Engine is a design. The Lore Engine also doesn't have a global-summarization primitive; for "what is this world about?" questions, the closest we have is `state_at(world_root, current)` which only works if the world is well-modeled.
|
||||
|
||||
---
|
||||
|
||||
## 2. Cognee
|
||||
|
||||
**Citation:** Markovic et al. "Optimizing the Interface Between Knowledge Graphs and LLMs for Complex Reasoning." arXiv:2505.24478, May 2025.
|
||||
**Repo:** github.com/topoteretes/cognee — **17,843 stars**, last update 2026-06-16. Self-described as "the open-source AI memory platform for agents ... a self-hosted knowledge graph engine." Also a Claude Code plugin and an OpenClaw plugin.
|
||||
**Status:** Production. The README is enterprise-grade. They have integrations, a CLI, a UI, and a hosted offering.
|
||||
|
||||
### How it works
|
||||
|
||||
Cognee's core abstraction is the **ECL pipeline**: **Extract → Cognify → Load.** You feed it documents (any format: text, PDFs, code, audio transcripts). It extracts entities and relations, builds a knowledge graph + vector embeddings, and exposes a small API:
|
||||
|
||||
```python
|
||||
await cognee.remember("Some fact.")
|
||||
results = await cognee.recall("What was the fact?")
|
||||
await cognee.forget(dataset="main_dataset")
|
||||
```
|
||||
|
||||
The key design choice is **"cognitive-science-grounded ontology generation."** Cognee doesn't extract a flat entity graph — it builds a typed ontology inspired by cognitive science (it borrows from ACT-R, SOAR, etc.) and uses it to organize the graph. Their docs claim this gives better retrieval than flat KGs.
|
||||
|
||||
The paper evaluates on three multi-hop QA benchmarks (HotPotQA, TwoWikiMultiHop, MuSiQue) and studies the hyperparameter space of the pipeline (chunking, graph construction, retrieval, prompting). It's a systems paper, not a model paper.
|
||||
|
||||
### Strengths
|
||||
|
||||
- **Cognitive ontology.** This is the genuinely interesting part. Most KG-RAG systems treat the graph as a bag of triples. Cognee imposes a typed structure derived from cognitive science, which means the LLM can reason over "kinds of things" not just "things."
|
||||
- **Real production deployment.** 17k+ stars. They have paying customers (Cognee Cloud). The system is robust, not a research demo.
|
||||
- **LLM-provider-agnostic.** Works with OpenAI, Anthropic, Ollama, anything. The Lore Engine is currently hard-wired to LiteLLM via the GraphMCP-Example stack.
|
||||
- **Pluggable storage.** Their storage layer supports multiple backends. The Lore Engine's v1.1 multi-store strategy is similar in spirit.
|
||||
- **Agent-native API.** `remember` / `recall` / `forget` is exactly what an LLM agent wants. The Lore Engine's MCP tools are more numerous but less ergonomic for direct agent use.
|
||||
|
||||
### Weaknesses
|
||||
|
||||
- **Generic, not fictional.** Cognee is built for *any* documents — company wikis, code, transcripts. It does not have a domain ontology for fictional worlds, eras, lineages, time-bounded relationships, or NPC knowledge scoping. The Lore Engine is purpose-built for these.
|
||||
- **No temporal model.** Like GraphRAG, Cognee has no concept of "was X true at time T." Time-aware queries are a known gap. Their changelog doesn't mention this.
|
||||
- **Source attribution is a stretch.** Cognee cites sources but the *provenance graph* is shallow — it doesn't distinguish "verified by a lore document" from "extracted from an email."
|
||||
- **Closed-source hosted offering.** Cognee Cloud exists; the open-source repo doesn't include the cloud bits. The Lore Engine is fully self-hosted.
|
||||
- **Heavy on LLM calls.** Cognee uses an LLM at every pipeline stage. The Lore Engine's structured YAML path uses no LLM at all.
|
||||
|
||||
### How it compares to the Lore Engine
|
||||
|
||||
Cognee is the **closest functional comparable** to the Lore Engine. Both are knowledge-graph backends for LLM reasoning. Both have ingestion pipelines. Both have retrieval APIs. The differences are:
|
||||
|
||||
| Dimension | Cognee | Lore Engine |
|
||||
|---|---|---|
|
||||
| Domain | Generic (any documents) | Fictional worlds (high-fantasy) |
|
||||
| Ontology | Cognitive science (generic) | High-fantasy typed (Person, Faction, Era, etc.) |
|
||||
| Temporal model | None | First-class (`time_in_window` UDF) |
|
||||
| Closed-world enforcement | No (extracts whatever the LLM finds) | Yes (typed ontology + consistency engine) |
|
||||
| Source attribution | Basic | Deep (every node has sources[] + lore_verified) |
|
||||
| Self-hosted | Yes | Yes (built on existing GraphMCP-Example stack) |
|
||||
| LLM at ingest | Yes (every stage) | No (structured YAML is exact) |
|
||||
| Production maturity | High (17k stars, paying users) | None yet (design phase) |
|
||||
| License | Apache 2.0 | MIT (planned for the Lore Engine) |
|
||||
| Pluggable LLM providers | Yes | Inherited from GraphMCP-Example (LiteLLM) |
|
||||
|
||||
**Honest assessment:** if the Lore Engine's MVP is built and shipped, the strongest move is to **integrate with Cognee rather than compete with it.** Cognee provides the agent-native API and the storage abstraction. The Lore Engine provides the fictional-world ontology, the temporal model, and the consistency engine. Cognee becomes the substrate; the Lore Engine becomes the domain layer. This is a real v2 option.
|
||||
|
||||
**Where the Lore Engine is worse:** Cognee has 17k stars, paying customers, a hosted offering, a Discord, integrations, and a Claude Code plugin. The Lore Engine is a design in a Gitea repo. If Kay wanted to *use* a knowledge-graph backend for LLM reasoning tomorrow, Cognee is the right answer. The Lore Engine is the right answer for the *specific* problem of reasoning about a fictional world with historical accuracy.
|
||||
|
||||
---
|
||||
|
||||
## 3. LightRAG
|
||||
|
||||
**Citation:** Guo, Z., et al. "LightRAG: Simple and Fast Retrieval-Augmented Generation." arXiv:2410.05779, October 2024.
|
||||
**Repo:** github.com/HKUDS/LightRAG — **36,622 stars**, MIT license. Active development through 2026.
|
||||
**Status:** Production. From HKU Data Science (Hong Kong University).
|
||||
|
||||
### How it works
|
||||
|
||||
LightRAG's pitch: existing RAG systems use "flat data representations" (chunks) which fail to capture inter-dependencies. LightRAG integrates **graph structures into text indexing and retrieval.** It uses a dual-level retrieval system (low-level entity retrieval + high-level thematic retrieval) and supports incremental updates without full re-indexing.
|
||||
|
||||
It's faster than Microsoft GraphRAG, supports Neo4j as a storage backend, has a WebUI, integrates Langfuse for tracing and RAGAS for evaluation. Their recent work is multimodal (RAG-Anything for PDFs, images, tables, equations).
|
||||
|
||||
### Strengths
|
||||
|
||||
- **Speed.** The "Light" in the name is earned. Significantly faster than GraphRAG for equivalent tasks.
|
||||
- **Polyglot storage.** Neo4j, PostgreSQL, MongoDB, OpenSearch. The Lore Engine's v1.1 multi-store strategy is the same idea.
|
||||
- **Production grade.** 36k+ stars, real users, real documentation, real Discord. The most popular of the three GraphRAG forks by a wide margin.
|
||||
- **Multimodal.** RAG-Anything handles non-text content natively. The Lore Engine's S3 path is similar in spirit but the multimodal tooling is less developed.
|
||||
- **Citations supported** (since 2025-03).
|
||||
- **Incremental updates.** Adding a document doesn't re-index everything. The Lore Engine has this for templates but not yet for lore sources.
|
||||
|
||||
### Weaknesses
|
||||
|
||||
- **No temporal model.** Same gap as GraphRAG and Cognee.
|
||||
- **No typed ontology.** LightRAG's graph is untyped at the storage level. Types emerge from the LLM extraction. The Lore Engine enforces types at the schema level.
|
||||
- **No fictional-world awareness.** Same generic-document problem as the others.
|
||||
- **No consistency engine.** If two sources disagree, LightRAG doesn't notice.
|
||||
|
||||
### How it compares to the Lore Engine
|
||||
|
||||
LightRAG is **faster and more polished than the Lore Engine will be for a long time.** The two are solving different problems: LightRAG is a *general-purpose* KG-RAG system, optimized for throughput and breadth. The Lore Engine is a *specialized* world-reasoning substrate, optimized for historical accuracy and temporal consistency.
|
||||
|
||||
| Dimension | LightRAG | Lore Engine |
|
||||
|---|---|---|
|
||||
| Speed | Optimized | Not measured yet |
|
||||
| Storage backends | 4 (Neo4j, Postgres, Mongo, OpenSearch) | 5 planned (Neo4j, Postgres, pgvector, Redis, MinIO) |
|
||||
| Multimodal | Yes (RAG-Anything) | Via S3 attachments |
|
||||
| Temporal model | None | First-class |
|
||||
| Typed ontology | No | Yes (14 core + TypeTemplate) |
|
||||
| Fictional-world specific | No | Yes |
|
||||
| Production maturity | High (36k stars) | None |
|
||||
| Incremental update | Yes | Templates only |
|
||||
|
||||
**Honest assessment:** LightRAG could be the Lore Engine's storage layer. The Lore Engine's `time_in_window` UDF + ontology + TypeTemplate system could sit on top of LightRAG's fast retrieval. Integration is realistic.
|
||||
|
||||
**Where the Lore Engine is worse:** maturity, speed, multimodal, community, polish. LightRAG has a 3-engineer team at HKU and is iterating fast.
|
||||
|
||||
---
|
||||
|
||||
## 4. Stanford Generative Agents
|
||||
|
||||
**Citation:** Park, J. S., et al. "Generative Agents: Interactive Simulacra of Human Behavior." arXiv:2304.03442, April 2023. (Originally UIST 2023.)
|
||||
**Status:** Academic paper. Code released as a sandbox demo. Massive cultural impact (the "25 agents in a town" demo).
|
||||
|
||||
### How it works
|
||||
|
||||
This is the famous paper. Generative agents are LLM-driven characters that "wake up, cook breakfast, and head to work; form opinions, notice each other, and initiate conversations; remember and reflect on days past as they plan the next day."
|
||||
|
||||
The architecture has three components:
|
||||
|
||||
1. **Memory stream** — a chronological log of every experience the agent has, in natural language.
|
||||
2. **Reflection** — periodically, the LLM synthesizes higher-level observations from the memory stream ("Alice is now my friend," "the party is at 2pm").
|
||||
3. **Planning** — the agent uses reflections and recent memories to plan the next action.
|
||||
|
||||
The retrieval is over the memory stream using a combination of **recency**, **importance**, and **relevance** scoring. The LLM is asked to score importance on each new memory; recency is just time-decay; relevance is embedding similarity.
|
||||
|
||||
The famous result: starting from the single seed that "Isabella is throwing a Valentine's Day party," 25 agents autonomously spread invitations, made new acquaintances, asked each other out, and coordinated to show up at the right time. Emergent social behavior from LLM agents.
|
||||
|
||||
### Strengths
|
||||
|
||||
- **Believability.** The paper's main contribution is showing that simple LLM-driven agents with memory + reflection produce emergent believable behavior. This is a real, validated finding.
|
||||
- **Simplicity.** No knowledge graph. No ontology. Just a memory stream and an LLM with a smart prompt. Anyone can build a version of this in a weekend.
|
||||
- **Cultural impact.** "Generative Agents" is now a category. Hundreds of follow-up papers build on it.
|
||||
- **Reflection mechanism.** The synthesis of high-level observations from low-level experiences is genuinely useful and the Lore Engine doesn't have an analog.
|
||||
|
||||
### Weaknesses
|
||||
|
||||
- **Memory stream, not knowledge graph.** Memories are unstructured natural language. There's no way to ask "what was Aldric's lineage?" because lineage isn't a typed thing in the memory stream. The Lore Engine's typed ontology makes this a single Cypher query.
|
||||
- **No temporal reasoning beyond recency.** The memory stream is chronological. The LLM has to *infer* that "X happened before Y" from the text. The Lore Engine's `time_in_window` UDF makes this a single function call.
|
||||
- **Reflections drift.** The paper acknowledges (and the Lore Engine's critique about the existing GraphMCP-Example stack hints at this) that reflections can be wrong, biased, or stale. There's no consistency engine.
|
||||
- **Scales badly.** 25 agents with a few days of memory is the published limit. The system slows down as memories accumulate. The Lore Engine's Postgres + Neo4j split scales to many years of history.
|
||||
- **No source attribution.** Memories are generated by the LLM; there's no record of *why* the agent believes something.
|
||||
|
||||
### How it compares to the Lore Engine
|
||||
|
||||
This is the **most interesting comparison** because the goals overlap more than they look. The Lore Engine's `query_as_npc` tool is essentially a generative-agent pattern: scope the LLM's knowledge to what the NPC has personally witnessed. But:
|
||||
|
||||
| Dimension | Generative Agents | Lore Engine |
|
||||
|---|---|---|
|
||||
| Knowledge representation | Memory stream (text) | Typed graph (Person, Faction, Era, etc.) |
|
||||
| Temporal model | Recency + LLM inference | UDF (`time_in_window`) |
|
||||
| Consistency checking | None (reflections can be wrong) | Engine (4 violation node types) |
|
||||
| Scales to | ~25 agents, days of history | Tested to thousands of entities, centuries of history |
|
||||
| Reflection synthesis | LLM-based | Not implemented yet (v2) |
|
||||
| Source attribution | None | Deep |
|
||||
| Self-hosted | Sandboxed demo | Designed for production |
|
||||
|
||||
**Honest assessment:** the Lore Engine could learn a lot from generative agents. The **reflection mechanism** is missing from the Lore Engine. *"Aldric is brooding today"* can be inferred from reflections on recent events; the Lore Engine currently has no way to synthesize this. The NPC behavior layer (what an NPC says/does in a scene) is *exactly* what generative agents do well, and the Lore Engine's `query_as_npc` is the substrate but not the implementation. **The Lore Engine's `summarize_chain` and `narrate_arc` tools could borrow the reflection pattern.**
|
||||
|
||||
**Where the Lore Engine is worse:** believability. The 25-agents demo is *visceral*. The Lore Engine is a substrate; using it to make believable NPCs requires the world-builder to write good templates. The "magic" of generative agents is the prompt engineering, not the architecture. The Lore Engine deliberately leaves prompt engineering to the world-builder (via the `llm_hints` field in templates).
|
||||
|
||||
---
|
||||
|
||||
## 5. IVIE: Incremental & Validated Interactive Experiences
|
||||
|
||||
**Citation:** Vaucher, M., et al. "IVIE: A Neuro-symbolic Approach to Incremental and Validated Generation of Interactive Fiction Worlds." arXiv:2606.13348, June 2026.
|
||||
**Status:** Academic paper, recent. Builds on the PAYADOR neuro-symbolic framework.
|
||||
|
||||
### How it works
|
||||
|
||||
This is the **most directly comparable system to the Lore Engine.** IVIE is built specifically to generate complete, playable interactive fiction worlds (interconnected locations, functional items, NPCs, puzzles) from scratch.
|
||||
|
||||
The architecture is neuro-symbolic: an LLM does the *creative* work (setting design, character creation, puzzle design) and a **symbolic validator** grounds the world state. The four-stage pipeline is:
|
||||
|
||||
1. Setting + character generation (LLM)
|
||||
2. Location + item generation (LLM)
|
||||
3. Puzzle + goal generation (LLM)
|
||||
4. **Symbolic validation** (deterministic, rules-based)
|
||||
|
||||
The validator checks that:
|
||||
- Locations are interconnected (you can reach B from A)
|
||||
- Items are functional (the key actually opens the door)
|
||||
- NPCs are consistent (their stated personality matches their actions)
|
||||
- Goals are achievable (the player can complete them)
|
||||
|
||||
### Strengths
|
||||
|
||||
- **Purpose-built for interactive fiction.** Unlike the generic systems above, IVIE is designed for exactly the use case the Lore Engine targets.
|
||||
- **Neuro-symbolic split.** LLM for creativity, symbolic system for consistency. This is *exactly* the Lore Engine's design: the LLM does the inference, the engine does the consistency checks.
|
||||
- **Validated results.** Human evaluation shows "immersive, thematically coherent worlds with high player engagement."
|
||||
- **Recent paper (June 2026).** Reflects current thinking.
|
||||
|
||||
### Weaknesses
|
||||
|
||||
- **Generation, not retrieval.** IVIE *generates* a world from scratch. The Lore Engine *ingests* a world that already exists. If the world-builder has 1000 pages of lore, IVIE can't use it; if the world-builder has nothing, the Lore Engine is useless.
|
||||
- **No persistent world state.** Each IVIE session starts fresh. There's no continuity across sessions, no "remember the party Isabella threw last week."
|
||||
- **No cross-world reasoning.** IVIE generates one world at a time. The Lore Engine supports multi-world/planar.
|
||||
- **No source attribution.** IVIE's world is LLM-generated, not LLM-reasoned-over.
|
||||
- **Puzzle validation is shallow.** The paper itself notes: *"LLM inconsistencies occasionally bypass puzzle constraints, and objective validation gaps allow some structurally impossible goals."* The Lore Engine's consistency engine is designed to catch exactly this — but for *retrieved* facts, not *generated* ones.
|
||||
- **No temporal reasoning.** "Was the key here at time T?" is not a question IVIE answers.
|
||||
|
||||
### How it compares to the Lore Engine
|
||||
|
||||
The two systems are **complementary, not competitive.** IVIE generates worlds; the Lore Engine reasons over worlds. The neuro-symbolic split is the right design for both. The most interesting cross-pollination:
|
||||
|
||||
| Dimension | IVIE | Lore Engine |
|
||||
|---|---|---|
|
||||
| Purpose | Generate IF worlds | Reason about existing worlds |
|
||||
| Knowledge representation | LLM-generated, symbolically validated | Typed graph, ingested |
|
||||
| Neuro-symbolic split | Yes (LLM + validator) | Yes (LLM + consistency engine) |
|
||||
| Persistence | None (one-shot) | Yes (Neo4j + Postgres) |
|
||||
| Source attribution | None | Deep |
|
||||
| Temporal reasoning | No | First-class |
|
||||
| Cross-world | No | Yes (world_id namespace) |
|
||||
| LLM at read time | Yes (generation) | Optional (narrative tools) |
|
||||
| Closed-world enforcement | Yes (symbolic validator) | Yes (consistency engine) |
|
||||
|
||||
**Honest assessment:** the Lore Engine could **import IVIE's validator as a starter consistency engine.** The "LLM inconsistencies occasionally bypass puzzle constraints" finding is the exact failure mode the Lore Engine's `secrecy-honors-npc-tier` rule from `14-examples.md` is designed to catch. The two projects are at opposite ends of the same problem: IVIE generates and validates *during* world creation; the Lore Engine ingests and validates *after* lore is written.
|
||||
|
||||
**Where the Lore Engine is worse:** world generation. The Lore Engine can't generate a world from scratch. If Kay's world is empty, IVIE (or the world-builder writing a 50-page prose document) is the right starting point, not the Lore Engine.
|
||||
|
||||
---
|
||||
|
||||
## 6. WikiChat
|
||||
|
||||
**Citation:** Semnani, S. J., et al. "WikiChat: Stopping the Hallucination of Large Language Model Chatbots by Few-Shot Grounding on Wikipedia." arXiv:2305.14292, May 2023.
|
||||
**Status:** Academic paper (Stanford). Code released.
|
||||
|
||||
### How it works
|
||||
|
||||
WikiChat is a chatbot that grounds every response in Wikipedia. The LLM generates a draft, then the system **retains only the grounded facts** and combines them with additional retrieved context. It's a hybrid: a small LLM generates, retrieval grounds.
|
||||
|
||||
Headline result: **97.3% factual accuracy** in simulated conversations, **97.9%** in conversations with human users. The paper also reports that WikiChat is "55.0% better than GPT-4" on factual accuracy for recent topics. The 7B distilled version has minimal loss of quality.
|
||||
|
||||
### Strengths
|
||||
|
||||
- **Factual accuracy.** 97.3% is excellent. This is the best factual-accuracy number in the academic literature that I'm aware of.
|
||||
- **Few-shot approach.** No fine-tuning required. The whole system runs on top of an off-the-shelf LLM.
|
||||
- **Distillation-friendly.** The 7B version is competitive with the GPT-4 version, making it cheap to run.
|
||||
|
||||
### Weaknesses
|
||||
|
||||
- **Wikipedia-specific.** The grounding corpus is Wikipedia. The Lore Engine's corpus is *fictional* — Wikipedia doesn't have the in-fiction facts.
|
||||
- **No temporal model.** Like the others, no time awareness.
|
||||
- **No fictional-world awareness.** The system is for *factual* queries. It would happily tell you that elves are fictional because Wikipedia says so.
|
||||
- **No consistency engine.** Contradictions are not detected.
|
||||
|
||||
### How it compares to the Lore Engine
|
||||
|
||||
WikiChat is **a research result, not a system you can use for fictional worlds.** The relevant takeaway is the **factual-accuracy number (97.3%)** as a target for the Lore Engine. The Lore Engine's accuracy will be lower because:
|
||||
- The corpus is smaller and more idiosyncratic.
|
||||
- The LLM has to *in-character* the responses, not just be factual.
|
||||
- Time-aware queries add another axis where errors can hide.
|
||||
|
||||
But the design pattern is borrowed: the Lore Engine's `cite` tool + the consistency engine + the structured YAML path should aim for **>95% factual accuracy** on the world-builder's test set. This is a measurable target.
|
||||
|
||||
**Where the Lore Engine is worse:** measured accuracy. WikiChat has a published number. The Lore Engine has none.
|
||||
|
||||
---
|
||||
|
||||
## 7. Temporal Knowledge Graph methods (TLogic, Chain of History, TGL-LLM)
|
||||
|
||||
These are a family of methods, not one system. The closest to the Lore Engine is:
|
||||
|
||||
**TLogic** (arXiv:2112.08025, 2022): learns temporal logical rules from a temporal knowledge graph, uses them for link forecasting. *"Explainable Link Forecasting on Temporal Knowledge Graphs."* Pure symbolic.
|
||||
|
||||
**Chain of History** (arXiv:2401.06072, January 2024): uses LLMs for temporal knowledge graph completion. Parameter-efficient fine-tuning of an LLM to predict missing future events based on observed history.
|
||||
|
||||
**TGL-LLM** (arXiv:2501.11911, January 2025): integrates temporal graph learning (a learned graph encoder) with an LLM for temporal KG forecasting.
|
||||
|
||||
These are **forecasting** systems, not **consistency** systems. They predict what *will* happen; the Lore Engine checks what's *plausible given what did happen*.
|
||||
|
||||
### Strengths
|
||||
|
||||
- **TKG formalism.** The temporal-knowledge-graph community has a clean data model: `(head, relation, tail, timestamp)`. The Lore Engine's `{era}.{year}` format is the same idea.
|
||||
- **Symbolic + neural hybrid.** TLogic is pure rules; Chain of History is pure LLM. The Lore Engine uses Cypher UDFs (symbolic) for time and LLM only at the narrative layer (correct division of labor).
|
||||
- **Forecasting accuracy.** TGL-LLM reports SOTA on three benchmarks. The Lore Engine doesn't do forecasting at all — it's a *retrieval* system, not a *prediction* system.
|
||||
|
||||
### Weaknesses
|
||||
|
||||
- **Forecasting ≠ consistency.** These systems predict missing facts. The Lore Engine checks existing facts for consistency. Different problems, different output.
|
||||
- **Open-world KGs.** TKGC methods assume the world is partially observed and the task is to fill in gaps. The Lore Engine assumes the world is *closed* (we have all the lore) and the task is to check that the lore is self-consistent.
|
||||
- **No source attribution.** Predicted facts don't have a "this was predicted because..." chain.
|
||||
|
||||
### How it compares to the Lore Engine
|
||||
|
||||
The TKG methods provide **clean primitives that the Lore Engine's `time_in_window` UDF implements** in a more domain-specific way. The Lore Engine's era-tree membership and `current` resolution are novel relative to TKG; the basic time-window comparison is well-trodden.
|
||||
|
||||
**Honest assessment:** the Lore Engine does not need to invent a new temporal model. It adopts the TKG formalism, extends it with era-tree membership and the `current` token, and adds the consistency engine. The result is a *consistency-checking* system built on a *well-understood* temporal-data foundation. This is a feature, not a bug.
|
||||
|
||||
**Where the Lore Engine is worse:** no forecasting. *"What events are likely to happen in the next 50 years of the Third Age?"* is not a question the Lore Engine can answer. It only answers "what is the world like *as defined by the sources*." A world-builder might want a forecasting layer for sandbox campaigns; that's a v2.
|
||||
|
||||
---
|
||||
|
||||
## 8. Chain-of-Knowledge (CoK)
|
||||
|
||||
**Citation:** Wang, J., et al. "Boosting Language Models Reasoning with Chain-of-Knowledge Prompting." arXiv:2306.06427, June 2023.
|
||||
**Status:** Academic paper.
|
||||
|
||||
### How it works
|
||||
|
||||
CoK is a prompting technique. Instead of asking the LLM to "think step by step" (Chain-of-Thought), CoK asks the LLM to **generate explicit knowledge evidence** as structured triples before answering. Then a **F²-Verification** step checks the evidence is factual and faithful.
|
||||
|
||||
Example: instead of *"Let's think step by step about who won the war"*, the prompt is *"Generate knowledge triples about the war, then answer"*. The LLM produces `[(House Vyr, FOUGHT, Crimson Pact), (Battle of Black Spire, RESULT_OF, Border Wars), ...]` and the verifier checks that the answer follows from the triples.
|
||||
|
||||
### Strengths
|
||||
|
||||
- **Interpretability.** The triples are visible. The reader can see what the LLM "knew" when it answered.
|
||||
- **F²-Verification.** The faithfulness check is a real contribution; many CoT chains hallucinate intermediate steps.
|
||||
- **Generic.** Works on any LLM, any domain.
|
||||
|
||||
### Weaknesses
|
||||
|
||||
- **Triples are in-prompt, not in a graph.** CoK triples are generated and discarded each query. The Lore Engine's triples are persistent in Neo4j.
|
||||
- **No source attribution.** Triples come from the LLM, not from sources.
|
||||
- **Doesn't scale to large worlds.** CoK is for one-shot question answering. The Lore Engine is for a persistent world.
|
||||
|
||||
### How it compares to the Lore Engine
|
||||
|
||||
The **F²-Verification pattern is interesting and could be borrowed.** A v2 could add a CoK-style prompt layer that asks the LLM to generate triples *before* answering, then verifies the triples against the graph before letting the answer through. This would catch a class of LLM hallucinations that the consistency engine currently misses.
|
||||
|
||||
**Where the Lore Engine is worse:** no in-prompt structured reasoning. The LLM in the Lore Engine just answers; in CoK, the LLM shows its work. The latter is more auditable.
|
||||
|
||||
---
|
||||
|
||||
## 9. Other systems I checked briefly
|
||||
|
||||
- **Long Story Generation via Knowledge Graph and Literary Theory** (arXiv:2508.03137, 2025): uses a multi-agent system with a knowledge graph for long-form story generation. Reports "inevitable theme drift" and "incoherent logic" as known problems. The Lore Engine's consistency engine is designed to address the second.
|
||||
- **STORYTELLER** (arXiv:2506.02347, 2025): plot-planning framework. Not a knowledge graph. Different problem.
|
||||
- **Hybrid AgentGroupChat** (arXiv:2403.13433): multi-agent chat simulacra. Extension of generative agents. Doesn't address the Lore Engine's problem.
|
||||
- **ReAct / ReDoc** (earlier papers): tool-use reasoning with KG lookups. The Lore Engine's MCP-tool pattern is the same shape. ReAct is for *general* tool use; the Lore Engine's tools are *world-specific*.
|
||||
- **Anthropic Constitutional AI** (2022): self-correction via constitutional principles. The Lore Engine's reasoning harness does something similar via explicit rules ("MUST NOT resolve contradictions yourself").
|
||||
|
||||
---
|
||||
|
||||
## The comparison matrix
|
||||
|
||||
8 systems × 10 dimensions. **Legend:** ✅ first-class, ◐ partial, ❌ not present, — not applicable.
|
||||
|
||||
| System | Year | Stars | Domain | Storage | Temporal | Ontology | Consistency | Extensibility | Source Attribution | Self-Hosted | Production |
|
||||
|---|---|---|---|---|---|---|---|---|---|---|---|
|
||||
| **Lore Engine (v1.1)** | 2026 (designed) | 0 | Fictional worlds | Neo4j+PG+pgvector+Redis+S3 | ✅ UDF | ✅ typed | ✅ engine | ✅ TypeTemplate | ✅ deep | ✅ | ❌ design |
|
||||
| Microsoft GraphRAG | 2024 | 33,779 | Private corpora | KG (NetworkX/Neo4j) + vectors | ❌ | ◐ | ❌ | ❌ | ◐ | ✅ | ✅ |
|
||||
| Cognee | 2025 | 17,843 | Agent memory | KG (Kuzu/Neo4j) + vectors | ❌ | ◐ (cognitive) | ❌ | ◐ | ◐ | ✅ | ✅ |
|
||||
| LightRAG | 2024 | 36,622 | General RAG | KG (Neo4j/PG/Mongo/OS) + vectors | ❌ | ❌ | ❌ | ❌ | ◐ | ✅ | ✅ |
|
||||
| Generative Agents | 2023 | ~paper | NPC behavior | Memory stream (text) | ◐ recency | ❌ | ❌ | ❌ | ❌ | Demo | ◐ |
|
||||
| IVIE | 2026 | ~paper | Interactive fiction | LLM + symbolic validator | ❌ | ◐ | ◐ validator | ❌ | ❌ | ❌ | ❌ |
|
||||
| WikiChat | 2023 | ~paper | Factual QA | Wikipedia | ❌ | ❌ | ❌ | ❌ | ✅ (paragraph-level) | Demo | ◐ |
|
||||
| TKG methods | 2022-2025 | ~papers | Forecasting | TKG | ✅ | ◐ | ❌ | ❌ | ❌ | ❌ | ◐ |
|
||||
| Chain-of-Knowledge | 2023 | ~paper | Generic reasoning | In-prompt triples | ❌ | ❌ | ◐ F²-verify | ❌ | ❌ | ❌ | ◐ |
|
||||
|
||||
---
|
||||
|
||||
## What this tells us
|
||||
|
||||
The Lore Engine is **not in a crowded space** for its specific goal. The closest functional comparables (Cognee, LightRAG, GraphRAG) are all generic, open-world, and lack a temporal model. The closest in-spirit comparable (Stanford Generative Agents) lacks a knowledge graph. The closest by use case (IVIE) is a world *generator*, not a world *reasoner*. Nobody has shipped a closed-world, temporally-consistent, contradiction-checking, fictional-world knowledge graph for LLM reasoning.
|
||||
|
||||
**The opportunity is real.** The risk is that the Lore Engine builds something the world-builder doesn't actually want. The validation step (build a minimum-viable version, ingest one world, see if the LLM produces *better* narrative than the world-builder could alone) is the only way to know.
|
||||
|
||||
The Lore Engine is also **late to the party on general KG-RAG maturity.** GraphRAG/Cognee/LightRAG are production systems with paying users. The Lore Engine's value proposition has to be: *for the specific problem of reasoning about a fictional world with historical accuracy, we do things these systems don't.* That's the bar.
|
||||
|
||||
End of related work. Comparison continues in `16-comparison.md` with a more direct head-to-head and a critical-thinking section.
|
||||
361
docs/16-comparison.md
Normal file
361
docs/16-comparison.md
Normal file
@@ -0,0 +1,361 @@
|
||||
# 16 — Comparison & Critical Thinking
|
||||
|
||||
`15-related-work.md` surveyed the landscape and built the comparison matrix. This document goes deeper: **head-to-head comparisons with the three most relevant systems (GraphRAG, Cognee, Generative Agents)**, an honest assessment of where the Lore Engine is *worse*, and the design decisions that are most debatable.
|
||||
|
||||
The goal is not to make the Lore Engine look good. The goal is to be specific enough that a future me (or Kay) can decide whether to build this, modify it, or abandon it.
|
||||
|
||||
---
|
||||
|
||||
## Head-to-head #1: Lore Engine vs. Microsoft GraphRAG
|
||||
|
||||
### The setup
|
||||
|
||||
Both systems use a knowledge graph to ground LLM reasoning. Both have ingestion pipelines. Both expose query APIs. They diverge sharply on what the query API looks like and what the data model is.
|
||||
|
||||
### What the Lore Engine does better
|
||||
|
||||
1. **Temporal model.** This is the single biggest differentiator. GraphRAG has no concept of "was X true at time T." The Lore Engine's `time_in_window` UDF, era-tree membership, and `current` token resolution are not in GraphRAG at any level. For a fictional world, this is a non-negotiable feature.
|
||||
|
||||
2. **Closed-world enforcement.** GraphRAG extracts whatever the LLM finds from the corpus. If the corpus has three people named "Aldric," the graph has three Aldric nodes and no way to know they're the same. The Lore Engine's `Person.name` uniqueness constraint, the alias system, the `lore_verified` flag, and the `loadKnownEntities` injection prevent this.
|
||||
|
||||
3. **Source attribution.** GraphRAG generates community summaries from chunks; the summaries lose the source provenance. The Lore Engine's `cite` tool always traces back to a specific `LoreSource` and chunk. The Lore Engine can answer *"according to which document?"* GraphRAG cannot.
|
||||
|
||||
4. **Consistency engine.** GraphRAG has no contradiction detection. The Lore Engine's 4-category consistency engine (Contradiction, Anachronism, Orphan, OntologyViolation) surfaces conflicts before the LLM hallucinates around them.
|
||||
|
||||
5. **Extensibility via TypeTemplate.** GraphRAG's graph is whatever the LLM extracted. The Lore Engine's `TypeTemplate` system lets the world-builder define new domain types as YAML. The 30+ tools GraphRAG doesn't have (`state_at`, `event_chain`, `list_lineage`, etc.) come from the typed ontology.
|
||||
|
||||
6. **NPC knowledge scoping.** The Lore Engine's `query_as_npc` tool (inherited from GraphMCP-Example) scopes the LLM's knowledge to what a specific character has witnessed. GraphRAG has nothing equivalent.
|
||||
|
||||
### What GraphRAG does better
|
||||
|
||||
1. **Global summarization.** GraphRAG's pre-generated community summaries are a genuine, validated contribution. *"What are the main themes in this world?"* is a question the Lore Engine cannot answer well today. The closest is `state_at(world_root, current)`, which only works if the world is fully modeled.
|
||||
|
||||
2. **Production maturity.** GraphRAG has 33,779 stars, a Microsoft Research blog post, a real community, and a v3.1.0 release. The Lore Engine has zero users.
|
||||
|
||||
3. **Polyglot storage.** GraphRAG supports multiple storage backends out of the box. The Lore Engine v1.1 plans 5 stores but only Neo4j is set up at design time.
|
||||
|
||||
4. **Multimodal.** GraphRAG's RAG-Anything integration handles images, tables, equations natively. The Lore Engine's S3-attachment model is rudimentary.
|
||||
|
||||
5. **Tooling.** GraphRAG has a WebUI, a CLI, prompt-tuning guides, and documentation at microsoft.github.io/graphrag. The Lore Engine has a design repo.
|
||||
|
||||
6. **Faster iteration on the user-facing API.** GraphRAG has shipped 3 major versions in 2 years. The Lore Engine is a single design.
|
||||
|
||||
7. **Existing user base.** Microsoft has customers. The Lore Engine has one (potential) world-builder.
|
||||
|
||||
### What neither does well
|
||||
|
||||
Both systems inherit the LLM's tendency to hallucinate. Both can be wrong about facts. Both are limited by the quality of the LLM. Both require the user to be skeptical of the output.
|
||||
|
||||
### Net assessment
|
||||
|
||||
**The Lore Engine solves a *different problem* from GraphRAG.** GraphRAG is for global sensemaking over private documents. The Lore Engine is for historical reasoning about a fictional world. The two could coexist; in fact, GraphRAG's community-detection idea could be borrowed for the Lore Engine's "what is this world about?" question (a v2 feature).
|
||||
|
||||
If the goal is to *answer questions about a fictional world with historical accuracy*, the Lore Engine is the right shape and GraphRAG is the wrong tool. If the goal is to *summarize a private document collection*, GraphRAG is the right tool and the Lore Engine is overkill.
|
||||
|
||||
The Lore Engine is *not* a better GraphRAG. It's a different system that addresses problems GraphRAG doesn't.
|
||||
|
||||
---
|
||||
|
||||
## Head-to-head #2: Lore Engine vs. Cognee
|
||||
|
||||
### The setup
|
||||
|
||||
This is the more dangerous comparison. Cognee is the closest functional comparable: a self-hosted, MIT-licensed, knowledge-graph backend for LLM reasoning, with an agent-native API, real production deployment, and a published paper.
|
||||
|
||||
### What the Lore Engine does better
|
||||
|
||||
1. **Domain specificity.** Cognee is generic; the Lore Engine is purpose-built for fictional worlds. For the specific question *"was House Vyr allied with the Crimson Pact in 340 TA?"*, the Lore Engine has a one-Cypher-call answer. Cognee has an LLM-extraction pipeline that may or may not produce the right entity, with no temporal model to filter by 340 TA.
|
||||
|
||||
2. **Temporal model.** As above. Cognee has no time. The Lore Engine's `time_in_window` UDF is a primitive the Lore Engine leans on heavily.
|
||||
|
||||
3. **Closed-world enforcement.** Cognee's "cognitive-science-grounded ontology" is generic (ACT-R, SOAR-inspired). It doesn't have a `Person` node with a `lifespan` property. It doesn't have a `Faction` node with a `founded` property. The Lore Engine's ontology is *specific* to worlds, and the consistency rules are *specific* to world-history problems.
|
||||
|
||||
4. **NPC knowledge scoping.** The Lore Engine has `query_as_npc` and the v1.1 NPC-tier-gated access control. Cognee has nothing equivalent. Cognee doesn't model *who knows what* at all.
|
||||
|
||||
5. **Source attribution.** The Lore Engine's `sources[]` array and `lore_verified` flag on every node are more rigorous than Cognee's paragraph-level citations.
|
||||
|
||||
6. **Multi-domain extensibility.** The Lore Engine's TypeTemplate system lets a world-builder add a thieves-guild mission type without code. Cognee's "cognitive ontology" is fixed at the library level.
|
||||
|
||||
7. **Cost.** The Lore Engine's structured YAML ingestion is free (no LLM). Cognee uses an LLM at every pipeline stage.
|
||||
|
||||
### What Cognee does better
|
||||
|
||||
1. **Production maturity.** 17,843 stars, paying customers (Cognee Cloud), real Discord, real integrations (Claude Code plugin, OpenClaw plugin). The Lore Engine has zero users.
|
||||
|
||||
2. **LLM-provider-agnostic.** Cognee works with OpenAI, Anthropic, Ollama, anything. The Lore Engine inherits GraphMCP-Example's LiteLLM integration, which is good but not as polished as Cognee's.
|
||||
|
||||
3. **Agent-native API.** `await cognee.remember(...)` and `await cognee.recall(...)` are beautiful Python async APIs. The Lore Engine's MCP-tool API is HTTP/JSON-RPC. Both are fine; Cognee's is more ergonomic for direct agent use.
|
||||
|
||||
4. **Multimodal.** Cognee ingests audio, images, video, code, anything. The Lore Engine's v1.1 plans multimodal via S3 attachments, but the implementation is not done.
|
||||
|
||||
5. **The Cognee paper benchmarks against standard QA.** The paper evaluates on HotPotQA, TwoWikiMultiHop, MuSiQue. The Lore Engine has no published benchmarks. When the Lore Engine is built, the *first thing* it should do is run on these benchmarks to get comparable numbers.
|
||||
|
||||
6. **Community and ecosystem.** Cognee has integrations, plugins, a hosted offering, a Discord. The Lore Engine has a Gitea repo.
|
||||
|
||||
7. **Operational tooling.** Cognee has a CLI, a UI, observability hooks, multi-tenant isolation. The Lore Engine's v1 has none of this.
|
||||
|
||||
### What neither does well
|
||||
|
||||
Both systems struggle with knowledge that *changes* over time. Cognee has no time. The Lore Engine's time model works for the data we have, but it can't predict what *will* happen. Neither system can answer *"what events are likely in the next 50 years?"* — that's a forecasting problem, not a retrieval problem.
|
||||
|
||||
Both systems are heavily dependent on the LLM's quality. If the LLM is bad at reading tables or bad at distinguishing speakers, both systems inherit that.
|
||||
|
||||
### Net assessment
|
||||
|
||||
**If the question is "should I use Cognee or the Lore Engine for my game-world reasoning?"** the answer is: *if the Lore Engine isn't built, use Cognee. If the Lore Engine is built, use the Lore Engine.*
|
||||
|
||||
Cognee is the better *foundation* for a knowledge-graph backend. The Lore Engine is the better *domain layer* for fictional-world reasoning. The right move in v2 is almost certainly to *build the Lore Engine on top of Cognee*: use Cognee's storage abstraction and agent-native API, and implement the Lore Engine's temporal model, ontology, and consistency engine as a Cognee extension.
|
||||
|
||||
This is a serious recommendation, not a hedge. **Cognee + Lore Engine > Lore Engine standalone**, for almost any real-world deployment. The Lore Engine as designed inherits from GraphMCP-Example because GraphMCP-Example is what Kay already has. If starting from scratch, Cognee would be the substrate.
|
||||
|
||||
### The honest take
|
||||
|
||||
Cognee was published in May 2025. The Lore Engine design started June 2026. The Lore Engine is *catching up* to where Cognee already is, on the dimensions of polish, community, and production-readiness. Where the Lore Engine is *ahead* is on domain specificity (fictional worlds, temporal model, NPC scoping) — but those are the things Cognee could add. The Lore Engine's moat is *not* the architecture; it's the *domain knowledge* baked into the ontology and the consistency rules. Cognee could replicate the temporal model in 6 months of focused work. What Cognee can't replicate overnight is the world-builder's specific lore.
|
||||
|
||||
The Lore Engine's strategic bet: **the world is the asset, the engine is the substrate.** A world-builder with 1000 pages of lore and the Lore Engine's MCP tools has something Cognee can't give them out of the box: a system that *knows* fictional-world ontology. Cognee gives them a system that knows *cognitive ontology*. For this specific problem, the fictional-world one is the right shape.
|
||||
|
||||
---
|
||||
|
||||
## Head-to-head #3: Lore Engine vs. Stanford Generative Agents
|
||||
|
||||
### The setup
|
||||
|
||||
Generative Agents is a research paper, not a system you can deploy. But the *idea* — memory stream + reflection + LLM-driven behavior — is what makes NPCs believable. The Lore Engine is the substrate; generative agents are the *behavior layer*.
|
||||
|
||||
### What the Lore Engine does better
|
||||
|
||||
1. **Knowledge representation.** Typed graph vs. memory stream. The Lore Engine can answer "who is Aldric's father?" with a single Cypher query. Generative agents would need the LLM to read the memory stream, find the relevant memory, and reason about it. The Lore Engine is faster, more reliable, and more auditable.
|
||||
|
||||
2. **Temporal precision.** `time_in_window` UDF vs. recency + LLM inference. The Lore Engine can answer "what was Aldric's relationship to Theron in 300 TA?" Generative agents can't separate "what Aldric remembers" from "what is true."
|
||||
|
||||
3. **Scales to large worlds.** Generative agents hit a wall at ~25 agents and a few days of memory. The Lore Engine scales to thousands of entities and centuries of history.
|
||||
|
||||
4. **Multi-agent support without combinatorial memory streams.** With 25 agents, each having a memory stream, the cross-agent memory is a real problem. The Lore Engine's shared Neo4j graph is the cross-agent state.
|
||||
|
||||
5. **Consistency checking.** Generative agents have no consistency engine. Reflections can be wrong. The Lore Engine flags anachronisms.
|
||||
|
||||
### What generative agents do better
|
||||
|
||||
1. **Believability.** This is the killer feature. The 25-agents demo is visceral. The Lore Engine's `query_as_npc` is the substrate but not the implementation. To make NPCs *behave* believably, you need the generative agents pattern. **The Lore Engine currently lacks the reflection mechanism.**
|
||||
|
||||
2. **Simplicity.** Generative agents is a few hundred lines of Python. The Lore Engine is a 14-doc design with a multi-service architecture. You can ship a generative-agents demo in a weekend; you can ship the Lore Engine in 19+ days.
|
||||
|
||||
3. **Planning and behavior synthesis.** Generative agents produce *plans* — sequences of actions the agent will take. The Lore Engine produces *facts* — what is true about the world. The behavior layer is missing from the Lore Engine.
|
||||
|
||||
4. **The "magic" of emergent behavior.** Generative agents produce emergent social dynamics from simple rules. The Lore Engine produces no emergent behavior; it's a database.
|
||||
|
||||
5. **Proven in the literature.** 4,000+ citations on the original paper. Hundreds of follow-ups. The patterns are well-understood.
|
||||
|
||||
### What neither does well
|
||||
|
||||
Neither handles the *world-generation* problem. Generative agents assume a world is given (the Smallville sandbox). The Lore Engine assumes a world is given (the YAML ingestion). IVIE generates worlds.
|
||||
|
||||
Neither has a strong story on **prophecy, deception, unreliable narration**. Generative agents' reflections are about *what the agent believes*, but the system doesn't model *what is true in the world separately from what the agent believes*. The Lore Engine's `lore_verified` flag is a crude version of this.
|
||||
|
||||
### Net assessment
|
||||
|
||||
The Lore Engine is the *substrate*; generative agents are the *behavior layer*. They are not in competition. **The Lore Engine should explicitly borrow the reflection mechanism from generative agents.** Add a v2 tool `reflect_on(entity, period)` that summarizes an entity's recent activity into higher-level observations, the way generative agents synthesize reflections from memory streams. This is a high-value, low-cost addition.
|
||||
|
||||
The Lore Engine's claim is not that it does what generative agents do. The claim is that it provides the *ground truth* that generative agents' memory streams lack. An NPC's memory stream is a *projection* of the world. The world-graph is the *source*. Generative agents + Lore Engine = an NPC that has a memory stream backed by a verified world-graph. The two together are stronger than either alone.
|
||||
|
||||
This is the most actionable cross-pollination in the whole related-work survey. **Build the reflection layer next.**
|
||||
|
||||
---
|
||||
|
||||
## Where the Lore Engine is *worse* — a direct list
|
||||
|
||||
This is the critical-thinking section. Where the design choices hurt the Lore Engine, and what we should do about it.
|
||||
|
||||
### 1. Maturity gap (worst weakness)
|
||||
|
||||
The Lore Engine is a design. GraphRAG, Cognee, and LightRAG are production systems with paying users, real communities, and years of iteration. **The Lore Engine is months of catch-up away from being useful**, and any user trying to deploy it today will find bugs, missing features, and rough edges. A user trying to deploy Cognee today will find a polished experience with documentation, integrations, and a Discord.
|
||||
|
||||
**Mitigation:** the Lore Engine's v2 plan is to *integrate with Cognee* rather than compete. If the Lore Engine is built as a Cognee extension, it inherits Cognee's production maturity. This is the single most important strategic move available.
|
||||
|
||||
### 2. Smaller corpus of validation
|
||||
|
||||
GraphRAG has Microsoft's research blog posts validating the approach on real datasets. Cognee has benchmarks (HotPotQA, TwoWikiMultiHop, MuSiQue). WikiChat has 97.3% factual accuracy. The Lore Engine has *zero validation*. The 5 question types from `07-reasoning-harness.md` are unproven.
|
||||
|
||||
**Mitigation:** Phase 10 of the v1 roadmap (see `09-roadmap.md`) is the reasoning-harness validation. The first build of the Lore Engine should run on a small hand-crafted world with 50 known facts, 10 known contradictions, 5 known anachronisms, and measure: how often does the LLM answer correctly? how often does it surface the contradiction? how often does it hallucinate? If those numbers aren't good, the design has a bug.
|
||||
|
||||
### 3. No global summarization
|
||||
|
||||
This is the worst *technical* gap. *"What is the shape of this world? What are the main themes?"* — the Lore Engine can't answer. GraphRAG's pre-generated community summaries are the right primitive and we don't have it.
|
||||
|
||||
**Mitigation:** borrow the community-detection + hierarchical-summarization pattern from GraphRAG. The Lore Engine can run Leiden on its Neo4j graph and pre-generate summaries; this is a v2 feature.
|
||||
|
||||
### 4. No forecasting
|
||||
|
||||
*"What events are likely to happen in the next 50 years of the Third Age?"* is a question the Lore Engine can't answer. The TKG methods (TLogic, Chain of History, TGL-LLM) can.
|
||||
|
||||
**Mitigation:** the Lore Engine is a *retrieval* system, not a *prediction* system. Adding forecasting changes the scope. v2 might add a `forecast_event` tool that uses a small TKG model; v1 doesn't need it.
|
||||
|
||||
### 5. No multimodal
|
||||
|
||||
The Lore Engine handles text and (via S3) attachments. GraphRAG/RAG-Anything handles images, tables, equations, audio. Cognee handles audio, video, code. The Lore Engine is text-only at the engine layer.
|
||||
|
||||
**Mitigation:** multimodal at the engine layer requires image/video understanding, which is a different research area. The Lore Engine can *store* multimodal content; it can't *reason* over it. A v2 might integrate a vision-language model for image-grounded questions.
|
||||
|
||||
### 6. Single-language extraction prompt
|
||||
|
||||
The lore-extractor's prompt is English-only. A Japanese or Spanish world-builder gets English entity names. The Lore Engine's `name` uniqueness constraint is locale-sensitive.
|
||||
|
||||
**Mitigation:** the extractor can be parameterized by language. The Cypher MERGE keys are still strings; the `aliases[]` field can hold multiple-script variants. v2.
|
||||
|
||||
### 7. No multi-tenant support
|
||||
|
||||
The Lore Engine is single-tenant. If a homelab wanted to host two worlds (one for each player's campaign), they'd need two separate stacks. Cognee supports multi-tenancy.
|
||||
|
||||
**Mitigation:** the `world_id` namespace is the right primitive. Multi-tenant would just be multiple `world_id` values in the same Neo4j, with row-level access control. v2.
|
||||
|
||||
### 8. No user-facing UI
|
||||
|
||||
Cognee has a CLI + WebUI. GraphRAG has a CLI + WebUI. LightRAG has a CLI + WebUI. The Lore Engine has... a design. A world-builder can ingest YAML, but they can't see the graph, browse contradictions, or review anachronisms without using Cypher.
|
||||
|
||||
**Mitigation:** a UI is in the v2 roadmap. v1 is CLI + curl. The world-builder who can write YAML can also run `cypher-shell`.
|
||||
|
||||
### 9. Weaker fallback for ambiguous queries
|
||||
|
||||
When the LLM's question is ambiguous, the Lore Engine's `lookup` tool returns a disambiguation list. GraphRAG just returns the top-K chunks. Cognee returns whatever the cognitive-ontology traversal finds. **The Lore Engine's behavior is better in principle but slower in practice** — every ambiguous query pays the cost of a round trip.
|
||||
|
||||
**Mitigation:** cache `lookup` results in Redis. The active context (per-session working set) is the right place to store resolved entities.
|
||||
|
||||
### 10. The 30-tool surface is high
|
||||
|
||||
We hit this in `10-critique.md`. The 30+ tools are at the LLM's tool-use ceiling. GraphRAG exposes fewer tools and lets the LLM reason with them. Cognee exposes 4 operations (`remember`, `recall`, `forget`, `improve`) and lets the cognitive-ontology routing decide.
|
||||
|
||||
**Mitigation:** Phase 10 will measure tool-selection accuracy with all 30 vs. collapsed to ~15. If collapsed is better, we collapse.
|
||||
|
||||
---
|
||||
|
||||
## Where the Lore Engine is *better* — a direct list
|
||||
|
||||
For balance.
|
||||
|
||||
### 1. Closed-world enforcement for fictional domains
|
||||
|
||||
This is the genuine, defensible advantage. No other system treats the world as a *closed, typed, time-bounded, contradiction-checked* graph. The Lore Engine's ontology is *specific* to the problem. The closest analog (Cognee's cognitive ontology) is generic.
|
||||
|
||||
### 2. First-class temporal model
|
||||
|
||||
The `time_in_window` UDF, era-tree membership, `current` token, and time-bounded edge properties are a real primitive that no other system has. For historical reasoning, this is non-negotiable.
|
||||
|
||||
### 3. Source attribution with confidence scoring
|
||||
|
||||
The `sources[]` array, `lore_verified` flag, `source_confidence` float, and `cite` tool together are more rigorous than anything in GraphRAG/Cognee/LightRAG. The Lore Engine can say "this claim has 3 sources, the most recent is from 2024, the confidence is 0.92." No other system does this.
|
||||
|
||||
### 4. Consistency engine
|
||||
|
||||
The 4-category consistency engine (Contradiction, Anachronism, Orphan, OntologyViolation) is unique. It surfaces problems *before* the LLM hallucinates around them. No other system does this at the engine level.
|
||||
|
||||
### 5. NPC knowledge scoping + tier-gated access control
|
||||
|
||||
The `query_as_npc` tool plus the v1.1 `npc_knowledge` template field is a real primitive for narrative games. The black-market example in `14-examples.md` shows it works for access control at the data layer. No other system has this.
|
||||
|
||||
### 6. TypeTemplate polymorphism
|
||||
|
||||
Adding a new domain type (thieves-guild missions, war campaigns) is a YAML exercise. No other system has this. GraphRAG/Cognee/LightRAG all require code changes for new entity types.
|
||||
|
||||
### 7. Multi-world/planar support
|
||||
|
||||
The `world_id` namespace and `Plane` label are the right primitives for multi-world campaigns. Most narrative games are single-world, but the high-fantasy genre is often multi-planar. The Lore Engine supports it; the others don't.
|
||||
|
||||
### 8. Organic bootstrap with structural-data surfacing
|
||||
|
||||
The `:Orphan` node is a primitive that says *"this entity is missing structural data — the chronicles are silent on it."* No other system surfaces this. The world-builder grows the world over time and the engine tells them what's incomplete.
|
||||
|
||||
### 9. Retcon preservation
|
||||
|
||||
The `retcon` Postgres table preserves history. No other system does this. Edits overwrite; retcons don't.
|
||||
|
||||
### 10. Time as a service, not a feature
|
||||
|
||||
The Lore Engine's design treats temporal reasoning as a *primitive* the rest of the system is built on. GraphRAG/Cognee/LightRAG treat time as a *property* of fields, not a system-wide invariant. The difference shows up in every consistency check.
|
||||
|
||||
---
|
||||
|
||||
## Debatable design decisions
|
||||
|
||||
These are choices I made in the design that I think are right, but could be wrong. Calling them out so Kay can push back.
|
||||
|
||||
### D1. The polymorphic `DomainEntity` wrapper is the right escape hatch, but it might be over-engineered
|
||||
|
||||
The `DomainEntity` + `Relation` + `TypeTemplate` system is 200 lines of Cypher plus a template-watcher service. It solves the "thieves-guild missions" problem. But:
|
||||
|
||||
- The world-builder has to learn YAML.
|
||||
- The template validation is non-trivial.
|
||||
- The dynamic tool generator adds complexity to the gateway.
|
||||
|
||||
**Alternative:** a simpler model where the world-builder writes a Go plugin that registers entity types. More code, less YAML. The trade-off is *world-builder complexity* vs *engine simplicity*.
|
||||
|
||||
**My read:** YAML is the right choice. The world-builder who can write a `family_tree.yaml` can write a `mission.yaml`. The YAML approach scales because the engine doesn't grow.
|
||||
|
||||
### D2. Five stores is a lot
|
||||
|
||||
The v1.1 storage strategy is Neo4j + Postgres + pgvector + Redis + S3. That's five services. On a homelab with 58GB RAM, it's fine. On a Raspberry Pi, it's not.
|
||||
|
||||
**Alternative:** collapse Postgres and pgvector (use the same Postgres instance), drop MinIO (use local filesystem for v1), and the storage layer is 3 services.
|
||||
|
||||
**My read:** the five-store split is the right *target architecture* and the v1 can ship with 3. The v1.1 plan in `09-roadmap.md` already says "MinIO can be deferred" and "pgvector can replace Neo4j's vector index."
|
||||
|
||||
### D3. The 30-tool surface is high
|
||||
|
||||
We've discussed this. 30 tools is at the empirical LLM ceiling.
|
||||
|
||||
**Alternative:** collapse `state_at` into `entity_context(comprehensive=true)`. Collapse `summarize_chain` into `narrate_arc(style=...)`. The number drops to ~20.
|
||||
|
||||
**My read:** start big, measure, collapse. The reasoning harness groups tools by function; if the LLM uses 8 of them 95% of the time, the long tail should be deprecated, not built.
|
||||
|
||||
### D4. The consistency engine might over-flag
|
||||
|
||||
We've discussed this. A world with rich, overlapping history will have many temporal overlaps that aren't real contradictions.
|
||||
|
||||
**Alternative:** default severity to `warn`, and only surface `error` for things the world-builder has explicitly marked as "must be consistent" (e.g., lineage).
|
||||
|
||||
**My read:** the design already has the right knobs (`severity`, `disable_rules[]`, `confidence_threshold`). The question is whether the world-builder uses them. Phase 7's test corpus needs to include "noisy" worlds to validate the defaults.
|
||||
|
||||
### D5. The structured YAML path is a heavy lift for the world-builder
|
||||
|
||||
Writing a `family_tree.yaml` is real work. The world-builder has to learn the schema, follow the strict format, and update it as the world evolves.
|
||||
|
||||
**Alternative:** use a *generative* world-builder tool that takes a markdown chapter and proposes a YAML diff, which the world-builder approves. The LLM in the loop makes structured authoring easier.
|
||||
|
||||
**My read:** this is a v2 tool. v1 is "world-builder writes YAML." The LLM-assisted world-builder is high-leverage but needs a working v1 to be useful.
|
||||
|
||||
### D6. The temporal model's `current` token is a global mutable
|
||||
|
||||
We've discussed this. The `:Now` config node is a single point of failure and a synchronization problem in multi-user scenarios.
|
||||
|
||||
**Alternative:** every session carries a `world_time` override. The `:Now` is the default but sessions can pin a different time. The active context (per-session) is the right place for this.
|
||||
|
||||
**My read:** the design has been updated in v1.1 to include `world_time` in the active context. The `:Now` is the fallback, not the only source of truth. **This fix is in scope of Phase 2.**
|
||||
|
||||
---
|
||||
|
||||
## The strategic question
|
||||
|
||||
Is the Lore Engine worth building?
|
||||
|
||||
**Strong case for:** the closest functional comparables (Cognee, LightRAG, GraphRAG) are all generic and lack a temporal model. The closest in-spirit comparable (Stanford Generative Agents) lacks a knowledge graph. The closest by use case (IVIE) generates worlds, doesn't reason over them. **No system in the literature does what the Lore Engine claims to do.** That's an opening.
|
||||
|
||||
**Strong case against:** GraphRAG has 33,779 stars and Microsoft Research. Cognee has 17,843 stars and a paying customer base. LightRAG has 36,622 stars and an HKU team. **The Lore Engine is competing with funded, shipping, polished systems for the same homelab resources.** The "niche" of closed-world fictional reasoning is real, but the niche may not be big enough to justify the 43-day build cost when Cognee (with a fictional-world ontology extension) could close 80% of the gap in 2 weeks.
|
||||
|
||||
**My recommendation:**
|
||||
|
||||
1. **Build the Lore Engine on top of Cognee, not on top of GraphMCP-Example.** The GraphMCP-Example substrate is a Kay-personal-project path. The Cognee substrate is a community-supported, MIT-licensed, production-grade path. The Lore Engine's *value* is the domain layer, not the substrate.
|
||||
|
||||
2. **Build the v1.1 polymorphic extension model first, not the v1 time model.** The polymorphic extension is the highest-leverage single feature; the time model is important but more incremental. If the Lore Engine has to ship one thing in v1, ship the TypeTemplate system. The time model can be a v1.1 follow-up.
|
||||
|
||||
3. **Validate before building more.** Build the MVP, ingest one small hand-crafted world, measure: can the LLM answer historical questions correctly? Does the consistency engine surface real problems? Does the TypeTemplate system work? **If the answer to any of these is "no," the design has a bug and the v2 should address it.** Don't build the v1.1 features on top of an unvalidated v1.
|
||||
|
||||
4. **The most leveraged next move** is a 1-week spike that builds the minimum-viable Lore Engine on Cognee and validates the core idea. If that spike succeeds, the 43-day plan makes sense. If it doesn't, the design needs a rethink before more code is written.
|
||||
|
||||
The critical-thinking section of the related work is the part I most want Kay to push back on. The comparison is honest; the strategic call is debatable. The right next move depends on what the spike shows.
|
||||
|
||||
---
|
||||
|
||||
## The summary
|
||||
|
||||
The Lore Engine is **a niche specialization of a solved problem** (KG-RAG) with **two genuinely novel primitives** (temporal model, TypeTemplate polymorphism) and **one strong domain story** (fictional-world reasoning). It is **miles behind** the leading systems in polish, community, and production maturity. It is **miles ahead** in domain specificity, source attribution, and consistency enforcement. It is **catching up to a moving target** (Cognee is iterating fast) and the right move might be to integrate rather than compete.
|
||||
|
||||
The honest recommendation: build the spike, measure, decide.
|
||||
Reference in New Issue
Block a user