docs: v1.1 — extensibility, multi-store, microservice decomposition
New docs (1623 lines): - 11-extensibility.md: polymorphic DomainEntity + TypeTemplate model. Adding a new domain (thieves-guild missions, war campaigns, etc.) is a YAML exercise, not a Go change. ~200 lines of Cypher, stable forever. - 12-storage-strategy.md: split into 5 stores (Neo4j, Postgres, pgvector, Redis, S3/MinIO). Each store holds what it's good at. Cross-store queries compose at the handler layer. Sagas handle multi-store write consistency. - 13-microservice-decomposition.md: split mcp-server's 1144-line main.go into a small gateway + pluggable handlers. Generic tool-handler protocol (GET /tools, POST /invoke). Macro/micro iteration at three speeds. - 14-examples.md: three end-to-end worked examples showing the v1.1 architecture in action. Thieves-guild mission (DomainEntity + Postgres), war-campaign tracker (DomainEntity + Postgres event log), black-market economy (NPC-tier-gated data + Postgres trade log). Updates to v1 docs: - 01-ontology: adds Plane, NPC, PC, Human labels (per Kay's Q2, Q3). Adds DomainEntity, Relation, TypeTemplate labels for v1.1. - 02-time-model: year-level precision is the default (Kay's Q1). - 09-roadmap: v1.1 phases (11-14), 20-day scope, recommended build order. - 10-critique: open questions all resolved; 4 new S1/S2 risks recorded (closed-world ontology, single-binary iteration, polymorphic overhead, cross-store consistency). Polymorphic extension added to 'good at' list. Design now v1.1: 14 docs, 4293 lines, ~280KB.
This commit is contained in:
Hermes Agent
@@ -4,6 +4,8 @@ A Neo4j-backed world ontology and modular MCP tool surface that lets an LLM reas
|
||||
|
||||
Built on top of the existing [GraphMCP-Example](https://git.homelab.local/kaykayyali/GraphMCP-Example) stack (Neo4j + Redis Streams + Go MCP server), with extensions for the things a *world* needs that a *message archive* does not.
|
||||
|
||||
**v1.1 (current):** adds polymorphic `DomainEntity` extension model, multi-store storage strategy, and a microservice decomposition plan. See `11-extensibility.md`, `12-storage-strategy.md`, `13-microservice-decomposition.md`, `14-examples.md`.
|
||||
|
||||
---
|
||||
|
||||
## Read in this order
|
||||
@@ -21,6 +23,10 @@ Built on top of the existing [GraphMCP-Example](https://git.homelab.local/kaykay
|
||||
| 08 | [Architecture](docs/08-architecture.md) | System diagram, data flow, service layout |
|
||||
| 09 | [Roadmap](docs/09-roadmap.md) | Phased build plan — MVP first, then layers |
|
||||
| 10 | [Critique](docs/10-critique.md) | Self-pressure-test: what could break, where this could fail |
|
||||
| 11 | [Extensibility](docs/11-extensibility.md) | **v1.1** — polymorphic `DomainEntity` + `TypeTemplate` model. New domains as YAML, not code. |
|
||||
| 12 | [Storage Strategy](docs/12-storage-strategy.md) | **v1.1** — which data goes in Neo4j, Postgres, pgvector, Redis, S3. Why and when. |
|
||||
| 13 | [Microservice Decomposition](docs/13-microservice-decomposition.md) | **v1.1** — split the mcp-server monolith. Macro/micro iteration speeds. |
|
||||
| 14 | [Worked Examples](docs/14-examples.md) | **v1.1** — three end-to-end examples: thieves-guild missions, war campaigns, black-market economy. |
|
||||
|
||||
## The 30-second pitch
|
||||
|
||||
|
||||
@@ -39,6 +39,45 @@ The world is modeled as a typed, temporal, source-attributed property graph. Thi
|
||||
| `Deity` | A named god, demigod, or divine being. | `name`, `domain[]`, `alignment`, `symbol` | `Aelar the Patient` |
|
||||
| `Spell` | A named magical effect. Has a `MagicSystem` parent (see lvl 1). | `name`, `level`, `school` | `Emberlance` |
|
||||
| `Material` | A specific substance with lore significance (orichalcum, soulglass, etc.). | `name`, `rarity` | `Soulglass` |
|
||||
| `Plane` (v1.1) | A named plane of existence, separate world, or parallel realm. Per Kay's answer (Q2), the engine supports multi-world/planar structures via a `world_id` namespace on every node. | `name`, `parent_plane` | `The Material Plane`, `The Fade` |
|
||||
|
||||
### New: Player-vs-NPC separation (v1.1)
|
||||
|
||||
Per Kay's answer (Q3), NPCs and PCs (and the humans behind PCs) are tracked separately.
|
||||
|
||||
| Label | Purpose | Key properties |
|
||||
|---|---|---|
|
||||
| `NPC` | An in-fiction character controlled by the engine / world-builder. Wraps or references a `Person` node. | `id`, `person_id` (Person), `controlled_by` (always "engine" or "world_builder"), `personality_summary` (free text), `voice_id` (TTS voice, optional) |
|
||||
| `PC` | An in-fiction character controlled by a human player. | `id`, `person_id` (Person), `controlled_by` (always "human"), `player_id` (Human, see below) |
|
||||
| `Human` | A real human, the player behind a PC. Tracks the player separately from the in-fiction character. Optional — only created if the world-builder wants to associate the two. | `id`, `name`, `discord_id?`, `email?`, `joined_at`, `notes` (world-builder's notes about the human, e.g. "prefers intrigue plots") |
|
||||
|
||||
The relationships:
|
||||
|
||||
```
|
||||
(:Human) -[:PLAYS]-> (:PC) -[:IS_PERSON]-> (:Person)
|
||||
(:NPC) -[:IS_PERSON]-> (:Person)
|
||||
```
|
||||
|
||||
This means:
|
||||
|
||||
- The in-fiction `Person` node is the canonical entity. NPCs and PCs both reference it.
|
||||
- NPC tier scoping (existing v1) works on the `Person` node. `query_as_npc` resolves to the `Person` via the `NPC` wrapper.
|
||||
- The human player (with their preferences, history, etc.) is a separate node. The world-builder can ask "what does this human player like?" without conflating it with "what is the in-fiction character like?"
|
||||
- For purely-NPC worlds (no PCs), no `Human` or `PC` nodes are created. The `NPC` and `Person` nodes are sufficient.
|
||||
|
||||
This is the answer to "associate the true human and some details about them where possible." The world-builder can attach as much or as little to the `Human` node as they want, and the in-fiction layer stays clean.
|
||||
|
||||
### New: Polymorphic domain entities (v1.1)
|
||||
|
||||
Per the extensibility question, the engine adds one new node label and one new edge label as a polymorphic substrate for *anything* the world-builder wants to model that isn't covered by the core types.
|
||||
|
||||
| Label | Purpose | Key properties |
|
||||
|---|---|---|
|
||||
| `DomainEntity` | A node representing a domain-specific concept (thieves-guild mission, war campaign, trade lot, ritual, spellbook, etc.). Defined by a `TypeTemplate`. | `id`, `type`, `name`, `world_id`, `properties: map<string, any>`, `template_id`, `sources[]`, `lore_verified` |
|
||||
| `Relation` | An edge between any two nodes (DomainEntity, Person, Faction, etc.) with typed properties. | `id`, `from_id`, `to_id`, `type`, `properties: map<string, any>`, `valid_from`, `valid_until`, `sources[]` |
|
||||
| `TypeTemplate` | The schema for a domain type. Stored as data, hot-reloadable. | `id`, `version`, `spec` (parsed YAML), `status` (active/draft/deprecated) |
|
||||
|
||||
The full design is in `11-extensibility.md`. The short version: a YAML file in `./templates/` defines a new domain type, the engine registers it, the world-builder writes instance YAMLs, and the LLM gets new tools automatically. No Go code change.
|
||||
|
||||
### New: Structural & temporal (lvl 1 — how things relate through time)
|
||||
|
||||
@@ -182,7 +221,7 @@ For the full Cypher that creates the schema, see `08-architecture.md#schema-boot
|
||||
|
||||
## What the ontology deliberately leaves out
|
||||
|
||||
- **Multi-world / planar.** Planes and parallel worlds get a `Plane` label, but only if the world actually has them. Adding labels for hypothetical concepts is a trap — it makes the schema look more complete than it is and encourages the LLM to invent.
|
||||
- **Multi-world / planar.** Planes and parallel worlds get a `Plane` label (added in v1.1, per Kay's Q2), and every node carries a `world_id` namespace. The engine supports them; the world-builder uses them only if the world actually has them.
|
||||
- **Abstractions like "the Will of the People" or "Destiny."** These are not nodes unless the world has a document that treats them as entities. Otherwise the LLM can claim them and the engine has nothing to verify against.
|
||||
- **In-world physics rules.** "Can a fireball ignite a dragon?" is not in this schema. It's a LLM-narrative question. The engine tells you what the world's books say about fireballs and dragons; it does not compute the outcome.
|
||||
|
||||
|
||||
@@ -23,7 +23,7 @@ All time references in the engine, on every edge, in every property, in every to
|
||||
Where:
|
||||
|
||||
- `era` is the slug of an `Era` node (e.g. `3rd_age`, `age_of_iron`, `long_winter`).
|
||||
- `unit` is the most specific time unit the source supports. Examples: `year_340`, `month_3`, `day_17`, `event_42` (referencing a `Date` node by ID).
|
||||
- `unit` is the most specific time unit the source supports. **Per Kay's answer (Q1), the engine supports year-level precision by default, with optional month/day/event ID when the source supports it.** Examples: `year_340`, `month_3`, `day_17`, `event_42` (referencing a `Date` node by ID).
|
||||
|
||||
**A bare `{era}` is valid** when no finer granularity is known — it means "at some point during this era."
|
||||
|
||||
@@ -201,7 +201,7 @@ All five are documented in detail with signatures in `05-mcp-tools.md`.
|
||||
|
||||
## Known limitations (and why we accept them)
|
||||
|
||||
- **A fact that was true for centuries cannot be precisely queried by century.** We only know the era. This is fine: most lore is at era granularity, and over-precision usually means fabrication.
|
||||
A fact that was true for centuries cannot be precisely queried by century — only at the era granularity we have. This is fine: most lore is at era granularity, and over-precision usually means fabrication. Year-level precision is supported for the cases where the source supports it (coronations, battles, deaths, births), and the engine prefers the most-specific-time-the-source-supports rather than guessing.
|
||||
- **Two sources giving the same era but different years produce a contradiction, not a synthesis.** The engine flags it; it does not pick a winner. See `04-consistency.md`.
|
||||
- **"Before the world began" or "after the end of time" are not representable.** A `null` `valid_from` with no parent `Era` is the closest, and we mark those as `mythic: true` to be clear they are cosmological claims, not historical ones.
|
||||
|
||||
|
||||
@@ -188,3 +188,88 @@ The **non-cuttable core is Phases 0–4 + Phase 7**. Schema, UDFs, time-aware to
|
||||
- **Do not skip the UDF unit tests.** Every time-aware query depends on `time_in_window` and `time_windows_overlap`. If they have a bug, every consistency check is wrong. Test first, then trust.
|
||||
- **Do not over-invest in prose extraction.** It's the path of least resistance because the LLM does the work. It's also the path with the highest error rate for the highest-stakes data. Structured YAML is the win.
|
||||
- **Do not try to support 100% of question types in the first build.** Ship the 5 patterns from `07-reasoning-harness.md` and iterate. The LLM is forgiving of missing tools if the existing ones are reliable.
|
||||
|
||||
---
|
||||
|
||||
## v1.1 phases (extensibility, storage, decomposition)
|
||||
|
||||
After the v1 MVP (Phases 0–4 of the original) is built and validated, v1.1 adds four phases that address the modularization question.
|
||||
|
||||
### Phase 11: Microservice decomposition (mechanical, ~5 days)
|
||||
|
||||
A pure refactor of the v1 single-binary `mcp-server` into a small gateway + per-handler files. No new functionality; just the file structure for Phase 12 to extend.
|
||||
|
||||
- [ ] Split `mcp-server/main.go` (1144 lines) into `main.go` + `handlers/{core,lineage,event,consistency,generation,worldbuilder}.go`.
|
||||
- [ ] Add an in-process tool registry. The gateway iterates the registry to build `tools/list`.
|
||||
- [ ] Add the HTTP-based tool handler protocol (`GET /tools`, `POST /invoke`) with a stub HTTP handler.
|
||||
- [ ] Add a `/healthz` aggregator that surfaces per-handler health.
|
||||
|
||||
**Verify:** all 30 v1 tools still work, `tools/list` returns the same list, and the gateway has < 600 lines of code.
|
||||
|
||||
### Phase 12: Polymorphic extension model (~7 days)
|
||||
|
||||
The big one. Adds the `DomainEntity`, `Relation`, and `TypeTemplate` labels, the template-watcher service, and the dynamic tool generator.
|
||||
|
||||
- [ ] Add the new labels and indexes from `11-extensibility.md#layer-2-the-domainentity-wrapper` to `neo4j-init.cypher`.
|
||||
- [ ] Build `services/template-watcher/`: watches `./templates/`, validates YAML, registers templates.
|
||||
- [ ] Build `services/template-registry/`: persists template specs in Postgres, hot-reloadable.
|
||||
- [ ] Implement the dynamic tool generator: a single generic handler that runs queries generated from `TypeTemplate` specs.
|
||||
- [ ] Add the `list_template_tools` MCP tool.
|
||||
- [ ] Ship the four example templates from `14-examples.md` (thieves-guild mission, war campaign, black-market lot, NPC secret knowledge).
|
||||
- [ ] Update the reasoning harness to mention template tools.
|
||||
|
||||
**Verify:** write `templates/thieves_guild/mission.yaml`, hit `POST /admin/templates/reload`, see 6 new tools in `tools/list`, ingest a mission, query it via `list_missions`, get a coherent answer. **No Go code change between "template added" and "tool available."**
|
||||
|
||||
### Phase 13: Multi-store storage (~5 days)
|
||||
|
||||
Adds Postgres, pgvector, MinIO, and the cross-store compose layer.
|
||||
|
||||
- [ ] Add Postgres, pgvector, and MinIO to docker-compose.
|
||||
- [ ] Implement the Postgres schema from `12-storage-strategy.md#schema-overview`.
|
||||
- [ ] Implement the cross-store compose layer in the relevant handlers.
|
||||
- [ ] Migrate `lore_chunk` and `domain_entity_summary` embeddings to pgvector.
|
||||
- [ ] Move lore source text to MinIO; keep metadata in Neo4j.
|
||||
- [ ] Add the saga pattern for multi-store writes (mission_log + DomainEntity + S3 attachment).
|
||||
|
||||
**Verify:** ingest a war campaign, query `campaign_strength` against Postgres event log, get a real-time answer that includes both Neo4j and Postgres data. Verify the saga rolls back cleanly on simulated mid-saga failure.
|
||||
|
||||
### Phase 14: External handler protocol (~3 days)
|
||||
|
||||
The HTTP-based plugin protocol, exercised with one external handler.
|
||||
|
||||
- [ ] Add auth + rate-limiting to the gateway's HTTP-handler dispatch.
|
||||
- [ ] Build one external handler as a proof — a Python `entity-linker` that uses a transformer model for ambiguous entity resolution.
|
||||
- [ ] Add a `register_external_handler` admin tool.
|
||||
|
||||
**Verify:** start the Python entity-linker, register it with the gateway, send an ambiguous `lookup` query, see the Python handler called, get a resolved entity.
|
||||
|
||||
### v1.1 total scope: ~20 days
|
||||
|
||||
v1.1 is similar in scope to v1 (Phases 0–4 = 19 days). It's the *next* MVP — the one that adds extensibility, multi-store, and decomposition. Both v1 and v1.1 are buildable; both produce a working system.
|
||||
|
||||
### v1.1 milestones (added to the Gitea repo)
|
||||
|
||||
| Milestone | Phase | Days |
|
||||
|---|---|---|
|
||||
| M7 — Gateway Refactor | Phase 11 | 5 |
|
||||
| M8 — Polymorphic Extensions | Phase 12 | 7 |
|
||||
| M9 — Multi-Store Storage | Phase 13 | 5 |
|
||||
| M10 — External Handlers | Phase 14 | 3 |
|
||||
|
||||
### What to cut from v1.1 if you're under time pressure
|
||||
|
||||
In strict order:
|
||||
|
||||
1. Phase 14 (External Handlers) — defer to a v1.2. The internal handler decomposition from Phase 11 is enough for the world-builder's needs.
|
||||
2. The saga pattern (Phase 13) — start with best-effort multi-store writes, no rollback. Add the saga when you actually see data corruption in production.
|
||||
3. MinIO — start with local filesystem for lore source text. Migrate to MinIO when you actually have a use case for cross-host blob storage.
|
||||
|
||||
### The recommended order: v1 → validate → v1.1
|
||||
|
||||
1. Build v1 Phases 0–4 (19 days). Validate: the time model works, structured ingestion is fast, the LLM can answer time-bounded questions correctly.
|
||||
2. Build v1.1 Phase 11 (5 days, mechanical). The mcp-server is now decomposable.
|
||||
3. Build v1.1 Phase 12 (7 days, polymorphic extensions). This is the phase that unlocks the "arbitrary new concept" question. **Ship this second because it's the highest-leverage single change after v1's data layer.**
|
||||
4. Build v1.1 Phase 13 (5 days, multi-store). The system now scales to the world's actual size.
|
||||
5. Build v1.1 Phase 14 (3 days, external handlers). Defer this if you can.
|
||||
|
||||
The full v1 + v1.1 is ~43 days. The MVP is 19. The extensibility is 26 (5 + 7 + 5 + 3 + Phases 5–10 from v1 = 11 days). Build the smallest thing that works; iterate from there.
|
||||
|
||||
@@ -148,13 +148,22 @@ A world-builder changes the lore. The engine must absorb the change without brea
|
||||
|
||||
These are decisions I couldn't make alone. The world-builder should answer them before Phase 1.
|
||||
|
||||
1. **How granular is the time model in practice?** Year-level precision, or just era? This affects the UDF implementation and the storage cost of the time-window index.
|
||||
2. **Are there multi-world / planar structures?** If so, we need a `Plane` label and a `world_id` namespace.
|
||||
3. **How are NPCs and PC players modeled?** Are players (the actual humans) tracked in the graph, or are only their in-fiction characters tracked? This affects the NPC-tier scoping logic.
|
||||
4. **What's the policy on retconning?** If the world-builder changes an established fact, do we mark the old edge as `retconned` (preserving history) or just `MERGE` over it (losing history)? The former is more honest; the latter is simpler.
|
||||
5. **How is the world bootstrapped?** Does the world-builder pre-define all eras and their boundaries, or do they emerge from ingested events? The former is more rigorous; the latter is more organic.
|
||||
6. **What's the confidence weighting formula?** When two sources disagree, the YAML wins by default. But what if both are prose? Prose from a 2nd-edition printing wins over a 1st-edition printing? Need a clear rule.
|
||||
7. **Are contradiction nodes user-facing?** Should the world-builder be able to *resolve* a contradiction (marking it as "this is the canonical version, the other source is wrong"), or do they stay flagged forever? The former requires a write tool; the latter is read-only.
|
||||
1. ~~**How granular is the time model in practice?**~~ **Resolved (Q1):** year-level precision is the default, with optional month/day/event precision when the source supports it. The UDF and the storage cost are unchanged.
|
||||
2. ~~**Are there multi-world / planar structures?**~~ **Resolved (Q2):** yes. The engine adds a `world_id` namespace and a `Plane` label. Multi-world queries are supported via `world_id` filters; planar relationships via `Plane` and `EXISTS_IN` edges.
|
||||
3. ~~**How are NPCs and PC players modeled?**~~ **Resolved (Q3):** separately. The `NPC`, `PC`, and `Human` labels in `01-ontology.md` cover this. The in-fiction `Person` is canonical; the wrappers track who controls it.
|
||||
4. ~~**What's the policy on retconning?**~~ **Resolved (Q4):** preserve history by default. Old edges/nodes are marked `retconned` with a snapshot in the `retcon` Postgres table (`12-storage-strategy.md#postgres-schema`). Explicit `DELETE` is the only way to remove something permanently.
|
||||
5. ~~**How is the world bootstrapped?**~~ **Resolved (Q5):** organically over a long period. The engine supports partial worlds (some eras defined, some not), and the consistency engine surfaces missing structural data as `:Orphan` nodes. No need to pre-define everything.
|
||||
6. ~~**What's the confidence weighting formula?**~~ **Resolved (Q6):** more recent source wins. The `source_uploaded_at` (or `source_published_at` when known) is the tiebreaker. The engine stores both. When two prose sources disagree and both are recent, the rule engine surfaces the contradiction; it does not pick a winner automatically.
|
||||
7. ~~**Are contradiction nodes user-facing?**~~ **Resolved (Q7):** the local engine is read-only for contradictions — the world-builder reviews them in a queue. An external source *may* be authorized to resolve contradictions later (e.g. a community lore-council with write access). The local engine never auto-resolves.
|
||||
|
||||
## Resolved-by-Kay decisions in v1.1
|
||||
|
||||
All 7 open questions are now resolved and reflected in:
|
||||
|
||||
- `01-ontology.md` — adds `Plane`, `NPC`, `PC`, `Human` labels
|
||||
- `02-time-model.md` — year-level precision is the default
|
||||
- `12-storage-strategy.md` — `retcon` Postgres table for retcon history
|
||||
- `09-roadmap.md` — Phase 0 (pre-flight) now includes resolving these
|
||||
|
||||
## What this design is good at
|
||||
|
||||
@@ -165,6 +174,7 @@ For balance:
|
||||
- **Structured ingestion.** The YAML path makes high-stakes data (lineage, era boundaries, faction rules) exact, not fuzzy.
|
||||
- **Modular tools.** Each tool does one job. Higher-level patterns are compositions, not mega-tools.
|
||||
- **Consistency surfacing.** The engine reports what it doesn't know as loudly as what it does.
|
||||
- **Polymorphic extension.** v1.1's `DomainEntity` + `TypeTemplate` model lets the world-builder add new domain types (thieves-guild missions, war campaigns, black markets) without code changes.
|
||||
|
||||
## What this design is not good at (yet)
|
||||
|
||||
@@ -172,11 +182,47 @@ For balance:
|
||||
- **Prophecy, deception, unreliable narration.** v1 doesn't model these as first-class.
|
||||
- **Forcing the LLM to behave.** The reasoning harness is a contract, not an enforcement mechanism.
|
||||
- **User experience for world-builders.** v1 is CLI + YAML. UI is a v2.
|
||||
- **Multi-world campaigns.** v1 is per-world.
|
||||
- **Versioning and retcon handling.** v1 mutates in place.
|
||||
- **Versioning and retcon handling at the v1 level.** v1 mutates in place; v1.1's `retcon` table preserves history but the in-graph nodes still get `MERGE`'d. A v2 might use temporal versioning on the graph itself.
|
||||
- **Auto-resolution of cross-source conflicts.** v1.1 surfaces them; the world-builder resolves.
|
||||
|
||||
## v1.1 critique additions
|
||||
|
||||
After the v1 review, the modularization question surfaced four new design risks worth recording.
|
||||
|
||||
### S1.4 (NEW, blocker) — Closed-world ontology ceiling
|
||||
|
||||
The v1 ontology has 14 hard-coded labels. A thieves-guild mission is forced into `Event`, a war campaign is forced into `Faction`-with-properties, a black-market trade log is forced into `Item`-with-properties. The LLM can *talk* about these things, but the engine can't *reason* over their structure.
|
||||
|
||||
**Fix:** v1.1 introduces the `DomainEntity` polymorphic wrapper + `TypeTemplate` data-defined schemas. See `11-extensibility.md`. This is the load-bearing change for "arbitrary new concept, define how it associates with larger constructs, but also have flexibility to get as detailed as we need."
|
||||
|
||||
**Status:** resolved in v1.1 design. Implementation is Phase 3 of v1.1 roadmap.
|
||||
|
||||
### S1.5 (NEW, blocker) — Single mcp-server binary blocks iteration
|
||||
|
||||
The existing GraphMCP-Example `mcp-server/main.go` is a 1144-line single file. Adding a new tool means editing main.go, recompiling, redeploying. The iteration loop for a world that's going to grow indefinitely is the cost of the entire program.
|
||||
|
||||
**Fix:** v1.1 decomposes the mcp-server into a small gateway + a set of pluggable handlers, with a tool-registry and a template-driven tool generator. See `13-microservice-decomposition.md`.
|
||||
|
||||
**Status:** resolved in v1.1 design. Implementation is Phase 1-3 of v1.1 (mechanical refactor, no new functionality).
|
||||
|
||||
### S2.5 (NEW, design risk) — The polymorphic wrapper adds query complexity
|
||||
|
||||
Every `DomainEntity` query is now polymorphic — the engine has to look up the template, get the field names, build the right query. The performance overhead is small for typed queries, but for `expand_context` and `graph_traverse`, the engine has to follow relations through the `Relation` label and re-resolve the template for each step.
|
||||
|
||||
**Fix:** the engine caches `TypeTemplate` lookups in Redis. The first time a template is referenced, its spec is loaded; subsequent queries use the cached version. Cache invalidation is on template reload (hot-reload event).
|
||||
|
||||
**Status:** acknowledged, fix designed, implementation in Phase 2 of v1.1.
|
||||
|
||||
### S2.6 (NEW, design risk) — Cross-store consistency is genuinely hard
|
||||
|
||||
When the world-builder writes a new mission, we touch Neo4j (entity, relations), Postgres (mission_log row), and S3 (any attachments). These three writes are not atomic. A partial failure leaves the world in an inconsistent state.
|
||||
|
||||
**Fix:** the saga pattern from `12-storage-strategy.md#the-cost-of-cross-store-transactions`. Each multi-store write is a saga; the engine records saga state in Postgres and rolls back partial failures.
|
||||
|
||||
**Status:** designed, implementation in Phase 4 of v1.1.
|
||||
|
||||
## Conclusion
|
||||
|
||||
The design is **viable for v1**, with a clear scope of 19 days for the MVP (Phases 0–4). The biggest risks are scale (entity resolution), over-flagging (consistency engine), and LLM misbehavior (harness enforcement). Each has a documented mitigation. The design is *intentionally a floor* — every doc ends with notes on what v2 should add, not what v1 should be stretched to cover.
|
||||
The design is **viable for v1**, with a clear scope of 19 days for the MVP (Phases 0–4), and **v1.1** extends it with the polymorphic extension model, the multi-store storage strategy, and the microservice decomposition. The 7 open questions are resolved. The biggest remaining risks are scale (entity resolution), over-flagging (consistency engine), cross-store consistency (sagas), and LLM misbehavior (harness enforcement). Each has a documented mitigation.
|
||||
|
||||
I would build this. The question is whether to build Phase 1–4 first and validate, or to scope-creep into the consistency engine (Phase 7) on the first sprint. Recommendation: **don't.** The MVP is the proof of concept. The consistency engine is the differentiator. Build them in order.
|
||||
I would build v1 first (Phases 0–4 of the original roadmap), validate the time model and structured ingestion, *then* layer v1.1 on top. The polymorphic extension model is the right shape for the world's growth, but it's the second thing to build, not the first.
|
||||
|
||||
397
docs/11-extensibility.md
Normal file
397
docs/11-extensibility.md
Normal file
@@ -0,0 +1,397 @@
|
||||
# 11 — Extensibility: Polymorphic Type Templates
|
||||
|
||||
The v1 ontology has 14 hard-coded labels. That's *fine for the first world* but it has a ceiling: a thieves-guild mission is forced into `:Event`, a war campaign is forced into `:Faction`-with-properties, a black-market trade log is forced into `:Item`-with-properties. The LLM can *talk* about these things, but the engine can't *reason* over their structure.
|
||||
|
||||
**"What missions has the Crimson Hand run in Mardsville over the last year, sorted by payout?"** is unanswerable today. The data lives in `summary` text fields, the relationships are implicit in prose, and the LLM has to reconstruct it via `semantic_search` and hope.
|
||||
|
||||
This document is the v1.1 fix: a **type-template system** that lets the world-builder define new domain types — missions, campaigns, trade lots, spellbooks, rituals, whatever — as *data*, not code. The core ontology stays small. The world grows without the engine growing.
|
||||
|
||||
## The principle: types are data, not schema
|
||||
|
||||
Today: a new domain type means a Go change, a Cypher change, a Neo4j migration, a docker rebuild.
|
||||
|
||||
Goal: a new domain type means a YAML file. The engine reads it. The LLM gets new tools automatically. No code.
|
||||
|
||||
This is the same pattern as WordPress custom post types, Salesforce custom objects, or Notion's database templates: the *platform* is fixed, the *content schema* is open.
|
||||
|
||||
## The architecture: 4 layers
|
||||
|
||||
```
|
||||
┌──────────────────────────────────────────────────────────────────┐
|
||||
│ LAYER 1: Core Ontology (fixed, in code) │
|
||||
│ ────────────────────────────────────── │
|
||||
│ Person, Faction, Location, Item, Era, Date, Lineage, │
|
||||
│ Culture, Deity, Language, MagicSystem, Title, Region, Material │
|
||||
│ + the time model + the consistency engine │
|
||||
│ + the MCP transport + the active context │
|
||||
│ This is ~3000 lines of Go and ~500 lines of Cypher. Stable. │
|
||||
└──────────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
│ every node is also a...
|
||||
▼
|
||||
┌──────────────────────────────────────────────────────────────────┐
|
||||
│ LAYER 2: Polymorphic "Domain Entity" wrapper │
|
||||
│ ────────────────────────────────────── │
|
||||
│ (:DomainEntity { │
|
||||
│ id, type, name, era_id, world_id, │
|
||||
│ properties: map<string, any>, ← free-form, type-checked │
|
||||
│ template_id: "thieves_guild_mission", │
|
||||
│ sources[], lore_verified, source_confidence │
|
||||
│ }) │
|
||||
│ Plus: (:Relation {from, to, type, properties, ...}) for │
|
||||
│ arbitrary edges between DomainEntities. │
|
||||
│ This is ~200 lines of Cypher. Stable. │
|
||||
└──────────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
│ defined as data in...
|
||||
▼
|
||||
┌──────────────────────────────────────────────────────────────────┐
|
||||
│ LAYER 3: Type Templates (data, hot-reloadable) │
|
||||
│ ────────────────────────────────────────── │
|
||||
│ YAML files in ./templates/ defining: │
|
||||
│ - what fields a domain type has (and their types) │
|
||||
│ - what relations it can have (and to which other types) │
|
||||
│ - which MCP tools auto-generate for it │
|
||||
│ - which ontology rules apply │
|
||||
│ - which LLM-inference hints (e.g. "treat this as a secret") │
|
||||
│ This is the part that grows. │
|
||||
└──────────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
│ instances of...
|
||||
▼
|
||||
┌──────────────────────────────────────────────────────────────────┐
|
||||
│ LAYER 4: Domain Instances (the world's actual data) │
|
||||
│ ────────────────────────────────────────────── │
|
||||
│ Mission #4471 ("The Pale Ledger Heist") │
|
||||
│ properties: {target: "merchant_guild_vault", payout: 500gp, │
|
||||
│ status: "completed", risk: "high"} │
|
||||
│ relations: GIVEN_BY → NPC (Vex the Silent) │
|
||||
│ LOCATED_IN → Location (Mardsville) │
|
||||
│ TARGETS → Person (Guildmaster Torren) │
|
||||
│ LOGGED_IN → TradeLog (lot #9821) │
|
||||
└──────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
The world-builder writes Layer 4 (instances) and Layer 3 (templates). The engine ships Layer 1 and Layer 2. The MCP tools and consistency rules auto-generate from the templates.
|
||||
|
||||
## Layer 1: Core ontology (unchanged from v1)
|
||||
|
||||
The 14 labels stay. They cover the *macro* structure of any world: who exists, where they are, when they lived, what they belonged to. This is stable.
|
||||
|
||||
## Layer 2: The `DomainEntity` wrapper
|
||||
|
||||
A single new node label and a single new edge label. This is the polymorphic backbone.
|
||||
|
||||
```cypher
|
||||
// A domain entity — anything that the world-builder wants to model
|
||||
// that isn't already covered by a core label.
|
||||
(:DomainEntity {
|
||||
id: "mission_4471",
|
||||
type: "ThievesGuildMission", // matches a :TypeTemplate.name
|
||||
name: "The Pale Ledger Heist",
|
||||
world_id: "arda_1st_age",
|
||||
|
||||
// Polymorphic, type-checked properties from the template
|
||||
properties: {
|
||||
target: "merchant_guild_vault",
|
||||
payout_gp: 500,
|
||||
status: "completed",
|
||||
risk: "high",
|
||||
secrecy: "guild_internal"
|
||||
},
|
||||
|
||||
// The world-builder can add a free-text summary for embedding
|
||||
summary: "Vex hired a crew to steal the ledger of merchant debts...",
|
||||
|
||||
// Standard provenance
|
||||
sources: ["crimson_hand_records.yaml"],
|
||||
lore_verified: true,
|
||||
source_confidence: 0.95,
|
||||
created_at: datetime(),
|
||||
updated_at: datetime()
|
||||
})
|
||||
|
||||
// A relation between any two nodes (DomainEntity, Person, Faction, etc.)
|
||||
(:Relation {
|
||||
id: "rel_abc",
|
||||
from_id: "mission_4471",
|
||||
to_id: "person_vex_silent",
|
||||
type: "GIVEN_BY", // matches a template's allowed_relations
|
||||
properties: { // typed by template
|
||||
at: "3rd_age.year_385",
|
||||
agreed_payout: 500
|
||||
},
|
||||
valid_from: "3rd_age.year_385",
|
||||
valid_until: null,
|
||||
sources: ["crimson_hand_records.yaml"],
|
||||
source_confidence: 0.9
|
||||
})
|
||||
```
|
||||
|
||||
Two new labels. ~200 lines of Cypher. Stable forever.
|
||||
|
||||
### Indexes for Layer 2
|
||||
|
||||
```cypher
|
||||
CREATE CONSTRAINT domain_entity_id IF NOT EXISTS
|
||||
FOR (d:DomainEntity) REQUIRE d.id IS UNIQUE;
|
||||
|
||||
CREATE CONSTRAINT relation_id IF NOT EXISTS
|
||||
FOR (r:Relation) REQUIRE r.id IS UNIQUE;
|
||||
|
||||
CREATE INDEX domain_entity_type IF NOT EXISTS
|
||||
FOR (d:DomainEntity) ON (d.type);
|
||||
|
||||
CREATE INDEX domain_entity_world IF NOT EXISTS
|
||||
FOR (d:DomainEntity) ON (d.world_id);
|
||||
|
||||
CREATE INDEX relation_type IF NOT EXISTS
|
||||
FOR (r:Relation) ON (r.type);
|
||||
|
||||
// For time-bounded relation queries (same pattern as core ontology)
|
||||
CREATE INDEX relation_time_window IF NOT EXISTS
|
||||
FOR (r:Relation) ON (r.valid_from, r.valid_until);
|
||||
```
|
||||
|
||||
## Layer 3: Type templates (the part that grows)
|
||||
|
||||
A type template is a YAML file in `./templates/<domain>/<type>.yaml` that defines:
|
||||
|
||||
- What fields the type has (name, type, required, indexed, secret).
|
||||
- What relations it can have (to which other types, with what properties).
|
||||
- What MCP tools the engine should auto-generate for it.
|
||||
- What ontology rules apply to it.
|
||||
- How the LLM should reason about it.
|
||||
|
||||
### Example: `thieves_guild_mission.yaml`
|
||||
|
||||
```yaml
|
||||
# This file defines the ThievesGuildMission domain type.
|
||||
# Drop it in ./templates/thieves_guild/ and the engine hot-reloads.
|
||||
|
||||
template:
|
||||
id: "thieves_guild_mission"
|
||||
version: "1.0"
|
||||
owner: "kaykayyali" # who maintains this template
|
||||
description: "A mission run by a thieves guild. Has a quest-giver, targets, payout, and outcome."
|
||||
|
||||
# Polymorphic node definition
|
||||
entity:
|
||||
label: "DomainEntity" # always DomainEntity in v1.1
|
||||
type_value: "ThievesGuildMission"
|
||||
|
||||
# Discriminator field — used by the LLM and the rule engine to
|
||||
# distinguish this from other domain types in the same world
|
||||
discriminator: "thieves_guild_mission"
|
||||
|
||||
properties:
|
||||
- { name: "mission_code", type: "string", required: true, unique: true }
|
||||
- { name: "target", type: "string", required: true, indexed: true }
|
||||
- { name: "payout_gp", type: "int", required: true, indexed: true }
|
||||
- { name: "status", type: "enum", values: ["planned", "active", "completed", "failed", "cancelled"] }
|
||||
- { name: "risk", type: "enum", values: ["low", "medium", "high", "extreme"] }
|
||||
- { name: "secrecy", type: "enum", values: ["public", "faction_internal", "guild_internal", "inner_circle_only"] }
|
||||
- { name: "target_location",type: "string", required: false, indexed: true }
|
||||
- { name: "started_at", type: "time_ref" } # canonical time string
|
||||
- { name: "completed_at", type: "time_ref" }
|
||||
- { name: "summary", type: "text", embedded: true } # embedded for semantic search
|
||||
|
||||
# Time model: this entity type has a temporal existence window
|
||||
temporal: true
|
||||
# valid_from defaults to started_at; valid_until defaults to completed_at + 30 days
|
||||
|
||||
# Allowed relations — typed, can reference core or domain types
|
||||
relations:
|
||||
- { name: "GIVEN_BY", to_type: "Person", cardinality: "exactly_one" }
|
||||
- { name: "GIVEN_BY", to_type: "NPC", cardinality: "exactly_one" } # an NPC wrapper
|
||||
- { name: "TARGETS", to_type: ["Person", "Faction", "Location", "DomainEntity"], cardinality: "many" }
|
||||
- { name: "PAID_BY", to_type: "Person", cardinality: "many" }
|
||||
- { name: "PAID_TO", to_type: "Person", cardinality: "many" }
|
||||
- { name: "LOGGED_IN", to_type: "DomainEntity", to_discriminator: "trade_log_entry", cardinality: "exactly_one" }
|
||||
- { name: "PART_OF", to_type: "DomainEntity", to_discriminator: "thieves_guild_campaign", cardinality: "many" }
|
||||
- { name: "WITNESSED_BY", to_type: "Person", cardinality: "many" }
|
||||
|
||||
# Default scoping: which NPCs can know about this entity
|
||||
# Uses the existing NPC tier system
|
||||
npc_knowledge:
|
||||
default_tier: "outer_circle" # only NPCs at outer_circle or closer can know
|
||||
field_overrides:
|
||||
- { field: "payout_gp", min_tier: "inner_circle" }
|
||||
- { field: "secrecy", min_tier: "inner_circle" }
|
||||
|
||||
# Auto-generated MCP tools for this type
|
||||
tools:
|
||||
- { name: "list_missions", params: ["filter_by", "sort_by", "limit"] }
|
||||
- { name: "get_mission", params: ["mission_code"] }
|
||||
- { name: "missions_by_target",params: ["target", "limit"] }
|
||||
- { name: "missions_in_era", params: ["era", "limit"] }
|
||||
- { name: "log_mission", params: ["mission_code", "properties", "relations"], write: true }
|
||||
# Tools are generated by the engine from this spec. The LLM sees them in tools/list.
|
||||
|
||||
# Ontology rules that should run for this type
|
||||
rules:
|
||||
- "no-anachronism-participation" # references the core rules
|
||||
- "secrecy-honors-npc-tier" # template-local rule
|
||||
- "payout-must-be-positive" # template-local rule
|
||||
|
||||
# LLM-inference hints — fed to the reasoning harness
|
||||
llm_hints:
|
||||
- "When a Mission has secrecy=inner_circle_only, the LLM MUST NOT reveal its
|
||||
details to an NPC with tier < inner_circle. Use the npc_knowledge field_overrides."
|
||||
- "When asked 'what is the Crimson Hand doing?', list_missions is a strong
|
||||
starting point. Sort by recency unless the user specifies otherwise."
|
||||
|
||||
# UI hints (for future world-builder UI)
|
||||
ui:
|
||||
icon: "🗡️"
|
||||
color: "#8b0000"
|
||||
list_columns: ["mission_code", "target", "status", "payout_gp", "completed_at"]
|
||||
detail_sections:
|
||||
- { title: "Mission", fields: ["mission_code", "target", "status", "risk"] }
|
||||
- { title: "People", fields_via_relations: ["GIVEN_BY", "TARGETS", "PAID_TO"] }
|
||||
- { title: "Payout", fields: ["payout_gp", "PAID_BY"] }
|
||||
- { title: "Timeline", fields: ["started_at", "completed_at"], relations: ["PART_OF"] }
|
||||
```
|
||||
|
||||
That's a single YAML file. No code. The engine:
|
||||
|
||||
1. Validates the schema on load.
|
||||
2. Creates the `(:TypeTemplate {id: "thieves_guild_mission", ...})` node.
|
||||
3. Registers the auto-generated MCP tools in `tools/list`.
|
||||
4. Wires the rules into the consistency engine.
|
||||
5. Re-indexes the polymorphic indexes.
|
||||
|
||||
Hot reload — no engine restart.
|
||||
|
||||
### What the engine does with a template
|
||||
|
||||
```
|
||||
On ./templates/thieves_guild/mission.yaml file change:
|
||||
1. YAML parser → schema validation
|
||||
2. Cypher write: MERGE (t:TypeTemplate {id: "thieves_guild_mission"}) SET t.<spec>
|
||||
3. Tool registry: for each entry in spec.tools, register a tool handler
|
||||
(the handler is a generic "domain tool runner" — same code, different params)
|
||||
4. Rule registry: for each entry in spec.rules, attach to consistency engine
|
||||
5. Hot-reload notification: send_message to subscribed LLM clients
|
||||
"Domain type thieves_guild_mission registered; 6 new tools available"
|
||||
```
|
||||
|
||||
**The MCP server is now a generic tool runner. It doesn't know about thieves guilds, war campaigns, or black markets. It knows about TypeTemplate nodes and dispatches generically.**
|
||||
|
||||
This is the architectural change that unlocks everything else.
|
||||
|
||||
## Layer 4: Domain instances (the world's data)
|
||||
|
||||
A world-builder writes a YAML file with concrete missions. The schema is enforced by the template.
|
||||
|
||||
```yaml
|
||||
# crimson_hand_missions.yaml
|
||||
# Ingested via POST /ingest/structured?type=thieves_guild_mission
|
||||
|
||||
template: "thieves_guild_mission"
|
||||
world_id: "arda_1st_age"
|
||||
instances:
|
||||
- mission_code: "M-4471"
|
||||
name: "The Pale Ledger Heist"
|
||||
target: "merchant_guild_vault"
|
||||
payout_gp: 500
|
||||
status: "completed"
|
||||
risk: "high"
|
||||
secrecy: "inner_circle_only"
|
||||
target_location: "mardsville"
|
||||
started_at: "3rd_age.year_385.month_3"
|
||||
completed_at: "3rd_age.year_385.month_4"
|
||||
summary: "Vex hired a crew of four to infiltrate the merchant guild's vault
|
||||
and steal the ledger of debts. The crew succeeded, but one member
|
||||
was captured and later executed."
|
||||
relations:
|
||||
- { type: "GIVEN_BY", to: "person_vex_silent" }
|
||||
- { type: "TARGETS", to: "faction_merchant_guild" }
|
||||
- { type: "LOGGED_IN", to: "trade_log_9821" }
|
||||
- { type: "PART_OF", to: "campaign_pale_quarter" }
|
||||
|
||||
- mission_code: "M-4472"
|
||||
name: "The Whisper Network"
|
||||
target: "intelligence_network"
|
||||
payout_gp: 1200
|
||||
status: "active"
|
||||
risk: "extreme"
|
||||
secrecy: "guild_internal"
|
||||
target_location: "valdorn"
|
||||
started_at: "3rd_age.year_386.month_1"
|
||||
relations:
|
||||
- { type: "GIVEN_BY", to: "person_alessia_dusk" }
|
||||
- { type: "TARGETS", to: "faction_crimson_pact" }
|
||||
```
|
||||
|
||||
The YAML is parsed, validated against the template, and written as `(:DomainEntity)` + `(:Relation)` nodes. No LLM. No code.
|
||||
|
||||
## How the LLM uses this
|
||||
|
||||
The LLM's tool list is now **dynamic**. After the template loads, `tools/list` returns:
|
||||
|
||||
```json
|
||||
{
|
||||
"tools": [
|
||||
// ... 30 core tools from the engine
|
||||
{ "name": "list_missions", "params": ["filter_by", "sort_by", "limit"], "domain": "thieves_guild_mission" },
|
||||
{ "name": "get_mission", "params": ["mission_code"], "domain": "thieves_guild_mission" },
|
||||
// ...
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
The LLM's reasoning harness (`07-reasoning-harness.md`) is updated: *"When the question matches a domain type, prefer the domain-specific tool over the generic ones. e.g. for 'what missions has the guild run,' use `list_missions` not `semantic_search`."*
|
||||
|
||||
The LLM doesn't have to know that "thieves guild missions" is a *new* domain. It sees a tool, it uses it. The tool happens to be auto-generated.
|
||||
|
||||
## The `TypeTemplate` node (in-graph representation)
|
||||
|
||||
For the consistency engine and the reasoning harness to reason over the templates, the templates live in the graph as data:
|
||||
|
||||
```cypher
|
||||
(:TypeTemplate {
|
||||
id: "thieves_guild_mission",
|
||||
version: "1.0",
|
||||
owner: "kaykayyali",
|
||||
spec_yaml: "...",
|
||||
spec: { ... parsed JSON ... },
|
||||
status: "active", // active | draft | deprecated
|
||||
registered_at: datetime()
|
||||
})
|
||||
```
|
||||
|
||||
The rule engine can read templates to generate type-aware rules. The LLM can read templates to understand the structure of a domain it's reasoning over.
|
||||
|
||||
## Versioning templates
|
||||
|
||||
Templates have a `version` field. When a template is updated:
|
||||
|
||||
- Old version stays in the graph as `(:TypeTemplate {id: "thieves_guild_mission", version: "0.9", status: "deprecated"})`.
|
||||
- New version is `(:TypeTemplate {id: "thieves_guild_mission", version: "1.0", status: "active"})`.
|
||||
- Existing instances reference the version they were ingested under.
|
||||
- The engine can replay ingestion against a newer template (with the world-builder's review).
|
||||
|
||||
This is the **retcon-preserving** answer to Kay's Q4. Domain types evolve; instances are not silently rewritten.
|
||||
|
||||
## What this enables
|
||||
|
||||
Adding a new domain is now:
|
||||
|
||||
1. Write `templates/<domain>/<type>.yaml` (~100 lines).
|
||||
2. POST to `POST /ingest/template` (hot-reload).
|
||||
3. Start writing instance YAMLs.
|
||||
4. Done.
|
||||
|
||||
No Go. No Cypher. No docker rebuild. The LLM gets new tools automatically.
|
||||
|
||||
That's the architecture. The next two docs decompose it further:
|
||||
- `12-storage-strategy.md` — what data goes in which store
|
||||
- `13-microservice-decomposition.md` — how the mcp-server is split so the LLM can iterate at micro and macro levels
|
||||
|
||||
## What this is NOT
|
||||
|
||||
- **Not a graph database replacement.** Neo4j stays as the macro world graph. `DomainEntity` is a node label, not a different DB.
|
||||
- **Not a no-code system.** The world-builder writes YAML, but the engine is still Go. The YAML is data, not code.
|
||||
- **Not a constraint-free system.** The template *defines* the constraints. The engine enforces them at ingest time and the consistency engine enforces them at write time. The LLM sees a typed system; it just doesn't have to write the type definition.
|
||||
- **Not a workaround for bad schema design.** If a domain really is `Person` or `Faction`, model it as such. The `DomainEntity` wrapper is for things that *aren't* core types.
|
||||
318
docs/12-storage-strategy.md
Normal file
318
docs/12-storage-strategy.md
Normal file
@@ -0,0 +1,318 @@
|
||||
# 12 — Storage Strategy: Which Data Goes Where
|
||||
|
||||
The v1 design treats Neo4j as the universal substrate. For v1.1, with polymorphic domain entities, vector embeddings, time-series events, and high-volume operational logs (trade lots, mission outcomes, campaign movements), **Neo4j alone is the wrong tool for the job.** Different data has different access patterns, and forcing them all into one graph makes the graph bad at everything.
|
||||
|
||||
This document is the storage role split: which database stores which kind of data, why, and how the engine queries across them.
|
||||
|
||||
## The principle: pick the right tool for the access pattern
|
||||
|
||||
Neo4j is excellent at: relationship traversal, graph pattern matching, recursive lineage, spatial aggregation. **It is mediocre at: full-text search, large property blobs, high-volume time-series ingestion, free-form JSON querying.**
|
||||
|
||||
If we force high-volume operational data (every trade, every mission step, every army movement) into Neo4j properties, the graph bloats, indexes fragment, and queries slow down. The right move is to store each kind of data where it naturally lives, and have the engine compose across stores.
|
||||
|
||||
## The five stores
|
||||
|
||||
| Store | What it holds | Why |
|
||||
|---|---|---|
|
||||
| **Neo4j** | The world graph. People, factions, locations, lineage, era trees, type templates, domain entities with relationships, time-bounded edges, ontology rules, violation nodes. | Graph traversal is the primary access pattern. |
|
||||
| **PostgreSQL** | Operational records with structured schemas. Trade logs, mission step logs, campaign event streams, audit trails, world-builder write history, session state, MCP tool call logs. | Relational, high-volume, time-series-friendly, transactional. |
|
||||
| **Qdrant** *(or pgvector)* | Vector embeddings of `LoreChunk`, `Message`, and `DomainEntity.summary` text. | Semantic search is the primary access pattern. |
|
||||
| **Redis** | Active MCP session state, per-session `world_time`, tool-call rate-limit counters, in-flight transaction state, ephemeral caches. | Sub-millisecond, ephemeral. |
|
||||
| **S3-compatible object store** (MinIO) | Full text of lore sources, images, audio, large attachments, archival snapshots. | Blob storage, cheap, durable. |
|
||||
|
||||
Existing GraphMCP-Example has Neo4j + Redis + the LLM proxy. We add **PostgreSQL** (the big new one), **Qdrant** (or pgvector, for self-contained deployments), and **MinIO** (or any S3 bucket).
|
||||
|
||||
## What goes in Neo4j
|
||||
|
||||
The macro world graph. Anything where the LLM will say *"traverse from A"* or *"find all X related to Y"* or *"is X connected to Y?"* — that lives in Neo4j.
|
||||
|
||||
- **Core entities:** Person, Faction, Location, Item, Era, Date, Lineage, Culture, Deity, Language, MagicSystem, Title, Region, Material.
|
||||
- **Time-bounded relations** between core entities: `RULED`, `MEMBER_OF`, `LOCATED_IN`, `PARTICIPATED_IN`, `ALLIED_WITH`, `POSSESSES`, etc. Always time-bounded. Always queryable via `time_in_window`.
|
||||
- **Polymorphic domain entities** (`:DomainEntity` with a `template_id`): a thieves-guild Mission, a war Campaign, a Spellbook, a TradeLot, a Ritual. The entity *itself* and its relations to other entities (Person, Faction, Location, other DomainEntities) live in Neo4j.
|
||||
- **Type templates** (`:TypeTemplate`): the YAML-defined schemas, stored as parsed JSON for the consistency engine and LLM to query.
|
||||
- **Violation nodes** (`:Contradiction`, `:Anachronism`, `:Orphan`, `:OntologyViolation`, `:ConsistencyRun`): the consistency engine's output.
|
||||
- **Lore source metadata** (`:LoreSource`): title, source_type, author, ingested_at, version. The *text* lives in object storage; the metadata is in Neo4j.
|
||||
- **Indexes:** all property indexes from `01-ontology.md` and `08-architecture.md`. Plus `(:DomainEntity).type`, `(:DomainEntity).world_id`, `(:Relation).type`, `(:Relation).valid_from/until`.
|
||||
|
||||
**What does NOT go in Neo4j:**
|
||||
- The full text of a lore source. (Goes in S3, with a pointer in Neo4j.)
|
||||
- The full text of a domain entity's `summary` (above some length threshold). (Goes in S3; short summaries stay in Neo4j for semantic-search embedding.)
|
||||
- The step-by-step log of a mission. (Goes in Postgres; only the *aggregate outcome* lives in Neo4j as the Mission node.)
|
||||
- Vector embeddings. (Goes in Qdrant; Neo4j's vector index is OK but not great for high-volume semantic search.)
|
||||
- High-volume time-series operational data.
|
||||
|
||||
## What goes in PostgreSQL
|
||||
|
||||
Operational records that are *append-mostly*, *high-volume*, and *not primarily about relationships*.
|
||||
|
||||
The shape that Postgres handles well: rows of typed columns, indexed on time, with foreign keys back to Neo4j IDs.
|
||||
|
||||
### Schema overview
|
||||
|
||||
```sql
|
||||
-- World, version, and migration state
|
||||
CREATE TABLE world (
|
||||
id TEXT PRIMARY KEY,
|
||||
name TEXT NOT NULL,
|
||||
current_era TEXT NOT NULL, -- canonical time string
|
||||
schema_version TEXT NOT NULL,
|
||||
created_at TIMESTAMPTZ DEFAULT now()
|
||||
);
|
||||
|
||||
-- Operational event log (every meaningful state change)
|
||||
CREATE TABLE lore_event (
|
||||
id BIGSERIAL PRIMARY KEY,
|
||||
world_id TEXT REFERENCES world(id),
|
||||
event_type TEXT NOT NULL, -- 'mission_logged', 'trade_completed', 'army_moved', ...
|
||||
entity_id TEXT, -- DomainEntity.id from Neo4j
|
||||
entity_type TEXT, -- discriminator
|
||||
occurred_at TIMESTAMPTZ NOT NULL,
|
||||
in_fiction_time TEXT, -- canonical time string
|
||||
payload JSONB NOT NULL, -- type-specific structured data
|
||||
sources TEXT[],
|
||||
actor_id TEXT, -- who/what triggered this
|
||||
created_at TIMESTAMPTZ DEFAULT now()
|
||||
);
|
||||
CREATE INDEX ON lore_event (world_id, occurred_at DESC);
|
||||
CREATE INDEX ON lore_event (entity_id, occurred_at DESC);
|
||||
CREATE INDEX ON lore_event (event_type, occurred_at DESC);
|
||||
CREATE INDEX ON lore_event USING GIN (payload);
|
||||
|
||||
-- Trade log (every lot, every transaction)
|
||||
CREATE TABLE trade_log (
|
||||
id BIGSERIAL PRIMARY KEY,
|
||||
world_id TEXT REFERENCES world(id),
|
||||
lot_id TEXT NOT NULL,
|
||||
item_id TEXT, -- DomainEntity.id of the Item or Material
|
||||
buyer_id TEXT, -- Person or Faction id
|
||||
seller_id TEXT,
|
||||
quantity NUMERIC,
|
||||
unit TEXT, -- 'gp', 'soulglass_shards', etc.
|
||||
unit_price NUMERIC,
|
||||
total_price NUMERIC,
|
||||
occurred_at TIMESTAMPTZ NOT NULL,
|
||||
in_fiction_time TEXT,
|
||||
location_id TEXT, -- Location.id
|
||||
secrecy TEXT, -- 'public', 'faction_internal', ...
|
||||
payload JSONB, -- type-specific extras
|
||||
sources TEXT[]
|
||||
);
|
||||
CREATE INDEX ON trade_log (world_id, occurred_at DESC);
|
||||
CREATE INDEX ON trade_log (lot_id);
|
||||
CREATE INDEX ON trade_log (buyer_id, occurred_at DESC);
|
||||
CREATE INDEX ON trade_log (seller_id, occurred_at DESC);
|
||||
CREATE INDEX ON trade_log (item_id, occurred_at DESC);
|
||||
CREATE INDEX ON trade_log (location_id, occurred_at DESC);
|
||||
CREATE INDEX ON trade_log USING GIN (payload);
|
||||
|
||||
-- Mission step log (per-mission timeline of events)
|
||||
CREATE TABLE mission_log (
|
||||
id BIGSERIAL PRIMARY KEY,
|
||||
mission_id TEXT NOT NULL, -- DomainEntity.id
|
||||
step_no INT NOT NULL,
|
||||
step_type TEXT, -- 'planned', 'briefed', 'infiltrated', 'completed', 'botched', 'paid'
|
||||
occurred_at TIMESTAMPTZ NOT NULL,
|
||||
in_fiction_time TEXT,
|
||||
party TEXT[], -- Person ids present
|
||||
location_id TEXT,
|
||||
outcome TEXT,
|
||||
notes TEXT,
|
||||
sources TEXT[],
|
||||
UNIQUE (mission_id, step_no)
|
||||
);
|
||||
|
||||
-- War campaign movement log
|
||||
CREATE TABLE campaign_event (
|
||||
id BIGSERIAL PRIMARY KEY,
|
||||
campaign_id TEXT NOT NULL, -- DomainEntity.id of the Campaign
|
||||
event_type TEXT, -- 'army_moved', 'battle', 'siege_begun', 'siege_lifted', ...
|
||||
occurred_at TIMESTAMPTZ NOT NULL,
|
||||
in_fiction_time TEXT,
|
||||
faction_id TEXT,
|
||||
location_id TEXT,
|
||||
army_size INT,
|
||||
casualties INT,
|
||||
outcome TEXT,
|
||||
payload JSONB,
|
||||
sources TEXT[]
|
||||
);
|
||||
CREATE INDEX ON campaign_event (campaign_id, occurred_at DESC);
|
||||
CREATE INDEX ON campaign_event (faction_id, occurred_at DESC);
|
||||
CREATE INDEX ON campaign_event (location_id, occurred_at DESC);
|
||||
|
||||
-- MCP tool call log (for the consistency monitor + audit)
|
||||
CREATE TABLE tool_call (
|
||||
id BIGSERIAL PRIMARY KEY,
|
||||
session_id TEXT,
|
||||
tool_name TEXT NOT NULL,
|
||||
arguments JSONB,
|
||||
result JSONB,
|
||||
duration_ms INT,
|
||||
error TEXT,
|
||||
called_at TIMESTAMPTZ DEFAULT now()
|
||||
);
|
||||
CREATE INDEX ON tool_call (tool_name, called_at DESC);
|
||||
CREATE INDEX ON tool_call (session_id, called_at DESC);
|
||||
|
||||
-- Retcon history (Kay's Q4)
|
||||
CREATE TABLE retcon (
|
||||
id BIGSERIAL PRIMARY KEY,
|
||||
world_id TEXT REFERENCES world(id),
|
||||
target_kind TEXT, -- 'entity' | 'relation' | 'property'
|
||||
target_id TEXT NOT NULL,
|
||||
before JSONB, -- snapshot of what was there
|
||||
after JSONB, -- what it was changed to
|
||||
reason TEXT,
|
||||
actor_id TEXT, -- world-builder id
|
||||
retconned_at TIMESTAMPTZ DEFAULT now(),
|
||||
sources TEXT[]
|
||||
);
|
||||
CREATE INDEX ON retcon (target_id, retconned_at DESC);
|
||||
|
||||
-- NPC dialogue history (for NPC knowledge scoping)
|
||||
CREATE TABLE dialogue_log (
|
||||
id BIGSERIAL PRIMARY KEY,
|
||||
world_id TEXT REFERENCES world(id),
|
||||
npc_id TEXT NOT NULL, -- Person.id
|
||||
session_id TEXT,
|
||||
message TEXT NOT NULL,
|
||||
in_fiction_time TEXT,
|
||||
occurred_at TIMESTAMPTZ DEFAULT now()
|
||||
);
|
||||
CREATE INDEX ON dialogue_log (npc_id, occurred_at DESC);
|
||||
```
|
||||
|
||||
These tables are **the operational backbone**. They're what gets high-volume writes, transactional integrity, and time-series queries.
|
||||
|
||||
## What goes in Qdrant (or pgvector)
|
||||
|
||||
Vector embeddings for semantic search. Three collections:
|
||||
|
||||
| Collection | Source | Dimension | Use |
|
||||
|---|---|---|---|
|
||||
| `lore_chunks` | `(:LoreChunk).text` | 768 | Semantic search over lore documents. |
|
||||
| `messages` | `(:Message).content` | 768 | Semantic search over dialogue. |
|
||||
| `domain_summaries` | `(:DomainEntity).summary` (if present) | 768 | Semantic search over domain entities. |
|
||||
|
||||
The first two inherit from GraphMCP-Example. The third is new and only populated for domain entities that opt in (the `embedded: true` field in the template).
|
||||
|
||||
**Qdrant vs pgvector:** Qdrant is faster and has better filtering. pgvector is simpler (one less service to run) and stays in the same DB family. For self-hosted homelab deployments where minimizing moving parts matters, **pgvector is the right call** for v1.1. We can swap to Qdrant later without changing the engine's API.
|
||||
|
||||
The lore-engine-Example already uses Neo4j's vector index. We can keep that for `lore_chunks` and add pgvector for `domain_summaries`, or migrate everything to pgvector. **Decision: pgvector for everything in v1.1.** Neo4j's vector index is fine but pgvector keeps our vector storage in one place.
|
||||
|
||||
## What goes in Redis
|
||||
|
||||
Ephemeral state. Lost on restart, not backed up.
|
||||
|
||||
- **Active session context** (replaces the in-process `sessionRegistry` in the existing MCP server). Each MCP session gets a Redis key with the active entity context, world_time override, and tool-call budget.
|
||||
- **Tool-call rate limits** (per session, per IP).
|
||||
- **Embedding cache** (frequently-searched queries → cached embedding).
|
||||
- **Pub/sub** for hot-reload notifications ("new template registered").
|
||||
- **In-flight transactions** (for multi-step writes that need atomicity across Neo4j + Postgres).
|
||||
|
||||
## What goes in S3 (MinIO)
|
||||
|
||||
- Full text of every `LoreSource` (the YAML/markdown the world-builder wrote).
|
||||
- Full text of every long `DomainEntity.summary`.
|
||||
- Attachments: images, audio, videos, large files referenced by lore.
|
||||
- Snapshots: world state exports, consistency engine reports, retcon bundles.
|
||||
|
||||
MinIO is the self-hosted S3. Same protocol, no AWS dependency.
|
||||
|
||||
## The cross-store query layer
|
||||
|
||||
The MCP tools the LLM uses compose across stores. The engine handles the cross-store joins; the LLM sees a unified response.
|
||||
|
||||
### Example: "What was the Crimson Hand's biggest heist in Mardsville last year?"
|
||||
|
||||
```
|
||||
LLM → tool: list_missions(filter_by={faction: "crimson_hand", location: "mardsville", since: "1_year_ago"}, sort_by="payout_gp", limit=5)
|
||||
|
||||
Engine:
|
||||
1. Neo4j: MATCH (m:DomainEntity {type: "ThievesGuildMission"})
|
||||
-[:TARGETS|F2]-> (loc:Location {name: "Mardsville"})
|
||||
-[:LOGGED_IN]-> (lot:DomainEntity {type: "TradeLogEntry"})
|
||||
-[:GIVEN_BY]-> (npc:Person {faction: "Crimson Hand"})
|
||||
RETURN m, lot, npc
|
||||
2. Postgres: SELECT * FROM mission_log WHERE mission_id IN (...) ORDER BY step_no
|
||||
3. Postgres: SELECT * FROM trade_log WHERE lot_id IN (...) AND occurred_at > ...
|
||||
4. Compose: return top 5 by payout_gp, with mission step timeline + trade details
|
||||
|
||||
LLM: gets a unified response. Doesn't know it crossed 3 stores.
|
||||
```
|
||||
|
||||
### Example: "What battles did the Vyrs lose?"
|
||||
|
||||
```
|
||||
LLM → tool: list_campaign_events(filter_by={faction: "house_vyr", outcome: "loss"})
|
||||
|
||||
Engine:
|
||||
1. Neo4j: get the Campaign nodes tied to house_vyr
|
||||
2. Postgres: SELECT * FROM campaign_event
|
||||
WHERE campaign_id IN (...) AND outcome = 'loss'
|
||||
ORDER BY occurred_at DESC
|
||||
3. Compose: return list with Neo4j faction details + Postgres battle details
|
||||
|
||||
LLM: unified response.
|
||||
```
|
||||
|
||||
The engine exposes **composed tools** like `list_missions`, `list_campaign_events`. The LLM calls one tool; the engine fans out across stores.
|
||||
|
||||
## The cross-store consistency story
|
||||
|
||||
The consistency engine operates across stores. A `:Contradiction` node in Neo4j can reference a Postgres row. An `OntologyRule` in Neo4j can include Cypher that joins with a Postgres query (via Neo4j's apoc.load.jdbc).
|
||||
|
||||
The rules that go cross-store:
|
||||
|
||||
- *"A `:DomainEntity` of type `TradeLot` referenced in Neo4j must have a corresponding row in `trade_log`."*
|
||||
- *"A mission marked `status: 'completed'` in Neo4j must have a `step_type = 'completed'` row in `mission_log`."*
|
||||
- *"A campaign event's `army_size` in Postgres must be within 10% of the `:DomainEntity` aggregate of the participating factions' `Person.count`."*
|
||||
|
||||
These rules are written in Cypher with `apoc.load.jdbc` calls. They run in the nightly batch.
|
||||
|
||||
## Why this is better than one big Neo4j
|
||||
|
||||
| Concern | Neo4j-only | Polyglot |
|
||||
|---|---|---|
|
||||
| High-volume writes (mission steps) | Bloats graph, slows down traversal | Postgres handles it cleanly |
|
||||
| Time-series queries (battles over time) | Requires traversal every query | Postgres `GROUP BY occurred_at` is fast |
|
||||
| Full-text search over millions of words | Slow, requires external index | pgvector or Qdrant, designed for it |
|
||||
| Vector search | OK, but coupled to graph | Dedicated vector store, decoupled |
|
||||
| Blob storage (full lore text, attachments) | Don't do this in Neo4j | S3, cheap, durable |
|
||||
| Sub-millisecond ephemeral state | Possible but ugly | Redis, designed for it |
|
||||
| Graph traversal and pattern matching | Excellent | Still excellent (Neo4j) |
|
||||
|
||||
The graph stays the graph. Operational data lives where it belongs. The LLM gets unified responses via composed tools.
|
||||
|
||||
## The cost: cross-store transactions
|
||||
|
||||
When the world-builder writes a new mission, we touch Neo4j (entity, relations), Postgres (mission_log row), and S3 (any attachments). **These three writes are not atomic.** A partial failure leaves the world in an inconsistent state.
|
||||
|
||||
**Mitigation: the saga pattern.**
|
||||
|
||||
```
|
||||
saga: log_mission:
|
||||
step 1: Postgres INSERT INTO mission_log
|
||||
step 2: Neo4j MERGE (:DomainEntity) + relations
|
||||
step 3: S3 PUT attachments (if any)
|
||||
step 4: Neo4j MERGE (:ConsistencyRun {saga_id: ...}) SET status = 'committed'
|
||||
|
||||
on failure at step 2: rollback step 1 (Postgres DELETE)
|
||||
on failure at step 3: mark mission as 'attachments_pending', retry later
|
||||
on failure at step 4: log to dead-letter queue, alert world-builder
|
||||
```
|
||||
|
||||
Sagas are more code than a single transaction, but they're correct. The alternative — putting everything in Neo4j and hoping — is the trap.
|
||||
|
||||
## What this is not
|
||||
|
||||
- **Not a microservices overhaul.** The 5 stores run in 1 docker-compose stack. The engine still looks like one system to the LLM.
|
||||
- **Not eventual-consistency-everywhere.** Most operations are single-store. The saga is for the multi-store cases.
|
||||
- **Not a "use Postgres for everything" anti-pattern.** We use Postgres for what it's good at, Neo4j for what it's good at, and the cross-store compose layer for the rest.
|
||||
- **Not free.** Postgres + Qdrant + MinIO is ~3 more services. On the 58GB host, this is fine (~1-2GB extra). On a Raspberry Pi, it would be wrong.
|
||||
|
||||
## Summary
|
||||
|
||||
The storage strategy is the part of the design that lets the engine scale to *the whole world*, not just the macro structure. Neo4j is the *nervous system* — the relations, the time model, the consistency engine. Postgres is the *muscle memory* — the high-volume operational data. Qdrant/pgvector is the *cortex* — the semantic search. Redis is the *short-term memory* — the session state. S3 is the *archive* — the durable storage.
|
||||
|
||||
Each store is the right tool for its job. The engine is the integration layer that makes them feel like one world.
|
||||
340
docs/13-microservice-decomposition.md
Normal file
340
docs/13-microservice-decomposition.md
Normal file
@@ -0,0 +1,340 @@
|
||||
# 13 — Microservice Decomposition: Iterate at Micro and Macro
|
||||
|
||||
The v1 design has a single `mcp-server` Go binary exposing 30 tools. Adding a new tool today means editing `main.go`, recompiling, redeploying. **The iteration loop is the cost of the entire program.** That's the wrong shape for a system the world-builder is going to extend indefinitely.
|
||||
|
||||
This document is the v1.1 decomposition: split `mcp-server` into a core router and a set of pluggable tool services. The LLM-facing API stays the same. The internal architecture becomes a network of small, independently-deployable services.
|
||||
|
||||
## The principle: core is stable, edges are extensible
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────────┐
|
||||
│ MCP GATEWAY (Go) │
|
||||
│ ────────────────────────────── │
|
||||
│ - JSON-RPC over HTTP + SSE │
|
||||
│ - Session registry (Redis-backed) │
|
||||
│ - Active context tracking │
|
||||
│ - Tool discovery proxy (delegates to registry) │
|
||||
│ - Tool call routing (parses request, dispatches to handler) │
|
||||
│ - ~500 lines of Go. Stable. │
|
||||
└─────────────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
│ in-process or HTTP
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────────────────┐
|
||||
│ TOOL REGISTRY (Postgres) │
|
||||
│ ────────────────────────────────────────── │
|
||||
│ One row per registered tool: │
|
||||
│ name, description, params_schema, handler_service, │
|
||||
│ handler_method, requires_auth, rate_limit, version │
|
||||
│ │
|
||||
│ Hot-reloadable. New tools appear via: │
|
||||
│ - template registration (auto-generates tools) │
|
||||
│ - manual POST /admin/tools │
|
||||
│ - plugin service startup (announces itself) │
|
||||
└─────────────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
│ dynamic dispatch
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────────────────┐
|
||||
│ TOOL HANDLER SERVICES (Go / Python / any) │
|
||||
│ ────────────────────────────────────────────────── │
|
||||
│ Each is a small, focused service that implements one category │
|
||||
│ of tools. New services can be added without touching the gateway. │
|
||||
│ │
|
||||
│ Phase-1 handlers (in-process Go): │
|
||||
│ - core-handler: was_true_at, true_during, state_at, │
|
||||
│ entities_present, timeline, lookup, │
|
||||
│ entity_context, expand_context │
|
||||
│ - lineage-handler: list_lineage, ancestors_of, descendants_of, │
|
||||
│ list_offspring, location_hierarchy │
|
||||
│ - event-handler: event_chain, events_during, lore_about, cite │
|
||||
│ - consistency-handler: get_contradictions, get_anachronisms, │
|
||||
│ get_ontology_violations, get_orphans, │
|
||||
│ run_consistency_check, latest_run, │
|
||||
│ flag_for_review, explain_violation, │
|
||||
│ add_ontology_rule, list_ontology_rules │
|
||||
│ - generation-handler: summarize_chain, narrate_arc │
|
||||
│ - worldbuilder-handler: add_entity, add_relation, │
|
||||
│ add_lore_source, add_template, etc. │
|
||||
│ │
|
||||
│ Hot-pluggable handlers (out-of-process, via HTTP): │
|
||||
│ - any HTTP service that speaks the tool-handler protocol │
|
||||
│ - tools register themselves with the gateway at startup │
|
||||
│ - can be written in Go, Python, Node, Rust, anything │
|
||||
└─────────────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
│ the handlers themselves
|
||||
│ compose across the data stores
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────────────────┐
|
||||
│ STORES (the data layer) │
|
||||
│ Neo4j · Postgres · pgvector · Redis · S3 (MinIO) │
|
||||
└─────────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
The **gateway** is the part that talks to the LLM. It's small, stable, and changes rarely. The **handlers** are the part that knows how to answer questions. They're small, focused, and can be added/replaced/swapped without touching the gateway. The **stores** are the data. They're the slowest-changing layer of all.
|
||||
|
||||
## The tool-handler protocol
|
||||
|
||||
A handler is a service that implements a tiny RPC protocol. The gateway calls it; the handler does the work; the gateway returns the result.
|
||||
|
||||
### The protocol
|
||||
|
||||
**Discovery:**
|
||||
```
|
||||
GET /tools
|
||||
→ 200 OK
|
||||
{
|
||||
"tools": [
|
||||
{ "name": "was_true_at", "description": "...", "params_schema": {...} },
|
||||
...
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
**Invocation:**
|
||||
```
|
||||
POST /invoke
|
||||
{
|
||||
"tool": "was_true_at",
|
||||
"arguments": { "relation": "RULED", "subject": "House Vyr", ... },
|
||||
"session_id": "abc123",
|
||||
"active_context": [...] // for handlers that need it
|
||||
}
|
||||
→ 200 OK
|
||||
{
|
||||
"result": { ... },
|
||||
"added_to_context": [...] // entities to add to session context
|
||||
}
|
||||
```
|
||||
|
||||
That's it. **Two endpoints. JSON in, JSON out.** A handler can be written in any language that speaks HTTP.
|
||||
|
||||
### Why this is the right seam
|
||||
|
||||
- **The gateway stays simple.** It just routes `tools/call` to a handler.
|
||||
- **Handlers are testable in isolation.** Mock the gateway, exercise the handler.
|
||||
- **A handler can be a single function or a 5000-line service.** Doesn't matter to the gateway.
|
||||
- **A handler can fail independently.** A bad handler doesn't crash the gateway; the gateway returns an error to the LLM and the LLM tries a different tool.
|
||||
- **Handlers can scale independently.** A heavy `summarize_chain` handler can be horizontally scaled without scaling the lightweight `lookup` handler.
|
||||
|
||||
## The dynamic tool generator
|
||||
|
||||
The most leveraged service in the v1.1 architecture is the **template-driven tool generator**. It watches `./templates/` for YAML changes and auto-generates tool registrations.
|
||||
|
||||
```
|
||||
On ./templates/thieves_guild/mission.yaml file change:
|
||||
|
||||
1. YAML parser validates against the template schema
|
||||
2. For each tool spec in the template:
|
||||
- Generate the JSON schema for params
|
||||
- Generate the Cypher / SQL / query code (data-defined, not hand-written)
|
||||
- Register the tool in the Postgres tool registry
|
||||
3. Notify the gateway: "new tools available"
|
||||
4. Gateway fetches the new tool list, merges with existing, hot-reloads
|
||||
5. LLM clients see the new tools in their next tools/list call
|
||||
```
|
||||
|
||||
The generated tool handler is a *generic* runner that knows the template spec. For each tool, the runner:
|
||||
|
||||
1. Parses the `arguments` against the template's `params` schema.
|
||||
2. Builds a query (Cypher for graph ops, SQL for time-series, composed for cross-store).
|
||||
3. Executes the query.
|
||||
4. Returns the result.
|
||||
|
||||
This is a *one-time* code generator. The runtime handler is shared across all generated tools. **Adding a new domain type means writing a template, not writing a handler.**
|
||||
|
||||
## The communication pattern: in-process first, HTTP second
|
||||
|
||||
In v1.1, all the Phase-1 handlers run **in the same Go binary as the gateway**. They're imported packages, not separate services. The HTTP-based plugin protocol is for handlers that *can't* run in-process — heavy ML models, Python ML toolkits, external services.
|
||||
|
||||
**Why in-process first:**
|
||||
- No network latency on the common path.
|
||||
- Shared connection pools (Neo4j driver, Postgres pool).
|
||||
- Simpler deployment (one binary, one docker image).
|
||||
- Easier debugging.
|
||||
|
||||
**Why HTTP second (for the future):**
|
||||
- A handler in Python (e.g. using spaCy for entity resolution) doesn't need to be rewritten in Go.
|
||||
- A heavy handler (e.g. ML inference for image generation) can be scaled independently.
|
||||
- An external service (e.g. a hosted embedding model) can be a handler.
|
||||
|
||||
The gateway treats in-process handlers and HTTP handlers identically. The dispatch is the same.
|
||||
|
||||
## The phase plan: which services when
|
||||
|
||||
The decomposition is a *progressive* split, not a big-bang rewrite.
|
||||
|
||||
### Phase 1 (MVP): single binary, multiple handler files
|
||||
|
||||
```
|
||||
mcp-server/ # one Go module
|
||||
├── main.go # gateway (~500 lines)
|
||||
├── handlers/
|
||||
│ ├── core.go # 9 core tools
|
||||
│ ├── lineage.go # 5 lineage/hierarchy tools
|
||||
│ ├── event.go # 4 lore/event tools
|
||||
│ ├── consistency.go # 11 consistency tools
|
||||
│ ├── generation.go # 2 generation tools
|
||||
│ ├── worldbuilder.go # 9 world-builder tools
|
||||
│ └── generated.go # template-driven tools (auto-registered)
|
||||
├── registry/
|
||||
│ └── tool_registry.go # in-process tool registry
|
||||
└── ...
|
||||
```
|
||||
|
||||
Same binary, multiple files. The `mcpTools` array from v1 is split into per-handler arrays, registered at startup.
|
||||
|
||||
**Deliverable:** the same 30+ tools, but the code is organized so the next phase's split is mechanical.
|
||||
|
||||
### Phase 2: extract the consistency engine to its own service
|
||||
|
||||
The consistency engine is *naturally* a separate service — it runs on a schedule, it has its own state (`:ConsistencyRun` nodes), and it has a different scaling profile from the read-path tools.
|
||||
|
||||
```
|
||||
services/
|
||||
├── mcp-gateway/ # gateway only
|
||||
├── consistency-runner/ # cron-based, runs rules
|
||||
├── consistency-monitor/ # HTTP service, exposes run_check
|
||||
```
|
||||
|
||||
**Deliverable:** the gateway no longer has any consistency logic. The consistency-runner and consistency-monitor are independently deployable.
|
||||
|
||||
### Phase 3: extract the template-driven tool generator
|
||||
|
||||
The template generator is a *hot-reload* service that needs to be able to restart tool registration without restarting the gateway. The right shape is a sidecar that watches `./templates/`, generates tool specs, and pushes them to the gateway via an admin API.
|
||||
|
||||
```
|
||||
services/
|
||||
├── mcp-gateway/
|
||||
├── template-watcher/ # watches ./templates/, generates tool specs
|
||||
├── template-registry/ # persists tool specs to Postgres
|
||||
```
|
||||
|
||||
**Deliverable:** world-builders can add a new domain type by writing YAML, posting to a webhook, and the gateway picks it up within 5 seconds. No gateway restart.
|
||||
|
||||
### Phase 4: HTTP-based handler protocol
|
||||
|
||||
The gateway is extended to support HTTP-based handlers. The first HTTP-based handler is a Python service that does heavy ML work the gateway shouldn't do — e.g. an entity-linking model that resolves ambiguous entity references at scale.
|
||||
|
||||
```
|
||||
services/
|
||||
├── mcp-gateway/ # now supports both in-process and HTTP handlers
|
||||
├── entity-linker/ # Python, uses a transformer model
|
||||
```
|
||||
|
||||
**Deliverable:** a Python handler is registered. The gateway dispatches to it transparently. The LLM doesn't know or care.
|
||||
|
||||
### Phase 5: external service handlers
|
||||
|
||||
Handlers can be hosted externally — a web service, a Cloudflare Worker, a serverless function. The auth and rate-limiting logic in the gateway makes this safe.
|
||||
|
||||
**Deliverable:** an external world-data provider (e.g. a community-maintained bestiary database) is a handler. The LLM can query it via the same MCP API.
|
||||
|
||||
## What this means for the world-builder
|
||||
|
||||
The world-builder's experience is unchanged at the API level:
|
||||
|
||||
- Write YAML → POST to `/ingest/template` → tools appear.
|
||||
- POST to `/ingest/structured` with instance YAML → data lands.
|
||||
- Query via MCP.
|
||||
|
||||
What changes is the *iteration loop*:
|
||||
|
||||
| | v1 (monolith) | v1.1 (decomposed) |
|
||||
|---|---|---|
|
||||
| Add a new domain type | Edit Go, edit Cypher, rebuild, redeploy | Write YAML, POST it. Done. |
|
||||
| Add a new tool for an existing type | Edit Go, rebuild, redeploy | Add a tool spec to the template. Done. |
|
||||
| Fix a bug in a tool | Find the right handler, rebuild, redeploy gateway | Fix in the handler service. Redeploy that service only. |
|
||||
| Add a heavy ML-based tool | Rewrite in Go or call out to Python via subprocess | Write a Python service, register as HTTP handler. |
|
||||
| Scale a single tool to 100x | Scale the whole gateway | Scale just that handler. |
|
||||
|
||||
The iteration loop drops from **hours** (code change, build, deploy) to **minutes** (YAML change, hot-reload). This is what makes the engine tractable for a world that's going to grow indefinitely.
|
||||
|
||||
## What this means for the LLM
|
||||
|
||||
The LLM doesn't know or care. It sees a single MCP server with a list of tools. The tool names, descriptions, and params are the same whether the tool is in-process or HTTP.
|
||||
|
||||
The LLM does get a *new* tool: `list_template_tools()`. This returns the auto-generated tools for all currently-loaded templates. The LLM can use this to discover what domain types are available without having to be told.
|
||||
|
||||
```json
|
||||
{
|
||||
"name": "list_template_tools",
|
||||
"description": "List all tools auto-generated from loaded type templates. Use this to discover what domain types are available in the current world.",
|
||||
"params": ["type_filter", "limit"]
|
||||
}
|
||||
```
|
||||
|
||||
## The handler-services directory layout
|
||||
|
||||
When the system is fully decomposed:
|
||||
|
||||
```
|
||||
services/
|
||||
├── mcp-gateway/ # the public MCP endpoint (~500 lines Go)
|
||||
├── template-watcher/ # watches templates, generates tool specs
|
||||
├── template-registry/ # persists tool specs
|
||||
├── core-tools/ # 9 core ontology tools
|
||||
├── lineage-tools/ # 5 lineage/hierarchy tools
|
||||
├── event-tools/ # 4 lore/event tools
|
||||
├── consistency-runner/ # nightly batch
|
||||
├── consistency-monitor/ # HTTP, exposes run_check
|
||||
├── generation-tools/ # 2 generation tools (LLM-in-loop)
|
||||
├── worldbuilder-tools/ # 9 write tools
|
||||
├── structured-ingestor/ # YAML → Neo4j + Postgres
|
||||
├── lore-extractor/ # LLM-based prose extraction
|
||||
├── dialogue-processor/ # in-fiction dialogue logs
|
||||
├── entity-linker/ # Python, ML-based entity resolution
|
||||
├── bestiary-external/ # external world-data provider
|
||||
└── ...
|
||||
```
|
||||
|
||||
Each service has its own Dockerfile, its own healthcheck, and its own test suite. The gateway is a thin router.
|
||||
|
||||
## The cost: more services to operate
|
||||
|
||||
Each service is a new line in `docker-compose.yml`, a new healthcheck, a new failure mode.
|
||||
|
||||
**Mitigation:** the gateway *aggregates* the healthchecks. The world-builder sees one health endpoint, not fifteen.
|
||||
|
||||
```
|
||||
GET /healthz on the gateway:
|
||||
{
|
||||
"gateway": "ok",
|
||||
"neo4j": "ok",
|
||||
"postgres": "ok",
|
||||
"pgvector": "ok",
|
||||
"redis": "ok",
|
||||
"minio": "ok",
|
||||
"handlers": {
|
||||
"core-tools": "ok",
|
||||
"lineage-tools": "ok",
|
||||
"consistency-monitor": "ok",
|
||||
"template-watcher": "degraded - retrying",
|
||||
...
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
A single degraded handler doesn't fail the healthcheck. The LLM can still use the tools that are working. A failed core tool does fail it.
|
||||
|
||||
## The decomposition is the answer to Kay's question
|
||||
|
||||
> *"We need to be able to take an arbitrary new concept, define how it associates with larger constructs, but also have flexibility to get as detailed as we need."*
|
||||
|
||||
The v1.1 decomposition answers this in three ways:
|
||||
|
||||
1. **New concept (macro):** a YAML template. No code change. Hot-reload. (~1 hour of world-builder work.)
|
||||
|
||||
2. **Detail on the concept (micro):** instance YAML with relations to other entities. The template defines the structure; the instance fills it in. (~10 minutes per instance.)
|
||||
|
||||
3. **New kind of data the template doesn't cover (escape hatch):** a new handler service in the language of choice. The gateway's HTTP-based handler protocol lets the world-builder or a third party add capabilities without modifying the engine. (~1 day of work for a small, focused handler.)
|
||||
|
||||
**The macro level is fast and declarative. The micro level is structured and bounded. The escape hatch is open and arbitrary.** Three layers, three iteration speeds.
|
||||
|
||||
## What this is NOT
|
||||
|
||||
- **Not a microservices fan-out.** The handlers can run in-process. The HTTP protocol is for *some* handlers, not all.
|
||||
- **Not a rewrite of GraphMCP-Example.** The existing tools (semantic_search, graph_traverse, query_as_npc, etc.) become the in-process `core-tools` handler. Same Go code, better file structure.
|
||||
- **Not a SaaS architecture.** The system runs as one docker-compose stack on one host. The decomposition is about *organizing* the code, not distributing it.
|
||||
- **Not a guarantee that all extensions are easy.** Some extensions still need code — anything that requires a new query language feature, a new database, or a new ML model. The escape hatch is for those, and it costs more.
|
||||
568
docs/14-examples.md
Normal file
568
docs/14-examples.md
Normal file
@@ -0,0 +1,568 @@
|
||||
# 14 — Worked Examples: How Extensions Look in Practice
|
||||
|
||||
This document shows, end-to-end, what it looks like to add three new domains to the Lore Engine:
|
||||
|
||||
1. **Thieves-guild missions** (Kay's first example)
|
||||
2. **War-campaign tracker** (Kay's second example)
|
||||
3. **Black-market economy** (Kay's third example)
|
||||
|
||||
Plus a fourth: **NPC secret knowledge** — how the existing NPC tier system integrates with the new template system.
|
||||
|
||||
For each, the deliverables are:
|
||||
|
||||
- The template YAML (the type definition).
|
||||
- An instance YAML (the actual data).
|
||||
- A sample LLM tool call (what the LLM sees and does).
|
||||
- A sample engine response (what the LLM gets back).
|
||||
- A note on the consistency rules that fire automatically.
|
||||
|
||||
The point of this document is to show that **adding a new domain is a YAML exercise, not a code exercise.** If the examples below feel like cheating, the design is working.
|
||||
|
||||
---
|
||||
|
||||
## Example 1: Thieves-guild Missions
|
||||
|
||||
### Context
|
||||
|
||||
The world has a thieves guild called the Crimson Hand, headquartered in Mardsville. The guild runs missions: heists, intelligence operations, assassinations, smuggling runs. The world-builder wants the LLM to be able to reason over the mission log: "What is the Crimson Hand's most profitable operation this year?" "Who is their best operative?" "What missions are currently active?"
|
||||
|
||||
The current ontology doesn't have a "Mission" type. The world-builder needs to add one.
|
||||
|
||||
### Step 1: Write the template
|
||||
|
||||
`./templates/thieves_guild/mission.yaml`:
|
||||
|
||||
```yaml
|
||||
template:
|
||||
id: "thieves_guild_mission"
|
||||
version: "1.0"
|
||||
owner: "kaykayyali"
|
||||
description: "A mission run by a thieves guild. Has a quest-giver, targets, payout, and outcome. Used by the Crimson Hand and similar organizations."
|
||||
|
||||
entity:
|
||||
label: "DomainEntity"
|
||||
type_value: "ThievesGuildMission"
|
||||
discriminator: "thieves_guild_mission"
|
||||
|
||||
properties:
|
||||
- { name: "mission_code", type: "string", required: true, unique: true }
|
||||
- { name: "target", type: "string", required: true, indexed: true }
|
||||
- { name: "payout_gp", type: "int", required: true, indexed: true }
|
||||
- { name: "status", type: "enum", values: ["planned", "active", "completed", "failed", "cancelled"], default: "planned" }
|
||||
- { name: "risk", type: "enum", values: ["low", "medium", "high", "extreme"], default: "medium" }
|
||||
- { name: "secrecy", type: "enum", values: ["public", "faction_internal", "guild_internal", "inner_circle_only"], default: "guild_internal" }
|
||||
- { name: "target_location",type: "string", required: false, indexed: true }
|
||||
- { name: "started_at", type: "time_ref" }
|
||||
- { name: "completed_at", type: "time_ref" }
|
||||
- { name: "summary", type: "text", embedded: true }
|
||||
|
||||
temporal: true
|
||||
# valid_from defaults to started_at; valid_until to completed_at + 30 days
|
||||
|
||||
relations:
|
||||
- { name: "GIVEN_BY", to_type: "Person", cardinality: "exactly_one" }
|
||||
- { name: "TARGETS", to_type: ["Person", "Faction", "Location", "DomainEntity"], cardinality: "many" }
|
||||
- { name: "PAID_BY", to_type: "Person", cardinality: "many" }
|
||||
- { name: "PAID_TO", to_type: "Person", cardinality: "many" }
|
||||
- { name: "LOGGED_IN", to_type: "DomainEntity", to_discriminator: "trade_log_entry", cardinality: "exactly_one" }
|
||||
- { name: "PART_OF", to_type: "DomainEntity", to_discriminator: "thieves_guild_campaign", cardinality: "many" }
|
||||
- { name: "WITNESSED_BY",to_type: "Person", cardinality: "many" }
|
||||
|
||||
npc_knowledge:
|
||||
default_tier: "outer_circle"
|
||||
field_overrides:
|
||||
- { field: "payout_gp", min_tier: "inner_circle" }
|
||||
- { field: "secrecy", min_tier: "inner_circle" }
|
||||
|
||||
tools:
|
||||
- { name: "list_missions", params: ["filter_by", "sort_by", "limit"] }
|
||||
- { name: "get_mission", params: ["mission_code"] }
|
||||
- { name: "missions_by_target", params: ["target", "limit"] }
|
||||
- { name: "missions_in_era", params: ["era", "limit"] }
|
||||
- { name: "active_missions", params: ["faction", "limit"] }
|
||||
- { name: "log_mission", params: ["mission_code", "properties", "relations"], write: true }
|
||||
|
||||
rules:
|
||||
- "no-anachronism-participation"
|
||||
- "secrecy-honors-npc-tier"
|
||||
- "payout-must-be-positive"
|
||||
- "completed-missions-have-completed-at"
|
||||
|
||||
llm_hints:
|
||||
- "When asked about a guild's recent activity, list_missions is a strong starting point. Sort by recency unless the user specifies otherwise."
|
||||
- "Mission secrecy controls who can know what. Don't reveal inner_circle_only details to an outer_circle NPC."
|
||||
|
||||
ui:
|
||||
icon: "🗡️"
|
||||
color: "#8b0000"
|
||||
list_columns: ["mission_code", "target", "status", "payout_gp", "completed_at"]
|
||||
```
|
||||
|
||||
That's the entire type definition. The world-builder drops it in `./templates/thieves_guild/`, then:
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:9000/admin/templates/reload
|
||||
# → {"reloaded": 1, "tools_registered": 6, "rules_attached": 4}
|
||||
```
|
||||
|
||||
Six new MCP tools, four consistency rules, registered in the engine. No Go code changed.
|
||||
|
||||
### Step 2: Write an instance
|
||||
|
||||
`./ingest/crimson_hand_missions.yaml`:
|
||||
|
||||
```yaml
|
||||
template: "thieves_guild_mission"
|
||||
world_id: "arda_1st_age"
|
||||
instances:
|
||||
- mission_code: "CH-4471"
|
||||
name: "The Pale Ledger Heist"
|
||||
target: "merchant_guild_vault"
|
||||
payout_gp: 500
|
||||
status: "completed"
|
||||
risk: "high"
|
||||
secrecy: "inner_circle_only"
|
||||
target_location: "mardsville"
|
||||
started_at: "3rd_age.year_385.month_3"
|
||||
completed_at: "3rd_age.year_385.month_4"
|
||||
summary: "Vex the Silent hired a crew of four to infiltrate the
|
||||
merchant guild's vault in Mardsville and steal the ledger
|
||||
of debts. The crew succeeded, but one member was captured
|
||||
and later executed by the city guard."
|
||||
relations:
|
||||
- { type: "GIVEN_BY", to: "person_vex_silent" }
|
||||
- { type: "TARGETS", to: "faction_merchant_guild" }
|
||||
- { type: "TARGETS", to: "location_mardsville" }
|
||||
- { type: "LOGGED_IN", to: "trade_log_9821" }
|
||||
- { type: "PAID_TO", to: "person_crew_member_2" }
|
||||
- { type: "PAID_TO", to: "person_crew_member_3" }
|
||||
- { type: "PART_OF", to: "campaign_pale_quarter" }
|
||||
|
||||
- mission_code: "CH-4472"
|
||||
name: "The Whisper Network"
|
||||
target: "intelligence_network"
|
||||
payout_gp: 1200
|
||||
status: "active"
|
||||
risk: "extreme"
|
||||
secrecy: "guild_internal"
|
||||
target_location: "valdorn"
|
||||
started_at: "3rd_age.year_386.month_1"
|
||||
summary: "Alessia Dusk is running a long-term operation to map
|
||||
the Crimson Pact's intelligence network in Valdorn.
|
||||
Estimated duration: 8 months."
|
||||
relations:
|
||||
- { type: "GIVEN_BY", to: "person_alessia_dusk" }
|
||||
- { type: "TARGETS", to: "faction_crimson_pact" }
|
||||
- { type: "LOGGED_IN", to: "trade_log_9844" }
|
||||
```
|
||||
|
||||
Ingest:
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:9000/ingest/structured \
|
||||
-F "file=@crimson_hand_missions.yaml"
|
||||
# → {"instances_written": 2, "relations_written": 14, "duration_ms": 47}
|
||||
```
|
||||
|
||||
### Step 3: LLM uses the new tools
|
||||
|
||||
The LLM is having a conversation with the user. The user asks:
|
||||
|
||||
> "What missions is the Crimson Hand currently running?"
|
||||
|
||||
The LLM's reasoning:
|
||||
|
||||
```
|
||||
Question type: Type 3 (world state at a time).
|
||||
Tools available: list_missions, active_missions, etc.
|
||||
Pick: active_missions(faction="crimson_hand")
|
||||
```
|
||||
|
||||
Tool call:
|
||||
|
||||
```json
|
||||
{
|
||||
"name": "active_missions",
|
||||
"arguments": {
|
||||
"faction": "crimson_hand",
|
||||
"limit": 10
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Engine response:
|
||||
|
||||
```json
|
||||
{
|
||||
"missions": [
|
||||
{
|
||||
"id": "mission_4472",
|
||||
"mission_code": "CH-4472",
|
||||
"name": "The Whisper Network",
|
||||
"target": "intelligence_network",
|
||||
"payout_gp": 1200,
|
||||
"status": "active",
|
||||
"risk": "extreme",
|
||||
"secrecy": "guild_internal",
|
||||
"target_location": "valdorn",
|
||||
"started_at": "3rd_age.year_386.month_1",
|
||||
"given_by": "Alessia Dusk",
|
||||
"targets": ["Crimson Pact"]
|
||||
}
|
||||
],
|
||||
"total": 1
|
||||
}
|
||||
```
|
||||
|
||||
LLM response to user:
|
||||
|
||||
> "The Crimson Hand has one active mission right now: *The Whisper Network*, a long-term intelligence operation in Valdorn. It was started by Alessia Dusk at the beginning of 386 TA, and is targeting the Crimson Pact's intelligence network. The mission is rated extreme risk and has an estimated duration of eight months. Source: crimson_hand_missions.yaml."
|
||||
|
||||
That answer is **not** in the v1 design. The data is in the polymorphic `DomainEntity` wrapper, the tool was auto-generated from the template, the engine composed the answer across Neo4j (the mission) and Postgres (the mission_log step entries). The LLM used one tool call.
|
||||
|
||||
### Consistency rules that fire
|
||||
|
||||
The four rules attached to the template run on every write:
|
||||
|
||||
1. **`no-anachronism-participation`** (core rule) — checks that the people involved in the mission existed at the time. Vex the Silent, the crew members, and Alessia Dusk all have lifespans covering 385-386 TA. **Pass.**
|
||||
|
||||
2. **`secrecy-honors-npc-tier`** (template rule) — checks that any NPC who knows about the mission has a tier ≥ the mission's `secrecy` field. The crew is at `outer_circle` tier; the mission is `inner_circle_only`. **Fail** — the rule fires, creates a `:OntologyViolation` node: *"Mission CH-4471 (secrecy=inner_circle_only) has WITNESSED_BY edges to NPCs at outer_circle tier."* The world-builder reviews.
|
||||
|
||||
3. **`payout-must-be-positive`** (template rule) — checks `payout_gp > 0`. Both missions pass.
|
||||
|
||||
4. **`completed-missions-have-completed-at`** (template rule) — checks that any mission with `status=completed` has a non-null `completed_at`. CH-4471 has it. **Pass.**
|
||||
|
||||
The world-builder can see the violation in their morning review queue: "Mission CH-4471 has a secrecy tier mismatch — the crew shouldn't know about it. Either bump the crew's tier or mark the mission as not witnessed by them."
|
||||
|
||||
---
|
||||
|
||||
## Example 2: War-campaign Tracker
|
||||
|
||||
### Context
|
||||
|
||||
The world has ongoing wars. The world-builder wants to track campaigns — multi-month military operations involving multiple factions, multiple battles, and shifting territorial control. The LLM should be able to answer: "Where is the Vyr-Crimson war right now?" "What battles has General Aldric won?" "What is the estimated strength of the Crimson Pact's army?"
|
||||
|
||||
### Template: `war_campaign.yaml`
|
||||
|
||||
```yaml
|
||||
template:
|
||||
id: "war_campaign"
|
||||
version: "1.0"
|
||||
owner: "kaykayyali"
|
||||
description: "A multi-faction military campaign. Tracks battles, sieges, troop movements, and territorial changes."
|
||||
|
||||
entity:
|
||||
label: "DomainEntity"
|
||||
type_value: "WarCampaign"
|
||||
discriminator: "war_campaign"
|
||||
|
||||
properties:
|
||||
- { name: "campaign_code", type: "string", required: true, unique: true }
|
||||
- { name: "name", type: "string", required: true }
|
||||
- { name: "status", type: "enum", values: ["planned", "active", "concluded", "truce", "escalated"], default: "active" }
|
||||
- { name: "started_at", type: "time_ref" }
|
||||
- { name: "ended_at", type: "time_ref" }
|
||||
- { name: "summary", type: "text", embedded: true }
|
||||
|
||||
temporal: true
|
||||
|
||||
relations:
|
||||
- { name: "PARTICIPATES", to_type: "Faction", cardinality: "many" }
|
||||
- { name: "COMMANDED_BY", to_type: "Person", cardinality: "many" }
|
||||
- { name: "OCCURS_IN", to_type: "Location", cardinality: "many" }
|
||||
- { name: "INCLUDES_BATTLE",to_type: "DomainEntity", to_discriminator: "battle", cardinality: "many" }
|
||||
- { name: "CAUSED_BY", to_type: "Event", cardinality: "many" }
|
||||
|
||||
tools:
|
||||
- { name: "list_campaigns", params: ["filter_by", "sort_by", "limit"] }
|
||||
- { name: "get_campaign", params: ["campaign_code"] }
|
||||
- { name: "campaign_battles", params: ["campaign_code", "limit"] }
|
||||
- { name: "campaign_strength", params: ["campaign_code", "faction", "as_of"] }
|
||||
- { name: "campaign_movements", params: ["campaign_code", "faction", "limit"] }
|
||||
- { name: "log_battle", params: ["campaign_code", "battle_data"], write: true }
|
||||
- { name: "log_movement", params: ["campaign_code", "faction", "from", "to", "army_size"], write: true }
|
||||
|
||||
rules:
|
||||
- "no-anachronism-participation"
|
||||
- "campaign-status-matches-end-date"
|
||||
- "battle-belongs-to-active-campaign"
|
||||
```
|
||||
|
||||
The interesting bit is `campaign_strength`. It's a *computed* tool — it doesn't return a static field, it queries across stores to give a real-time answer.
|
||||
|
||||
When the LLM calls `campaign_strength(campaign_code="VC-001", faction="house_vyr", as_of="3rd_age.year_386.month_5")`, the engine:
|
||||
|
||||
1. Neo4j: get the campaign node, get `PARTICIPATES` factions, get `COMMANDED_BY` generals.
|
||||
2. Postgres: query `campaign_event` for all `army_moved` events involving House Vyr up to the `as_of` time, with their `army_size` and `casualties`.
|
||||
3. Compose: compute current strength = initial_strength - casualties + reinforcements.
|
||||
4. Return the number with citations.
|
||||
|
||||
The world-builder never has to update a "current strength" field — it's always computed from the event log.
|
||||
|
||||
### Instance: a campaign + its events
|
||||
|
||||
`./ingest/vyr_crimson_war.yaml`:
|
||||
|
||||
```yaml
|
||||
template: "war_campaign"
|
||||
world_id: "arda_1st_age"
|
||||
instances:
|
||||
- campaign_code: "VC-001"
|
||||
name: "The Border Wars"
|
||||
status: "active"
|
||||
started_at: "3rd_age.year_380.month_1"
|
||||
summary: "The ongoing border conflict between House Vyr and the
|
||||
Crimson Pact, fought primarily over the Northern Reaches."
|
||||
relations:
|
||||
- { type: "PARTICIPATES", to: "faction_house_vyr" }
|
||||
- { type: "PARTICIPATES", to: "faction_crimson_pact" }
|
||||
- { type: "COMMANDED_BY", to: "person_aldric_raventhorne" }
|
||||
- { type: "COMMANDED_BY", to: "person_general_kael" }
|
||||
- { type: "OCCURS_IN", to: "region_northern_reaches" }
|
||||
```
|
||||
|
||||
And the campaign events — these are **not** `DomainEntity` instances. They go directly into Postgres `campaign_event`:
|
||||
|
||||
```yaml
|
||||
# This is a Postgres-direct ingest, not a DomainEntity ingest.
|
||||
template: "campaign_event" # a special template that writes to Postgres
|
||||
world_id: "arda_1st_age"
|
||||
events:
|
||||
- campaign_id: "campaign_VC-001"
|
||||
event_type: "army_moved"
|
||||
occurred_at: "3rd_age.year_380.month_2"
|
||||
faction_id: "faction_house_vyr"
|
||||
location_id: "location_thornwall_keep"
|
||||
army_size: 5000
|
||||
outcome: "successful"
|
||||
|
||||
- campaign_id: "campaign_VC-001"
|
||||
event_type: "battle"
|
||||
occurred_at: "3rd_age.year_380.month_4"
|
||||
location_id: "location_black_spire_pass"
|
||||
faction_id: "faction_house_vyr"
|
||||
opponent_id: "faction_crimson_pact"
|
||||
army_size: 5000
|
||||
casualties: 800
|
||||
outcome: "victory"
|
||||
|
||||
- campaign_id: "campaign_VC-001"
|
||||
event_type: "army_moved"
|
||||
occurred_at: "3rd_age.year_386.month_3"
|
||||
faction_id: "faction_house_vyr"
|
||||
location_id: "location_mardsville"
|
||||
army_size: 3200
|
||||
outcome: "in_progress"
|
||||
```
|
||||
|
||||
Postgres is the right place for these because they're high-volume time-series data. The campaign node in Neo4j is the *summary*; the events in Postgres are the *detail*.
|
||||
|
||||
### LLM usage
|
||||
|
||||
User: "How is the Border Wars going for House Vyr?"
|
||||
|
||||
LLM: `campaign_strength(campaign_code="VC-001", faction="house_vyr", as_of="3rd_age.year_386.month_6")`
|
||||
|
||||
Engine: computes from the Postgres event log. Returns:
|
||||
|
||||
```json
|
||||
{
|
||||
"campaign": "VC-001",
|
||||
"faction": "house_vyr",
|
||||
"as_of": "3rd_age.year_386.month_6",
|
||||
"initial_strength": 5000,
|
||||
"current_strength": 2400,
|
||||
"casualties_total": 2600,
|
||||
"reinforcements_received": 0,
|
||||
"recent_battles": [
|
||||
{ "name": "Battle of Black Spire", "outcome": "victory", "date": "380.4" },
|
||||
{ "name": "Siege of Frosthollow", "outcome": "loss", "date": "382.2" },
|
||||
...
|
||||
],
|
||||
"current_position": "Mardsville (advancing)"
|
||||
}
|
||||
```
|
||||
|
||||
LLM: "House Vyr's campaign is in trouble. They started with 5,000 troops and are down to 2,400 after six years of fighting. They won the opening Battle of Black Spire but lost the Siege of Frosthollow in 382 TA. Their current position is Mardsville, where they appear to be advancing. They haven't received any reinforcements. Sources: vyr_crimson_war.yaml."
|
||||
|
||||
This answer combines Neo4j (the campaign node, the faction relations) with Postgres (the event log) and produces a real-time answer. **No static "current strength" field is ever wrong** — it's always computed from the source data.
|
||||
|
||||
---
|
||||
|
||||
## Example 3: Black-market Economy
|
||||
|
||||
### Context
|
||||
|
||||
The world has a black market for rare materials: soulglass, orichalcum, dragonbone, dream-essence. The market is controlled by a few mastermind smugglers, with trade logs known only to them. The LLM should be able to reason over the market — but the data should be **access-controlled by NPC tier**. A commoner NPC cannot know who the mastermind smuggler is. A noble NPC can know the market exists but not the prices. Only the inner circle knows the actual trade log.
|
||||
|
||||
This is the example that requires the NPC tier system *and* the new template system *and* the access-control rules.
|
||||
|
||||
### Template: `black_market_lot.yaml`
|
||||
|
||||
```yaml
|
||||
template:
|
||||
id: "black_market_lot"
|
||||
version: "1.0"
|
||||
owner: "kaykayyali"
|
||||
description: "A lot of rare or illicit material being traded on the black market."
|
||||
|
||||
entity:
|
||||
label: "DomainEntity"
|
||||
type_value: "BlackMarketLot"
|
||||
discriminator: "black_market_lot"
|
||||
|
||||
properties:
|
||||
- { name: "lot_code", type: "string", required: true, unique: true }
|
||||
- { name: "material", type: "string", required: true, indexed: true }
|
||||
- { name: "quantity", type: "int", required: true }
|
||||
- { name: "unit_price_gp", type: "int", required: true, indexed: true }
|
||||
- { name: "total_price_gp", type: "int", required: true }
|
||||
- { name: "status", type: "enum", values: ["available", "reserved", "sold", "seized"], default: "available" }
|
||||
- { name: "secrecy", type: "enum", values: ["market_public", "merchant_knows", "smuggler_inner", "mastermind_only"], default: "smuggler_inner" }
|
||||
- { name: "summary", type: "text", embedded: true }
|
||||
|
||||
temporal: true
|
||||
|
||||
relations:
|
||||
- { name: "SOLD_BY", to_type: "Person", cardinality: "exactly_one" } # the seller
|
||||
- { name: "SOLD_TO", to_type: "Person", cardinality: "many" }
|
||||
- { name: "STORED_AT", to_type: "Location",cardinality: "exactly_one" }
|
||||
- { name: "PART_OF", to_type: "DomainEntity", to_discriminator: "smuggling_network", cardinality: "many" }
|
||||
|
||||
npc_knowledge:
|
||||
default_tier: "outer_circle" # only the smuggler inner circle knows about lots
|
||||
field_overrides:
|
||||
- { field: "unit_price_gp", min_tier: "smuggler_inner" }
|
||||
- { field: "total_price_gp", min_tier: "smuggler_inner" }
|
||||
- { field: "SOLD_TO", min_tier: "mastermind_only" } # who bought is mastermind-only
|
||||
|
||||
tools:
|
||||
- { name: "list_lots", params: ["material", "status", "limit"] }
|
||||
- { name: "get_lot", params: ["lot_code"] }
|
||||
- { name: "market_prices", params: ["material", "as_of"] }
|
||||
- { name: "log_trade", params: ["lot_code", "buyer_id", "amount"], write: true }
|
||||
- { name: "smuggler_network", params: ["smuggler_id"], min_tier: "smuggler_inner" }
|
||||
|
||||
rules:
|
||||
- "no-anachronism-participation"
|
||||
- "lot-must-be-tied-to-seller"
|
||||
- "buyer-must-be-recorded-on-sale"
|
||||
- "secrecy-honors-npc-tier"
|
||||
|
||||
llm_hints:
|
||||
- "Black market data is access-controlled. Don't reveal lot details to an NPC whose tier is below the lot's secrecy field."
|
||||
- "Market prices aggregate: market_prices is the public-safe version; get_lot reveals details only to those who can know."
|
||||
```
|
||||
|
||||
### The access-control flow
|
||||
|
||||
This is the part that doesn't exist in v1. The engine uses the NPC tier system plus the template's `npc_knowledge` field to enforce access control.
|
||||
|
||||
```
|
||||
LLM: query_as_npc(npc_name="Garrick the Cooper", question="What's the price of soulglass these days?")
|
||||
```
|
||||
|
||||
The engine:
|
||||
|
||||
1. Looks up Garrick. He's a `Person` with `tier: "commoner"`.
|
||||
2. The LLM is asking about black market data, so the engine checks the relevant template's `npc_knowledge.default_tier`. For `black_market_lot`, that's `outer_circle`. Garrick is a commoner. He doesn't have access.
|
||||
3. The engine returns:
|
||||
|
||||
```json
|
||||
{
|
||||
"answer": "no_access",
|
||||
"required_tier": "outer_circle",
|
||||
"npc_tier": "commoner",
|
||||
"reason": "Garrick is a commoner and does not have access to black market data."
|
||||
}
|
||||
```
|
||||
|
||||
The LLM is told (via the reasoning harness) to **never** return black market data to Garrick, even if the data exists in the graph. Garrick doesn't know. The engine enforces that.
|
||||
|
||||
For a different NPC:
|
||||
|
||||
```
|
||||
LLM: query_as_npc(npc_name="Captain Yara", question="What's the going rate for soulglass?")
|
||||
```
|
||||
|
||||
Captain Yara is a `Person` with `tier: "smuggler_inner"`. She can know about the market. The engine:
|
||||
|
||||
1. Looks up Captain Yara. `tier: "smuggler_inner"`.
|
||||
2. The template requires `outer_circle` for the lot list. Yara qualifies. She can also see `unit_price_gp` (requires `smuggler_inner`).
|
||||
3. The engine calls `market_prices(material="soulglass", as_of="current")`:
|
||||
|
||||
```json
|
||||
{
|
||||
"material": "soulglass",
|
||||
"current_price_per_unit_gp": 250,
|
||||
"lots_available": 3,
|
||||
"lots_recently_sold": [
|
||||
{ "lot_code": "BML-0042", "quantity": 5, "sale_price_gp": 1250, "buyer": "[REDACTED - mastermind_only]" }
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
The buyer is redacted — that field requires `mastermind_only` tier, and Yara is `smuggler_inner`. The LLM sees the redaction and is told not to attempt to identify the buyer.
|
||||
|
||||
The whole access-control story is data-defined. The NPC tier system is a single Person property. The template's `npc_knowledge` field defines what each tier can know. The engine enforces it on every tool call. The LLM sees either the data, a redaction, or an access-denied response.
|
||||
|
||||
### Postgres tables for trade logs
|
||||
|
||||
The black market writes to the `trade_log` Postgres table. Every sale is a row. Only the mastermind smuggler can query it; the LLM sees aggregates.
|
||||
|
||||
```sql
|
||||
SELECT material, AVG(unit_price), COUNT(*)
|
||||
FROM trade_log
|
||||
WHERE location_id = 'mardsville' AND occurred_at > '3rd_age.year_385'
|
||||
GROUP BY material;
|
||||
```
|
||||
|
||||
The `market_prices` tool runs this query, applies NPC tier filtering, and returns a result that respects access control.
|
||||
|
||||
---
|
||||
|
||||
## Example 4: NPC secret knowledge (showing the tier integration)
|
||||
|
||||
This is the simplest example, but it shows how a core engine concept (NPC tier) is *extended* by templates.
|
||||
|
||||
The existing `Person.tier` property is a string. The existing `query_as_npc` tool uses it to scope knowledge. In v1.1, templates can declare *their own* tiers and have their own tier-overrides per field.
|
||||
|
||||
```yaml
|
||||
# In a "secret_knowledge" template:
|
||||
npc_knowledge:
|
||||
default_tier: "outer_circle"
|
||||
field_overrides:
|
||||
- { field: "secret", min_tier: "inner_circle" }
|
||||
- { field: "method", min_tier: "outer_circle" }
|
||||
- { field: "target_identity", min_tier: "inner_circle" }
|
||||
```
|
||||
|
||||
The engine combines the Person's tier with the template's tier requirements. A commoner sees only `method`. An outer_circle member sees `method` + `secret`. An inner_circle member sees everything.
|
||||
|
||||
This is the same pattern as column-level access control in a database, but expressed as data, and applied at the tool layer where the LLM is.
|
||||
|
||||
---
|
||||
|
||||
## What this means in practice
|
||||
|
||||
For a world-builder, adding a new domain is now:
|
||||
|
||||
1. **Day 1: define the template** (~2 hours of YAML).
|
||||
2. **Day 1 afternoon: write the first few instances** (~30 minutes each).
|
||||
3. **Day 2: the LLM can already answer questions about it.**
|
||||
4. **Ongoing: refine the template, add more instances, add more rules.**
|
||||
|
||||
Compare to v1 (the original GraphMCP-Example design):
|
||||
|
||||
1. **Week 1: design the schema, write the Go code, write the Cypher migration, deploy.**
|
||||
2. **Week 2: the LLM can answer questions about it.**
|
||||
3. **Ongoing: write more code for each new query type.**
|
||||
|
||||
The iteration loop drops from weeks to hours. That's the v1.1 win.
|
||||
|
||||
## What this is NOT
|
||||
|
||||
- **Not a generic CMS.** The templates are typed, the engine validates them, and the consistency engine enforces constraints. The world-builder doesn't get to make a freeform database.
|
||||
- **Not a no-code tool.** Writing the template YAML is a *real skill*. A bad template produces a bad domain. The world-builder still has to think about structure.
|
||||
- **Not a substitute for the core ontology.** If a concept really is a `Person` or a `Faction`, model it as such. The `DomainEntity` wrapper is for things that *aren't* core types — domains that are *additional* to the macro world.
|
||||
- **Not a guarantee of query performance.** A template with 10,000 instances of a complex type, queried 100 times per second, will need indexing and possibly denormalization. The engine is fast enough for the world-builder's interactive use; it's not a high-throughput transactional system.
|
||||
Reference in New Issue
Block a user