lore-engine/docs/11-extensibility.md

11 — Extensibility: Polymorphic Type Templates

The v1 ontology has 14 hard-coded labels. That's fine for the first world but it has a ceiling: a thieves-guild mission is forced into :Event, a war campaign is forced into :Faction-with-properties, a black-market trade log is forced into :Item-with-properties. The LLM can talk about these things, but the engine can't reason over their structure.

"What missions has the Crimson Hand run in Mardsville over the last year, sorted by payout?" is unanswerable today. The data lives in summary text fields, the relationships are implicit in prose, and the LLM has to reconstruct it via semantic_search and hope.

This document is the v1.1 fix: a type-template system that lets the world-builder define new domain types — missions, campaigns, trade lots, spellbooks, rituals, whatever — as data, not code. The core ontology stays small. The world grows without the engine growing.

The principle: types are data, not schema

Today: a new domain type means a Go change, a Cypher change, a Neo4j migration, a docker rebuild.

Goal: a new domain type means a YAML file. The engine reads it. The LLM gets new tools automatically. No code.

This is the same pattern as WordPress custom post types, Salesforce custom objects, or Notion's database templates: the platform is fixed, the content schema is open.

The architecture: 4 layers

┌──────────────────────────────────────────────────────────────────┐
│  LAYER 1: Core Ontology (fixed, in code)                        │
│  ──────────────────────────────────────                         │
│  Person, Faction, Location, Item, Era, Date, Lineage,           │
│  Culture, Deity, Language, MagicSystem, Title, Region, Material  │
│  + the time model + the consistency engine                      │
│  + the MCP transport + the active context                       │
│  This is ~3000 lines of Go and ~500 lines of Cypher. Stable.    │
└──────────────────────────────────────────────────────────────────┘
                              │
                              │  every node is also a...
                              ▼
┌──────────────────────────────────────────────────────────────────┐
│  LAYER 2: Polymorphic "Domain Entity" wrapper                   │
│  ──────────────────────────────────────                         │
│  (:DomainEntity {                                                 │
│    id, type, name, era_id, world_id,                             │
│    properties: map<string, any>,    ← free-form, type-checked     │
│    template_id: "thieves_guild_mission",                          │
│    sources[], lore_verified, source_confidence                   │
│  })                                                                │
│  Plus: (:Relation {from, to, type, properties, ...}) for         │
│  arbitrary edges between DomainEntities.                          │
│  This is ~200 lines of Cypher. Stable.                            │
└──────────────────────────────────────────────────────────────────┘
                              │
                              │  defined as data in...
                              ▼
┌──────────────────────────────────────────────────────────────────┐
│  LAYER 3: Type Templates (data, hot-reloadable)                  │
│  ──────────────────────────────────────────                     │
│  YAML files in ./templates/ defining:                            │
│    - what fields a domain type has (and their types)             │
│    - what relations it can have (and to which other types)       │
│    - which MCP tools auto-generate for it                         │
│    - which ontology rules apply                                   │
│    - which LLM-inference hints (e.g. "treat this as a secret")   │
│  This is the part that grows.                                     │
└──────────────────────────────────────────────────────────────────┘
                              │
                              │  instances of...
                              ▼
┌──────────────────────────────────────────────────────────────────┐
│  LAYER 4: Domain Instances (the world's actual data)             │
│  ──────────────────────────────────────────────                 │
│  Mission #4471 ("The Pale Ledger Heist")                          │
│    properties: {target: "merchant_guild_vault", payout: 500gp,   │
│                 status: "completed", risk: "high"}                │
│    relations: GIVEN_BY → NPC (Vex the Silent)                     │
│                LOCATED_IN → Location (Mardsville)                 │
│                TARGETS → Person (Guildmaster Torren)              │
│                LOGGED_IN → TradeLog (lot #9821)                   │
└──────────────────────────────────────────────────────────────────┘

The world-builder writes Layer 4 (instances) and Layer 3 (templates). The engine ships Layer 1 and Layer 2. The MCP tools and consistency rules auto-generate from the templates.

Layer 1: Core ontology (unchanged from v1)

The 14 labels stay. They cover the macro structure of any world: who exists, where they are, when they lived, what they belonged to. This is stable.

Layer 2: The DomainEntity wrapper

A single new node label and a single new edge label. This is the polymorphic backbone.

// A domain entity — anything that the world-builder wants to model
// that isn't already covered by a core label.
(:DomainEntity {
  id: "mission_4471",
  type: "ThievesGuildMission",          // matches a :TypeTemplate.name
  name: "The Pale Ledger Heist",
  world_id: "arda_1st_age",
  
  // Polymorphic, type-checked properties from the template
  properties: {
    target: "merchant_guild_vault",
    payout_gp: 500,
    status: "completed",
    risk: "high",
    secrecy: "guild_internal"
  },
  
  // The world-builder can add a free-text summary for embedding
  summary: "Vex hired a crew to steal the ledger of merchant debts...",
  
  // Standard provenance
  sources: ["crimson_hand_records.yaml"],
  lore_verified: true,
  source_confidence: 0.95,
  created_at: datetime(),
  updated_at: datetime()
})

// A relation between any two nodes (DomainEntity, Person, Faction, etc.)
(:Relation {
  id: "rel_abc",
  from_id: "mission_4471",
  to_id: "person_vex_silent",
  type: "GIVEN_BY",                      // matches a template's allowed_relations
  properties: {                          // typed by template
    at: "3rd_age.year_385",
    agreed_payout: 500
  },
  valid_from: "3rd_age.year_385",
  valid_until: null,
  sources: ["crimson_hand_records.yaml"],
  source_confidence: 0.9
})

Two new labels. ~200 lines of Cypher. Stable forever.

Indexes for Layer 2

CREATE CONSTRAINT domain_entity_id IF NOT EXISTS
FOR (d:DomainEntity) REQUIRE d.id IS UNIQUE;

CREATE CONSTRAINT relation_id IF NOT EXISTS
FOR (r:Relation) REQUIRE r.id IS UNIQUE;

CREATE INDEX domain_entity_type IF NOT EXISTS
FOR (d:DomainEntity) ON (d.type);

CREATE INDEX domain_entity_world IF NOT EXISTS
FOR (d:DomainEntity) ON (d.world_id);

CREATE INDEX relation_type IF NOT EXISTS
FOR (r:Relation) ON (r.type);

// For time-bounded relation queries (same pattern as core ontology)
CREATE INDEX relation_time_window IF NOT EXISTS
FOR (r:Relation) ON (r.valid_from, r.valid_until);

Layer 3: Type templates (the part that grows)

A type template is a YAML file in ./templates/<domain>/<type>.yaml that defines:

• What fields the type has (name, type, required, indexed, secret).
• What relations it can have (to which other types, with what properties).
• What MCP tools the engine should auto-generate for it.
• What ontology rules apply to it.
• How the LLM should reason about it.

Example: thieves_guild_mission.yaml

# This file defines the ThievesGuildMission domain type.
# Drop it in ./templates/thieves_guild/ and the engine hot-reloads.

template:
  id: "thieves_guild_mission"
  version: "1.0"
  owner: "kaykayyali"            # who maintains this template
  description: "A mission run by a thieves guild. Has a quest-giver, targets, payout, and outcome."

# Polymorphic node definition
entity:
  label: "DomainEntity"           # always DomainEntity in v1.1
  type_value: "ThievesGuildMission"
  
  # Discriminator field — used by the LLM and the rule engine to
  # distinguish this from other domain types in the same world
  discriminator: "thieves_guild_mission"
  
  properties:
    - { name: "mission_code",   type: "string",  required: true,  unique: true }
    - { name: "target",         type: "string",  required: true,  indexed: true }
    - { name: "payout_gp",      type: "int",     required: true,  indexed: true }
    - { name: "status",         type: "enum",    values: ["planned", "active", "completed", "failed", "cancelled"] }
    - { name: "risk",           type: "enum",    values: ["low", "medium", "high", "extreme"] }
    - { name: "secrecy",        type: "enum",    values: ["public", "faction_internal", "guild_internal", "inner_circle_only"] }
    - { name: "target_location",type: "string",  required: false, indexed: true }
    - { name: "started_at",     type: "time_ref" }   # canonical time string
    - { name: "completed_at",   type: "time_ref" }
    - { name: "summary",        type: "text",    embedded: true }  # embedded for semantic search
  
  # Time model: this entity type has a temporal existence window
  temporal: true
  # valid_from defaults to started_at; valid_until defaults to completed_at + 30 days

# Allowed relations — typed, can reference core or domain types
relations:
  - { name: "GIVEN_BY",   to_type: "Person",        cardinality: "exactly_one" }
  - { name: "GIVEN_BY",   to_type: "NPC",           cardinality: "exactly_one" }   # an NPC wrapper
  - { name: "TARGETS",    to_type: ["Person", "Faction", "Location", "DomainEntity"], cardinality: "many" }
  - { name: "PAID_BY",    to_type: "Person",        cardinality: "many" }
  - { name: "PAID_TO",    to_type: "Person",        cardinality: "many" }
  - { name: "LOGGED_IN",  to_type: "DomainEntity",  to_discriminator: "trade_log_entry", cardinality: "exactly_one" }
  - { name: "PART_OF",    to_type: "DomainEntity",  to_discriminator: "thieves_guild_campaign", cardinality: "many" }
  - { name: "WITNESSED_BY", to_type: "Person",      cardinality: "many" }

# Default scoping: which NPCs can know about this entity
# Uses the existing NPC tier system
npc_knowledge:
  default_tier: "outer_circle"           # only NPCs at outer_circle or closer can know
  field_overrides:
    - { field: "payout_gp",   min_tier: "inner_circle" }
    - { field: "secrecy",     min_tier: "inner_circle" }

# Auto-generated MCP tools for this type
tools:
  - { name: "list_missions",     params: ["filter_by", "sort_by", "limit"] }
  - { name: "get_mission",       params: ["mission_code"] }
  - { name: "missions_by_target",params: ["target", "limit"] }
  - { name: "missions_in_era",   params: ["era", "limit"] }
  - { name: "log_mission",       params: ["mission_code", "properties", "relations"], write: true }
  # Tools are generated by the engine from this spec. The LLM sees them in tools/list.

# Ontology rules that should run for this type
rules:
  - "no-anachronism-participation"   # references the core rules
  - "secrecy-honors-npc-tier"        # template-local rule
  - "payout-must-be-positive"        # template-local rule

# LLM-inference hints — fed to the reasoning harness
llm_hints:
  - "When a Mission has secrecy=inner_circle_only, the LLM MUST NOT reveal its
     details to an NPC with tier < inner_circle. Use the npc_knowledge field_overrides."
  - "When asked 'what is the Crimson Hand doing?', list_missions is a strong
     starting point. Sort by recency unless the user specifies otherwise."

# UI hints (for future world-builder UI)
ui:
  icon: "🗡️"
  color: "#8b0000"
  list_columns: ["mission_code", "target", "status", "payout_gp", "completed_at"]
  detail_sections:
    - { title: "Mission",   fields: ["mission_code", "target", "status", "risk"] }
    - { title: "People",    fields_via_relations: ["GIVEN_BY", "TARGETS", "PAID_TO"] }
    - { title: "Payout",    fields: ["payout_gp", "PAID_BY"] }
    - { title: "Timeline",  fields: ["started_at", "completed_at"], relations: ["PART_OF"] }

That's a single YAML file. No code. The engine:

1. Validates the schema on load.
2. Creates the (:TypeTemplate {id: "thieves_guild_mission", ...}) node.
3. Registers the auto-generated MCP tools in tools/list.
4. Wires the rules into the consistency engine.
5. Re-indexes the polymorphic indexes.

Hot reload — no engine restart.

What the engine does with a template

On ./templates/thieves_guild/mission.yaml file change:
  1. YAML parser → schema validation
  2. Cypher write: MERGE (t:TypeTemplate {id: "thieves_guild_mission"}) SET t.<spec>
  3. Tool registry: for each entry in spec.tools, register a tool handler
     (the handler is a generic "domain tool runner" — same code, different params)
  4. Rule registry: for each entry in spec.rules, attach to consistency engine
  5. Hot-reload notification: send_message to subscribed LLM clients
     "Domain type thieves_guild_mission registered; 6 new tools available"

The MCP server is now a generic tool runner. It doesn't know about thieves guilds, war campaigns, or black markets. It knows about TypeTemplate nodes and dispatches generically.

This is the architectural change that unlocks everything else.

Layer 4: Domain instances (the world's data)

A world-builder writes a YAML file with concrete missions. The schema is enforced by the template.

# crimson_hand_missions.yaml
# Ingested via POST /ingest/structured?type=thieves_guild_mission

template: "thieves_guild_mission"
world_id: "arda_1st_age"
instances:
  - mission_code: "M-4471"
    name: "The Pale Ledger Heist"
    target: "merchant_guild_vault"
    payout_gp: 500
    status: "completed"
    risk: "high"
    secrecy: "inner_circle_only"
    target_location: "mardsville"
    started_at: "3rd_age.year_385.month_3"
    completed_at: "3rd_age.year_385.month_4"
    summary: "Vex hired a crew of four to infiltrate the merchant guild's vault
              and steal the ledger of debts. The crew succeeded, but one member
              was captured and later executed."
    relations:
      - { type: "GIVEN_BY",  to: "person_vex_silent" }
      - { type: "TARGETS",   to: "faction_merchant_guild" }
      - { type: "LOGGED_IN", to: "trade_log_9821" }
      - { type: "PART_OF",   to: "campaign_pale_quarter" }
  
  - mission_code: "M-4472"
    name: "The Whisper Network"
    target: "intelligence_network"
    payout_gp: 1200
    status: "active"
    risk: "extreme"
    secrecy: "guild_internal"
    target_location: "valdorn"
    started_at: "3rd_age.year_386.month_1"
    relations:
      - { type: "GIVEN_BY",  to: "person_alessia_dusk" }
      - { type: "TARGETS",   to: "faction_crimson_pact" }

The YAML is parsed, validated against the template, and written as (:DomainEntity) + (:Relation) nodes. No LLM. No code.

How the LLM uses this

The LLM's tool list is now dynamic. After the template loads, tools/list returns:

{
  "tools": [
    // ... 30 core tools from the engine
    { "name": "list_missions", "params": ["filter_by", "sort_by", "limit"], "domain": "thieves_guild_mission" },
    { "name": "get_mission",   "params": ["mission_code"], "domain": "thieves_guild_mission" },
    // ...
  ]
}

The LLM's reasoning harness (07-reasoning-harness.md) is updated: "When the question matches a domain type, prefer the domain-specific tool over the generic ones. e.g. for 'what missions has the guild run,' use list_missions not semantic_search."

The LLM doesn't have to know that "thieves guild missions" is a new domain. It sees a tool, it uses it. The tool happens to be auto-generated.

The TypeTemplate node (in-graph representation)

For the consistency engine and the reasoning harness to reason over the templates, the templates live in the graph as data:

(:TypeTemplate {
  id: "thieves_guild_mission",
  version: "1.0",
  owner: "kaykayyali",
  spec_yaml: "...",
  spec: { ... parsed JSON ... },
  status: "active",         // active | draft | deprecated
  registered_at: datetime()
})

The rule engine can read templates to generate type-aware rules. The LLM can read templates to understand the structure of a domain it's reasoning over.

Versioning templates

Templates have a version field. When a template is updated:

• Old version stays in the graph as (:TypeTemplate {id: "thieves_guild_mission", version: "0.9", status: "deprecated"}).
• New version is (:TypeTemplate {id: "thieves_guild_mission", version: "1.0", status: "active"}).
• Existing instances reference the version they were ingested under.
• The engine can replay ingestion against a newer template (with the world-builder's review).

This is the retcon-preserving answer to Kay's Q4. Domain types evolve; instances are not silently rewritten.

What this enables

Adding a new domain is now:

1. Write templates/<domain>/<type>.yaml (~100 lines).
2. POST to POST /ingest/template (hot-reload).
3. Start writing instance YAMLs.
4. Done.

No Go. No Cypher. No docker rebuild. The LLM gets new tools automatically.

That's the architecture. The next two docs decompose it further:

• 12-storage-strategy.md — what data goes in which store
• 13-microservice-decomposition.md — how the mcp-server is split so the LLM can iterate at micro and macro levels

What this is NOT

• Not "add anything as a DomainEntity." Some things are primitives and should remain in the core ontology. A Plane is a primitive — a first-class graph node with relations. Don't make a plane a DomainEntity with a template_id: 'plane'. The engine reasons over planes via graph traversal, not via type-template.
• Not a free-for-all. The type-template system is for content types (missions, campaigns, trade lots, rituals). The core types and the plane types are the engine's contract; they change via the ontology docs, not via runtime templates.

Extending the plane taxonomy (v1.2)

The plane model from 17-planes.md is open-ended: any kind value is valid, and any plane-relation type (REFLECTS, LAYER_OF, ADJACENT_TO, ACCESSIBLE_VIA, or new ones) is just a Cypher edge. The world-builder can add their cosmology's planes without code changes.

Adding a custom plane

To add a "Realm of Dreams" demiplane to a setting:

MERGE (s:Setting {id: 'eberron'})-[:HAS_PLANE]->(p:Plane {id: 'eberron.realm_of_dreams'})
SET p.name = 'The Realm of Dreams',
    p.kind = 'demiplane',
    p.summary = 'A demiplane entered only through sleep or specific fey magic.',
    p.accessible = true,
    p.alignment_tendency = 'chaotic_neutral'

The engine does not need to know kind: 'demiplane' in advance — the value is a string the LLM can reason over. The plugin layer exposes tools like find_planes_by_kind(kind='demiplane') that match on the string.

Adding a custom plane-relation

To add BORDERS (a less-common D&D cosmology relation: two planes that share a physical border):

MATCH (a:Plane {id: 'eberron.shadowfell'}), (b:Plane {id: 'eberron.feywild'})
MERGE (a)-[:BORDERS]->(b)

The plugin's allowlist of relation types is data-driven (per the existing extensibility pattern). The world-builder adds the new relation to the allowlist via the OntologyRule mechanism, and the LLM starts seeing it.

Per-setting customization

The plane taxonomy is per-setting because plane-ids are setting-scoped (eberron.material vs mardonari.material). A setting that needs more planes just adds them. A setting that doesn't need a Shadowfell doesn't have one. The LLM's list_planes(setting_id='eberron') returns whatever exists for that setting.

• Not a constraint-free system. The template defines the constraints. The engine enforces them at ingest time and the consistency engine enforces them at write time. The LLM sees a typed system; it just doesn't have to write the type definition.
• Not a workaround for bad schema design. If a domain really is Person or Faction, model it as such. The DomainEntity wrapper is for things that aren't core types.