lore-engine-poc/docs/merge/00-inventory.md

Phase 0 — GraphMCP-Example Substrate Inventory

Canonical catalog of every moving part in the GraphMCP-Example substrate (/root/GraphMCP-Example, pinned commit 064daa9).

Phase 0 of the lore-engine × GraphMCP merge. Companion to the merge architecture: planning-artifacts/architecture.md.

Story: lore-engine-merge-prds/_bmad-output/planning-artifacts/stories/S1-phase-0-inventory.md Enforced by: tests/test_inventory_completeness.py (RED→GREEN gate)

This inventory is the gate: nothing else in the merge ships until every worker, MCP tool, and Redis stream here is enumerated and verified. Downstream phases (S2–S7) read this doc to decide what to preserve, what to replace, and what to deprecate.

Dual-LLM arbitration note. Three workers run in twin replicas sharing the same Dockerfile and Go binary: entity-extractor/entity-extractor-2, lore-extractor/lore-extractor-2, encounter-processor/encounter-processor-2. The -2 replicas target a different LLM endpoint (the Lemonade NPU at 100.77.136.12:11434, model qwen3.5) than the primary (local Ollama CPU at ollama-cpu:11435, model qwen2.5:3b). They are documented here as one logical pair per worker.

Throughput note. Expected per-stream throughput is TBD for all four streams — no baseline measurement exists in the repo. Add real numbers once the stack has run under load for a week.

---

1. Workers (10 Go binaries)

1.1 discord-connector

• Container: discord-connector
• Source: services/discord-connector/main.go (357 lines)
• Purpose: Streams live Discord MESSAGE_CREATE events into raw.discord; groups messages by channel + time window, emits one Encounter-shaped message to raw.encounters when the window closes.
• Env vars:
  • DISCORD_TOKEN (required) — bot token
  • DISCORD_GUILD_ID (required) — server ID
  • DISCORD_CHANNELS (* or comma-separated IDs, default *)
  • BACKFILL_LIMIT (default 100, set 0 to skip backfill)
  • GROUPING_TIMEOUT_MINS (default 15) — conversation window length
  • REDIS_URL (default redis://redis:6379)
  • REDIS_STREAM (default raw.messages — note: written stream)
  • ENCOUNTER_STREAM (default raw.encounters)
• Streams read: none (gateway source; reads from Discord gateway)
• Streams written: raw.discord (every message), raw.encounters (one per closed window)
• Cypher queries emitted: none (Redis-only worker)
• LLM call sites: none
• Notable implementation details:
  • Dedup via SET NX discord:seen:<id> EX 604800 (7-day TTL).
  • Window keys: discord:group:<channelID>:{first_ts,last_ts,authors} with TTL = GROUPING_TIMEOUT_MINS.
  • Flush lock: discord:group:<channelID>:flushing (30s TTL) prevents duplicate encounter emission during rolling restarts.
  • Backfill rate-limits with time.Sleep(500ms) between pagination calls.

1.2 discord-filter

• Container: discord-filter
• Source: services/discord-filter/main.go (395 lines)
• Purpose: Consumes raw.discord, embeds each message via Ollama, runs ANN search against the lore_chunk_embeddings vector index, and promotes lore-relevant messages to raw.messages. Always writes a :DiscordMessage node regardless of promotion.
• Env vars:
  • REDIS_URL (default redis://redis:6379)
  • IN_STREAM (default raw.discord)
  • OUT_STREAM (default raw.messages)
  • REDIS_GROUP (default discord-filter)
  • CONSUMER_NAME (default discord-filter-1)
  • NEO4J_URL (default bolt://neo4j:7687)
  • EMBED_URL (default http://ollama-gpu:11434)
  • EMBED_MODEL (default nomic-embed-text)
  • SIMILARITY_THRESHOLD (default 0.72) — cosine floor vs lore chunks
  • TOP_K (default 3) — ANN neighbors
• Streams read: raw.discord
• Streams written: raw.messages (only promoted messages)
• Cypher queries emitted:
  • MATCH (p:Person) WHERE p.source = 'lore' RETURN p.name (refreshed every 5 min, name cache)
  • MERGE (m:DiscordMessage {id: $id}) ON CREATE SET ... SET m.promoted, m.match_score, m.match_reason
  • CALL db.index.vector.queryNodes('lore_chunk_embeddings', $topK, $embedding) YIELD node, score RETURN score ORDER BY score DESC LIMIT 1 (ANN against lore chunks)
• LLM call sites: HTTP POST to $EMBED_URL/v1/embeddings (every message). No chat-completion calls.
• Decision logic: promote if score >= SIMILARITY_THRESHOLD (reason embedding:<score>) OR if any cached lore Person name is a substring of the lower-cased message (reason name_match:<name>).

1.3 lore-watcher

• Container: lore-watcher
• Source: services/lore-watcher/main.go (233 lines)
• Purpose: Watches ./lore-data/ on the host (bind-mounted at /data/lore) for .md file creates/changes. Hashes each file with SHA-256, skips unchanged, and POSTs the file as multipart to the ingestion worker's /ingest/lore endpoint. State persists in .lore-watcher-state.json inside the watch dir.
• Env vars:
  • WATCH_DIR (default /data/lore)
  • INGEST_URL (default http://ingestion-worker:8080/ingest/lore)
  • DEBOUNCE_MS (default 500)
• Streams read: none
• Streams written: indirectly triggers raw.lore via ingestion-worker (the watcher doesn't write the stream itself)
• Cypher queries emitted: none
• LLM call sites: none
• Notable: uses fsnotify per-file time.AfterFunc debouncing, ignores dotfiles, .swp, ~, .tmp, 4913 (vim swap), and any non-.md extension. Recursively adds new subdirectories to the watch set. Content fingerprint is sha256 (lowercase hex).

1.4 ingestion-worker

• Container: ingestion-worker
• Source: services/ingestion-worker/main.go (699 lines)
• Purpose: Consumes raw.messages, chunks each message (default 512 chars, 64 overlap), embeds via GPU Ollama, writes :Message and :Chunk nodes with float32 embeddings. Also exposes an HTTP server on port 8080 with POST /ingest/lore — accepts a markdown upload, parses it into :LoreDocument + :LoreChunk nodes, publishes the doc to raw.lore.
• Env vars:
  • REDIS_URL, REDIS_STREAM (default raw.messages), REDIS_GROUP (default ingestion), CONSUMER_NAME (default ingestion-worker-1)
  • NEO4J_URL, NEO4J_USER, NEO4J_PASSWORD
  • EMBED_URL (default http://ollama-gpu:11434), EMBED_MODEL (default nomic-embed-text)
  • CHUNK_SIZE (default 512), CHUNK_OVERLAP (default 64)
  • HTTP_PORT (default 8080)
  • LORE_STREAM (default raw.lore)
  • LOG_LEVEL (default info)
• Streams read: raw.messages
• Streams written: raw.lore (when /ingest/lore is called)
• Cypher queries emitted (constants at top of file):
  • Message chunk write: MERGE (m:Message {id: $msgID}) → MERGE (c:Chunk {id: $chunkID}) → MERGE (m)-[:HAS_CHUNK]->(c)
  • Lore doc write: MERGE (d:LoreDocument {id: $docID}) → MERGE (c:LoreChunk {id: $chunkID}) → MERGE (d)-[:HAS_CHUNK]->(c)
• LLM call sites: HTTP POST to $EMBED_URL/v1/embeddings (every chunk). No chat-completion calls.
• HTTP server: POST /ingest/lore (multipart upload), also serves per-message POST from lore-watcher.

1.5 entity-extractor (and -2)

• Containers: entity-extractor, entity-extractor-2
• Source: services/entity-extractor/main.go (567 lines) — both replicas use the same binary; only env vars differ
• Purpose: Consumes raw.messages, calls the LLM with an entity + relation extraction prompt (Person/Location/Faction/Event/Item/Creature, 11 relation types), writes :Entity nodes with dynamic labels via APOC and :Message-[:MENTIONS]->:Entity edges. Also writes :Person-[:POSTED]->:Message for the author and merges relations via apoc.merge.relationship. Exclusive relation types (default ALLIED_WITH,ENEMY_OF) supersede prior outgoing edges from the same source.
• Env vars (primary):
  • REDIS_STREAM (default raw.messages), REDIS_GROUP (extraction), CONSUMER_NAME (entity-extractor-1)
  • LLM_URL (default http://ollama-cpu:11435), LLM_MODEL (default qwen2.5:3b)
  • PROMPT_FILE (optional override of the default system prompt)
  • SUPERSEDE_RELATIONS (default ALLIED_WITH,ENEMY_OF)
• Env vars (-2 replica):
  • LLM_URL=http://100.77.136.12:11434 (remote Lemonade NPU)
  • LLM_MODEL=qwen3.5
  • CONSUMER_NAME=entity-extractor-2
• Streams read: raw.messages
• Streams written: none
• Cypher queries emitted:
  • MERGE (m:Message {id: $msgID}) + UNWIND $entities AS ent → MERGE (e {name: ent.name}) ON CREATE SET e.type, e.source → CALL apoc.create.addLabels(e, [ent.type]) → MERGE (m)-[:MENTIONS]->(node)
  • MERGE (p:Person {id: $authorID}) ON CREATE SET p.name → MERGE (m:Message {id: $msgID}) MERGE (p)-[:POSTED]->(m)
  • MATCH (a {name: $from}) MATCH (b {name: $to}) → CALL apoc.merge.relationship(a, $rel, {}, {}, b) YIELD rel SET rel.since, rel.msg_id
  • Supersede: MATCH (a {name: $from})-[r]->() WHERE type(r) = $rel AND NOT coalesce(r.superseded, false) SET r.superseded = true, r.superseded_by = $msgID
• LLM call sites: HTTP POST to $LLM_URL/v1/chat/completions with the default extraction prompt (Person/Location/Faction/Event/Item/Creature + 11 relation types). One call per message.
• Prompt: defined inline as defaultSystemPrompt in services/entity-extractor/main.go:72-116.

1.6 lore-extractor (and -2)

• Containers: lore-extractor, lore-extractor-2
• Source: services/lore-extractor/main.go (656 lines) — both replicas share the binary
• Purpose: Consumes raw.lore, calls the LLM with a lore-focused extraction prompt (Person/Location/Event/Faction/Item/Creature + same 11 relation types), writes entities with e.lore_verified = true, links them back to :LoreDocument via FEATURES, and detects :Contradiction nodes between documents that disagree.
• Env vars (primary):
  • REDIS_STREAM (default raw.lore), REDIS_GROUP (lore-extraction), CONSUMER_NAME (lore-extractor-1)
  • LLM_URL (default http://ollama-cpu:11435), LLM_MODEL (default qwen2.5:3b)
  • PROMPT_FILE (optional override)
• Env vars (-2 replica):
  • LLM_URL=http://100.77.136.12:11434, LLM_MODEL=qwen3.5
  • CONSUMER_NAME=lore-extractor-2
• Streams read: raw.lore
• Streams written: none
• Cypher queries emitted:
  • Pre-check: MATCH (d:LoreDocument)-[:FEATURES]->(e) RETURN ... to know what entities already exist for the doc
  • MERGE (d:LoreDocument {id: $docID}) → UNWIND $entities AS ent → MERGE (e {name: ent.name}) ON MATCH SET e.lore_verified = true → MERGE (d)-[:FEATURES]->(node)
  • Relation merge (same APOC pattern as entity-extractor)
  • Contradiction detection: MATCH (a)-[r1]->(x) MATCH (a)-[r2]->(y) where both rels have the same canonical predicate but different targets → MERGE (contra:Contradiction {subject, claim_a, claim_b}) → MERGE (a)-[:HAS_CONTRADICTION]->(contra)
• LLM call sites: HTTP POST to $LLM_URL/v1/chat/completions with the default lore-extraction prompt. One call per lore document.
• Prompt: defined inline as defaultSystemPrompt in services/lore-extractor/main.go:53-.

1.7 encounter-processor (and -2)

• Containers: encounter-processor, encounter-processor-2
• Source: services/encounter-processor/main.go (530 lines) — both replicas share the binary
• Purpose: Consumes raw.encounters, creates :Encounter nodes with :WITNESSED edges for each participant (linking via Person.name or fallback Person.id), OCCURRED_AT to a :Location, and calls the LLM on the encounter summary to extract FEATURED entity links.
• Env vars (primary):
  • REDIS_STREAM (default raw.encounters), REDIS_GROUP (encounter-processing), CONSUMER_NAME (encounter-processor-1)
  • LLM_URL (default http://ollama-cpu:11435), LLM_MODEL (default qwen2.5:3b)
• Env vars (-2 replica):
  • LLM_URL=http://100.77.136.12:11434, LLM_MODEL=qwen3.5
  • CONSUMER_NAME=encounter-processor-2
• Streams read: raw.encounters
• Streams written: none
• Cypher queries emitted:
  • MERGE (enc:Encounter {id: $id}) then cascade entity normalisation
    • link (MATCH (e), MATCH (e {name: $canonical}))
  • MERGE (p:Person {name: $name}) MERGE (enc:Encounter {id: $encID}) MERGE (p)-[w:WITNESSED]->(enc)
  • MATCH (loc {name: $canonical}) MATCH (enc:Encounter {id: $encID}) MERGE (enc)-[:OCCURRED_AT]->(loc) (fallback: MERGE (loc:Location {name: $location}) MERGE (enc)-[:OCCURRED_AT]->(loc))
  • MATCH (enc:Encounter {id: $encID}) + UNWIND $entities AS ent → MERGE (e {name: ent.name}) MERGE (enc)-[:FEATURED]->(node)
• LLM call sites: HTTP POST to $LLM_URL/v1/chat/completions with the encounterSystemPrompt. One call per encounter to extract FEATURED entities.
• Prompt: defined inline as encounterSystemPrompt in services/encounter-processor/main.go:85-.

1.8 mcp-server

• Container: mcp-server
• Source: services/mcp-server/main.go (1435 lines)
• Purpose: Exposes the MCP tool surface (11 tools) to clients over HTTP + SSE on port 9000. Embeds user queries via GPU Ollama, runs Cypher traversals and ANN searches against Neo4j.
• Env vars:
  • NEO4J_URL (default bolt://neo4j:7687), NEO4J_USER, NEO4J_PASSWORD
  • EMBED_URL (default http://ollama-gpu:11434), EMBED_MODEL (default nomic-embed-text)
  • MCP_PORT (default 9000)
  • MAX_CONTEXT_TOKENS (default 4000) — caps context for fast TTFT
  • LOG_LEVEL (default info)
• Streams read: none (synchronous query layer over Neo4j)
• Streams written: none
• Cypher queries emitted: see the per-tool rows in §2 — every tool handler runs its own Cypher.
• LLM call sites: HTTP POST to $EMBED_URL/v1/embeddings (every query that needs semantic search — semantic_search, query_as_npc).
• Wire protocol: MCP 2024-11-05 over GET /sse + POST /message?sessionId=X.

---

2. MCP Tools (11 tools)

All tool definitions live in mcp-server/main.go in the mcpTools slice declared at lines 137–268. The dispatcher at lines 316–341 maps each name to a handler.

Tool	Input schema (required)	Output shape	Handler location
semantic_search	query: string, limit?: int (default 5)	List of {id, content, score, source} chunks/messages	services/mcp-server/main.go:317 → handleSemanticSearch at :616
graph_traverse	entity: string, depth?: int (1–3, default 2)	{nodes: [...], relationships: [...]}	services/mcp-server/main.go:319 → handleGraphTraverse at :698
get_context	message_id: string	{message, chunks, entities, relations}	services/mcp-server/main.go:321 → handleGetContext at :735
get_person_profile	name: string	{topics, interests, message_history, co_occurring_persons}	services/mcp-server/main.go:323 → handleGetPersonProfile at :771
query_as_npc	npc_name: string, question: string, limit?: int (default 5)	{chunks: [...], encounters: [...]} — scoped to NPC's WITNESSED edges	services/mcp-server/main.go:325 → handleQueryAsNPC at :922
log_encounter	title: string, participants: string (CSV), summary: string, location?: string, type?: string (default conversation)	{id, title, participants, ...} of the created :Encounter	services/mcp-server/main.go:327 → handleLogEncounter at :812
get_unresolved	type?: string, limit?: int (default 30)	List of provisional entities (`:Person	Location
get_contradictions	subject?: string, limit?: int (default 20)	List of :Contradiction rows with claim_a/claim_b	services/mcp-server/main.go:331 → handleGetContradictions at :1225
list_encounters	limit?: int (default 10)	Encounters ordered by recency	services/mcp-server/main.go:333 → handleListEncounters at :1305
search_encounters	query?: string, location?: string, participant?: string, limit?: int (default 10)	Filtered encounters	services/mcp-server/main.go:335 → handleSearchEncounters at :1337
get_encounter	id: string	{encounter, participants, featured_entities}	services/mcp-server/main.go:337 → handleGetEncounter at :1402

Server identity: name graphmcp, version 1.0.0, protocol 2024-11-05. Set in dispatch() at services/mcp-server/main.go:359-360.

---

3. Redis Streams (4 streams)

All four streams use the same Redis instance (redis://redis:6379, AOF on with everysec fsync, maxmemory 1gb noeviction). Retention is governed by Redis memory limits rather than explicit XADD MAXLEN ~; no per-stream trimming is configured.

3.1 raw.discord

• Producers: discord-connector (every live MESSAGE_CREATE, plus every backfilled message). One XADD per message at services/discord-connector/main.go:218-229.
• Consumers: discord-filter (consumer group discord-filter).
• Message fields: id, content, author, timestamp, source=discord, channel_id, channel_name.
• Retention policy: implicit — bounded by Redis maxmemory 1gb noeviction (stream blocks writes if exhausted).
• Expected throughput: TBD — measure under realistic Discord load.

3.2 raw.messages

• Producers:
  • discord-filter (only promoted messages — those passing the lore similarity threshold or matching a known lore Person name). XADD at services/discord-filter/main.go:274-287.
  • The legacy discord-connector config still names raw.messages as its REDIS_STREAM default — see services/discord-connector/main.go:42; in the live stack the connector actually writes to raw.discord (the compose file overrides REDIS_STREAM: raw.discord for the connector). Treat the env var as historical.
• Consumers:
  • ingestion-worker (consumer group ingestion)
  • entity-extractor + entity-extractor-2 (consumer group extraction, two consumers in the same group share load)
• Message fields: id, content, author, timestamp, source, channel_id, channel_name (for Discord-sourced messages).
• Retention policy: implicit, Redis maxmemory.
• Expected throughput: TBD.

3.3 raw.lore

• Producers: ingestion-worker (via POST /ingest/lore HTTP handler, which calls rdb.XAdd at services/ingestion-worker/main.go:534). In practice the lore-watcher triggers these uploads but does not write the stream directly.
• Consumers: lore-extractor + lore-extractor-2 (consumer group lore-extraction, two consumers).
• Message fields: document-level fields set by the ingestion HTTP handler (docID, title, filename, content, etc.).
• Retention policy: implicit, Redis maxmemory.
• Expected throughput: TBD — bounded by human-driven drops of markdown files into ./lore-data/.

3.4 raw.encounters

• Producers:
  • discord-connector (one XADD per closed conversation window, at services/discord-connector/main.go:298-312)
  • Future: mcp-server log_encounter tool may also publish here, but the current handler writes directly to Neo4j and does NOT XADD.
• Consumers: encounter-processor + encounter-processor-2 (consumer group encounter-processing, two consumers).
• Message fields: id, title, type=conversation, location=channel_name, participants (CSV), summary, timestamp=first_message_ts.
• Retention policy: implicit, Redis maxmemory.
• Expected throughput: TBD.

---

4. Topology snapshot

                ┌──────────────────────────┐
                │   Discord Gateway         │
                └────────┬─────────────────┘
                         │ MESSAGE_CREATE / backfill
                         ▼
                ┌──────────────────────────┐
                │   discord-connector       │ ── raw.discord
                └────────┬─────────────────┘
                         ▼
                ┌──────────────────────────┐     ┌────────────────────────┐
                │   discord-filter          │ ──▶ │   raw.messages (XADD)  │
                │   (embed + ANN + name)    │     └────┬───────────────────┘
                └──────────────────────────┘          │
                                                     │
                            ┌────────────────────────┴────────────┐
                            ▼                                     ▼
                   ┌────────────────────┐              ┌────────────────────┐
                   │ ingestion-worker   │              │ entity-extractor   │
                   │  group: ingestion  │              │  group: extraction │
                   │                    │              │ + entity-extractor-2│
                   │ writes Chunk nodes │              │ (writes Entities,  │
                   │ + raw.lore (HTTP)  │              │  Mentions, rels)   │
                   └─────────┬──────────┘              └────────────────────┘
                             │ raw.lore
                             ▼
                    ┌────────────────────┐         ┌────────────────────┐
                    │ lore-extractor     │         │ encounter-processor│
                    │  group:            │         │  group: encounter- │
                    │   lore-extraction  │         │   processing       │
                    │ + lore-extractor-2 │         │ + encounter-       │
                    │ (FEATURES, Contra) │         │   processor-2      │
                    └────────────────────┘         └────────────────────┘
                             ▲                                ▲
                             │                                │
                             │                                │ raw.encounters
                             │                                │
                             │ (indirect)                     │
                  lore-watcher → POST /ingest/lore            │
                                                               │
                     discord-connector ──────────────┬────────┘
                     (15-min conversation window)    │
                                                    │
                                              raw.encounters

                       ┌────────────────────┐
                       │   mcp-server       │ ◀── HTTP+SSE from clients
                       │  (11 MCP tools,    │     (e.g. lore-engine bot)
                       │   reads Neo4j      │
                       │   only — no        │
                       │   stream I/O)      │
                       └────────────────────┘

---

5. Cross-reference

• Phase 0 story: lore-engine-merge-prds/_bmad-output/planning-artifacts/stories/S1-phase-0-inventory.md
• PRD: meta/prd.md
• Architecture: planning-artifacts/architecture.md
• Epics: meta/epics.md
• Test enforcing this inventory: tests/test_inventory_completeness.py
• Source pinned at GraphMCP-Example commit 064daa9.