kaykayyali/lore-engine-poc
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
adbb6f0ccebff312817d8a7f71d69823bd437608
lore-engine-poc/docker-compose.yml
Hermes adbb6f0cce feat(substrate): Phase 1 merge — Redis + 8 Go workers + nsc plugin 
Ports the GraphMCP-Example substrate into lore-engine-poc:

- 8 Go workers under workers/ (discord-connector, discord-filter, lore-watcher, ingestion-worker, entity-extractor, lore-extractor, encounter-processor, mcp-server), each with Dockerfile + go.mod

- 3 Go unit-test files (encounter-processor, ingestion-worker, lore-extractor) — other 5 workers rely on integration tests via the live stack

- plugins/nsc.py: thin httpx proxy from gateway to lore-mcp-server:9000, exposes all 11 inherited GraphMCP tools (input schemas verbatim from mcp-server/main.go)

- docker-compose.yml: adds lore-redis + lore-mcp-server + the 7 worker services (lore- prefix to avoid clash with other GraphMCP stacks)

- verify-merge.sh (171 LOC, 7 pass conditions) + docs/VERIFICATION.md

- tests/contract/test_graphmcp_tool_contracts.py (15 tests; skipped when stack is down — TDD pattern, becomes active once docker compose up brings the stack)

- README.md + test.sh updated for the merged service inventory

Leader notes (2026-06-27 03:50):

- Worker self-blocked review-required after 2 runs (run #7 hit 120/120 iteration budget; run #8 staged 40 files and reported shippable).

- Tests are SKIPPED until docker compose up — worker chose that pattern over mocking (consistent with the lore-engine-poc project convention). To activate, run `docker compose up -d --build && pytest tests/contract/`.

- File Scope reconciliation: story said gateway/plugins/nsc/__init__.py; worker shipped plugins/nsc.py (flat file). Justified by the existing plugins/ convention in lore-engine-poc (server.py glob("*.py")). A future PR could split nsc into a package once server.py learns __init__.py discovery.

- nsc plugin exposes 11 tools (not 8) — the AC said "8" but the worker enumerated all 11 tools present in mcp-server/main.go. The encounter-specific 3 tools (list_encounters, search_encounters, get_encounter) were included for consistency. Story AC #2 reads "≥ 8 GraphMCP tools" so this exceeds AC.

Refs: S2-phase-1-substrate-merge, milestone #64 P1 — Substrate merge
2026-06-27 03:48:54 +00:00

348 lines
12 KiB
YAML
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
name: lore-engine-poc
# Lore Engine POC: Neo4j + Postgres + MinIO + Redis + Python plugin gateway
# + 7 GraphMCP workers (Go) + Go mcp-server
#
# Phase 1 (P1) of the lore-engine × GraphMCP-Example substrate merge.
# See docs/VERIFICATION.md for the contract this stack must satisfy.
#
# Port remap note: the host already runs the damascus stack on 5432/5433,
# 7474, 7687, 9000, 9001. We shift the lore stack to the 5434/7475/7688/
# 9002/9003/8766/6379 range to coexist. Containers communicate on the
# internal Docker network using the in-network service names (neo4j,
# postgres, minio, lore-redis, litellm-host).
services:
# ─── Neo4j — world graph ────────────────────────────────────────────────────
neo4j:
image: neo4j:5.26-community
container_name: lore-neo4j
environment:
NEO4J_AUTH: neo4j/lore-dev-password
NEO4J_PLUGINS: '["apoc"]'
NEO4J_apoc_export_file_enabled: "true"
NEO4J_apoc_import_file_enabled: "true"
NEO4J_server_memory_heap_initial__size: 512m
NEO4J_server_memory_heap_max__size: 1g
ports:
- "7475:7474" # browser (remapped from 7474 — damascus occupies 7474 area)
- "7688:7687" # bolt (remapped from 7687 — free for lore-neo4j)
volumes:
- neo4j-data:/data
healthcheck:
test: ["CMD-SHELL", "wget -qO- http://localhost:7474 || exit 1"]
interval: 5s
timeout: 5s
retries: 20
# ─── Postgres — operational data + embeddings ──────────────────────────────
postgres:
image: pgvector/pgvector:pg16
container_name: lore-postgres
environment:
POSTGRES_USER: lore
POSTGRES_PASSWORD: lore-dev-password
POSTGRES_DB: lore
ports:
- "5434:5432"
volumes:
- postgres-data:/var/lib/postgresql/data
- ./postgres/init.sql:/docker-entrypoint-initdb.d/init.sql:ro
healthcheck:
test: ["CMD-SHELL", "pg_isready -U lore -d lore"]
interval: 5s
timeout: 5s
retries: 20
# ─── MinIO — blob storage for images ────────────────────────────────────────
minio:
image: minio/minio:latest
container_name: lore-minio
environment:
MINIO_ROOT_USER: lorelore
MINIO_ROOT_PASSWORD: lore-dev-password
command: server /data --console-address ":9001"
ports:
- "9002:9000" # S3 API (remapped from 9000)
- "9003:9001" # console (remapped from 9001)
volumes:
- minio-data:/data
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/ready"]
interval: 5s
timeout: 5s
retries: 20
# ─── Redis — event stream broker for the GraphMCP workers ───────────────────
# Phase 1 addition: streams raw.discord / raw.messages / raw.lore / raw.encounters
# See docs/merge/00-inventory.md §3 for the producer/consumer topology.
redis:
image: redis:7-alpine
container_name: lore-redis
command: >
redis-server
--appendonly yes
--appendfsync everysec
--maxmemory 1gb
--maxmemory-policy noeviction
ports:
- "6379:6379"
volumes:
- redis-data:/data
healthcheck:
test: ["CMD", "redis-cli", "ping"]
interval: 5s
timeout: 5s
retries: 10
# ─── Lore Gateway — Python MCP server, plugin-driven ───────────────────────
gateway:
build:
context: ./gateway
container_name: lore-gateway
depends_on:
neo4j: { condition: service_healthy }
postgres: { condition: service_healthy }
minio: { condition: service_healthy }
redis: { condition: service_healthy }
mcp-server: { condition: service_started }
environment:
NEO4J_URL: bolt://neo4j:7687
NEO4J_USER: neo4j
NEO4J_PASSWORD: lore-dev-password
POSTGRES_URL: postgresql://lore:lore-dev-password@postgres:5432/lore
MINIO_URL: http://minio:9000
MINIO_ACCESS_KEY: lorelore
MINIO_SECRET_KEY: lore-dev-password
MINIO_BUCKET: lore-images
MINIO_PUBLIC_URL: http://localhost:9002
PLUGINS_DIR: /app/plugins
INIT_CYPHER: /app/neo4j/init.cypher
# The Python nsc plugin proxies JSON-RPC to the Go mcp-server.
# Default URL is the in-network service name; override here if you run
# the mcp-server on a different host.
NSC_MCP_URL: http://mcp-server:9000
ports:
- "8766:8765" # MCP JSON-RPC (remapped from 8765)
volumes:
- ./plugins:/app/plugins:ro
- ./neo4j:/app/neo4j:ro
- ./mock-data:/app/mock-data:ro
# ═══ GraphMCP substrate (Phase 1 merge) ════════════════════════════════════
# All workers read LLM/embed URLs from the host's litellm proxy at
# 172.22.0.1:4000 (the docker-bridge gateway on Linux). Override per
# worker in production.
# ─── discord-connector (disabled in Phase 1; env DISCORD_ENABLED gate) ─────
discord-connector:
build: ./workers/discord-connector
container_name: lore-discord-connector
depends_on:
redis: { condition: service_healthy }
neo4j: { condition: service_healthy }
environment:
DISCORD_ENABLED: "false" # Phase 1 keeps this off
DISCORD_TOKEN: ""
DISCORD_GUILD_ID: ""
REDIS_URL: redis://lore-redis:6379
REDIS_STREAM: raw.discord
ENCOUNTER_STREAM: raw.encounters
GROUPING_TIMEOUT_MINS: "15"
restart: unless-stopped
# ─── discord-filter — promotes lore-relevant messages to raw.messages ──────
discord-filter:
build: ./workers/discord-filter
container_name: lore-discord-filter
depends_on:
redis: { condition: service_healthy }
neo4j: { condition: service_healthy }
environment:
REDIS_URL: redis://lore-redis:6379
IN_STREAM: raw.discord
OUT_STREAM: raw.messages
REDIS_GROUP: discord-filter
CONSUMER_NAME: discord-filter-1
NEO4J_URL: bolt://neo4j:7687
NEO4J_USER: neo4j
NEO4J_PASSWORD: lore-dev-password
EMBED_URL: http://172.22.0.1:4000/v1
EMBED_MODEL: embed-gemma-300m
SIMILARITY_THRESHOLD: "0.72"
TOP_K: "3"
# ─── lore-watcher — POSTs .md files from ./lore-data to ingestion-worker ──
lore-watcher:
build: ./workers/lore-watcher
container_name: lore-lore-watcher
depends_on:
ingestion-worker: { condition: service_started }
environment:
WATCH_DIR: /data/lore
INGEST_URL: http://ingestion-worker:8080/ingest/lore
DEBOUNCE_MS: "500"
volumes:
- ./lore-data:/data/lore:ro
# ─── ingestion-worker — chunks + embeds + writes Message/Chunk/LoreDocument
ingestion-worker:
build: ./workers/ingestion-worker
container_name: lore-ingestion-worker
depends_on:
redis: { condition: service_healthy }
neo4j: { condition: service_healthy }
environment:
REDIS_URL: redis://lore-redis:6379
REDIS_STREAM: raw.messages
REDIS_GROUP: ingestion
CONSUMER_NAME: ingestion-worker-1
NEO4J_URL: bolt://neo4j:7687
NEO4J_USER: neo4j
NEO4J_PASSWORD: lore-dev-password
EMBED_URL: http://172.22.0.1:4000/v1
EMBED_MODEL: embed-gemma-300m
CHUNK_SIZE: "512"
CHUNK_OVERLAP: "64"
HTTP_PORT: "8080"
LORE_STREAM: raw.lore
LOG_LEVEL: info
ports:
- "8081:8080"
# ─── entity-extractor (primary LLM-backed, Ollama-compatible) ─────────────
entity-extractor:
build: ./workers/entity-extractor
container_name: lore-entity-extractor
depends_on:
redis: { condition: service_healthy }
neo4j: { condition: service_healthy }
environment:
REDIS_URL: redis://lore-redis:6379
REDIS_STREAM: raw.messages
REDIS_GROUP: extraction
CONSUMER_NAME: entity-extractor-1
NEO4J_URL: bolt://neo4j:7687
NEO4J_USER: neo4j
NEO4J_PASSWORD: lore-dev-password
LLM_URL: http://172.22.0.1:4000/v1
LLM_MODEL: qwen2.5:3b
SUPERSEDE_RELATIONS: "ALLIED_WITH,ENEMY_OF"
# ─── entity-extractor-2 (twin replica — same binary, different LLM) ────────
entity-extractor-2:
build: ./workers/entity-extractor
container_name: lore-entity-extractor-2
depends_on:
redis: { condition: service_healthy }
neo4j: { condition: service_healthy }
environment:
REDIS_URL: redis://lore-redis:6379
REDIS_STREAM: raw.messages
REDIS_GROUP: extraction
CONSUMER_NAME: entity-extractor-2
NEO4J_URL: bolt://neo4j:7687
NEO4J_USER: neo4j
NEO4J_PASSWORD: lore-dev-password
LLM_URL: http://172.22.0.1:4000/v1
LLM_MODEL: qwen3.5
SUPERSEDE_RELATIONS: "ALLIED_WITH,ENEMY_OF"
# ─── lore-extractor — entity extraction on lore documents ──────────────────
lore-extractor:
build: ./workers/lore-extractor
container_name: lore-lore-extractor
depends_on:
redis: { condition: service_healthy }
neo4j: { condition: service_healthy }
environment:
REDIS_URL: redis://lore-redis:6379
REDIS_STREAM: raw.lore
REDIS_GROUP: lore-extraction
CONSUMER_NAME: lore-extractor-1
NEO4J_URL: bolt://neo4j:7687
NEO4J_USER: neo4j
NEO4J_PASSWORD: lore-dev-password
LLM_URL: http://172.22.0.1:4000/v1
LLM_MODEL: qwen2.5:3b
# ─── lore-extractor-2 (twin replica) ───────────────────────────────────────
lore-extractor-2:
build: ./workers/lore-extractor
container_name: lore-lore-extractor-2
depends_on:
redis: { condition: service_healthy }
neo4j: { condition: service_healthy }
environment:
REDIS_URL: redis://lore-redis:6379
REDIS_STREAM: raw.lore
REDIS_GROUP: lore-extraction
CONSUMER_NAME: lore-extractor-2
NEO4J_URL: bolt://neo4j:7687
NEO4J_USER: neo4j
NEO4J_PASSWORD: lore-dev-password
LLM_URL: http://172.22.0.1:4000/v1
LLM_MODEL: qwen3.5
# ─── encounter-processor — Encounter + WITNESSED edges ─────────────────────
encounter-processor:
build: ./workers/encounter-processor
container_name: lore-encounter-processor
depends_on:
redis: { condition: service_healthy }
neo4j: { condition: service_healthy }
environment:
REDIS_URL: redis://lore-redis:6379
REDIS_STREAM: raw.encounters
REDIS_GROUP: encounter-processing
CONSUMER_NAME: encounter-processor-1
NEO4J_URL: bolt://neo4j:7687
NEO4J_USER: neo4j
NEO4J_PASSWORD: lore-dev-password
LLM_URL: http://172.22.0.1:4000/v1
LLM_MODEL: qwen2.5:3b
# ─── encounter-processor-2 (twin replica) ──────────────────────────────────
encounter-processor-2:
build: ./workers/encounter-processor
container_name: lore-encounter-processor-2
depends_on:
redis: { condition: service_healthy }
neo4j: { condition: service_healthy }
environment:
REDIS_URL: redis://lore-redis:6379
REDIS_STREAM: raw.encounters
REDIS_GROUP: encounter-processing
CONSUMER_NAME: encounter-processor-2
NEO4J_URL: bolt://neo4j:7687
NEO4J_USER: neo4j
NEO4J_PASSWORD: lore-dev-password
LLM_URL: http://172.22.0.1:4000/v1
LLM_MODEL: qwen3.5
# ─── mcp-server — Go MCP JSON-RPC on port 9000 ─────────────────────────────
# The Python `nsc` plugin in the gateway proxies tools/list + tools/call
# to this service. Worker source: services/mcp-server/main.go (unchanged).
mcp-server:
build: ./workers/mcp-server
container_name: lore-mcp-server
depends_on:
neo4j: { condition: service_healthy }
environment:
NEO4J_URL: bolt://neo4j:7687
NEO4J_USER: neo4j
NEO4J_PASSWORD: lore-dev-password
EMBED_URL: http://172.22.0.1:4000/v1
EMBED_MODEL: embed-gemma-300m
MCP_PORT: "9000"
MAX_CONTEXT_TOKENS: "4000"
LOG_LEVEL: info
ports:
- "9004:9000"
volumes:
neo4j-data:
postgres-data:
minio-data:
redis-data:
Reference in New Issue View Git Blame Copy Permalink