docs(adr): 0008 — graph backend is Neo4j (overrides Cognee's Kuzu default)

Cognee's default graph DB is Kuzu (PR #1022, June 2025). We override to Neo4j for battle-testedness + Java UDFs. The time model (time_in_window/overlap, era tree, current token) now ships as a Neo4j Java UDF — queryable inline in Cypher, the contract was_true_at depends on. Kuzu has no Java UDF mechanism, so on Kuzu the time model would have been app-layer Python called outside the query. The POC's pure-Python port becomes the reference impl + test oracle. Resolves the 'verify which backend' hedge that was never executed: - 08-architecture.md: storage line, diagram, time-model impl, hosting - 22-cognee-boundary.md: storage ownership - CONTEXT.md: Cognee entry pinned to Neo4j - 00-overview.md: ADR index Post-cognify consistency hook (Q10 second half): Cognee exposes no built-in hook. 04-consistency.md updated — live sweep runs as a final Task in run_custom_pipeline (post-ingest, <100ms); nightly full sweep runs as external cron. Both write the same :ConsistencyRun shape. Co-Authored-By: Claude <noreply@anthropic.com>
2026-06-17 22:55:21 -04:00
parent a5cf55bbf0
commit 0f939b3bdb
6 changed files with 104 additions and 12 deletions
--- a/CONTEXT.md
+++ b/CONTEXT.md
@@ -61,7 +61,7 @@ A declared revision that supersedes — never deletes — an old fact. Old edges
 A record of one execution of the consistency engine (started_at, rules_run, violations_found). The LLM checks `latest_run()` before answering historical questions.

 **Cognee**:
-The substrate — the MIT-licensed framework that owns storage (Neo4j/Kuzu), extraction, embedding, and the `remember`/`recall`/`forget` API. Pinned at v1.1.2 (ADR 0006).
+The substrate — the MIT-licensed framework that owns extraction, embedding, and the `remember`/`recall`/`forget` API. Pinned at v1.1.2 (ADR 0006). Graph backend pinned to Neo4j, overriding Cognee's Kuzu default (ADR 0008).

 ## Avoid list (global)

--- a/docs/00-overview.md
+++ b/docs/00-overview.md
@@ -108,5 +108,6 @@ If you want to challenge it: jump to `10-critique.md` first. I tried to break it
 - `0005-primary-llm-minimax-m3.md` — primary LLM is Minimax-M3
 - `0006-cognee-version-pin.md` — Cognee pinned at 1.1.2; harness is the upgrade gate
 - `0007-graph-model-ontology-contract.md` — ontology enforced via `graph_model=`, not RDF/OWL
+- `0008-graph-backend-neo4j.md` — Neo4j (not Cognee's Kuzu default); time model ships as Java UDF

 **Build plan (`plan/`):** the 10 slice specs and an index.
--- a/docs/04-consistency.md
+++ b/docs/04-consistency.md
@@ -125,15 +125,26 @@ These produce `:Orphan` nodes with a `reason` field. The LLM is told "we don't k

 ## Running the consistency engine

-Two modes:
+Cognee exposes no built-in post-`cognify` hook (ADR 0008). The
+consistency engine attaches via two mechanisms:

-### Live (synchronous)
+### Live (in-process, post-ingest)

-Some checks run on every write. `MATCH (p:Person {lore_verified: true}) SET ...` triggers a quick anachronism scan on the affected entity. Latency target: <100ms.
+A fast sweep runs as the final `Task` in `run_custom_pipeline(tasks=[...])`,
+right after extraction. It scans only the entities touched by the
+ingest, materializes `Contradiction`/`Anachronism`/`Orphan` nodes
+while the pipeline context is warm, and targets <100ms. This is
+what makes a freshly-ingested chunk's violations visible before the
+LLM is ever asked about them.

-### Batch (nightly)
+### Batch (nightly, external)

-The full rule set runs at 03:00 wall-clock time. A new `:ConsistencyRun` node records the run, and any new violations are materialized.
+The full rule set runs at 03:00 wall-clock as an external scheduled
+job (cron), independent of any single ingest — re-scans the whole
+graph for long-tail rules and cross-entity patterns the live sweep
+doesn't cover. A new `:ConsistencyRun` node records the run, and
+any new violations are materialized. Both modes write the same
+`:ConsistencyRun` shape.

 ```
 (:ConsistencyRun {
--- a/docs/08-architecture.md
+++ b/docs/08-architecture.md
@@ -1,6 +1,6 @@
 # 08 — Architecture

-The Lore Engine is a **domain layer** on top of [Cognee](https://github.com/topoteretes/cognee), a self-hosted, MIT-licensed knowledge-graph framework. Cognee provides the storage abstraction (Neo4j or Kuzu + vector index), the LLM-based extraction pipeline, the embedding pipeline, and the agent-native `remember/recall/forget` API. The Lore Engine adds a typed high-fantasy ontology, a temporal model, a consistency engine, NPC knowledge scoping, and a TypeTemplate polymorphic extension system on top.
+The Lore Engine is a **domain layer** on top of [Cognee](https://github.com/topoteretes/cognee), a self-hosted, MIT-licensed knowledge-graph framework. Cognee provides the storage abstraction (**Neo4j** + vector index — pinned by ADR 0008, overriding Cognee's Kuzu default), the LLM-based extraction pipeline, the embedding pipeline, and the agent-native `remember/recall/forget` API. The Lore Engine adds a typed high-fantasy ontology, a temporal model, a consistency engine, NPC knowledge scoping, and a TypeTemplate polymorphic extension system on top.

 Cognee is the **plumbing**; the Lore Engine is the **domain layer** that the world-builder cares about.

@@ -30,7 +30,7 @@ Cognee is the **plumbing**; the Lore Engine is the **domain layer** that the wor
                              ┌─────────────────────────┐
                              │     Cognee Storage      │
                              │  ─────────────────────  │
-                              │  Neo4j or Kuzu (graph)  │
+                              │  Neo4j (graph)          │
                              │  + Vector store         │
                              │  + Postgres (metadata)  │
                              │  ─────────────────────  │
@@ -282,7 +282,7 @@ Implementation notes:
 - Resolves `current` against the `:Now` config node.
 - Walks the era tree for parent-era membership.
 - Treats `null` as open-ended.
- Implementation: a Cognee plugin (Python) registered at startup, OR a Neo4j user-defined function (Java) if Cognee is using Neo4j as the storage backend. The Phase 2 spike (see `09-roadmap.md`) determines which.
+- Implementation: a **Neo4j user-defined function (Java)** — the graph backend is Neo4j (ADR 0008), so the time model ships as a Java UDF queryable inline in Cypher, not as Python application-layer code. The POC's pure-Python `time_in_window` port is the reference implementation and test oracle for the UDF.

 ### `time_windows_overlap(from_a, until_a, from_b, until_b) → bool`

@@ -298,7 +298,7 @@ The `time_in_window` and `time_windows_overlap` UDFs are the only ones the engin

 ## Hosting & deployment

-The Lore Engine on Cognee is **one process** that runs the Cognee MCP server, the storage adapter (Neo4j or Kuzu), and the Lore Engine extension. The hosting story is:
+The Lore Engine on Cognee is **one process** that runs the Cognee MCP server, the storage adapter (Neo4j), and the Lore Engine extension. The hosting story is:

 - **Cognee** — runs in a Docker container, exposes the MCP server on `:8000` (or whatever port the deployment chooses), manages the storage adapter. The Lore Engine extension is a Python package installed into the Cognee image.
 - **Storage backend** — Cognee supports Neo4j 5.x (most common in production), Kuzu (embedded, single-binary), and others via the `cognee-neo4j` / `cognee-kuzu` adapters. The Lore Engine does not need to know which one is in use.
--- a/docs/22-cognee-boundary.md
+++ b/docs/22-cognee-boundary.md
@@ -11,8 +11,10 @@ when someone asks "could we swap Cognee for X?"

 ## What Cognee owns

- **Storage.** The graph database (Neo4j or Kuzu, Cognee's
-  choice) and the vector store (pgvector or Qdrant).
+- **Storage.** The graph database — **Neo4j**, pinned by ADR 0008
+  (Cognee's default is Kuzu, but we override for battle-testedness
+  and for the Java UDFs the time model needs). The vector store
+  (pgvector or Qdrant) is Cognee's choice.
 - **Ingestion pipeline.** The `cognee.add` /
  `cognee.cognify` lifecycle, including chunking and embedding.
 - **Extraction.** The LLM call that turns chunks into
--- a/docs/adr/0008-graph-backend-neo4j.md
+++ b/docs/adr/0008-graph-backend-neo4j.md
@@ -0,0 +1,78 @@
+# Graph backend is Neo4j, not Cognee's Kuzu default
+
+**Status:** accepted.
+
+Cognee's default graph database is **Kuzu** (an in-process,
+file-based C++ graph DB — set as default in Cognee PR #1022,
+June 2025). The Lore Engine overrides that default and pins
+to **Neo4j** (`GRAPH_DATABASE_PROVIDER="neo4j"`).
+
+## Why Neo4j
+
+- **Battle-tested.** Neo4j is the production-grade, self-hosted
+  graph database with years of real deployments. For a substrate
+  decision — the thing the whole engine sits on — we want the
+  one that won't surprise us at scale, not the zero-setup local
+  default.
+- **Java UDFs.** The time model (`time_in_window`,
+  `time_windows_overlap`, era-tree membership, `current` token
+  resolution) is the load-bearing primitive of the whole engine.
+  On Neo4j it ships as a **Java user-defined function** — fast,
+  queryable inline in Cypher, unit-tested against 50+ cases. Kuzu
+  has no Java UDF mechanism; on Kuzu the time model would have to
+  be pure-Python application-layer code, called *outside* the
+  query — which breaks the "time check is a single Cypher clause"
+  contract that `was_true_at` depends on (see `00-overview.md`'s
+  question-flow diagram). The POC's pure-Python `time_in_window`
+  port becomes a *reference implementation and test oracle*, not
+  the production path.
+- **Cypher + APOC.** The consistency engine's Category C rules
+  (ADR 0007's neighbor — declarative `OntologyRule`) lean on
+  Cognee's documented Cypher + APOC `apoc.util.validate`
+  pattern. APOC is a Neo4j plugin; Kuzu has no equivalent.
+
+## What we give up
+
+- **Zero-setup.** Kuzu is a file on disk; Neo4j is a server to
+  run (`docker compose up neo4j`). Acceptable for a production
+  engine; the quickstart (`21-quickstart.md`) ships a compose
+  file that brings it up.
+- **Memory footprint.** Neo4j is heavier than Kuzu. Acceptable;
+  the engine isn't targeting embedded/single-binary deployment.
+
+## What this does NOT decide
+
+- **Vector store.** Separate from the graph backend; Cognee
+  supports pgvector and Qdrant. Out of scope here.
+- **Cognee's other defaults.** Only the graph provider is
+  overridden; everything else rides Cognee's defaults.
+
+## The post-cognify consistency hook
+
+Cognee exposes no built-in post-`cognify` hook. The consistency
+runner attaches via one of:
+
+1. **In-process custom `Task`** — append the consistency sweep
+   as the final `Task` in `run_custom_pipeline(tasks=[...])`,
+   after extraction. Runs immediately after ingest; materializes
+   `Contradiction`/`Anachronism`/`Orphan` nodes while the
+   pipeline context is still warm. (Recommended for the
+   post-ingest sweep.)
+2. **Sequential await** — `await cognify()` then `await
+   run_consistency(scope="new")`. Simpler; no pipeline plumbing.
+3. **External cron** — the full nightly re-sweep (03:00) runs as
+   an external scheduled job over the whole graph, independent of
+   any single ingest. (Recommended for the long-tail rules + the
+   `ConsistencyRun` record.)
+
+The engine uses **(1) + (3)**: a fast in-process sweep after
+every cognify for freshness, plus a nightly external sweep for
+completeness. Both write the same `:ConsistencyRun` node shape.
+
+## Cross-references
+
+- `docs/02-time-model.md` — the time model the UDF implements
+- `docs/04-consistency.md` — the consistency engine + its run modes
+- `docs/adr/0006-cognee-version-pin.md` — version policy
+- `docs/adr/0007-graph-model-ontology-contract.md` — the other Cognee-API decision
+- `docs/22-cognee-boundary.md` — storage ownership