feat(bmad): add canonical _kit (templates + sample) + ingest validation
Some checks failed
test / contract-and-unit (push) Failing after 14s

BMAD-onboarding kit for the Damascus orchestrator:

- docs/adding-a-new-project.md — full onboarding guide covering layout,
  required story section headers, common pitfalls (with the four classes
  of bug that have cost real cycles here: Path.rglob doesn't follow
  symlinks, architecture.md must be at planning-artifacts/architecture.md
  exactly, missing section headers burn 3 retries each, etc.)
- bmad/_kit/ — read-only reference material (templates + sample)
  - templates/{prd,architecture,epics,story}.md
  - sample/hello-bmad/_bmad-output/ — one fully-formed worked example
    (2-story FastAPI project, valid end-to-end)
  - README.md — kit-level contract
- scripts/test-ingest.sh — pre-flight validation that catches the four
  bug classes before any DB write. Verified against the live orchestrator
  container: passes on the sample, fails (correctly) on a hand-broken tree
  with both missing-section AND symlink bugs in one run.
- docker-compose.yml — replace /home/kaykayyali/_bmad bind (which
  doesn't exist on this server) with ./bmad/_kit. Kit now ships with
  the repo.
- .gitignore — re-include bmad/_kit/ so it travels with the repo while
  keeping the existing 'bmad/ is ephemeral mount content' contract.

Verified end-to-end: 'damascus ingest --project hello-bmad' succeeded
on the live orchestrator, _find_bmad_story resolved both stories.

The 'architecture.md is ingested as a work item' quirk is documented in
docs/adding-a-new-project.md §'Common pitfalls' with a one-liner fix.

Refs: t_5aa80e4b (parallel dashboard work — committed separately)
This commit is contained in:
damascus-heartbeat
2026-06-26 06:03:39 +00:00
parent cfcd571928
commit 82b9758be6
13 changed files with 1310 additions and 3 deletions

10
.gitignore vendored
View File

@@ -33,8 +33,14 @@ Thumbs.db
specs/
data/specs/
# BMAD output dirs are read-only mounts from other projects — not our code
bmad/
# BMAD output dirs are read-only mounts from other projects — not our code.
# The _kit subdir is the canonical reference kit shipped with this repo
# (templates, samples, README) — re-included below. Everything else under
# bmad/ (e.g. bmad/wh40k-pc/, bmad/restitution/) is still treated as
# ephemeral mount content.
bmad/*
!bmad/_kit/
!bmad/_kit/**
# Hermes evidence dirs (e2e screenshots + logs regenerated by tests)
.hermes/evidence/

63
bmad/_kit/README.md Normal file
View File

@@ -0,0 +1,63 @@
# BMAD Kit — Damascus Orchestrator
> **This directory is read-only reference material** for new projects onboarding to the Damascus orchestrator. Copy from here, never add to it.
## Contents
```
bmad/_kit/
├── README.md ← this file
├── templates/
│ ├── prd.md ← Product Requirements Document template
│ ├── architecture.md ← Architecture doc template (lives at planning-artifacts/architecture.md)
│ ├── epics.md ← Epics + story summary template
│ └── story.md ← Per-story brief template (required section headers)
└── sample/
└── hello-bmad/ ← one fully-formed worked example
└── _bmad-output/
├── planning-artifacts/
│ ├── architecture.md
│ └── stories/
│ ├── S1-hello-endpoint.md
│ └── S2-list-endpoints.md
└── meta/
└── prd.md
```
## How to use
For a real onboarding, see `docs/adding-a-new-project.md` in the repo root. The short version:
```bash
# 1. Copy the sample as a starting point
cp -r bmad/_kit/sample/hello-bmad /root/my-project
# 2. Rename + edit
cd /root/my-project
mv _bmad-output/meta/prd.md{,.bak} # edit in place
# 3. Validate before going live
cd /root/damascus-orchestrator
./scripts/test-ingest.sh /root/my-project/_bmad-output my-project
# 4. Wire the bind mount + real ingest (see docs/adding-a-new-project.md)
```
## Maintenance contract
**Don't add to `_kit/`.** The kit is the canonical reference — adding to it creates drift. If you find a new template pattern is needed, the right move is:
1. Document the gap in `docs/adding-a-new-project.md` under "Common pitfalls" or "Open decisions"
2. If the orchestrator needs a new capability, file an issue against `kaykayyali/damascus-orchestrator`
3. If the gap is project-specific, copy + adapt from `_kit/templates/` into your project's `_bmad-output/`, don't modify the kit
## When the orchestrator changes
The kit must stay in sync with `src/damascus/phases.py` (which parses story sections) and `src/damascus/cli.py` (which does the ingest glob). When either changes:
1. Update `templates/story.md` section list to match
2. Update `scripts/test-ingest.sh` validation to match
3. Update `docs/adding-a-new-project.md` "Common pitfalls" to match
4. Update the worked sample (`sample/hello-bmad/`) to match
This is a manual chore. There's no automated lint linking the kit to the orchestrator code.

View File

@@ -0,0 +1,70 @@
# PRD — Hello BMAD
**Project**: `kaykayyali/hello-bmad` (sample project — not a real app)
**Author**: Worked example for Damascus orchestrator BMAD onboarding
**Date**: 2026-06-25
**Status**: Sample / template
---
## 1. Goal
A tiny REST API that returns a "hello" JSON response. Two endpoints: `GET /hello` and `GET /hello/list`. This is a **worked example** for the BMAD-onboarding docs — not a real product.
## 2. Personas
| Persona | What they want |
|---|---|
| **A future agent onboarding a project** | A complete, runnable example of BMAD output that the Damascus orchestrator can ingest without errors. |
That's it. One persona. This is a teaching example.
## 3. User Stories (v1)
### P0 — must have for v1
- **U1**: As the demo agent, I want a `GET /hello` endpoint that returns `{"message": "hello, world"}` so I can verify the orchestrator ingested + built + ran a project.
- **U2**: As the demo agent, I want `GET /hello/list` to return an array of strings so I can verify multi-endpoint support.
### Out of scope for v1
- Auth, persistence, deployment. Just the two endpoints.
## 4. Functional Requirements
### 4.1 `GET /hello`
- Returns 200 + JSON body `{"message": "hello, world"}`
- No request body, no query params
### 4.2 `GET /hello/list`
- Returns 200 + JSON array `["alpha", "beta", "gamma"]`
- No request body, no query params
## 5. Non-Functional Requirements
| NFR | Requirement |
|---|---|
| Tech stack | Python 3.11 + FastAPI |
| Tests | pytest with at least 2 tests (one per endpoint) |
| Build time | < 5s (it's two routes) |
## 6. Acceptance Criteria (v1 ships when ALL are true)
- [ ] `curl localhost:8000/hello` returns `{"message": "hello, world"}`
- [ ] `curl localhost:8000/hello/list` returns `["alpha", "beta", "gamma"]`
- [ ] `pytest tests/` passes
- [ ] Both routes are documented in OpenAPI (FastAPI does this automatically)
## 7. Risks
None — it's a sample project.
## 8. Out of Scope
Everything except the two endpoints.
## 9. Open Questions
None. Resolved by being a 2-route sample.

View File

@@ -0,0 +1,78 @@
# Architecture — Hello BMAD
**Date**: 2026-06-25
**Companion to**: `meta/prd.md`
---
## 1. System context
```
┌─────────────────────────┐
│ hello-bmad (FastAPI) │
│ port 8000 │
│ │
│ GET /hello │
│ GET /hello/list │
└─────────────────────────┘
│ HTTP
┌─────────────────────────┐
│ curl / pytest / agent │
└─────────────────────────┘
```
## 2. Component diagram
```
hello-bmad/
├── main.py ← FastAPI app + route definitions
├── tests/
│ └── test_main.py ← pytest tests for both routes
├── requirements.txt ← fastapi, uvicorn, pytest, httpx
└── Dockerfile ← optional — orchestrator runs pytest, not the server
```
## 3. State shape
None — pure stateless request handlers. No DB, no in-memory state.
## 4. External contracts
| Contract | Endpoint | Args | Returns |
|---|---|---|---|
| `GET /hello` | HTTP GET | none | `{"message": "hello, world"}` |
| `GET /hello/list` | HTTP GET | none | `["alpha", "beta", "gamma"]` |
FastAPI generates the OpenAPI schema automatically. No external APIs consumed.
## 5. Tech stack
| Layer | Choice | Why |
|---|---|---|
| Framework | FastAPI | Smallest viable Python API framework |
| Server | uvicorn | Standard ASGI server for FastAPI |
| Tests | pytest + httpx | Industry standard, async-friendly |
## 6. Deployment
The orchestrator runs `pytest tests/` as the test command — no deployment needed for a sample. The build phase will run the tests and report green if the implementation is correct.
## 7. Failure modes
None relevant for a sample.
## 8. Security
None — local-only sample.
## 9. Open decisions (resolved)
1. **Two routes only**: simpler than one route with parameters, demonstrates multi-endpoint patterns.
2. **No DB**: keeps the example to ~50 lines of code.
3. **JSON array for /list**: shows that the orchestrator handles non-object return types.
## 10. References
- [FastAPI docs](https://fastapi.tiangolo.com/) — for any implementer who needs a refresher

View File

@@ -0,0 +1,38 @@
# S1 — Hello endpoint
**Epic**: E1
**Status**: pending
**Branch**: `feat/S1-hello-endpoint`
## Goal
Implement `GET /hello` in a FastAPI app. Returns `{"message": "hello, world"}` with HTTP 200. No request body, no query params.
## Acceptance Criteria
- [ ] `GET /hello` returns HTTP 200 + JSON body `{"message": "hello, world"}`
- [ ] The endpoint is registered with FastAPI's `@app.get("/hello")` decorator
- [ ] `pytest tests/test_main.py::test_hello_endpoint` passes
- [ ] The OpenAPI schema generated by FastAPI includes the `/hello` route
## TDD Plan
1. Write `test_hello_endpoint` asserting `client.get("/hello").json() == {"message": "hello, world"}`. Confirm it fails (no implementation yet).
2. Run `pytest tests/test_main.py -k hello` — confirm RED.
3. Add the `@app.get("/hello")` route with the stub return.
4. Run the test again — confirm GREEN.
## File Scope
- `main.py`
- `tests/test_main.py`
## Test Command
```bash
python -m pytest tests/test_main.py::test_hello_endpoint -q
```
## Ambiguities
(none)

View File

@@ -0,0 +1,38 @@
# S2 — Hello list endpoint
**Epic**: E1
**Status**: pending
**Branch**: `feat/S2-list-endpoint`
## Goal
Implement `GET /hello/list` in the same FastAPI app from S1. Returns a JSON array `["alpha", "beta", "gamma"]` with HTTP 200. Demonstrates that the orchestrator handles non-object return types.
## Acceptance Criteria
- [ ] `GET /hello/list` returns HTTP 200 + JSON body `["alpha", "beta", "gamma"]`
- [ ] The endpoint is registered with FastAPI's `@app.get("/hello/list")` decorator
- [ ] `pytest tests/test_main.py::test_hello_list_endpoint` passes
- [ ] `pytest tests/` (both tests together) passes — confirms no regression on S1
## TDD Plan
1. Write `test_hello_list_endpoint` asserting `client.get("/hello/list").json() == ["alpha", "beta", "gamma"]`. Confirm it fails (no implementation yet).
2. Run `pytest tests/test_main.py -k hello_list` — confirm RED.
3. Add the `@app.get("/hello/list")` route with the stub return.
4. Run `pytest tests/` — confirm both S1 and S2 GREEN.
## File Scope
- `main.py`
- `tests/test_main.py`
## Test Command
```bash
python -m pytest tests/ -q
```
## Ambiguities
(none)

View File

@@ -0,0 +1,96 @@
# Architecture — <Project Name>
> **Template**: copy this file to `<project>/_bmad-output/planning-artifacts/architecture.md`. **This file MUST live at `planning-artifacts/architecture.md` exactly** — the orchestrator's spec-refiner hardcodes this path. If you put it elsewhere, your refiner runs blind.
**Date**: <YYYY-MM-DD>
**Companion to**: `meta/prd.md`
---
## 1. System context
<ASCII diagram showing how this project fits with its dependencies / external systems. Use box-and-arrow.>
```
┌──────────────────────┐ ┌──────────────────────┐
│ <This project> │ ───> │ <Dependency> │
│ │ HTTP │ │
└──────────────────────┘ └──────────────────────┘
```
## 2. Component diagram
```
src/
├── main.ts ← entry point
├── <subsystem>/ ← <responsibility>
│ ├── index.ts
│ └── ...
```
## 3. State shape
<TypeScript / Python / Go type definitions for the project's core data model. Be concrete.>
```typescript
type CoreEntity = {
id: string;
// ...
};
```
## 4. External contracts
| Contract | Endpoint / tool / function | Args | Returns |
|---|---|---|---|
| <API name> | `POST /api/v1/<thing>` | `{...}` | `{...}` |
| <MCP tool> | `<tool_name>(args)` | `<args>` | `<return shape>` |
| <Library fn> | `<lib.func>(input)` | `<input>` | `<output>` |
**Critical**: link out to canonical source-of-truth docs (URLs) for every external contract. Don't paraphrase what the API does — point at the spec.
## 5. Tech stack
| Layer | Choice | Why |
|---|---|---|
| Build | <Vite / Webpack / Cargo> | <reason> |
| Framework | <React / FastAPI / Actix> | <reason> |
| UI | <MUI / Tailwind / raw> | <reason> |
| State | <Redux / useReducer / context> | <reason> |
| Storage | <Postgres / SQLite / None> | <reason> |
| Auth | <JWT / session / none> | <reason> |
## 6. Deployment
- **Where**: <host / cluster / serverless>
- **How**: <docker compose / k8s / static + CDN>
- **CI/CD**: <GitHub Actions / Gitea Actions / manual>
- **Rollback**: <strategy>
## 7. Failure modes
| Failure | User-visible behavior | Recovery |
|---|---|---|
| <Dependency down> | <error state> | <retry / fallback> |
| <DB unreachable> | <error state> | <reconnect with backoff> |
## 8. Security
- <Auth model>
- <Secret handling>
- <Network exposure (public / tailnet-only / LAN-only)>
## 9. Open decisions (resolved)
If you made policy/UX/architecture calls that downstream agents might second-guess, list them here:
1. **<Decision>**: <what you chose + why>
2. **<Decision>**: <what you chose + why>
This preempts the spec-refiner from asking the same questions on every story.
## 10. References
- <Link to upstream API spec>
- <Link to related architecture doc>
- <Link to deployment runbook>

View File

@@ -0,0 +1,58 @@
# Epics & Stories — <Project Name>
> **Template**: copy this file to `<project>/_bmad-output/meta/epics.md`. (Or put it at `planning-artifacts/epics.md` if you want the refiner to read it as part of the brief — but then it'll also be ingested as a work item; pick one.)
**Date**: <YYYY-MM-DD>
**Companion to**: `meta/prd.md`, `planning-artifacts/architecture.md`
---
## Epic E1 — <Epic Title>
> <One-sentence summary of what this epic delivers>
**Acceptance for epic**:
- [ ] <Criterion 1>
- [ ] <Criterion 2>
| Story | Title | Acceptance |
|---|---|---|
| **S1** | <title> | <one-line acceptance> |
| **S2** | <title> | <one-line acceptance> |
---
## Epic E2 — <Epic Title>
> <One-sentence summary>
**Acceptance for epic**:
- [ ] <Criterion>
| Story | Title | Acceptance |
|---|---|---|
| **S3** | <title> | <one-line acceptance> |
| **S4** | <title> | <one-line acceptance> |
---
## Story sizing guide for the orchestrator
- **S1-S<N>**: <rough size estimate each>
- Realistically with retries and review cycles: <N hours>
**Dependencies**:
- E2 must finish before E3 starts (need E2's output to author E3)
- E3 can run in parallel with E4 (independent UI work)
**Suggested ordering for orchestrator**: E1 → E2 → E3 → E4. Reasoning: <why this order>.
---
## Story count summary
- **E1** (<name>): <N> stories
- **E2** (<name>): <N> stories
- **Total**: <N> stories
Estimated <N> hours of focused worker time. Realistically with retries and review cycles: <N> days of unattended orchestration.

View File

@@ -0,0 +1,84 @@
# PRD — <Project Name>
> **Template**: copy this file to `<project>/_bmad-output/meta/prd.md` and fill in. **Do NOT put the PRD in `planning-artifacts/`** — it will be ingested as a work item. Keep it in `meta/`.
**Project**: `kaykayyali/<project-repo>`
**Author**: <your name or agent id>
**Date**: <YYYY-MM-DD>
**Status**: Draft v1 — pending review
---
## 1. Goal
<One paragraph: what is this project, who is it for, what's the smallest end-state we can ship in v1?>
## 2. Personas
| Persona | What they want |
|---|---|
| **<Primary user>** | <primary need> |
| **<Secondary user>** | <secondary need> |
## 3. User Stories (v1)
### P0 — must have for v1
- **U1**: As <persona>, I <action> so that <outcome>.
- **U2**: As <persona>, I <action> so that <outcome>.
### P1 — nice-to-have for v1
- **U3**: As <persona>, I <action> so that <outcome>.
### Out of scope for v1
- <Feature X — explicitly not building>
- <Feature Y — explicitly not building>
## 4. Functional Requirements
### 4.1 <Subsystem / capability>
<Bullet list of what the system must do. Be specific enough that an engineer can estimate.>
### 4.2 <Another subsystem>
<...>
## 5. Non-Functional Requirements
| NFR | Requirement | How verified |
|---|---|---|
| **Performance** | <latency/throughput target> | <how to measure> |
| **Availability** | <uptime target> | <how to monitor> |
| **Bundle size** | <size budget> | <where to assert> |
| **Mobile** | <mobile-friendly or not> | <viewport to test> |
## 6. Acceptance Criteria (v1 ships when ALL are true)
- [ ] <criterion 1 — testable>
- [ ] <criterion 2 — testable>
- [ ] <criterion 3 — testable>
## 7. Risks
| Risk | Mitigation |
|---|---|
| <Risk 1> | <how to reduce / detect> |
| <Risk 2> | <mitigation> |
## 8. Out of Scope (for the record)
- <Feature not building — and why>
- <Tech choice not making — and why>
## 9. Open Questions
- <Question 1 — to resolve before kickoff>
- <Question 2 — to resolve during epic 1>
## 10. Reference Links
- <Link to related docs>
- <Link to upstream API contract>

View File

@@ -0,0 +1,82 @@
# S<n> — <Short Title>
> **Template**: copy this file to `<project>/_bmad-output/planning-artifacts/stories/S<n>-<slug>.md` for each story.
>
> **Required**: every story MUST have all six H2 section headers below (`## Goal`, `## Acceptance Criteria`, `## TDD Plan`, `## File Scope`, `## Test Command`, `## Ambiguities`). The spec-refiner parses them literally. A missing section → `verdict=spec_wrong` and 3 retries wasted.
**Epic**: <E1|E2|...>
**Status**: pending
**Branch**: `feat/<branch-name>`
---
## Goal
<One paragraph: what the implementation should achieve. Be concrete — "add a button" is bad, "add a 'Save' button to the entity detail panel that POSTs to /api/v1/entities/{id}/save and shows a toast on success" is good.>
## Acceptance Criteria
- [ ] <Criterion 1 — testable. "The button POSTs and the toast appears within 1s" beats "The button works.">
- [ ] <Criterion 2>
- [ ] <Criterion 3>
- [ ] (Optional) <Criterion 4 — nice-to-have for this story>
## TDD Plan
1. <Failing test 1 — what to write first, what behavior it asserts>
2. <Failing test 2>
3. <Failing test 3>
The TDD Plan is what the implementer writes BEFORE any production code. Each test should fail with the current code, then pass after the implementation lands.
## File Scope
- `<path/to/file-1>`
- `<path/to/file-2>`
- `<path/to/file-3>`
**Critical**: list every file the implementer may touch. The orchestrator enforces this list — if the implementer adds a file outside this scope, the reviewer fails it. Be honest: if a story needs 5 files, list 5. Don't artificially narrow scope to "look small."
## Test Command
```bash
<exact shell command that proves the story is done>
```
The test command runs after the implementation. Exit 0 = story done. Non-zero = retry.
Examples by project type:
- **Frontend**: `cd ui && npm run build && npx playwright test tests/e2e/<story>.spec.ts`
- **Backend**: `pytest tests/<story>.py -q`
- **Full-stack**: `bash scripts/verify.sh` (which builds + tests + runs E2E)
- **Docs-only**: `markdownlint <file.md>` or `grep -q "<expected section>" <file.md>`
## Ambiguities
<Open questions for a human. Either resolve them yourself in this section (preferred — saves an `awaiting_human` round-trip) or list them as bullets for the spec-refiner to surface.>
Examples:
- "Filter combination: AND or OR? Answer: AND-composed."
- "Persistence: localStorage or session-only? Answer: session-only per PRD §3."
- "Edge case: what if the API returns 5xx? Answer: show a generic error toast."
If no ambiguities: write `(none)`. Don't leave the section blank.
---
## Definition of done (for the implementer)
- All acceptance criteria pass
- `npm run build` (or equivalent) exits 0
- The test command exits 0
- No new files outside the declared File Scope
- Branch pushed to origin with a single clean commit (or a small set of conventional commits)
- PR opened against main with title matching `<type>(<scope>): <description>` (Conventional Commits)
## Notes for the reviewer
<Anything the reviewer should know before approving — test coverage concerns, design tradeoffs, links to related stories.>
## Out of scope (explicit)
<Things this story is NOT doing — preempt "why didn't you also do X" questions from reviewers.>

View File

@@ -93,7 +93,14 @@ services:
- ./wiki:/opt/damascus/llm-wiki
# Mount the host's BMAD output dirs under /opt/damascus/bmad/<project>/
- /root/restitution/_bmad-output:/opt/damascus/bmad/restitution/_bmad-output:ro
- /home/kaykayyali/_bmad:/opt/damascus/bmad/_kit:ro
- /root/mindmaps-prds/_bmad-output:/opt/damascus/bmad/mindmaps/_bmad-output:ro
# BMAD kit — templates, samples, and reference docs. Ships with the
# orchestrator repo at bmad/_kit/. Read-only.
- ./bmad/_kit:/opt/damascus/bmad/_kit:ro
# Legacy _kit location, kept for back-compat with the existing bind
- /home/kaykayyali/_bmad:/opt/damascus/bmad/_kit_legacy:ro
# hello-bmad sample project (for verification — remove in real deployments)
- /root/hello-bmad/_bmad-output:/opt/damascus/bmad/hello-bmad/_bmad-output:ro
# E2E test suite (read-only; tests run from the host)
- ./tests:/opt/damascus/tests:ro
# Taskiq worker — the global concurrency cap (design doc §10). For sync

View File

@@ -0,0 +1,427 @@
# Adding a New Project to the Damascus Orchestrator
> **Audience**: an engineer or agent onboarding a new project so its stories get picked up by the orchestrator's `spec → build → review → merged` cycle.
>
> **Time estimate**: 30 minutes for a small project (≤10 stories); 23 hours for a multi-epic project (≥30 stories).
---
## TL;DR
```bash
# 1. Have your BMAD output ready at /root/<project>/_bmad-output/
# (see "Layout" section below)
ls /root/my-project/_bmad-output/planning-artifacts/stories/ # should show S1-..., S2-..., etc.
# 2. Validate locally — does NOT touch the DB
./scripts/test-ingest.sh /root/my-project/_bmad-output my-project
# 3. Wire the bind mount in docker-compose.yml
# (see "Step 3 — Wire the bind mount" below)
docker compose up -d --force-recreate --no-deps orchestrator
# 4. Real ingest
docker exec damascus-orchestrator-orchestrator-1 \
damascus ingest --project my-project
# 5. Watch the first story run through the cycle
hermes kanban --board my-project list
# or set up a watchdog (see "Monitoring" below)
```
If anything goes wrong at step 2, fix the BMAD output. If step 4 fails or the stories don't have the right section headers, fix the BMAD output. **Do not edit the orchestrator code.**
---
## What "BMAD" means here
The Damascus orchestrator doesn't run BMAD agents or BMAD workflow skills directly. What it does is **ingest pre-written BMAD planning artifacts** (PRDs, architecture docs, epics, per-story briefs) and turn each `.md` file into a `work_items` row that the orchestrator's cycle picks up.
The relationship:
```
┌─────────────────────────┐ ┌──────────────────────────┐
│ BMAD planning output │ │ Damascus orchestrator │
│ (you write this) │ │ (picks this up) │
│ │ │ │
│ _bmad-output/ │ │ work_items table │
│ planning-artifacts/ │ ───> │ phase=spec rows │
│ architecture.md │ ingest │ one per .md file │
│ <epic>.md │ │ │
│ stories/ │ │ cycle processes them: │
│ S1-...md │ │ spec → build → review │
│ S2-...md │ │ → merged │
└─────────────────────────┘ └──────────────────────────┘
```
If you have a real BMAD project (with `bmad-auto` skill or BMAD agents generating the artifacts), great — point the orchestrator at the output. If you're writing the artifacts by hand (the common case for ≤30 stories), use the templates in `bmad/_kit/templates/` and follow this doc.
---
## Layout
The orchestrator expects a specific directory layout **inside** the container at `/opt/damascus/bmad/<project>/_bmad-output/`. The host path that bind-mounts to it is whatever you choose (we use `/root/<project>/_bmad-output/` by convention; see `docker-compose.yml` for the actual mapping).
```
_bmad-output/ ← root of your project's BMAD output
├── planning-artifacts/ ← INGESTED as work_items (one per .md)
│ ├── architecture.md ← REQUIRED — read by spec-refiner
│ ├── epics.md ← OPTIONAL — meta doc, may live here or in meta/
│ └── stories/ ← where your per-story briefs live
│ ├── S1-...md ← required section headers (see "Story format")
│ ├── S2-...md
│ └── ...
└── meta/ ← NOT ingested — pure reference docs
├── prd.md
├── epics.md ← if not in planning-artifacts/
└── ...
```
**Why split `meta/` from `planning-artifacts/`?**
The orchestrator's `damascus ingest` (in `src/damascus/cli.py`) globs every `.md` under `planning-artifacts/` and treats each as a story. If you put your PRD there, the orchestrator will try to "implement the PRD" as a feature. Keep meta documents (PRD, long epics doc) in `meta/` so they're reference material, not work items.
**Why must `architecture.md` live at `planning-artifacts/architecture.md` exactly?**
The spec-refiner reads it via `_find_architecture()` in `src/damascus/phases.py`, which hardcodes that path. There's no `meta/architecture.md` fallback. If you forget this, your refiner runs blind and produces weak specs.
---
## Story format — required section headers
Every story `.md` file **must** have these H2 section headers. The orchestrator's spec-refiner (`phases.py:55-78`) parses them out and rejects the story as `spec_wrong` if any are missing:
```markdown
# S<n> — <short title>
**Epic**: <E1|E2|...>
**Status**: pending
**Branch**: `feat/<branch-name>`
## Goal
<one paragraph — what the implementation should achieve>
## Acceptance Criteria
- [ ] <testable criterion 1>
- [ ] <testable criterion 2>
- [ ] <testable criterion 3>
## TDD Plan
1. <failing test 1 — what to write before any code>
2. <failing test 2>
3. <failing test 3>
## File Scope
- `<path/to/file-1>`
- `<path/to/file-2>`
- `<path/to/file-3>`
## Test Command
```bash
<exact shell command that proves the story is done>
```
## Ambiguities
<list of open questions for a human, or "(none)" if you resolved them all>
```
**What happens if a section is missing**: spec-refiner returns `verdict=spec_wrong, missing=['TDD Plan']` and the row gets retried up to 3 times before burning out. **Don't ship stories without these headers.**
**Tip**: copy from `bmad/_kit/templates/story.md` and fill in. Don't hand-author the section names — they're parsed literally.
### Where to put per-story briefs
Two valid layouts:
**Layout A (canonical, recommended)**:
```
planning-artifacts/stories/S<n>-<slug>.md
```
**Layout B (canonical BMAD layout)** — when your toolchain generates stories here:
```
implementation-artifacts/stories/S<n>-<slug>.md
```
Layout B alone **does not work** — `phases.py:_find_bmad_story` only scans `planning-artifacts/`. If your toolchain puts stories in `implementation-artifacts/`, you need a **bind mount that copies or symlinks** them into `planning-artifacts/stories/` inside the container. Or move them.
**Don't use a symlink on the host that `Path.rglob` would have to follow.** Python's `pathlib.Path.rglob` (which the spec-refiner uses) does **not** follow symlinks by default in Python ≤3.12. The orchestrator runs Python 3.12. Use a real copy or a bind mount, not a symlink.
---
## Project repo on disk
The orchestrator needs the project's source repo cloned into `/workspace/projects/<project>/` **inside the container**. The cycle's build phase (`phases.py:build()`) clones it from Gitea on first run if it doesn't exist:
```
If /workspace/projects/<project>/ doesn't exist when the build phase claims a row,
the build returns verdict=tests_failed, error="project repo not found at..."
```
So your **Gitea repo must exist before the first row's build phase fires**. The `damascus ingest` step doesn't require the repo (ingest only writes to `work_items`), but the build phase does.
### Setup checklist
- [ ] Gitea repo exists at `kaykayyali/<project>` (private, with the user's default branch — usually `main`)
- [ ] Either:
- The build phase is allowed to clone from Gitea at first run (it will — uses `DAMASCUS_GITEA_TOKEN` env var), OR
- You pre-clone to `/workspace/projects/<project>/` inside the container via the `projects` named volume
### Worktree behavior
The build phase creates a worktree at `/workspace/worktrees/<project>/<story-id>` for each story. The worktree branch name is `feat/<story-id>`. The orchestrator opens a PR against the project's main branch (uses `git_ops.ensure_worktree()` in `src/damascus/git_ops.py`).
---
## Step-by-step onboarding
### Step 1 — Author the BMAD output
Two paths:
**(a) Hand-author**: copy `bmad/_kit/templates/` to a working dir, fill in the markdown. Use `bmad/_kit/sample/hello-bmad/` as a worked example.
**(b) Use BMAD agents (if you have them)**: run your BMAD `bmad-create-prd` / `bmad-create-architecture` / `bmad-create-story` workflows, point the output at `_bmad-output/`.
Either way, end up with:
```
/root/my-project/_bmad-output/
├── planning-artifacts/
│ ├── architecture.md ← required
│ └── stories/
│ ├── S1-setup-scaffold.md
│ ├── S2-add-feature-x.md
│ └── ...
└── meta/ ← optional
├── prd.md
└── epics.md
```
### Step 2 — Validate with `scripts/test-ingest.sh`
```bash
cd /root/damascus-orchestrator
./scripts/test-ingest.sh /root/my-project/_bmad-output my-project
```
This dry-runs the orchestrator's ingest **without writing to the DB**. It checks:
- All required sections present in every story
- `architecture.md` is in the right place
- No symlinks (which `Path.rglob` won't follow)
- The orchestrator's `find_bmad_story` actually finds each story when the refiner looks for it
Exit code 0 = ready to ingest. Non-zero = fix the BMAD output and re-run.
### Step 3 — Wire the bind mount in `docker-compose.yml`
Add to the `orchestrator` service's `volumes:` list:
```yaml
volumes:
# ... existing mounts ...
- /root/my-project/_bmad-output:/opt/damascus/bmad/my-project/_bmad-output:ro
```
The pattern: `/root/<host-dir>/_bmad-output``/opt/damascus/bmad/<project>/_bmad-output`.
`my-project` (the right-hand side) must match the project name you'll pass to `damascus ingest`.
Then recreate the orchestrator container so it picks up the new mount:
```bash
docker compose up -d --force-recreate --no-deps orchestrator
```
Verify the mount worked:
```bash
docker exec damascus-orchestrator-orchestrator-1 \
ls /opt/damascus/bmad/my-project/_bmad-output/planning-artifacts/stories/ | head -10
```
### Step 4 — Real ingest
```bash
docker exec damascus-orchestrator-orchestrator-1 \
damascus ingest --project my-project
```
Expected output: `ingested N stories for my-project` (where N = your story count).
Verify:
```bash
docker exec damascus-orchestrator-orchestrator-1 \
damascus list --project my-project --limit 5
```
All rows should show `phase=spec`. If any show `phase=awaiting_human`, the spec-refiner asked questions — see "Handling human questions" below.
### Step 5 — Let the cycle run
The orchestrator's scheduler fires `damascus cycle` every 60 seconds (see `orchestrator-scheduler` logs). Each cycle claims one row, advances it through `spec → build → review → merged`. With 1 worker thread, expect one row every ~5-15 minutes depending on story complexity.
To watch live:
```bash
docker logs -f damascus-orchestrator-orchestrator-scheduler-1
docker logs -f damascus-orchestrator-orchestrator-1
```
To inspect a specific row:
```bash
docker exec damascus-orchestrator-orchestrator-1 \
damascus show <work-item-id>
```
---
## Monitoring (recommended)
Set up a board watchdog so you get Discord pings on state changes (new tasks, blocked, done):
```bash
# 1. Copy the template
cp /root/.hermes/skills/devops/kanban-orchestrator/scripts/board-watchdog.sh \
~/.hermes/scripts/my-project-watchdog.sh
# 2. Edit the BOARD= line at the top
sed -i 's|^BOARD=.*|BOARD="my-project"|' ~/.hermes/scripts/my-project-watchdog.sh
# 3. Create the cron (no_agent, Discord-delivered)
hermes cron create "every 1m" \
"Watch my-project board; deliver state changes to Discord." \
--no-agent \
--script my-project-watchdog.sh \
--deliver discord
```
The watchdog is silent when the board is stable, pings Discord when rows transition (claimed → done → blocked). See `bmad/_kit/sample/hello-bmad/` or the existing `damascus-orchestrator-watchdog.sh` for a worked example.
---
## Handling human questions
When the spec-refiner asks a clarifying question, the row enters `phase=awaiting_human` and a `human_issues` row opens. You can see them:
```bash
docker exec damascus-orchestrator-orchestrator-1 \
damascus questions
```
Or via the dashboard at `https://<host>:9110/` (the React UI shows open human issues with full markdown rendering and inline answer forms — see `t_5aa80e4b` if that feature is in flight on your version).
To answer:
```bash
# 1. Get the issue ID
docker exec damascus-orchestrator-orchestrator-1 \
damascus questions
# 2. Answer it
docker exec damascus-orchestrator-orchestrator-1 \
damascus answer <issue-uuid> "your answer text"
# 3. The next cycle resumes the row, re-runs the refiner with your answer in context
```
To answer in bulk (when the same question comes up repeatedly), write the answer into the story's `## Ambiguities` section in the BMAD output and re-ingest. The refiner reads the ambiguities as guidance.
---
## Common pitfalls (learned the hard way)
### 1. `Path.rglob` doesn't follow symlinks
If you symlink `planning-artifacts/stories``../implementation-artifacts/stories`, the orchestrator's `find_bmad_story` will not find your stories (Python 3.12 default). Use a real copy or a bind mount.
### 2. `architecture.md` must be at `planning-artifacts/architecture.md` exactly
The spec-refiner hardcodes this path. Putting it at `meta/architecture.md` breaks it silently — the refiner runs without architecture context and produces weak specs.
### 3. Missing story section headers → `spec_wrong`
Stories without all six required sections (`Goal`, `Acceptance Criteria`, `TDD Plan`, `File Scope`, `Test Command`, `Ambiguities`) get `verdict=spec_wrong` and burn 3 retries. Use the template.
### 4. Stories in `implementation-artifacts/stories/` don't ingest
The ingest command only globs `planning-artifacts/**/*.md`. Either move the stories, or bind-mount `implementation-artifacts/` into the container's `planning-artifacts/`.
### 5. The build phase clones from Gitea — make sure the repo exists first
If your Gitea repo doesn't exist or has the wrong default branch, the first build will fail. Verify with:
```bash
curl -s -H "Authorization: token $TOKEN" \
"https://git.homelab.local/api/v1/repos/kaykayyali/my-project" | jq .default_branch
```
### 6. Worktree branch collisions
If two stories try to use the same branch name (default `feat/<story-id>`), the second one's worktree setup fails with a branch-already-exists error. Pick unique story IDs.
### 7. `tokens` API key vs `token` header
When calling the Gitea API manually, the header is `Authorization: token <PAT>`, not `Authorization: Bearer`. Gitea's auth is quirky.
### 8. `architecture.md` gets ingested as a work item (orchestrator quirk)
The orchestrator's `damascus ingest` command globs every `.md` under `planning-artifacts/`. Since `architecture.md` must live there (rule #2), it gets ingested too — as a story with `story_id="architecture"`. This is harmless (the spec-refiner skips it gracefully) but pollutes the work_items table.
**Fix after first ingest**:
```bash
docker exec damascus-orchestrator-db-1 \
psql -U damascus damascus -c \
"DELETE FROM work_items WHERE project='<your-project>' AND story_id='architecture';"
```
Or pre-empt it by renaming: `mv planning-artifacts/architecture.md planning-artifacts/_architecture.md` — but then the refiner won't find it (rule #2). Better to ingest then delete.
---
## Reference: directory layout for the `_kit`
The `bmad/_kit/` directory in this repo contains:
```
bmad/_kit/
├── README.md ← this directory's contract
├── templates/
│ ├── prd.md ← copy + fill for your project's PRD
│ ├── architecture.md ← copy + fill for your project's arch doc
│ ├── epics.md ← copy + fill for the epics summary
│ └── story.md ← copy + fill for each per-story brief
└── sample/
└── hello-bmad/ ← one fully-formed worked example
└── _bmad-output/
├── planning-artifacts/
│ ├── architecture.md
│ └── stories/
│ ├── S1-hello-world.md
│ └── S2-add-endpoint.md
└── meta/
└── prd.md
```
The `_kit` is **read-only reference material**. New projects should **copy** from it, never add to it. If you find yourself wanting to add a new template, that means the orchestrator needs a new capability — file an issue against `kaykayyali/damascus-orchestrator`.
---
## See also
- `bmad/_kit/README.md` — kit-level contract
- `bmad/_kit/sample/hello-bmad/` — worked example
- `src/damascus/cli.py` (`ingest_cmd` function) — the actual ingest logic
- `src/damascus/phases.py` — phase functions (`build`, `refine_spec`, etc.)
- `docs/VERIFICATION.md` — how to verify the orchestrator works after a change
- `wiki/concepts/state-resume-protocol.md` — how the cycle resumes after crashes

260
scripts/test-ingest.sh Executable file
View File

@@ -0,0 +1,260 @@
#!/usr/bin/env bash
# test-ingest.sh — Validate a BMAD project's _bmad-output/ tree BEFORE
# running the real `damascus ingest`. Catches the four classes of bug
# that have cost real cycles on this orchestrator:
#
# 1. Missing required section headers in story files
# (orchestrator's spec-refiner returns `spec_wrong` and burns
# 3 retries per story)
# 2. Symlinks in the tree that Path.rglob won't follow
# (Python 3.12 default — orchestrator's find_bmad_story uses rglob)
# 3. architecture.md missing from planning-artifacts/architecture.md
# (spec-refiner hardcodes this path)
# 4. Story files in implementation-artifacts/ not mirrored to
# planning-artifacts/stories/ (orchestrator only ingests from
# planning-artifacts/)
#
# Usage:
# ./scripts/test-ingest.sh /root/<project>/_bmad-output <project-name>
#
# --check-only run only the local tree validation; don't contact
# the orchestrator container
#
# Exit codes:
# 0 tree is valid and ready to ingest
# 1 validation failure (printed to stderr)
# 2 orchestrator container unreachable (only when not --check-only)
#
# This script does NOT write to the DB. It only validates shape.
set -euo pipefail
BMAD_ROOT="${1:-}"
PROJECT_NAME="${2:-}"
if [ -z "$BMAD_ROOT" ] || [ -z "$PROJECT_NAME" ]; then
echo "usage: $0 <path-to-_bmad-output> <project-name> [--check-only]" >&2
exit 1
fi
CHECK_ONLY=false
if [ "${3:-}" = "--check-only" ]; then
CHECK_ONLY=true
fi
# Resolve to absolute path
BMAD_ROOT=$(cd "$BMAD_ROOT" 2>/dev/null && pwd || { echo "ERROR: $BMAD_ROOT is not a directory" >&2; exit 1; })
echo "=== test-ingest.sh ==="
echo "BMAD root: $BMAD_ROOT"
echo "Project: $PROJECT_NAME"
echo "Mode: $([ "$CHECK_ONLY" = true ] && echo 'check-only (no orchestrator contact)' || echo 'full (will contact orchestrator)')"
echo ""
# ── Check 1: required layout ──────────────────────────────────────────
echo "── Check 1: required layout ──"
FAILED_CHECKS=0
REQUIRED_PATHS=(
"$BMAD_ROOT/planning-artifacts"
"$BMAD_ROOT/planning-artifacts/architecture.md"
)
for p in "${REQUIRED_PATHS[@]}"; do
if [ ! -e "$p" ]; then
echo " ✗ MISSING: $p" >&2
echo " The orchestrator hardcodes this path. Without it, the spec-refiner runs blind." >&2
FAILED_CHECKS=$((FAILED_CHECKS + 1))
else
echo "$p"
fi
done
# Stories must be under planning-artifacts/ OR mirrored there from implementation-artifacts/
STORIES_DIR="$BMAD_ROOT/planning-artifacts/stories"
if [ ! -d "$STORIES_DIR" ]; then
echo " ✗ MISSING: $STORIES_DIR" >&2
echo " Per-story briefs must be at planning-artifacts/stories/ for the orchestrator to ingest them." >&2
FAILED_CHECKS=$((FAILED_CHECKS + 1))
else
echo "$STORIES_DIR"
STORY_COUNT=$(find "$STORIES_DIR" -maxdepth 1 -name '*.md' -type f | wc -l | tr -d ' ')
if [ "$STORY_COUNT" -eq 0 ]; then
echo " ✗ No story files found in $STORIES_DIR" >&2
FAILED_CHECKS=$((FAILED_CHECKS + 1))
else
echo " ✓ Found $STORY_COUNT story file(s)"
fi
# Check if there's also an implementation-artifacts/ that needs to be in sync
IMPL_STORIES="$BMAD_ROOT/../implementation-artifacts/stories"
if [ -d "$IMPL_STORIES" ] && [ ! -L "$STORIES_DIR" ]; then
IMPL_COUNT=$(find "$IMPL_STORIES" -maxdepth 1 -name '*.md' -type f | wc -l | tr -d ' ')
if [ "$IMPL_COUNT" -ne "$STORY_COUNT" ]; then
echo " ⚠ WARNING: implementation-artifacts/stories/ has $IMPL_COUNT files, planning-artifacts/stories/ has $STORY_COUNT." >&2
echo " If you use the standard BMAD layout, copy or bind-mount the stories into planning-artifacts/stories/." >&2
fi
fi
fi
# ── Check 2: no symlinks that rglob won't follow ──────────────────────
echo ""
echo "── Check 2: symlink audit (Path.rglob won't follow these in Python 3.12) ──"
SYM_COUNT=0
SYM_FILES=()
while IFS= read -r -d '' link; do
SYM_COUNT=$((SYM_COUNT + 1))
SYM_FILES+=("$link")
done < <(find "$BMAD_ROOT" -type l -print0 2>/dev/null || true)
if [ "$SYM_COUNT" -gt 0 ]; then
for link in "${SYM_FILES[@]}"; do
echo " ✗ SYMLINK: $link$(readlink "$link")" >&2
done
echo " Replace with a real copy or a bind mount (see docs/adding-a-new-project.md)." >&2
FAILED_CHECKS=$((FAILED_CHECKS + 1))
else
echo " ✓ No symlinks in the tree"
fi
# ── Check 3: required story section headers ───────────────────────────
echo ""
echo "── Check 3: required section headers in every story ──"
REQUIRED_SECTIONS=(
"## Goal"
"## Acceptance Criteria"
"## TDD Plan"
"## File Scope"
"## Test Command"
"## Ambiguities"
)
BAD_COUNT=0
while IFS= read -r story; do
story_basename=$(basename "$story")
missing=()
for section in "${REQUIRED_SECTIONS[@]}"; do
if ! grep -qF "$section" "$story"; then
missing+=("$section")
fi
done
if [ "${#missing[@]}" -gt 0 ]; then
BAD_COUNT=$((BAD_COUNT + 1))
echo "$story_basename — missing sections: ${missing[*]}" >&2
else
echo "$story_basename"
fi
done < <(find "$STORIES_DIR" -maxdepth 1 -name '*.md' -type f)
if [ "$BAD_COUNT" -gt 0 ]; then
echo "" >&2
echo " $BAD_COUNT story file(s) have missing sections." >&2
echo " The orchestrator's spec-refiner returns 'spec_wrong' for each one and burns 3 retries." >&2
echo " Fix: copy from bmad/_kit/templates/story.md and re-run." >&2
FAILED_CHECKS=$((FAILED_CHECKS + 1))
fi
# ── Check 4: every story has a non-empty Test Command ────────────────
echo ""
echo "── Check 4: Test Command has a real shell command ──"
EMPTY_CMD_COUNT=0
while IFS= read -r story; do
story_basename=$(basename "$story")
# Extract everything between "## Test Command" and the next ## heading
cmd=$(awk '/^## Test Command/{flag=1; next} /^## /{flag=0} flag' "$story" | sed '/^```/d; /^$/d' | head -5)
if [ -z "$(echo "$cmd" | tr -d '[:space:]')" ]; then
EMPTY_CMD_COUNT=$((EMPTY_CMD_COUNT + 1))
echo "$story_basename — Test Command is empty" >&2
fi
done < <(find "$STORIES_DIR" -maxdepth 1 -name '*.md' -type f)
if [ "$EMPTY_CMD_COUNT" -gt 0 ]; then
echo "" >&2
echo " $EMPTY_CMD_COUNT story file(s) have empty Test Commands." >&2
echo " The orchestrator will run 'echo no test command' which always passes — your story ships unverified." >&2
FAILED_CHECKS=$((FAILED_CHECKS + 1))
else
echo " ✓ All Test Commands populated"
fi
# ── Optional Check 5: live orchestrator dry-run ───────────────────────
if [ "$CHECK_ONLY" = false ] && [ "$FAILED_CHECKS" -eq 0 ]; then
echo ""
echo "── Check 5: live orchestrator dry-run ──"
# Check the orchestrator container is reachable
if ! docker exec damascus-orchestrator-orchestrator-1 true 2>/dev/null; then
echo " ✗ Orchestrator container not reachable" >&2
echo " Either bring it up ('docker compose up -d orchestrator') or re-run with --check-only" >&2
exit 2
fi
# Verify the bind mount is in place inside the container
CONTAINER_PATH="/opt/damascus/bmad/$PROJECT_NAME/_bmad-output"
if ! docker exec damascus-orchestrator-orchestrator-1 test -d "$CONTAINER_PATH" 2>/dev/null; then
echo "$CONTAINER_PATH not visible inside orchestrator container" >&2
echo " Add a bind mount to docker-compose.yml:" >&2
echo " - $BMAD_ROOT:$CONTAINER_PATH:ro" >&2
echo " Then 'docker compose up -d --force-recreate --no-deps orchestrator'" >&2
exit 1
fi
echo " ✓ Bind mount visible inside container at $CONTAINER_PATH"
# Run the actual dry-run ingest
echo ""
echo " Running: damascus ingest --project $PROJECT_NAME --dry-run"
if ! docker exec damascus-orchestrator-orchestrator-1 \
damascus ingest --project "$PROJECT_NAME" --dry-run 2>&1; then
echo " ✗ Dry-run ingest failed" >&2
exit 1
fi
echo ""
echo " Now verifying _find_bmad_story can locate each story (the real bottleneck):"
CANNOT_FIND=0
while IFS= read -r story; do
story_basename=$(basename "$story" .md)
# The orchestrator's match is: story_id in f.stem
# story_id comes from Path(f).stem during ingest (the filename without .md)
if ! docker exec damascus-orchestrator-orchestrator-1 \
python3 -c "
from pathlib import Path
import sys
p = Path('$CONTAINER_PATH')
sid = '$story_basename'
found = any(sid in f.stem for f in p.rglob('*.md'))
sys.exit(0 if found else 1)
" 2>/dev/null; then
CANNOT_FIND=$((CANNOT_FIND + 1))
echo "$story_basename — _find_bmad_story won't find this!" >&2
else
echo "$story_basename"
fi
done < <(find "$STORIES_DIR" -maxdepth 1 -name '*.md' -type f)
if [ "$CANNOT_FIND" -gt 0 ]; then
echo "" >&2
echo " $CANNOT_FIND story file(s) cannot be located by the spec-refiner." >&2
echo " This is the symlink-or-missing-section bug. Check:" >&2
echo " - Are there symlinks in the tree? Path.rglob won't follow them." >&2
echo " - Are the story files actually under planning-artifacts/stories/?" >&2
exit 1
fi
fi
echo ""
if [ "$FAILED_CHECKS" -gt 0 ]; then
echo "=== $FAILED_CHECKS check(s) FAILED — fix the issues above and re-run ===" >&2
exit 1
fi
echo "=== All checks passed ==="
echo ""
echo "Next step: docker exec damascus-orchestrator-orchestrator-1 \\"
echo " damascus ingest --project $PROJECT_NAME"