damascus-orchestrator/docs/adding-a-new-project.md

Adding a New Project to the Damascus Orchestrator

Audience: an engineer or agent onboarding a new project so its stories get picked up by the orchestrator's spec → build → review → merged cycle.

Time estimate: 30 minutes for a small project (≤10 stories); 2–3 hours for a multi-epic project (≥30 stories).

---

TL;DR

# 1. Have your BMAD output ready at /root/<project>/_bmad-output/
#    (see "Layout" section below)
ls /root/my-project/_bmad-output/planning-artifacts/stories/   # should show S1-..., S2-..., etc.

# 2. Validate locally — does NOT touch the DB
./scripts/test-ingest.sh /root/my-project/_bmad-output my-project

# 3. Wire the bind mount in docker-compose.yml
#    (see "Step 3 — Wire the bind mount" below)
docker compose up -d --force-recreate --no-deps orchestrator

# 4. Real ingest
docker exec damascus-orchestrator-orchestrator-1 \
    damascus ingest --project my-project

# 5. Watch the first story run through the cycle
hermes kanban --board my-project list
# or set up a watchdog (see "Monitoring" below)

If anything goes wrong at step 2, fix the BMAD output. If step 4 fails or the stories don't have the right section headers, fix the BMAD output. Do not edit the orchestrator code.

---

What "BMAD" means here

The Damascus orchestrator doesn't run BMAD agents or BMAD workflow skills directly. What it does is ingest pre-written BMAD planning artifacts (PRDs, architecture docs, epics, per-story briefs) and turn each .md file into a work_items row that the orchestrator's cycle picks up.

The relationship:

┌─────────────────────────┐         ┌──────────────────────────┐
│  BMAD planning output   │         │  Damascus orchestrator    │
│  (you write this)       │         │  (picks this up)          │
│                         │         │                           │
│  _bmad-output/          │         │  work_items table         │
│   planning-artifacts/   │  ───>   │   phase=spec rows         │
│    architecture.md      │ ingest  │   one per .md file        │
│    <epic>.md            │         │                           │
│   stories/              │         │  cycle processes them:    │
│    S1-...md             │         │   spec → build → review   │
│    S2-...md             │         │   → merged                │
└─────────────────────────┘         └──────────────────────────┘

If you have a real BMAD project (with bmad-auto skill or BMAD agents generating the artifacts), great — point the orchestrator at the output. If you're writing the artifacts by hand (the common case for ≤30 stories), use the templates in bmad/_kit/templates/ and follow this doc.

---

Layout

The orchestrator expects a specific directory layout inside the container at /opt/damascus/bmad/<project>/_bmad-output/. The host path that bind-mounts to it is whatever you choose (we use /root/<project>/_bmad-output/ by convention; see docker-compose.yml for the actual mapping).

_bmad-output/                          ← root of your project's BMAD output
├── planning-artifacts/                ← INGESTED as work_items (one per .md)
│   ├── architecture.md                ← REQUIRED — read by spec-refiner
│   ├── epics.md                       ← OPTIONAL — meta doc, may live here or in meta/
│   └── stories/                       ← where your per-story briefs live
│       ├── S1-...md                   ← required section headers (see "Story format")
│       ├── S2-...md
│       └── ...
└── meta/                              ← NOT ingested — pure reference docs
    ├── prd.md
    ├── epics.md                       ← if not in planning-artifacts/
    └── ...

Why split meta/ from planning-artifacts/?

The orchestrator's damascus ingest (in src/damascus/cli.py) globs every .md under planning-artifacts/ and treats each as a story. If you put your PRD there, the orchestrator will try to "implement the PRD" as a feature. Keep meta documents (PRD, long epics doc) in meta/ so they're reference material, not work items.

Why must architecture.md live at planning-artifacts/architecture.md exactly?

The spec-refiner reads it via _find_architecture() in src/damascus/phases.py, which hardcodes that path. There's no meta/architecture.md fallback. If you forget this, your refiner runs blind and produces weak specs.

---

Story format — required section headers

Every story .md file must have these H2 section headers. The orchestrator's spec-refiner (phases.py:55-78) parses them out and rejects the story as spec_wrong if any are missing:

# S<n> — <short title>

**Epic**: <E1|E2|...>
**Status**: pending
**Branch**: `feat/<branch-name>`

## Goal

<one paragraph — what the implementation should achieve>

## Acceptance Criteria

- [ ] <testable criterion 1>
- [ ] <testable criterion 2>
- [ ] <testable criterion 3>

## TDD Plan

1. <failing test 1 — what to write before any code>
2. <failing test 2>
3. <failing test 3>

## File Scope

- `<path/to/file-1>`
- `<path/to/file-2>`
- `<path/to/file-3>`

## Test Command

```bash
<exact shell command that proves the story is done>

Ambiguities

<list of open questions for a human, or "(none)" if you resolved them all>

**What happens if a section is missing**: spec-refiner returns `verdict=spec_wrong, missing=['TDD Plan']` and the row gets retried up to 3 times before burning out. **Don't ship stories without these headers.**

**Tip**: copy from `bmad/_kit/templates/story.md` and fill in. Don't hand-author the section names — they're parsed literally.

### Where to put per-story briefs

Two valid layouts:

**Layout A (canonical, recommended)**:

planning-artifacts/stories/S-.md

**Layout B (canonical BMAD layout)** — when your toolchain generates stories here:

implementation-artifacts/stories/S-.md

Layout B alone **does not work** — `phases.py:_find_bmad_story` only scans `planning-artifacts/`. If your toolchain puts stories in `implementation-artifacts/`, you need a **bind mount that copies or symlinks** them into `planning-artifacts/stories/` inside the container. Or move them.

**Don't use a symlink on the host that `Path.rglob` would have to follow.** Python's `pathlib.Path.rglob` (which the spec-refiner uses) does **not** follow symlinks by default in Python ≤3.12. The orchestrator runs Python 3.12. Use a real copy or a bind mount, not a symlink.

---

## Project repo on disk

The orchestrator needs the project's source repo cloned into `/workspace/projects/<project>/` **inside the container**. The cycle's build phase (`phases.py:build()`) clones it from Gitea on first run if it doesn't exist:

If /workspace/projects// doesn't exist when the build phase claims a row, the build returns verdict=tests_failed, error="project repo not found at..."

So your **Gitea repo must exist before the first row's build phase fires**. The `damascus ingest` step doesn't require the repo (ingest only writes to `work_items`), but the build phase does.

### Setup checklist

- [ ] Gitea repo exists at `kaykayyali/<project>` (private, with the user's default branch — usually `main`)
- [ ] Either:
  - The build phase is allowed to clone from Gitea at first run (it will — uses `DAMASCUS_GITEA_TOKEN` env var), OR
  - You pre-clone to `/workspace/projects/<project>/` inside the container via the `projects` named volume

### Worktree behavior

The build phase creates a worktree at `/workspace/worktrees/<project>/<story-id>` for each story. The worktree branch name is `feat/<story-id>`. The orchestrator opens a PR against the project's main branch (uses `git_ops.ensure_worktree()` in `src/damascus/git_ops.py`).

---

## Step-by-step onboarding

### Step 1 — Author the BMAD output

Two paths:

**(a) Hand-author**: copy `bmad/_kit/templates/` to a working dir, fill in the markdown. Use `bmad/_kit/sample/hello-bmad/` as a worked example.

**(b) Use BMAD agents (if you have them)**: run your BMAD `bmad-create-prd` / `bmad-create-architecture` / `bmad-create-story` workflows, point the output at `_bmad-output/`.

Either way, end up with:

/root/my-project/_bmad-output/ ├── planning-artifacts/ │ ├── architecture.md ← required │ └── stories/ │ ├── S1-setup-scaffold.md │ ├── S2-add-feature-x.md │ └── ... └── meta/ ← optional ├── prd.md └── epics.md

### Step 2 — Validate with `scripts/test-ingest.sh`

```bash
cd /root/damascus-orchestrator
./scripts/test-ingest.sh /root/my-project/_bmad-output my-project

This dry-runs the orchestrator's ingest without writing to the DB. It checks:

• All required sections present in every story
• architecture.md is in the right place
• No symlinks (which Path.rglob won't follow)
• The orchestrator's find_bmad_story actually finds each story when the refiner looks for it

Exit code 0 = ready to ingest. Non-zero = fix the BMAD output and re-run.

Step 3 — Wire the bind mount in docker-compose.yml

Add to the orchestrator service's volumes: list:

volumes:
  # ... existing mounts ...
  - /root/my-project/_bmad-output:/opt/damascus/bmad/my-project/_bmad-output:ro

The pattern: /root/<host-dir>/_bmad-output → /opt/damascus/bmad/<project>/_bmad-output.

my-project (the right-hand side) must match the project name you'll pass to damascus ingest.

Then recreate the orchestrator container so it picks up the new mount:

docker compose up -d --force-recreate --no-deps orchestrator

Verify the mount worked:

docker exec damascus-orchestrator-orchestrator-1 \
    ls /opt/damascus/bmad/my-project/_bmad-output/planning-artifacts/stories/ | head -10

Step 4 — Real ingest

docker exec damascus-orchestrator-orchestrator-1 \
    damascus ingest --project my-project

Expected output: ingested N stories for my-project (where N = your story count).

Verify:

docker exec damascus-orchestrator-orchestrator-1 \
    damascus list --project my-project --limit 5

All rows should show phase=spec. If any show phase=awaiting_human, the spec-refiner asked questions — see "Handling human questions" below.

Step 5 — Let the cycle run

The orchestrator's scheduler fires damascus cycle every 60 seconds (see orchestrator-scheduler logs). Each cycle claims one row, advances it through spec → build → review → merged. With 1 worker thread, expect one row every ~5-15 minutes depending on story complexity.

To watch live:

docker logs -f damascus-orchestrator-orchestrator-scheduler-1
docker logs -f damascus-orchestrator-orchestrator-1

To inspect a specific row:

docker exec damascus-orchestrator-orchestrator-1 \
    damascus show <work-item-id>

---

Monitoring (recommended)

Set up a board watchdog so you get Discord pings on state changes (new tasks, blocked, done):

# 1. Copy the template
cp /root/.hermes/skills/devops/kanban-orchestrator/scripts/board-watchdog.sh \
   ~/.hermes/scripts/my-project-watchdog.sh

# 2. Edit the BOARD= line at the top
sed -i 's|^BOARD=.*|BOARD="my-project"|' ~/.hermes/scripts/my-project-watchdog.sh

# 3. Create the cron (no_agent, Discord-delivered)
hermes cron create "every 1m" \
    "Watch my-project board; deliver state changes to Discord." \
    --no-agent \
    --script my-project-watchdog.sh \
    --deliver discord

The watchdog is silent when the board is stable, pings Discord when rows transition (claimed → done → blocked). See bmad/_kit/sample/hello-bmad/ or the existing damascus-orchestrator-watchdog.sh for a worked example.

---

Handling human questions

When the spec-refiner asks a clarifying question, the row enters phase=awaiting_human and a human_issues row opens. You can see them:

docker exec damascus-orchestrator-orchestrator-1 \
    damascus questions

Or via the dashboard at https://<host>:9110/ (the React UI shows open human issues with full markdown rendering and inline answer forms — see t_5aa80e4b if that feature is in flight on your version).

To answer:

# 1. Get the issue ID
docker exec damascus-orchestrator-orchestrator-1 \
    damascus questions

# 2. Answer it
docker exec damascus-orchestrator-orchestrator-1 \
    damascus answer <issue-uuid> "your answer text"

# 3. The next cycle resumes the row, re-runs the refiner with your answer in context

To answer in bulk (when the same question comes up repeatedly), write the answer into the story's ## Ambiguities section in the BMAD output and re-ingest. The refiner reads the ambiguities as guidance.

---

Common pitfalls (learned the hard way)

1. Path.rglob doesn't follow symlinks

If you symlink planning-artifacts/stories → ../implementation-artifacts/stories, the orchestrator's find_bmad_story will not find your stories (Python 3.12 default). Use a real copy or a bind mount.

2. architecture.md must be at planning-artifacts/architecture.md exactly

The spec-refiner hardcodes this path. Putting it at meta/architecture.md breaks it silently — the refiner runs without architecture context and produces weak specs.

3. Missing story section headers → spec_wrong

Stories without all six required sections (Goal, Acceptance Criteria, TDD Plan, File Scope, Test Command, Ambiguities) get verdict=spec_wrong and burn 3 retries. Use the template.

4. Stories in implementation-artifacts/stories/ don't ingest

The ingest command only globs planning-artifacts/**/*.md. Either move the stories, or bind-mount implementation-artifacts/ into the container's planning-artifacts/.

5. The build phase clones from Gitea — make sure the repo exists first

If your Gitea repo doesn't exist or has the wrong default branch, the first build will fail. Verify with:

curl -s -H "Authorization: token $TOKEN" \
    "https://git.homelab.local/api/v1/repos/kaykayyali/my-project" | jq .default_branch

6. Worktree branch collisions

If two stories try to use the same branch name (default feat/<story-id>), the second one's worktree setup fails with a branch-already-exists error. Pick unique story IDs.

7. tokens API key vs token header

When calling the Gitea API manually, the header is Authorization: token <PAT>, not Authorization: Bearer. Gitea's auth is quirky.

8. architecture.md gets ingested as a work item (orchestrator quirk)

The orchestrator's damascus ingest command globs every .md under planning-artifacts/. Since architecture.md must live there (rule #2), it gets ingested too — as a story with story_id="architecture". This is harmless (the spec-refiner skips it gracefully) but pollutes the work_items table.

Fix after first ingest:

docker exec damascus-orchestrator-db-1 \
    psql -U damascus damascus -c \
    "DELETE FROM work_items WHERE project='<your-project>' AND story_id='architecture';"

Or pre-empt it by renaming: mv planning-artifacts/architecture.md planning-artifacts/_architecture.md — but then the refiner won't find it (rule #2). Better to ingest then delete.

---

Reference: directory layout for the _kit

The bmad/_kit/ directory in this repo contains:

bmad/_kit/
├── README.md                            ← this directory's contract
├── templates/
│   ├── prd.md                           ← copy + fill for your project's PRD
│   ├── architecture.md                  ← copy + fill for your project's arch doc
│   ├── epics.md                         ← copy + fill for the epics summary
│   └── story.md                         ← copy + fill for each per-story brief
└── sample/
    └── hello-bmad/                      ← one fully-formed worked example
        └── _bmad-output/
            ├── planning-artifacts/
            │   ├── architecture.md
            │   └── stories/
            │       ├── S1-hello-world.md
            │       └── S2-add-endpoint.md
            └── meta/
                └── prd.md

The _kit is read-only reference material. New projects should copy from it, never add to it. If you find yourself wanting to add a new template, that means the orchestrator needs a new capability — file an issue against kaykayyali/damascus-orchestrator.

---

See also

• bmad/_kit/README.md — kit-level contract
• bmad/_kit/sample/hello-bmad/ — worked example
• src/damascus/cli.py (ingest_cmd function) — the actual ingest logic
• src/damascus/phases.py — phase functions (build, refine_spec, etc.)
• docs/VERIFICATION.md — how to verify the orchestrator works after a change
• wiki/concepts/state-resume-protocol.md — how the cycle resumes after crashes