Index /docs markdown into recall (searchable docs)

Round-3: investigated against the LIVE MAIN DB (this design worktree's .env points at DB_DATABASE=flower — the real shared DB; I only ran read-only SELECTs, never migrate/RefreshDatabase). Branch flower/docs-recall-fix3, 1 test-only commit c3c895c, suite green (508 passed, 1 skip), pint clean. NOT merged. ## The docs are NOT actually 0-chunks — the feature works Live evidence: - All 7 /docs (RepoDocs 117–123) each HAVE a surviving chunk (ids 10716–10722), correct text (2.5k–6.4k chars), tags ["doc","section:doc"], a meili_id set, and 1 INDEXED chunk_embedding. - recall_search(query:"idea plan dispatch provenance epic depends-on gating…", sources:[doc]) returns doc #119 (briefs-and-dispatch) RANKED #1, score 0.996, full content. So recall over /docs is live-working right now. - No self-delete: EmbedChunks' cleanup keeps the chunk (sectionTag == the stored section key). New DocsChunkSurvivesTest proves survival across fresh / short / double-run / pre-existing-legacy-chunk cases (all pass). ## Why "0 chunks" was reported (the real gotcha) chunkable_type. RepoDoc is NOT in the Eloquent morphMap (only dispatch_request is), so doc chunks store the FQCN `App\Models\RepoDoc`. A count of `chunks WHERE chunkable_type='doc'` returns 0 even though the chunks exist. recall itself is correct — SOURCE_MODELS['doc'=>RepoDoc::class] filters by that same FQCN — which is why recall finds them while a naive 'doc' count shows 0. My test codifies this (asserts getMorphClass()===RepoDoc::class, count('doc')===0, count(RepoDoc::class)===1). So: no chunk bug, nothing to fix in EmbedChunks/buildDocChunks. ## The ONE real remaining gap (ops, not code) The /docs/{slug} recall pointer (hit.doc = {slug,url:/docs/…}) is MERGED + TESTED (RecallToolsTest::…links_flower_docs green, enrichHit doc block present at RecallService ~L1185) but does NOT appear in the LIVE recall hit above → the served flower MCP is running RecallService code from BEFORE the #45 pointer commit (0122654). Fix = reload the served flower MCP / clear opcache on MAIN (reconnect the stdio server or restart the HTTP MCP), NOT a code change. ## Notes for whoever merges next - fix2 (no-embedder → build chunks anyway, merged) was a harmless robustness improvement but was NOT MAIN's cause (MAIN has 1 active embedder). Leave it. - Optional follow-up (recommend, do NOT rush): add a 'doc' morphMap alias so chunkable_type is consistently 'doc' everywhere — removes this footgun — but it needs a one-time data migration of the 64 existing FQCN doc chunks, so it's its own small brief, not a hotfix. - Bottom line: docs → recall is functionally working on MAIN today; the only visible-to-you gap is the /docs pointer, which a served-MCP reload surfaces.

Round-2 fix on branch flower/docs-recall-fix2 (off master, has the association fix) — full suite green (504 passed, 1 pre-existing skip, 3597 assertions), pint clean. NOT merged. 1 commit 01b7218. ## Repro-first (your instruction) Wrote a test that fails the way MAIN does: DocsChunkWithoutEmbedderTest seeds NO active Embedder, then EmbedChunks::dispatchSync(project) — asserted a doc Chunk is built. It FAILED (0 chunks), reproducing MAIN. My earlier end-to-end test seeded an embedder, which masked exactly this. Confirmed the trace: buildDocChunks IS wired into buildChunks; but EmbedChunks::handle() (line ~70) early-returns when Embedder::where(is_active) is EMPTY — BEFORE buildChunks() runs — so 0 chunk rows are built for ANY source. That's the gap (not association, not docSections). ## Fix (touches a Horizon job — flagged below) handle() now builds + persists the chunk rows FIRST, regardless of embedder availability; it only defers the embed pass + Meili upsert (with a warning) when no embedder is active. So new content is never invisible: rows exist for the DB-search fallback and embed+index on the next embedder-active run. The with-embedder path is unchanged (build → ensureIndex → embed → upsert). Repro test now passes; with-embedder end-to-end still passes; EmbedChunksTest unchanged-green. ## ⚠️ TWO things to do on merge 1. **Graceful Horizon reload** — this changes app/Jobs/EmbedChunks.php (a Horizon-run job). `php artisan horizon:terminate` after merge. 2. **Verify an ACTIVE EMBEDDER exists on MAIN** — this is very likely the real ops root cause. Logic check: if MAIN had an active embedder, the OLD code would NOT have early-returned, buildDocChunks WOULD have built #119's chunk, and docs would already be chunked. They're at 0 → MAIN almost certainly has NO active embedder right now, so EmbedChunks has been no-op'ing for ALL new content (not just docs). Run on MAIN: `Embedder::where('is_active', true)->exists()`. - My fix makes dispatchSync(16) BUILD the doc chunk rows even with no embedder (your "chunks>0" check passes). - BUT recall_search over /docs needs those chunks EMBEDDED into Meili → requires an active embedder. So: activate/confirm an embedder → next EmbedChunks run embeds the now-built doc chunks → the /docs/{slug} recall pointer resolves. (If an active embedder already exists, the 0-chunks had another cause and a re-run will surface it — but the repro points squarely at no-embedder.) Net: the code no longer silently drops all chunk-building when embedders are momentarily/again absent; and the operator-side check is whether MAIN has an active embedder at all.

Bug FIXED on branch flower/docs-recall-fix (off master) — full suite green (501 passed, 1 pre-existing skip, 3578 assertions), pint clean. NOT merged. 1 commit 2a3bf1a (Brief: #45). ## Diagnosis Your root cause was right: the RepoDocs were mis-associated so EmbedChunks' whereHas('project', is_indexed) chunked 0 of them. I reproduced end-to-end (a test that RUNS index-docs then EmbedChunks::dispatchSync) and confirmed: the association code was correct GIVEN a good project resolution — the failure was in HOW the command picked the project. It resolved by config slug/name ('flower'), which on MAIN didn't land on the indexed flower project #16 (a name/slug collision picking a different/unindexed project, and/or config-cache staleness), so the docs got tied to the wrong/absent project → skipped by EmbedChunks. ## Fix Resolve the docs' project the way the repo-doc ingest does — by PATH OWNERSHIP: the project whose root_path is an ancestor of the docs dir. On MAIN that's flower #16 (owns …/flower/resources/docs), deterministically, regardless of slug/config. Refinements: - MOST-SPECIFIC owner wins (…/flower beats a broad …/code ancestor registered at the parent dir) — avoids mis-resolving to a parent project. - Indexed match preferred (both on path-ownership ties and in the config-ref fallback). - HEAL: on each run, drop any docs-prefix repo_docs NOT owned by the resolved project (incl. your existing null/wrong-project rows), so a re-index re-creates them correctly associated + re-embeds. ## Verified end-to-end (your explicit ask) tests/Feature/Docs/DocsRecallEndToEndTest — runs `flower:index-docs` then `EmbedChunks::dispatchSync($project->id)` and asserts a `doc` chunk IS created (0 before the fix). Plus command tests: project association is set; most-specific-owner resolution over a broad ancestor; the heal path removes a mis-associated row + re-associates to the indexed project. ## For you (merge + confirm on MAIN) Merge, then `php artisan flower:index-docs --force`. Output now prints the resolved target, e.g. "… N healed, N upserted → project flower (#16, indexed)". Expect chunks>0 for the doc RepoDocs and the /docs/{slug} recall pointer to resolve. (No Horizon-job code changed — EmbedChunks still untouched; the command creates/associates RepoDocs and dispatches the existing job.)

Built + DONE on branch flower/docs-recall (off master) — full suite green (491 passed, 1 pre-existing skip, 3506 assertions), pint clean. NOT merged. 2 commits, each `Brief: #45`: - 71901af — flower:index-docs command: indexes resources/docs/*.md into repo_docs (one RepoDoc per doc; slug/section/title/body via DocsService; idempotent via content_hash; prunes removed docs; dispatches EmbedChunks). + migration (repo_docs.doc_slug + section) + config.docs.{path,project,path_prefix} + DocsService::body() + hourly schedule entry. - 0122654 — RecallService: a doc hit whose RepoDoc has a doc_slug carries a doc pointer {slug,title,section,url:/docs/{slug}}; ordinary repo docs unchanged. Rides the existing pipeline exactly as specced: creating RepoDoc rows lets the UNCHANGED EmbedChunks::buildDocChunks() chunk+embed them as `doc`. Tests: command (upsert/idempotent/force/prune/embed-dispatch), EmbedChunks builds a `doc` chunk for a flower-docs RepoDoc, recall_search(sources:[doc]) returns it with the /docs pointer. ## ⚠️ Runtime / Horizon flags (as requested) - **EmbedChunks and every other Horizon-run job are UNCHANGED** — I did NOT modify any queued/pipeline job code. `git diff` confirms app/Jobs/EmbedChunks.php untouched. So **no Horizon graceful-reload is needed for job code.** (The command DISPATCHES the existing EmbedChunks job — using it, not changing it.) - **Migration to run on MAIN after merge:** `php artisan migrate` (adds nullable repo_docs.doc_slug + section). Additive/backward-compatible; existing README/docs rows keep doc_slug=null. - **RecallService change is request-time** (MCP server + web), live on the next request after merge — not a Horizon job. - **Scheduler:** added `flower:index-docs` hourly to routes/console.php — schedule:work (Solo proc 968) re-reads each minute, so it picks up without a reload; flagging since it's a new scheduled command. - **Config:** flower.docs.path defaults to resource_path('docs'); project defaults to slug 'flower'. If the flower project isn't is_indexed, the command warns and EmbedChunks skips the docs until it is. ## For the orchestrator (live verification on MAIN, post-merge) I did NOT run real-data verification in the worktree (isolated DB; per CLAUDE.md live real-data belongs on MAIN). After merge on MAIN: `php artisan migrate` → `php artisan flower:index-docs` (should upsert ~7 RepoDocs for the flower /docs) → a real `recall_search("using the flower MCP", sources:[doc])` should return the doc with a `/docs/using-flower-mcp` pointer. Then the hourly schedule keeps it fresh.

Created + specced → PLANNED. The #37 follow-up: index resources/docs/*.md as the existing `doc` chunkable so recall_search surfaces the docs, hits linking to /docs/{slug}. Rides on the existing RepoDoc/doc + EmbedChunks pipeline + DocsService front-matter parsing; idempotent (content-hash). Addresses todo #328. Single dispatch, no blocking questions.

## Goal Make the `/docs` pages (the #37 file-based markdown under `resources/docs/*.md`) **searchable via `recall_search`**, so agents and the operator find documentation the same way they find sessions, briefs, and commits — the natural #37 follow-up. ## Context flower already has a **`doc` chunkable source** + a `RepoDoc` model + the `EmbedChunks` pipeline (it indexes project `docs/**` markdown — e.g. a lounge HANDOFF surfaces as `source=doc`). But the flower user-facing docs live in `resources/docs/` (a non-standard location), so they're almost certainly NOT indexed today. Todo #328 flagged this: 3 dogfood reports show agents expect recall to know repo docs. ## Scope (single dispatch) 1. **Ingest `resources/docs/*.md` as `doc` chunks.** Feed each doc through the existing doc-indexing pipeline — reuse `DocsService` to parse front matter (title / section / slug / body) so metadata is accurate — chunk + embed as the `doc` chunkable type. Prefer extending the existing `RepoDoc`/`doc` path; a small dedicated indexer only if it doesn't cleanly extend. 2. **Link hits back to the reader.** A `recall_search` hit for one of these carries a pointer to **`/docs/{slug}`** (the #37 reader URL) + the doc title/section — not a raw file path — so it's one click to open. 3. **Keep it fresh.** Re-index on change: reuse the repo-doc sync / scheduled path, or a `flower:index-docs` command that upserts by content-hash (idempotent, like brief/commit chunking). ## Acceptance - After indexing, `recall_search("<a phrase from a /docs page>", sources:[doc])` returns the matching doc, ranked, with a `/docs/{slug}` pointer + title/section. - Re-indexing is idempotent (content-hash guarded — no duplicate chunks). - Tests: a docs file is chunked + embedded as `doc`; a search returns it with the /docs pointer. `php artisan test` green + pint. `Brief: #45` trailer. ## Provenance Follow-up to #37 (/docs pages) + todo #328 (index repo docs — 3 dogfood reports asked for it). Created + specced by flower-design (2026-07-01). Rides on the existing `doc` chunkable + `EmbedChunks` + `DocsService`. No blocking questions.

Follow-up to #37 (/docs pages): chunk + embed resources/docs/*.md as the existing `doc` chunkable type so recall_search surfaces the documentation, with hits linking to /docs/{slug}. Rides on the existing RepoDoc/doc + EmbedChunks pipeline + DocsService parsing. Addresses todo #328 (agents repeatedly expect recall to know repo docs).