Pipeline memory/scale hardening: refactor indexing/embedding/summarizing to O(changed) not O(corpus) — deep review + adversarial review

You are being dispatched from flower Brief #193: Pipeline memory/scale hardening: refactor indexing/embedding/summarizing to O(changed) not O(corpus) — deep review + adversarial review

Recall pointer:
- Use recall_brief with id 193 for the full folder if you need provenance.

Target:
- project: flower (/Users/mikeferrara/Documents/code/flower)
- branch: flower/193-filesort-fix
- worktree: not specified
- kind: fresh

Current brief spec:
## ⚠️ REOPENED 2026-07-04 — real-data regression, REVERTED (revert commit 2bc99d4, reverting merge e5b6082)
The first #193 O(changed) refactor PASSED an independent Claude adversarial review AND the full sqlite suite (945 green) — but FAILED on real MySQL: the Phase-A reconcile paging query `select * from commits where exists (select * from projects ...)` triggers MySQL error 1038 "Out of sort memory". It filesorts WIDE rows (chunks/commits carry large text — chunks.text avg 1.4KB, max 565KB) and Herd's default `sort_buffer_size` (262144 = 256KB) can't hold them. Every scheduled flower:embed failed ~1/2min. Reverted to restore the old O(corpus) code + the 4G php.ini stopgap (which holds). LESSON: the review + tests verified PHP-memory O(changed) correctly but both missed the MySQL SERVER-SIDE sort limit — because sqlite has no sort_buffer and the test corpus is tiny. Real-MySQL verification is mandatory.

## Fix (re-ship #193 correctly)
1. Re-apply the O(changed) refactor — the branch `flower/193-pipeline-memory-hardening` still has it (two-phase paged reconcile+sync, `meili_synced_at` watermark, config memory_limit). Rebase it onto current master (which now has #155/#199 but not #193).
2. FIX THE FILESORT: reconcile paging queries must NOT filesort wide rows. Likely fixes: `->select([...only the columns reconcile needs...])` so the big `text`/body columns aren't pulled into the sort; confirm `chunkById` actually uses the PK index (the whereHas/`exists` subquery may force a plan change → filesort — check EXPLAIN); `->reorder('id')` if an inherited orderBy is the culprit. Apply to ALL source reconcile queries (commits, segments, briefs, todos, scratchpads, docs), not just commits.
3. Do NOT depend on raising MySQL `sort_buffer_size` — it's shared server config (the auto-mode guard correctly blocks mutating it) and not version-controlled.
4. VERIFY ON REAL MYSQL BEFORE DECLARING DONE (mandatory — this is exactly what the first attempt skipped): run `flower:embed` against MAIN's real corpus (~11.5k chunks; note `meili_synced_at` is currently NULL for all rows since the column survived the revert → first pass re-syncs everything, EXPLAIN the reconcile queries) and confirm zero error 1038 + bounded PHP memory + the backlog drains.
5. Keep the Goal #4 O(changed) regression test; if feasible add coverage that exercises the reconcile query shape against wide rows.
6. `php artisan test` green + `./vendor/bin/pint`. `Brief: #193` trailer. Worktree-pinned; never edit MAIN. Phase-2 review before re-merge.

## Provenance
Shipped (merge e5b6082) → reverted (2bc99d4) same day after real-data failure. Original root cause + O(changed) design in the brief history. auto_dispatch_on_planned stays true.

Recent/key trace events:
[10] note_added flower-193-worker: Phase 1 complete — EmbedChunks made O(changed), not O(corpus). Branch `flower/193-pipeline-memory-hardening` (rebased onto master 811c4e2 so it carries #189; the worktree HEAD b285444 predated it). 3 commits, each `Brief: #193`:
- 50a24fe — migration `chunks.meili_synced_at` (nullable ts, indexed; 28-char index name; after() is a sqlite no-op) + config `embed.memory_limit` (1024M) / `embed.reconcile_page_size` / `embed.sync_page_size` (200).
- aab1d2a — the EmbedChunks refactor.
- 0c220b0 — the anti-recurrence regression test (Goal #4).

CONFIRMED + EXTENDED ROOT CAUSE: `flower:embed --queue` is scheduled everyTwoMinutes with projectId=null (FlowerServiceProvider:147), i.e. over the WHOLE corpus. `handle()` re-processed everything each run — `buildChunks()` loaded all ~11.5k chunks into one Collection; an N+1 `currentEmbedding()` per chunk; every already-indexed chunk's stored vector loaded into `$vectorsByChunk` (~9.5k); the Meili upsert built a doc per chunk carrying those vectors (held twice) and re-upserted ALL of them. Peak = O(total corpus) → 512MB OOM in guzzle psr7 (the allocation that tips a full heap). #189/Meili/FLOWER-K each capped a single PAYLOAD, not this accumulation.

WHAT CHANGED (app/Jobs/EmbedChunks.php): split into two decoupled, paged phases whose peak memory is O(page + changed), independent of corpus size.
- Phase A `reconcileChunks()`: each source type (segments/briefs/docs/commits/todos/scratchpads) now pages with `chunkById(reconcile_page_size)` and updateOrCreates its chunk rows in place — no corpus-wide Collection, no relation-setting. delete-on-change + stale-section pruning preserved verbatim.
- Phase B `embedAndIndexPending()`: selects ONLY dirty chunks at the QUERY level via `needsSyncQuery()` (never-synced OR `updated_at > meili_synced_at` OR fewer `indexed` embeddings than active embedders — a correlated count(distinct) subquery), pages with `chunkById(sync_page_size)` eager-loading each page's `project/worktree/embeddings` (the N+1 is gone), embeds only the pending ones under the #189 byte bounds, upserts that page's docs to Meili INCREMENTALLY, then stamps `meili_id + meili_synced_at` via the BASE query builder (so updated_at is NOT bumped — otherwise the chunk would perpetually re-select itself) and marks embeddings indexed. Already-indexed, unchanged chunks are never loaded and never re-upserted — the ~9.5k stored vectors are no longer materialized.
- `advanceSessionStates()` now takes the bounded set of touched segment ids (accumulated across pages) instead of a corpus-wide chunk collection.
- Config-driven per-job `memory_limit` applied at handle() start (folds the Herd php84 4G stopgap into version-controlled config; skipped under phpunit so it can't lower the suite's limit). The 4G php.ini stopgap can be reverted once this merges.

HOW MEMORY IS NOW BOUNDED: reconcile hydrates ≤ reconcile_page_size source rows at a time; embed/index hydrates ≤ sync_page_size chunks + their embeddings, embeds/upserts per page, frees between pages. In steady state the dirty set is ~0, so a run does ~0 embed/upsert work regardless of how large the corpus is. Vectors are only ever loaded for the small changed set being re-upserted.

GOAL #4 (anti-recurrence test, tests/Feature/Search/EmbedChunksTest.php::test_embed_work_is_bounded_by_changed_not_total_corpus): black-box invariant — a steady-state run over a fully-indexed corpus does 0 embeds + 0 upserts, and a single new chunk costs exactly 1 embed + 1 upsert whether the corpus is 10 or 51 (constant → O(changed)). If the O(corpus) shape returned, steady-state would re-upsert the whole corpus and the +1 counts would grow with corpus size — the test would fail.

GOAL #5 (review summarize/ingest/watch for the same shape): reviewed — no analogous fix needed. `IngestSession` and `SegmentSession` are per-session jobs (SegmentSession already chunks map/reduce per FLOWER-K; IngestSession batches event inserts at 500). `flower:watch` (ScansHarnessSessions) is a bounded fan-out: it iterates session-file refs, dedups on a signature key set, and dispatches ONE IngestSession per changed session — it never accumulates transcript content. EmbedChunks was the sole O(corpus)-per-run job.

BEHAVIOR PRESERVED: idempotency; graceful no-key path (rows still built for the DB fallback, embed/index deferred with a count-only log); re-embed-on-text/hash-change; reuse-mysql-vectors-after-meili-fail (meili_id/meili_synced_at stay NULL on a failed upsert → reselected → reuse stored vector, no re-embed); session-state advancement (Indexed/Embedded/Error self-heal); indexed-project scoping; #189 request byte bounds + the Meili payload bound (untouched).

QUALITY GATES: `MEILISEARCH_KEY=LARAVEL-HERD ~/bin/php artisan test` → 927 tests, 925 passed, 2 skipped (pre-existing env-gated), 0 failed. Pint clean on all changed files. Migration is sqlite-portable (whole suite runs it under RefreshDatabase). Did NOT run migrate against MAIN's shared MySQL, did NOT start any daemon, did NOT merge — leaving Phase-2 adversarial review + merge + Horizon reload + real-data verification to the orchestrator.

NOTE for reviewers: the `updated_at > meili_synced_at` clause uses strict `>` (>= would re-select every chunk every run since the sync stamp is >= the row's updated_at). Pure metadata-only changes that don't touch text AND land in the same clock-second as the last sync are eventually-consistent rather than instant; text/hash changes are always caught via the embedding-incompleteness clause. Files: app/Jobs/EmbedChunks.php, config/flower.php, database/migrations/2026_07_04_120000_add_meili_synced_at_to_chunks_table.php, tests/Feature/Search/EmbedChunksTest.php.
[11] dispatched flower-193-worker: Dispatch request #78 marked done.
[12] status_change flower-193-worker: (no body)
[13] review_requested flower-orchestrator: Phase-2 dual-harness adversarial review (dogfooding the review flow, per the brief's process): one Claude reviewer + one Codex reviewer independently verify the O(changed) embed refactor on branch flower/193-pipeline-memory-hardening BEFORE the orchestrator merges. Both must PASS. Focus: does it ACTUALLY bound memory (steady-state loads 0 unchanged chunks), the strict `>` vs `>=` meili_synced_at clause (stale-index risk?), re-embed-on-change preserved, idempotency + graceful no-key path, migration MySQL-safe, and whether the Goal #4 test truly asserts O(changed).
[14] status_change flower-orchestrator: (no body)
[15] review_passed flower-orchestrator: PASS — Independent Claude adversarial review PASS: traced all 8 risk points + ran the 21 EmbedChunks tests. Memory provably O(page+changed) (steady-state loads zero unchanged chunks); strict `updated_at > meili_synced_at` is SAFE (content changes caught by the embedding-incompleteness clause, not the timestamp); deletes remain the separate pruner's job (no regression); Goal #4 test is a genuine invariant. Non-blocking P2: keyless-degraded state does O(corpus) *work* (bounded memory) each run; steady-state correlated count() subquery is modest recurring CPU. Codex cross-harness reviewer was blocked by a sentry-MCP boot hang (flagged separately) — single deep Claude review + orchestrator merge validation stood in.
[16] status_change flower-orchestrator: (no body)
[17] merged flower-orchestrator: Merged to master (commits 50a24fe/aab1d2a/0c220b0). `add_meili_synced_at_to_chunks_table` migration ran on MAIN MySQL (141ms). Full combined suite 945 tests / 944 passed / 1 skipped / 0 failed. Horizon reloaded (SIGTERM master pid 22434 → Solo auto-restart) so the new O(changed) EmbedChunks code is live. Next: verify on real data that the embed backlog drains with bounded memory + no OOM. Follow-up: the Herd php84 4G php.ini stopgap can now be reverted (new code caps per-job memory_limit to 1024M default, peak is tens of MB).
[18] spec_snapshot flower-orchestrator: Refactor flower's ingest→summarize→embed→Meili pipeline so per-run memory is **O(changed), not O(corpus)**.

ROOT CAUSE (confirmed on real data): `App\Jobs\EmbedChunks` re-processes the ENTIRE corpus every ~2 min — `buildChunks()` loads all ~11.5k chunks (all segments/commits/briefs/docs/todos/scratchpads) into memory + an N+1 currentEmbedding() query per chunk, and the Meili upsert re-loads every already-indexed chunk's vector (~9.5k) into `$vectorsByChunk` + re-builds a Meili document per chunk carrying those vectors + re-upserts ALL of them, even though almost nothing changed. Vectors are ~25–60KB each in PHP × ~9.5k held twice ≈ 0.5–1 GB → exceeds 512MB; the crash surfaces in guzzle psr7 only because that's the allocation that tips an already-full heap. Prior fixes (FLOWER-K summarize chunking, Meili payload bound, #189 embed request bound) each capped a single PAYLOAD, not this per-run accumulation, so it recurs and worsens as data grows.

GOALS:
1. Only build/embed/upsert content that is NEW or CHANGED since last run — filter at the QUERY level (not load-all-then-check-in-PHP); page/cursor (chunkById/lazy); upsert to Meili incrementally per batch; free memory between batches; stop re-loading + re-upserting unchanged vectors/documents. Fix the N+1.
2. Config-driven embed memory_limit (a 4G stopgap is live in Herd php84 php.ini — make it a proper per-job config value; note the stopgap can be reverted once the real fix lands).
3. Keep bounded/streamed HTTP payloads (#189 request bound + the Meili bound); add response-side safety if feasible.
4. A REGRESSION TEST that fails if a pipeline job's peak memory / loaded-row-count scales with total corpus size (seed a large corpus, assert bounded). This is what stops the recurrence.
5. Apply the same "process everything every run" review to SegmentSession/summarize + flower:watch/ingest and fix analogously.

PROCESS (operator-directed):
- Phase 1 — dedicated Claude agent: deep-review the whole pipeline, confirm/extend this root cause, write a concrete refactor design, then implement it. Worktree-pinned; NEVER edit MAIN; `php artisan test` green + pint; preserve behavior (idempotency, graceful no-key path, session state advancement, re-embed-on-change). Decompose into reviewable steps if large.
- Phase 2 — adversarial review dogfooding our own review flow: one Claude reviewer + one Codex reviewer independently review (correctness, does-it-actually-bound-memory, regressions, edge cases) via brief_request_review / brief_review BEFORE the orchestrator merges. Reconcile, merge, reload Horizon, verify on real data (backlog drains, peak memory bounded, no OOMs).

Full evidence + detail in the brief note. `Brief: #193` trailer.
[19] refinement flower-orchestrator: ## ⚠️ REOPENED 2026-07-04 — real-data regression, REVERTED (revert commit 2bc99d4, reverting merge e5b6082)
The first #193 O(changed) refactor PASSED an independent Claude adversarial review AND the full sqlite suite (945 green) — but FAILED on real MySQL: the Phase-A reconcile paging query `select * from commits where exists (select * from projects ...)` triggers MySQL error 1038 "Out of sort memory". It filesorts WIDE rows (chunks/commits carry large text — chunks.text avg 1.4KB, max 565KB) and Herd's default `sort_buffer_size` (262144 = 256KB) can't hold them. Every scheduled flower:embed failed ~1/2min. Reverted to restore the old O(corpus) code + the 4G php.ini stopgap (which holds). LESSON: the review + tests verified PHP-memory O(changed) correctly but both missed the MySQL SERVER-SIDE sort limit — because sqlite has no sort_buffer and the test corpus is tiny. Real-MySQL verification is mandatory.

## Fix (re-ship #193 correctly)
1. Re-apply the O(changed) refactor — the branch `flower/193-pipeline-memory-hardening` still has it (two-phase paged reconcile+sync, `meili_synced_at` watermark, config memory_limit). Rebase it onto current master (which now has #155/#199 but not #193).
2. FIX THE FILESORT: reconcile paging queries must NOT filesort wide rows. Likely fixes: `->select([...only the columns reconcile needs...])` so the big `text`/body columns aren't pulled into the sort; confirm `chunkById` actually uses the PK index (the whereHas/`exists` subquery may force a plan change → filesort — check EXPLAIN); `->reorder('id')` if an inherited orderBy is the culprit. Apply to ALL source reconcile queries (commits, segments, briefs, todos, scratchpads, docs), not just commits.
3. Do NOT depend on raising MySQL `sort_buffer_size` — it's shared server config (the auto-mode guard correctly blocks mutating it) and not version-controlled.
4. VERIFY ON REAL MYSQL BEFORE DECLARING DONE (mandatory — this is exactly what the first attempt skipped): run `flower:embed` against MAIN's real corpus (~11.5k chunks; note `meili_synced_at` is currently NULL for all rows since the column survived the revert → first pass re-syncs everything, EXPLAIN the reconcile queries) and confirm zero error 1038 + bounded PHP memory + the backlog drains.
5. Keep the Goal #4 O(changed) regression test; if feasible add coverage that exercises the reconcile query shape against wide rows.
6. `php artisan test` green + `./vendor/bin/pint`. `Brief: #193` trailer. Worktree-pinned; never edit MAIN. Phase-2 review before re-merge.

## Provenance
Shipped (merge e5b6082) → reverted (2bc99d4) same day after real-data failure. Original root cause + O(changed) design in the brief history. auto_dispatch_on_planned stays true.
[20] status_change flower-orchestrator: (no body)
[21] note_added flower-orchestrator: RE-APPLIED + WORKING (operator go-ahead 2026-07-04). Un-reverted (commit 0289f3a "Reapply Merge…"), config:clear, migration no-op (column survived the revert). Raised `sort_buffer_size` 256KB→128MB via `SET PERSIST` (operator-authorized; persists across MySQL restarts). Horizon reloaded (new master 58484). VERIFIED ON REAL DATA: inline `flower:embed` over the full corpus completed clean ("Done"), zero error 1038, ALL 11,554 chunks now have meili_synced_at set (first full backfill done), failed_jobs=0. So #193's O(changed) code is LIVE + working; steady-state is now truly O(changed) and the 512MB PHP-OOM root cause is fixed.

REMAINING SCOPE (keep #193 active, lower urgency now that the pipeline is healthy): fix the Phase-A reconcile filesort per spec step 2 (`->select([narrow cols])` / index-ordered paging / EXPLAIN) so the 128MB sort_buffer stopgap can be dropped. Verify the fix on REAL MySQL (the mandatory step). Codex Phase-2 reviewer: retry with a FRESH session — operator confirms a new session clears the sentry-MCP boot hang.

Recommended linked context:
{
    "todos": [],
    "scratchpads": [
        {
            "id": 386,
            "solo_scratchpad_id": "1078",
            "name": "flower-orchestrator (daemon 25) — reset handoff (2026-07-04 #3)",
            "archived": false,
            "revision": 1
        }
    ]
}

Execution notes:
- Treat the brief as the source of truth.
- Keep work scoped to this dispatch request.
- Use brief_append / brief_update_status when reporting material progress; as your final dispatched-worker step, call brief_dispatch_complete with dispatch_request_id (or brief_id) and actor_ref.
- Codex workers should verify mutating Flower tools with tool_search query `brief_append brief_dispatch_complete flower_feedback` (limit 20) when tool availability is in doubt; report raw SEE/LOAD vs NOT visible instead of silently using local fallbacks.
- Add a git commit trailer `Brief: #193` to every commit for this brief so flower can exact-link commits back to the brief.

You are being dispatched from flower Brief #193: Pipeline memory/scale hardening: refactor indexing/embedding/summarizing to O(changed) not O(corpus) — deep review + adversarial review

Recall pointer:
- Use recall_brief with id 193 for the full folder if you need provenance.

Target:
- project: flower (/Users/mikeferrara/Documents/code/flower)
- branch: flower/193-pipeline-memory-hardening
- worktree: not specified
- kind: fresh

Current brief spec:
Refactor flower's ingest→summarize→embed→Meili pipeline so per-run memory is **O(changed), not O(corpus)**.

ROOT CAUSE (confirmed on real data): `App\Jobs\EmbedChunks` re-processes the ENTIRE corpus every ~2 min — `buildChunks()` loads all ~11.5k chunks (all segments/commits/briefs/docs/todos/scratchpads) into memory + an N+1 currentEmbedding() query per chunk, and the Meili upsert re-loads every already-indexed chunk's vector (~9.5k) into `$vectorsByChunk` + re-builds a Meili document per chunk carrying those vectors + re-upserts ALL of them, even though almost nothing changed. Vectors are ~25–60KB each in PHP × ~9.5k held twice ≈ 0.5–1 GB → exceeds 512MB; the crash surfaces in guzzle psr7 only because that's the allocation that tips an already-full heap. Prior fixes (FLOWER-K summarize chunking, Meili payload bound, #189 embed request bound) each capped a single PAYLOAD, not this per-run accumulation, so it recurs and worsens as data grows.

GOALS:
1. Only build/embed/upsert content that is NEW or CHANGED since last run — filter at the QUERY level (not load-all-then-check-in-PHP); page/cursor (chunkById/lazy); upsert to Meili incrementally per batch; free memory between batches; stop re-loading + re-upserting unchanged vectors/documents. Fix the N+1.
2. Config-driven embed memory_limit (a 4G stopgap is live in Herd php84 php.ini — make it a proper per-job config value; note the stopgap can be reverted once the real fix lands).
3. Keep bounded/streamed HTTP payloads (#189 request bound + the Meili bound); add response-side safety if feasible.
4. A REGRESSION TEST that fails if a pipeline job's peak memory / loaded-row-count scales with total corpus size (seed a large corpus, assert bounded). This is what stops the recurrence.
5. Apply the same "process everything every run" review to SegmentSession/summarize + flower:watch/ingest and fix analogously.

PROCESS (operator-directed):
- Phase 1 — dedicated Claude agent: deep-review the whole pipeline, confirm/extend this root cause, write a concrete refactor design, then implement it. Worktree-pinned; NEVER edit MAIN; `php artisan test` green + pint; preserve behavior (idempotency, graceful no-key path, session state advancement, re-embed-on-change). Decompose into reviewable steps if large.
- Phase 2 — adversarial review dogfooding our own review flow: one Claude reviewer + one Codex reviewer independently review (correctness, does-it-actually-bound-memory, regressions, edge cases) via brief_request_review / brief_review BEFORE the orchestrator merges. Reconcile, merge, reload Horizon, verify on real data (backlog drains, peak memory bounded, no OOMs).

Full evidence + detail in the brief note. `Brief: #193` trailer.

Recent/key trace events:
[1] participant_joined flower-orchestrator: (no body)
[2] note_added flower-orchestrator: ## Why this brief exists
Multiple days of recurring 512MB OOMs in the pipeline (FLOWER-K summarize, Meili-413 payload, FLOWER-1R/1Q/J embed — feedback #90 / briefs #189 et al). Each fix bounded a single HTTP *payload* and the problem came back in a new spot as the corpus grew. This brief targets the **shared root cause**, not the next symptom. Operator directive (2026-07-04): raise memory_limit as an immediate stopgap AND engineer the pipeline so this can't keep recurring; do it as a dedicated Claude deep-review/refactor followed by an adversarial review dogfooding our own review system (one Claude reviewer + one Codex reviewer).

## Confirmed root cause (orchestrator investigation, evidence below)
`App\Jobs\EmbedChunks` processes the **entire corpus in one job invocation every ~2 min** (scheduler dispatches it with projectId=null = all indexed projects):
- `buildChunks()` loads ALL chunks for ALL content types into one in-memory Collection: `SessionSegment::...->get()` (all 3,001 segments) + all 3,953 commits + 187 briefs + all docs/todos/scratchpads → ~11,472 Chunk rows (15.2 MB of text) materialized every run, plus an N+1 `currentEmbedding()` query per chunk.
- The Meili upsert re-materializes the WHOLE corpus: for every already-indexed chunk it loads the stored vector into `$vectorsByChunk` (9,463 of them), then `$documents = $chunks->map(documentFor(...vectors))` builds a Meili doc per chunk carrying those vectors, then `upsertDocuments($documents)` re-upserts all 11,472 — even though almost nothing changed since the last run.
- Vectors are ~25–60 KB each as PHP float arrays; ~9,500 held (twice: `$vectorsByChunk` + `$documents`) ≈ 500 MB–1 GB → exceeds the 512 MB limit. The crash surfaces in guzzle psr7 (`Utils.php`/`Stream.php`) only because that is the allocation that tips an already-full heap; bounding the request payload (#189) therefore did not help.
- Memory is **O(total indexed corpus)** and grows every day → recurring + worsening OOM. This "process everything every run" shape likely repeats on the summarize + ingest sides too (in scope to review).

## Evidence (real MySQL, 2026-07-04 ~05:15Z)
chunks=11,472 (avg text 1,386 chars, max 565,908, total 15.2 MB) · segments=3,001 · commits=3,953 · briefs=187 · chunk_embeddings: indexed=9,463 / embedded=960 / pending=772. Fatal: `Allowed memory size of 536870912 bytes exhausted` in guzzlehttp/psr7 Utils/Stream, ~1 per 2 min, MaxAttemptsExceeded on EmbedChunks.

## Goals (the durable fix)
1. **Memory O(changed/batch), not O(corpus).** Only build/embed/upsert content that is NEW or CHANGED since last run — filter at the QUERY level (don't load-all-then-check-in-PHP). Page/cursor through work (`chunkById`/lazy), upsert to Meili incrementally per batch, and free memory between batches. Stop re-loading + re-upserting unchanged vectors/documents.
2. **Config-driven memory_limit safety net** for the embed job (stopgap already applied; fold it in properly).
3. **Bounded/streamed HTTP payloads** (keep #189's request bound + the Meili bound; add response-side safety if feasible).
4. **Regression guard:** a test/asserting invariant that fails if a pipeline job's memory or loaded-row-count scales with total corpus size (e.g. seed a large corpus, assert peak memory / query row counts stay bounded per run). This is the thing that stops the recurrence.
5. Review the **summarize (SegmentSession)** and **ingest (flower:watch)** stages for the same "process everything every run" shape and fix analogously.

## Process (operator-directed)
- **Phase 1 — deep review + refactor:** dedicated Claude agent. Read the whole pipeline (flower:watch/ingest → SegmentSession/summarize → EmbedChunks/embed → MeiliIndexManager), confirm/extend this root-cause, write a concrete refactor design, then implement it on a branch (worktree-pinned). Keep `php artisan test` green + pint. Preserve behavior (idempotency, graceful no-key path, session state advancement, re-embed-on-change).
- **Phase 2 — adversarial review (dogfood our review system):** one Claude reviewer + one Codex reviewer independently review the refactor (correctness, does-it-actually-bound-memory, regressions, edge cases) via flower's brief review flow (brief_request_review / brief_review) before the orchestrator merges. Reconcile findings, then merge + reload Horizon + verify on real data (backlog drains, peak memory bounded, no OOMs).

## Constraints
Worktree-pinned; never edit MAIN. `Brief: #<this>` trailer. Migrations sqlite-portable. This is a big change — decompose into reviewable steps if needed.
[3] link_added flower-orchestrator: (no body)
[4] plan_proposed flower-orchestrator: Refactor flower's ingest→summarize→embed→Meili pipeline so per-run memory is **O(changed), not O(corpus)**.

ROOT CAUSE (confirmed on real data): `App\Jobs\EmbedChunks` re-processes the ENTIRE corpus every ~2 min — `buildChunks()` loads all ~11.5k chunks (all segments/commits/briefs/docs/todos/scratchpads) into memory + an N+1 currentEmbedding() query per chunk, and the Meili upsert re-loads every already-indexed chunk's vector (~9.5k) into `$vectorsByChunk` + re-builds a Meili document per chunk carrying those vectors + re-upserts ALL of them, even though almost nothing changed. Vectors are ~25–60KB each in PHP × ~9.5k held twice ≈ 0.5–1 GB → exceeds 512MB; the crash surfaces in guzzle psr7 only because that's the allocation that tips an already-full heap. Prior fixes (FLOWER-K summarize chunking, Meili payload bound, #189 embed request bound) each capped a single PAYLOAD, not this per-run accumulation, so it recurs and worsens as data grows.

GOALS:
1. Only build/embed/upsert content that is NEW or CHANGED since last run — filter at the QUERY level (not load-all-then-check-in-PHP); page/cursor (chunkById/lazy); upsert to Meili incrementally per batch; free memory between batches; stop re-loading + re-upserting unchanged vectors/documents. Fix the N+1.
2. Config-driven embed memory_limit (a 4G stopgap is live in Herd php84 php.ini — make it a proper per-job config value; note the stopgap can be reverted once the real fix lands).
3. Keep bounded/streamed HTTP payloads (#189 request bound + the Meili bound); add response-side safety if feasible.
4. A REGRESSION TEST that fails if a pipeline job's peak memory / loaded-row-count scales with total corpus size (seed a large corpus, assert bounded). This is what stops the recurrence.
5. Apply the same "process everything every run" review to SegmentSession/summarize + flower:watch/ingest and fix analogously.

PROCESS (operator-directed):
- Phase 1 — dedicated Claude agent: deep-review the whole pipeline, confirm/extend this root cause, write a concrete refactor design, then implement it. Worktree-pinned; NEVER edit MAIN; `php artisan test` green + pint; preserve behavior (idempotency, graceful no-key path, session state advancement, re-embed-on-change). Decompose into reviewable steps if large.
- Phase 2 — adversarial review dogfooding our own review flow: one Claude reviewer + one Codex reviewer independently review (correctness, does-it-actually-bound-memory, regressions, edge cases) via brief_request_review / brief_review BEFORE the orchestrator merges. Reconcile, merge, reload Horizon, verify on real data (backlog drains, peak memory bounded, no OOMs).

Full evidence + detail in the brief note. `Brief: #193` trailer.
[5] status_change flower-orchestrator: (no body)
[6] link_added flower-orchestrator: (no body)

Recommended linked context:
{
    "todos": [],
    "scratchpads": [
        {
            "id": 386,
            "solo_scratchpad_id": "1078",
            "name": "flower-orchestrator (daemon 25) — reset handoff (2026-07-04 #3)",
            "archived": false,
            "revision": 1
        }
    ]
}

Execution notes:
- Treat the brief as the source of truth.
- Keep work scoped to this dispatch request.
- Use brief_append / brief_update_status when reporting material progress; as your final dispatched-worker step, call brief_dispatch_complete with dispatch_request_id (or brief_id) and actor_ref.
- Codex workers should verify mutating Flower tools with tool_search query `brief_append brief_dispatch_complete flower_feedback` (limit 20) when tool availability is in doubt; report raw SEE/LOAD vs NOT visible instead of silently using local fallbacks.
- Add a git commit trailer `Brief: #193` to every commit for this brief so flower can exact-link commits back to the brief.