Embed pipeline OOM: bound EmbedChunks HTTP payload + guard oversized chunks (512MB exhaustion, backlog climbing)

You are being dispatched from flower Brief #189: Embed pipeline OOM: bound EmbedChunks HTTP payload + guard oversized chunks (512MB exhaustion, backlog climbing)

Recall pointer:
- Use recall_brief with id 189 for the full folder if you need provenance.

Target:
- project: flower (/Users/mikeferrara/Documents/code/flower)
- branch: flower/189-embed-oom-bound-payload
- worktree: not specified
- kind: fresh

Current brief spec:
## Symptom (recall_health=CRITICAL, actively degrading)
`App\Jobs\EmbedChunks` is OOM-ing at the **512MB PHP memory limit while reading an HTTP body/stream in the embed step** (Sentry FLOWER-1R = `Illuminate/Http/Client/Response.php`, FLOWER-1Q = `guzzlehttp/psr7/Stream.php`, FLOWER-J = `EmbedChunks` MaxAttemptsExceeded). Jobs fail ~1 per 8 min → `failed_jobs` 0→3+, and the chunk-embed **backlog is climbing 77→683→1059** (not draining). New/large sessions' chunks are not indexed → recall degrades for them. NOT app-down (flower.test serves; ingest fresh).

## Root cause
A giant chunk/session embed payload (request to the embedding provider and/or the Meilisearch write, or reading their response) blows the 512MB limit → job crashes → retries → max-attempts → chunk never indexes → backlog grows. Same cluster the predecessor saw at cycle 91 (FLOWER-1A/1B/J + Meili-413), now actively degrading during a live summarization wave.

## Fix (this is the summarize-side FLOWER-K fix, applied to the embed side)
1. **Bound the embed HTTP payload:** cap per-call batch size and/or per-chunk byte size in `App\Jobs\EmbedChunks` so a single request/response body cannot approach 512MB; split oversized batches into smaller calls. Put the thresholds in `config/flower.php` under a new **`embed.*`** block (env-overridable), mirroring the existing `summarize.*` shape — look at how FLOWER-K did it and follow that pattern.
2. **Guard oversized chunks:** if a single chunk's content is pathologically large, truncate/skip-with-warn (log + mark) instead of OOM-ing the whole job. Mirror the summarize reduce-on-oversize guard.
3. **Stream / bound the HTTP response** rather than loading the whole body into memory where feasible (`Http\Client\Response` reads the full body). Raising the embed-worker `memory_limit` is a stopgap ONLY — do the payload bounding, not just a bigger limit.

## Explicitly NOT the worker's job (orchestrator owns these on MAIN after merge)
- Do NOT reload Horizon, run `queue:retry`, or drain the backlog — the orchestrator does that on MAIN after merging (Horizon caches job code at boot; needs a graceful `horizon:terminate` reload).
- Do NOT edit MAIN or touch `.env`. Put defaults in version-controlled `config/flower.php` (worktree `.env`s drift).

## Constraints
- Work ONLY inside your assigned worktree; NEVER edit under MAIN (`/Users/mikeferrara/Documents/code/flower`). Use relative paths. Commit trailer `Brief: #189` on every commit.

## Verify
- Unit/feature test proving a pathologically large chunk/batch is bounded/split and does not exhaust memory (assert batch/byte caps; keep sqlite-portable). 
- `MEILISEARCH_KEY=LARAVEL-HERD ~/bin/php artisan test` green; pint on changed files.
- Note in your final report which config keys you added so the orchestrator can tune them on MAIN.

## Meta
Severity: HIGH (degrading core pipeline — indexing). Autonomous. Source: feedback #90 / fix-spec Solo 1077 / escalation Solo 1076 / signal #62 (routed by flower-ops daemon 23). Sentry: FLOWER-1R / 1Q / J.

Recent/key trace events:
[1] participant_joined flower-orchestrator: (no body)
[2] note_added flower-orchestrator: ## Symptom (recall_health=CRITICAL, actively degrading)
`App\Jobs\EmbedChunks` is OOM-ing at the **512MB PHP memory limit while reading an HTTP body/stream in the embed step** (Sentry FLOWER-1R = `Illuminate/Http/Client/Response.php`, FLOWER-1Q = `guzzlehttp/psr7/Stream.php`, FLOWER-J = `EmbedChunks` MaxAttemptsExceeded). Jobs fail ~1 per 8 min → `failed_jobs` 0→3+, and the chunk-embed **backlog is climbing 77→683→1059** (not draining). New/large sessions' chunks are not indexed → recall degrades for them. NOT app-down (flower.test serves; ingest fresh).

## Root cause
A giant chunk/session embed payload (request to the embedding provider and/or the Meilisearch write, or reading their response) blows the 512MB limit → job crashes → retries → max-attempts → chunk never indexes → backlog grows. Same cluster the predecessor saw at cycle 91 (FLOWER-1A/1B/J + Meili-413), now actively degrading during a live summarization wave.

## Fix (this is the summarize-side FLOWER-K fix, applied to the embed side)
1. **Bound the embed HTTP payload:** cap per-call batch size and/or per-chunk byte size in `App\Jobs\EmbedChunks` so a single request/response body cannot approach 512MB; split oversized batches into smaller calls. Put the thresholds in `config/flower.php` under a new **`embed.*`** block (env-overridable), mirroring the existing `summarize.*` shape — look at how FLOWER-K did it and follow that pattern.
2. **Guard oversized chunks:** if a single chunk's content is pathologically large, truncate/skip-with-warn (log + mark) instead of OOM-ing the whole job. Mirror the summarize reduce-on-oversize guard.
3. **Stream / bound the HTTP response** rather than loading the whole body into memory where feasible (`Http\Client\Response` reads the full body). Raising the embed-worker `memory_limit` is a stopgap ONLY — do the payload bounding, not just a bigger limit.

## Explicitly NOT the worker's job (orchestrator owns these on MAIN after merge)
- Do NOT reload Horizon, run `queue:retry`, or drain the backlog — the orchestrator does that on MAIN after merging (Horizon caches job code at boot; needs a graceful `horizon:terminate` reload).
- Do NOT edit MAIN or touch `.env`. Put defaults in version-controlled `config/flower.php` (worktree `.env`s drift).

## Verify
- Unit/feature test proving a pathologically large chunk/batch is bounded/split and does not exhaust memory (assert batch/byte caps; keep sqlite-portable). 
- `MEILISEARCH_KEY=LARAVEL-HERD ~/bin/php artisan test` green; pint on changed files.
- Note in your final report which config keys you added so the orchestrator can tune them on MAIN.

## Meta
Severity: HIGH (degrading core pipeline — indexing). Autonomous. Source: feedback #90 / fix-spec Solo 1077 / escalation Solo 1076 / signal #62 (routed by flower-ops daemon 23). Sentry: FLOWER-1R / 1Q / J. Add commit trailer `Brief: #<this>`.
[3] link_added flower-orchestrator: (no body)
[4] plan_proposed flower-orchestrator: ## Symptom (recall_health=CRITICAL, actively degrading)
`App\Jobs\EmbedChunks` is OOM-ing at the **512MB PHP memory limit while reading an HTTP body/stream in the embed step** (Sentry FLOWER-1R = `Illuminate/Http/Client/Response.php`, FLOWER-1Q = `guzzlehttp/psr7/Stream.php`, FLOWER-J = `EmbedChunks` MaxAttemptsExceeded). Jobs fail ~1 per 8 min → `failed_jobs` 0→3+, and the chunk-embed **backlog is climbing 77→683→1059** (not draining). New/large sessions' chunks are not indexed → recall degrades for them. NOT app-down (flower.test serves; ingest fresh).

## Root cause
A giant chunk/session embed payload (request to the embedding provider and/or the Meilisearch write, or reading their response) blows the 512MB limit → job crashes → retries → max-attempts → chunk never indexes → backlog grows. Same cluster the predecessor saw at cycle 91 (FLOWER-1A/1B/J + Meili-413), now actively degrading during a live summarization wave.

## Fix (this is the summarize-side FLOWER-K fix, applied to the embed side)
1. **Bound the embed HTTP payload:** cap per-call batch size and/or per-chunk byte size in `App\Jobs\EmbedChunks` so a single request/response body cannot approach 512MB; split oversized batches into smaller calls. Put the thresholds in `config/flower.php` under a new **`embed.*`** block (env-overridable), mirroring the existing `summarize.*` shape — look at how FLOWER-K did it and follow that pattern.
2. **Guard oversized chunks:** if a single chunk's content is pathologically large, truncate/skip-with-warn (log + mark) instead of OOM-ing the whole job. Mirror the summarize reduce-on-oversize guard.
3. **Stream / bound the HTTP response** rather than loading the whole body into memory where feasible (`Http\Client\Response` reads the full body). Raising the embed-worker `memory_limit` is a stopgap ONLY — do the payload bounding, not just a bigger limit.

## Explicitly NOT the worker's job (orchestrator owns these on MAIN after merge)
- Do NOT reload Horizon, run `queue:retry`, or drain the backlog — the orchestrator does that on MAIN after merging (Horizon caches job code at boot; needs a graceful `horizon:terminate` reload).
- Do NOT edit MAIN or touch `.env`. Put defaults in version-controlled `config/flower.php` (worktree `.env`s drift).

## Constraints
- Work ONLY inside your assigned worktree; NEVER edit under MAIN (`/Users/mikeferrara/Documents/code/flower`). Use relative paths. Commit trailer `Brief: #189` on every commit.

## Verify
- Unit/feature test proving a pathologically large chunk/batch is bounded/split and does not exhaust memory (assert batch/byte caps; keep sqlite-portable). 
- `MEILISEARCH_KEY=LARAVEL-HERD ~/bin/php artisan test` green; pint on changed files.
- Note in your final report which config keys you added so the orchestrator can tune them on MAIN.

## Meta
Severity: HIGH (degrading core pipeline — indexing). Autonomous. Source: feedback #90 / fix-spec Solo 1077 / escalation Solo 1076 / signal #62 (routed by flower-ops daemon 23). Sentry: FLOWER-1R / 1Q / J.
[5] status_change flower-orchestrator: (no body)

Recommended linked context:
{
    "todos": [],
    "scratchpads": []
}

Execution notes:
- Treat the brief as the source of truth.
- Keep work scoped to this dispatch request.
- Use brief_append / brief_update_status when reporting material progress; as your final dispatched-worker step, call brief_dispatch_complete with dispatch_request_id (or brief_id) and actor_ref.
- Codex workers should verify mutating Flower tools with tool_search query `brief_append brief_dispatch_complete flower_feedback` (limit 20) when tool availability is in doubt; report raw SEE/LOAD vs NOT visible instead of silently using local fallbacks.
- Add a git commit trailer `Brief: #189` to every commit for this brief so flower can exact-link commits back to the brief.