EmbedChunks: bound giant-session embed payloads (Meili 413 + PHP 512MB OOM) — Mode-2 embed root cause

You are being dispatched from flower Brief #35: EmbedChunks: bound giant-session embed payloads (Meili 413 + PHP 512MB OOM) — Mode-2 embed root cause

Recall pointer:
- Use recall_brief with id 35 for the full folder if you need provenance.

Target:
- project: flower (/Users/mikeferrara/Documents/code/flower)
- branch: flower/embed-payload-bounds
- worktree: not specified
- kind: fresh

Current brief spec:
## Problem
Giant sessions fail at the **embed/index** stage (distinct from + downstream of the summarization
timeout). Ops cycle 91 (last 40 min): 6 EmbedChunks + 3 SegmentSession failed_jobs.
- **Meili HTTP 413 "payload too large"** (3×): `EmbedChunks` sends a batch/document to Meili that
  exceeds its payload limit → rejected.
- **512MB PHP OOM** serializing the giant payload: FLOWER-1B (meilisearch-php JSON serialize,
  ~125MB) + FLOWER-1A (Sentry `AbstractSerializer` capturing the failure, ~186MB).
- EmbedChunks maxes retries → sessions left un-indexed. **This is the concrete root cause of the
  Mode-2 embed gap flagged cycle 86.** So the giants fail at BOTH stages: summarize (FLOWER-16/Y) and
  embed/index (this).

## Fix (design — orchestrator's call per ops)
1. **Batch Meili upserts by BYTE SIZE, not just count.** Split documents into batches whose
   serialized size stays well under Meili's payload limit (target ~8 MB, make it a config key).
   Fixes the 413 AND bounds the meilisearch-php JSON serialize memory (FLOWER-1B).
2. **Cap/truncate oversized chunk TEXT.** No single chunk document should be enormous. Enforce a max
   text size per chunk doc (truncate with a marker) so one giant chunk can't blow the limit on its
   own. First VERIFY how giants produce oversized chunks — chunking should already bound size; if the
   chunker is leaking oversized chunks, fix that; otherwise cap at embed time.
3. **Stop the Sentry OOM (FLOWER-1A).** When EmbedChunks throws, Sentry serializes the exception
   context INCLUDING the giant payload → OOM. Scrub/truncate the payload from the exception context
   (Sentry `max_value_length` / `before_send`, or simply don't attach the giant document to the
   thrown exception) so error-handling can never OOM.
4. **(Optional safety) raise the embed worker `memory_limit`** as margin — but the payload-bounding
   above is the real fix, not this.

## Acceptance
- EmbedChunks splits Meili upserts into byte-sized batches under the payload limit; a giant session
  indexes without a 413.
- No 512MB OOM on giant-session embed (bounded serialize batches + oversized-chunk cap).
- A simulated embed failure does NOT OOM Sentry capture (payload scrubbed from context).
- Regression test: an oversized-chunk / large-batch scenario embeds/splits correctly.
- Keep `php artisan test` green + pint. `Brief: #35` trailer. **Job code → Horizon reload after merge.**

## Relations
Downstream of the giant-session summarization timeout (FLOWER-16/Y); root cause of the Mode-2 embed
gap (cycle 86; brief #30 reconciled the *state*, this fixes the *cause*). Belongs to the
ingest-optimization thread (brief #14).

## Provenance
Ops cycle 91 (2026-07-01): FLOWER-1A (Sentry serialize OOM), FLOWER-1B (meilisearch-php serialize
OOM), FLOWER-J (EmbedChunks retry storm, 21 events), Meili 413. Detail: kv sentry:triaged:flower-1a/1b/j.
(FLOWER-1C = the harmless "--dry-run does not exist" mis-invocation — no action.)

Recent/key trace events:
[1] participant_joined flower-orchestrator: (no body)
[2] note_added flower-orchestrator: Ops cycle 91: giant sessions fail at embed/index — Meili HTTP 413 (payload too large) + 512MB PHP OOM serializing the payload (meilisearch-php JSON + Sentry capture). This is the concrete root cause of the Mode-2 embed gap (cycle 86). Ops reporting, fix design deferred to orchestrator.
[3] plan_proposed flower-orchestrator: ## Problem
Giant sessions fail at the **embed/index** stage (distinct from + downstream of the summarization
timeout). Ops cycle 91 (last 40 min): 6 EmbedChunks + 3 SegmentSession failed_jobs.
- **Meili HTTP 413 "payload too large"** (3×): `EmbedChunks` sends a batch/document to Meili that
  exceeds its payload limit → rejected.
- **512MB PHP OOM** serializing the giant payload: FLOWER-1B (meilisearch-php JSON serialize,
  ~125MB) + FLOWER-1A (Sentry `AbstractSerializer` capturing the failure, ~186MB).
- EmbedChunks maxes retries → sessions left un-indexed. **This is the concrete root cause of the
  Mode-2 embed gap flagged cycle 86.** So the giants fail at BOTH stages: summarize (FLOWER-16/Y) and
  embed/index (this).

## Fix (design — orchestrator's call per ops)
1. **Batch Meili upserts by BYTE SIZE, not just count.** Split documents into batches whose
   serialized size stays well under Meili's payload limit (target ~8 MB, make it a config key).
   Fixes the 413 AND bounds the meilisearch-php JSON serialize memory (FLOWER-1B).
2. **Cap/truncate oversized chunk TEXT.** No single chunk document should be enormous. Enforce a max
   text size per chunk doc (truncate with a marker) so one giant chunk can't blow the limit on its
   own. First VERIFY how giants produce oversized chunks — chunking should already bound size; if the
   chunker is leaking oversized chunks, fix that; otherwise cap at embed time.
3. **Stop the Sentry OOM (FLOWER-1A).** When EmbedChunks throws, Sentry serializes the exception
   context INCLUDING the giant payload → OOM. Scrub/truncate the payload from the exception context
   (Sentry `max_value_length` / `before_send`, or simply don't attach the giant document to the
   thrown exception) so error-handling can never OOM.
4. **(Optional safety) raise the embed worker `memory_limit`** as margin — but the payload-bounding
   above is the real fix, not this.

## Acceptance
- EmbedChunks splits Meili upserts into byte-sized batches under the payload limit; a giant session
  indexes without a 413.
- No 512MB OOM on giant-session embed (bounded serialize batches + oversized-chunk cap).
- A simulated embed failure does NOT OOM Sentry capture (payload scrubbed from context).
- Regression test: an oversized-chunk / large-batch scenario embeds/splits correctly.
- Keep `php artisan test` green + pint. `Brief: #35` trailer. **Job code → Horizon reload after merge.**

## Relations
Downstream of the giant-session summarization timeout (FLOWER-16/Y); root cause of the Mode-2 embed
gap (cycle 86; brief #30 reconciled the *state*, this fixes the *cause*). Belongs to the
ingest-optimization thread (brief #14).

## Provenance
Ops cycle 91 (2026-07-01): FLOWER-1A (Sentry serialize OOM), FLOWER-1B (meilisearch-php serialize
OOM), FLOWER-J (EmbedChunks retry storm, 21 events), Meili 413. Detail: kv sentry:triaged:flower-1a/1b/j.
(FLOWER-1C = the harmless "--dry-run does not exist" mis-invocation — no action.)

Recommended linked context:
{
    "todos": [],
    "scratchpads": []
}

Execution notes:
- Treat the brief as the source of truth.
- Keep work scoped to this dispatch request.
- Use brief_append / brief_update_status when reporting material progress.
- Add a git commit trailer `Brief: #35` to every commit for this brief so flower can exact-link commits back to the brief.