Priority queue(s) for ingest Should we break up the 'bulk ingest on n

You are being dispatched from flower Brief #173: Priority queue(s) for ingest

Should we break up the 'bulk ingest on n

Recall pointer:
- Use recall_brief with id 173 for the full folder if you need provenance.

Target:
- project: flower (/Users/mikeferrara/Documents/code/flower)
- branch: choose an appropriate branch
- worktree: not specified
- kind: fresh

Current brief spec:
## Goal
Protect near-real-time recall for ACTIVE sessions during episodic bulk-ingest surges (new-project indexing or operator `depth` bumps), and stop a single stuck/huge session's retry cycle from blocking fresh ingest — by isolating ingest into purpose-built Horizon lanes.

## Grounding
Builds on the shipped #19 / FLOWER-K split (config/horizon.php): a **fast** supervisor (IngestSession / EmbedChunks / sync / refine) + an **isolated long** supervisor (SegmentSession on redis-long). Ingest is FINITE / operator-driven (#14) — big backfills fire on depth bumps or new-project indexing then self-drain — so the value here is protecting live recall DURING those surges, not steady state. #14 explicitly deferred "small-first re-prioritization of the whole parsed backlog (invasive; risks starving live ingest)" to v2+; this brief is that v2 item, resolved as LANE ISOLATION rather than in-queue reordering.

## Decisions (operator, brief Q&A #34/#35, 2026-07-04)
- Isolation: **dedicated low-priority 'backfill' Horizon lane (few workers), separate from the live-ingest lane** (NOT in-queue recency reordering).
- Retries: **dedicated retry/backoff side-lane** so a stuck huge session's retries can't block fresh ingest.

## Scope
1. **Backfill lane.** Route bulk/backfill ingest (newly-indexed projects, depth-bump backfills) to a dedicated low-priority Horizon lane with few workers, distinct from the live-ingest lane that handles active/current-session ingest. Classify a job backfill-vs-live at dispatch (e.g. origin = initial-import / bulk vs live flower:watch scan, and/or session recency). Live lane is preferred for workers/scheduling so active-session recall stays near-real-time during a backfill surge.
2. **Retry side-lane.** Failed ingest jobs retry on a dedicated backoff lane rather than re-queuing on the live/backfill lane, so a stuck/huge session's retry cycle never blocks fresh ingest. Bounded attempts + backoff; terminal after N (surfaces via #14's flower:ingest-backlog stuck-tail).
3. Lane + worker settings config-driven and env-overridable (config/horizon.php + config/flower.php; mirror the fast/long pattern; document worker counts). Redis connection/queue naming consistent with existing redis-long usage.

## Out of scope
- In-queue recency reordering (operator chose lane isolation instead).
- Summarize/embed internals (own briefs: #35 embed bounding, #53 oversized summarize, #19 long queue).
- Ingest-health VISIBILITY/metrics — that is #174 + #112.

## Acceptance
- A backfill-classified ingest job runs on the backfill lane; a live-session ingest job runs on the live lane; under a simulated backfill surge, live-session ingest is not starved (workers reserved for the live lane).
- A failing ingest job retries on the retry side-lane with backoff (not the main lane); bounded attempts then terminal; does not block a fresh live job.
- Config-driven lane/worker settings (env-overridable) + tests (classification/routing + retry-lane targeting + no-starve invariant). `php artisan test` green + pint. `Brief: #173` trailer.

## Provenance
Operator note (2026-07-04) → flower-refine grounding (append) → operator Q&A #34/#35 → this spec. Related: #174 (ingest health/visibility), #112 (planned ingest visibility UI), #19/FLOWER-K (queue split), #14 (backlog visibility + deferred re-prioritization).

Recent/key trace events:
[1] participant_joined operator:mike: (no body)
[2] note_added operator:mike: Priority queue(s) for ingest

Should we break up the 'bulk ingest on newly indexed project' work from the 'ingesting currently active sessions' work?

What I don't want is our current sessions that are working together/parallel to not have real-time recall (or close to it) because we're backlogged ingesting some 6 months of history on a large project or something like that. 

I also wonder if we should push _retries_ on ingest over to a backup/side queue (maybe not the bulk queue) so that _other_ active/fresh ingest can try to get in there rather than get stuck behind a retry cycle on a huge session that won't finish?
[3] status_change operator:mike: (no body)
[4] participant_joined flower-refine: (no body)
[5] refinement flower-refine: Refine grounding (flower-refine). This builds on shipped infra, not greenfield:

- #19 / FLOWER-K (done): Horizon is ALREADY split into a **fast** queue (IngestSession / EmbedChunks / sync / refine) + an **isolated long** queue (SegmentSession on redis-long). That's the queue-separation foundation to extend.
- #14 "Optimize ingest at depth" (complete) EXPLICITLY DEFERRED to v2+: "Small-first re-prioritization of the whole parsed backlog (invasive; risks starving live ingest — the thing to avoid)." #173 is essentially that deferred v2 item, reframed as live-vs-bulk isolation.
- #14 finding: ingest is FINITE / operator-driven — big backfills fire on depth bumps or newly-indexed projects, then self-drain. So #173's value is protecting near-real-time recall for ACTIVE sessions DURING those episodic surges, not steady state.

Two separable asks in the note: (a) split live-session ingest from bulk/new-project backfill; (b) isolate ingest RETRIES so a stuck huge session's retry cycle can't block fresh ingest.

Recommendation lean: a dedicated low-priority 'backfill' lane (few workers) distinct from the live-ingest lane (mirrors the fast/long pattern), with live sessions preferred; retries → bounded backoff on a side lane. Surfacing the design forks to the operator as async questions before proposing a canonical spec. Related: #174 (ingest health/visibility), #112 (planned ingest visibility UI).
[6] agent_question flower-refine: To keep live/active-session ingest from being starved by bulk backfill (new-project index or depth bumps), how should we isolate them? (We already have a fast + isolated-long Horizon split from #19.)
[7] agent_question flower-refine: Should ingest RETRIES move off the main path so a stuck/huge session's retry cycle can't block fresh ingest?
[8] operator_answer operator:mike: Dedicated low-priority 'backfill' Horizon lane (few workers), separate from the live-ingest lane
[9] operator_answer operator:mike: Dedicated retry/backoff side-lane
[10] spec_snapshot flower-refine: Priority queue(s) for ingest

Should we break up the 'bulk ingest on newly indexed project' work from the 'ingesting currently active sessions' work?

What I don't want is our current sessions that are working together/parallel to not have real-time recall (or close to it) because we're backlogged ingesting some 6 months of history on a large project or something like that. 

I also wonder if we should push _retries_ on ingest over to a backup/side queue (maybe not the bulk queue) so that _other_ active/fresh ingest can try to get in there rather than get stuck behind a retry cycle on a huge session that won't finish?
[11] refinement flower-refine: ## Goal
Protect near-real-time recall for ACTIVE sessions during episodic bulk-ingest surges (new-project indexing or operator `depth` bumps), and stop a single stuck/huge session's retry cycle from blocking fresh ingest — by isolating ingest into purpose-built Horizon lanes.

## Grounding
Builds on the shipped #19 / FLOWER-K split (config/horizon.php): a **fast** supervisor (IngestSession / EmbedChunks / sync / refine) + an **isolated long** supervisor (SegmentSession on redis-long). Ingest is FINITE / operator-driven (#14) — big backfills fire on depth bumps or new-project indexing then self-drain — so the value here is protecting live recall DURING those surges, not steady state. #14 explicitly deferred "small-first re-prioritization of the whole parsed backlog (invasive; risks starving live ingest)" to v2+; this brief is that v2 item, resolved as LANE ISOLATION rather than in-queue reordering.

## Decisions (operator, brief Q&A #34/#35, 2026-07-04)
- Isolation: **dedicated low-priority 'backfill' Horizon lane (few workers), separate from the live-ingest lane** (NOT in-queue recency reordering).
- Retries: **dedicated retry/backoff side-lane** so a stuck huge session's retries can't block fresh ingest.

## Scope
1. **Backfill lane.** Route bulk/backfill ingest (newly-indexed projects, depth-bump backfills) to a dedicated low-priority Horizon lane with few workers, distinct from the live-ingest lane that handles active/current-session ingest. Classify a job backfill-vs-live at dispatch (e.g. origin = initial-import / bulk vs live flower:watch scan, and/or session recency). Live lane is preferred for workers/scheduling so active-session recall stays near-real-time during a backfill surge.
2. **Retry side-lane.** Failed ingest jobs retry on a dedicated backoff lane rather than re-queuing on the live/backfill lane, so a stuck/huge session's retry cycle never blocks fresh ingest. Bounded attempts + backoff; terminal after N (surfaces via #14's flower:ingest-backlog stuck-tail).
3. Lane + worker settings config-driven and env-overridable (config/horizon.php + config/flower.php; mirror the fast/long pattern; document worker counts). Redis connection/queue naming consistent with existing redis-long usage.

## Out of scope
- In-queue recency reordering (operator chose lane isolation instead).
- Summarize/embed internals (own briefs: #35 embed bounding, #53 oversized summarize, #19 long queue).
- Ingest-health VISIBILITY/metrics — that is #174 + #112.

## Acceptance
- A backfill-classified ingest job runs on the backfill lane; a live-session ingest job runs on the live lane; under a simulated backfill surge, live-session ingest is not starved (workers reserved for the live lane).
- A failing ingest job retries on the retry side-lane with backoff (not the main lane); bounded attempts then terminal; does not block a fresh live job.
- Config-driven lane/worker settings (env-overridable) + tests (classification/routing + retry-lane targeting + no-starve invariant). `php artisan test` green + pint. `Brief: #173` trailer.

## Provenance
Operator note (2026-07-04) → flower-refine grounding (append) → operator Q&A #34/#35 → this spec. Related: #174 (ingest health/visibility), #112 (planned ingest visibility UI), #19/FLOWER-K (queue split), #14 (backlog visibility + deferred re-prioritization).
[12] status_change flower-refine: (no body)

Recommended linked context:
{
    "todos": [],
    "scratchpads": []
}

Execution notes:
- Treat the brief as the source of truth.
- Keep work scoped to this dispatch request.
- Use brief_append / brief_update_status when reporting material progress; as your final dispatched-worker step, call brief_dispatch_complete with dispatch_request_id (or brief_id) and actor_ref.
- Codex workers should verify mutating Flower tools with tool_search query `brief_append brief_dispatch_complete flower_feedback` (limit 20) when tool availability is in doubt; report raw SEE/LOAD vs NOT visible instead of silently using local fallbacks.
- Add a git commit trailer `Brief: #173` to every commit for this brief so flower can exact-link commits back to the brief.