flower
/
All briefs
complete draft feedback flower
from feedback #144 · ROOT CAUSE for fb#138 found — NOT orphaned timers /...

Daemon spawn command exceeds Solo's 16KB agent-metadata cap → breaks timers/pokes/scratchpads fleet-wide (fix: deliver charter out-of-band)

canonical · plan

Spec

markdown

hand-off · dispatch

Dispatch

Auto-dispatch

when it reaches planned

Design-loop

design pass before build

This brief is complete — dispatch is closed.

#142 done fresh flower · flower/271-daemon-spawn-command-size
agent: codex
You are being dispatched from flower Brief #271: Daemon spawn command exceeds Solo's 16KB agent-metadata cap → breaks timers/pokes/scratchpads fleet-wide (fix: deliver charter out-of-band)

Recall pointer:
- Use recall_brief with id 271 for the full folder if you need provenance.

Target:
- project: flower (/Users/mikeferrara/Documents/code/flower)
- branch: flower/271-daemon-spawn-command-size
- worktree: not specified
- kind: fresh

Current brief spec:
# Fix: daemon spawn command exceeds Solo's 16KB agent-metadata cap → breaks timers/pokes/scratchpads fleet-wide

## Root cause (confirmed live — see feedback #144)
Solo persists each spawned agent's **entire launch command line** into `agent-channels.db` → `agents.metadata.command`, and caps serialized agent metadata at **16384 bytes**. Daemons are currently spawned with their **full rendered charter/boot-packet passed inline as the launch command**, so:
- Orchestrator spawn command = **19766 bytes** (measured, Solo proc 1179) → exceeds the cap → Solo REJECTS the metadata write → **every** `timer_*` and `scratchpad_append` op for that agent fails with `-32602 metadata must be at most 16384 bytes when serialized`.
- `flower-ops-2` = 14952 bytes → just under the cap → ops's timers happen to work (by luck, ~1.4KB from breaking).
- Worker agents spawned with a bare name + `send_input` kickoff = 143 bytes → timers fully work.

Deterministic on packet size → it **recurs on every orchestrator spawn/reset** (observed across the 40→43→45 reset chain). NOT orphaned-timer accumulation; a Solo restart does NOT durably fix it. Impact: no Solo self-poll timers, no `daemon_poke` delivery, no Solo scratchpad writes for any daemon whose packet ≥ 16KB (all orchestrators today; ops/refine too once v7 charters push them over). The fleet currently limps on a background-shell self-poll workaround.

## The single seam (all daemon spawns funnel here)
`App\Services\Daemons\SpawnDaemonBridge::spawn()` (`app/Services/Daemons/SpawnDaemonBridge.php:166-171`):
```php
$soloProcess = $this->solo->spawnAgent(
    projectId: (string) $plan['solo_project']['id'],
    agentToolId: (string) $plan['agent_tool_id'],
    name: (string) $plan['agent_name'],
    agentArgs: [(string) $plan['packet']],   // <-- entire packet becomes the launch command
);
```
Reset-successor spawns reuse this via `DaemonResetService::spawnSuccessorAttempt()` → `$this->spawn->spawn(...)` (`DaemonResetService.php:365`), so fixing this ONE seam covers initial spawns AND resets.

## Fix — deliver the packet out-of-band; keep the launch command short
Change the spawn so the launch command passed to Solo is a **short bootstrap** (a few hundred bytes) and the full packet reaches the daemon another way. **Recommended (proven — it's exactly how dispatched workers stay at 143 bytes):**
- Write `$plan['packet']` to a **file** (unique per spawn, e.g. keyed by actor_ref + a non-time token, under a temp/known dir the daemon can read by absolute path) and pass a short agentArg such as: `Read <absolute-packet-path> and follow it as your daemon boot packet.` The daemon's first action reads the file and proceeds exactly as today.
- **Keep the packet CONTENT byte-identical — only the DELIVERY changes.** Charter, reset-handshake, grounding steps all still reach the daemon unchanged.
- Handle cleanup sensibly (leaving the small file for provenance is fine; or remove after boot).
- Acceptable alternative if cleaner in this codebase: deliver via `send_input` after PTY readiness. But the file approach avoids PTY-readiness races — prefer it.
- Note: scripts must not rely on `Date.now()`-style nondeterminism only if that matters for tests; use a stable unique token (actor_ref + daemon id) for the filename, not a raw timestamp, so tests are deterministic.

Do NOT regress: the daemon must still boot, check in, and (for resets) run the successor handshake exactly as before.

## Acceptance
- After the change, a freshly-spawned **orchestrator** (the largest packet) yields `agents.metadata.command` well under 16384 bytes, and `timer_list` / `timer_set` / `scratchpad_append` SUCCEED for it.
  - Worker verification: a unit/feature test that inspects the `SoloClient::spawnAgent` call (mock the client) and asserts `agentArgs` is the SHORT bootstrap string, NOT the full packet, AND that the packet content is written to the delivery file (content preserved). Assert the bootstrap length is comfortably < 1KB.
  - Live end-to-end verification (spawn a real daemon on MAIN, check `processes.command` length + run `timer_list`) is the **orchestrator's** job post-merge — call it out in your brief_append.
- Rendered packet content unchanged (same charter/boot steps); only delivery differs.
- `SpawnPacketServiceTest` / any `SpawnDaemonBridge` test updated for the new delivery; `MEILISEARCH_KEY=LARAVEL-HERD ANTHROPIC_API_KEY= ~/bin/php artisan test` GREEN; `~/bin/php ./vendor/bin/pint` on changed files.
- `Brief: #271` trailer on every commit. Use `~/bin/php` (PHP 8.4).

## Touch points
`app/Services/Daemons/SpawnDaemonBridge.php` (the seam), `app/Services/Daemons/SpawnPacketService.php` (if a delivery helper fits there), `DaemonResetService.php` (confirm reset path inherits the fix — it should, via the shared bridge), the SoloClient (`app/Services/Solo/*` — confirm `spawnAgent` arg handling / how agentArgs become the command). Also check but likely out of scope: `DirectDispatchService.php:211` `spawnAgent` (workers already stay small — confirm unaffected). `CompactionPacketService` renders a packet too — if it is delivered as a spawn command anywhere, apply the same treatment; if it's delivered via send_input already, no change.

Authority: operator-directed **TOP PRIORITY** (Mike, 2026-07-05). Dispatch to Codex. This restores Solo timers/pokes/scratchpads fleet-wide once merged + all daemons cycle onto the fixed spawn path.

Recent/key trace events:
[1] participant_joined flower-orchestrator: (no body)
[2] note_added flower-orchestrator: Operator-directed TOP PRIORITY (Mike, 2026-07-05): brief + dispatch to Codex + merge. Root cause confirmed live in bug #144. Full spec set via brief_update_spec.
[3] link_added flower-orchestrator: (no body)
[4] plan_proposed flower-orchestrator: # Fix: daemon spawn command exceeds Solo's 16KB agent-metadata cap → breaks timers/pokes/scratchpads fleet-wide

## Root cause (confirmed live — see feedback #144)
Solo persists each spawned agent's **entire launch command line** into `agent-channels.db` → `agents.metadata.command`, and caps serialized agent metadata at **16384 bytes**. Daemons are currently spawned with their **full rendered charter/boot-packet passed inline as the launch command**, so:
- Orchestrator spawn command = **19766 bytes** (measured, Solo proc 1179) → exceeds the cap → Solo REJECTS the metadata write → **every** `timer_*` and `scratchpad_append` op for that agent fails with `-32602 metadata must be at most 16384 bytes when serialized`.
- `flower-ops-2` = 14952 bytes → just under the cap → ops's timers happen to work (by luck, ~1.4KB from breaking).
- Worker agents spawned with a bare name + `send_input` kickoff = 143 bytes → timers fully work.

Deterministic on packet size → it **recurs on every orchestrator spawn/reset** (observed across the 40→43→45 reset chain). NOT orphaned-timer accumulation; a Solo restart does NOT durably fix it. Impact: no Solo self-poll timers, no `daemon_poke` delivery, no Solo scratchpad writes for any daemon whose packet ≥ 16KB (all orchestrators today; ops/refine too once v7 charters push them over). The fleet currently limps on a background-shell self-poll workaround.

## The single seam (all daemon spawns funnel here)
`App\Services\Daemons\SpawnDaemonBridge::spawn()` (`app/Services/Daemons/SpawnDaemonBridge.php:166-171`):
```php
$soloProcess = $this->solo->spawnAgent(
    projectId: (string) $plan['solo_project']['id'],
    agentToolId: (string) $plan['agent_tool_id'],
    name: (string) $plan['agent_name'],
    agentArgs: [(string) $plan['packet']],   // <-- entire packet becomes the launch command
);
```
Reset-successor spawns reuse this via `DaemonResetService::spawnSuccessorAttempt()` → `$this->spawn->spawn(...)` (`DaemonResetService.php:365`), so fixing this ONE seam covers initial spawns AND resets.

## Fix — deliver the packet out-of-band; keep the launch command short
Change the spawn so the launch command passed to Solo is a **short bootstrap** (a few hundred bytes) and the full packet reaches the daemon another way. **Recommended (proven — it's exactly how dispatched workers stay at 143 bytes):**
- Write `$plan['packet']` to a **file** (unique per spawn, e.g. keyed by actor_ref + a non-time token, under a temp/known dir the daemon can read by absolute path) and pass a short agentArg such as: `Read <absolute-packet-path> and follow it as your daemon boot packet.` The daemon's first action reads the file and proceeds exactly as today.
- **Keep the packet CONTENT byte-identical — only the DELIVERY changes.** Charter, reset-handshake, grounding steps all still reach the daemon unchanged.
- Handle cleanup sensibly (leaving the small file for provenance is fine; or remove after boot).
- Acceptable alternative if cleaner in this codebase: deliver via `send_input` after PTY readiness. But the file approach avoids PTY-readiness races — prefer it.
- Note: scripts must not rely on `Date.now()`-style nondeterminism only if that matters for tests; use a stable unique token (actor_ref + daemon id) for the filename, not a raw timestamp, so tests are deterministic.

Do NOT regress: the daemon must still boot, check in, and (for resets) run the successor handshake exactly as before.

## Acceptance
- After the change, a freshly-spawned **orchestrator** (the largest packet) yields `agents.metadata.command` well under 16384 bytes, and `timer_list` / `timer_set` / `scratchpad_append` SUCCEED for it.
  - Worker verification: a unit/feature test that inspects the `SoloClient::spawnAgent` call (mock the client) and asserts `agentArgs` is the SHORT bootstrap string, NOT the full packet, AND that the packet content is written to the delivery file (content preserved). Assert the bootstrap length is comfortably < 1KB.
  - Live end-to-end verification (spawn a real daemon on MAIN, check `processes.command` length + run `timer_list`) is the **orchestrator's** job post-merge — call it out in your brief_append.
- Rendered packet content unchanged (same charter/boot steps); only delivery differs.
- `SpawnPacketServiceTest` / any `SpawnDaemonBridge` test updated for the new delivery; `MEILISEARCH_KEY=LARAVEL-HERD ANTHROPIC_API_KEY= ~/bin/php artisan test` GREEN; `~/bin/php ./vendor/bin/pint` on changed files.
- `Brief: #271` trailer on every commit. Use `~/bin/php` (PHP 8.4).

## Touch points
`app/Services/Daemons/SpawnDaemonBridge.php` (the seam), `app/Services/Daemons/SpawnPacketService.php` (if a delivery helper fits there), `DaemonResetService.php` (confirm reset path inherits the fix — it should, via the shared bridge), the SoloClient (`app/Services/Solo/*` — confirm `spawnAgent` arg handling / how agentArgs become the command). Also check but likely out of scope: `DirectDispatchService.php:211` `spawnAgent` (workers already stay small — confirm unaffected). `CompactionPacketService` renders a packet too — if it is delivered as a spawn command anywhere, apply the same treatment; if it's delivered via send_input already, no change.

Authority: operator-directed **TOP PRIORITY** (Mike, 2026-07-05). Dispatch to Codex. This restores Solo timers/pokes/scratchpads fleet-wide once merged + all daemons cycle onto the fixed spawn path.
[5] status_change flower-orchestrator: (no body)

Recommended linked context:
{
    "todos": [],
    "scratchpads": []
}

Execution notes:
- Treat the brief as the source of truth.
- Keep work scoped to this dispatch request.
- Use brief_append / brief_update_status when reporting material progress; as your final dispatched-worker step, call brief_dispatch_complete with dispatch_request_id (or brief_id) and actor_ref.
- Codex workers should verify mutating Flower tools with tool_search query `brief_append brief_dispatch_complete flower_feedback` (limit 20) when tool availability is in doubt; report raw SEE/LOAD vs NOT visible instead of silently using local fallbacks.
- Add a git commit trailer `Brief: #271` to every commit for this brief so flower can exact-link commits back to the brief.
- Need an operator call while working this brief? A question ABOUT THIS BRIEF -> brief_ask(271, ...); a standalone decision not tied to the brief -> decision_ask(...). Both expose the full affordance set (confirm | single_choice | multi_choice | text, options + recommended, allow_write_in); prefer async questions over blocking and set is_blocking only when you truly cannot proceed.
- Cited-refs index (Brief #244): when a report / checkpoint / DONE summary cites numbered entities, append a compact `Refs:` block at the END mapping each `#N` to its REAL stored title + status — `#<num>: <title> (<status>)`, one per line (typed prefixes: bare `#N` = brief, `fb#N` feedback, `dec#N` decision, `sess#N` session, `dr#N` dispatch_request). Resolve any title you did not already pull with recall_refs instead of guessing; skip the block on trivial one-liners.

provenance · append-only

Trace

live
or paste a screenshot uploading…
  1. link added 11h ago
    agent · system:commit-trailer
  2. participant joined 11h ago
    system · system:commit-trailer
  3. merged 11h ago

    Merged `flower/271-daemon-spawn-command-size` → master @ **ab135e5** (worker commit 6d4efe4). Full suite GREEN on MAIN: 1276 tests, 1274 passed, 2 skipped, 7852 assertions. Only SpawnDaemonBridge.php + 2 tests; no migration, no charter/template change. Fix: rendered packet written to `storage/app/private/daemon-spawn-packets/daemon-<id>-<actor_ref>.md`; Solo receives a <1KB bootstrap arg (`Read <path> and follow it…`). Worker proc 1182 closed. ROLLOUT NOTE: the fix is in code + unit-verified, but only takes effect for a daemon SPAWNED BY a process running #271 code. The standing flower MCP servers (orchestrator/ops/refine) + Horizon cache pre-#271 code, so live timer restoration requires those spawner processes to reload onto #271 first (a fresh artisan/web spawn path runs current code). Live end-to-end verification (real spawn → check processes.command length + timer_list) tracked as orchestrator follow-up.

    agent · flower-orchestrator
  4. status change 11h ago
    agent · flower-271-worker
  5. dispatched 11h ago

    Dispatch request #142 marked done.

    agent · flower-271-worker
  6. note added 11h ago

    Implemented daemon spawn packet out-of-band delivery on branch `flower/271-daemon-spawn-command-size`, commit `6d4efe4acb305f22465ca5c0b17f06b6580b0e00` (`Brief: #271`). `SpawnDaemonBridge::spawn()` now writes the rendered packet byte-for-byte to `storage/app/private/daemon-spawn-packets/daemon-<id>-<actor_ref>.md` and passes Solo only a short bootstrap arg (`Read <absolute-path> and follow it as your daemon boot packet.`). The bootstrap arg is asserted under 1KB and the full packet is asserted to be absent from `SoloClient::spawnAgent` agentArgs. Packet content is preserved exactly in the delivery file; initial daemon spawns, epic lead spawns, and reset-successor spawns all inherit the shared bridge path. Confirmed `SoloClient::spawnAgent()` still passes `agentArgs` through as `--arg`, so this directly shortens the command Solo persists. Confirmed DirectDispatch remains separate and unaffected; CompactionPacketService renders copy/paste packets and does not live-spawn through this seam. Verification run: - `~/bin/php artisan test --filter=SpawnDaemonBridgeTest` => passed, 11 tests / 85 assertions. - `~/bin/php artisan test tests/Feature/Daemons/SpawnPacketServiceTest.php tests/Feature/Daemons/DaemonResetServiceTest.php tests/Feature/Mcp/DaemonResetToolsTest.php` => passed, 26 tests / 331 assertions. - `~/bin/php artisan test --filter=SpawnEpicLeadCommandTest` => passed, 3 tests / 14 assertions. - `MEILISEARCH_KEY=LARAVEL-HERD ANTHROPIC_API_KEY= ~/bin/php artisan test` => passed, 1276 tests, 1261 passed, 15 skipped, 7799 assertions. - `~/bin/php ./vendor/bin/pint app/Services/Daemons/SpawnDaemonBridge.php tests/Feature/Daemons/SpawnDaemonBridgeTest.php tests/Feature/Daemons/SpawnEpicLeadCommandTest.php` => passed. - `git diff --check` => clean. Live end-to-end verification is intentionally left for the orchestrator after merge on MAIN: spawn a real daemon, inspect the persisted Solo command length, and verify `timer_list` / `timer_set` / `scratchpad_append` succeed for the spawned daemon.

    agent · flower-271-worker
  7. participant joined 11h ago
    system · flower-271-worker
  8. dispatched 11h ago

    Dispatch request #142 queued for flower.

    agent · flower-orchestrator
  9. status change 11h ago
    agent · flower-orchestrator
  10. status change 11h ago
    agent · flower-orchestrator
  11. plan proposed 11h ago

    # Fix: daemon spawn command exceeds Solo's 16KB agent-metadata cap → breaks timers/pokes/scratchpads fleet-wide ## Root cause (confirmed live — see feedback #144) Solo persists each spawned agent's **entire launch command line** into `agent-channels.db` → `agents.metadata.command`, and caps serialized agent metadata at **16384 bytes**. Daemons are currently spawned with their **full rendered charter/boot-packet passed inline as the launch command**, so: - Orchestrator spawn command = **19766 bytes** (measured, Solo proc 1179) → exceeds the cap → Solo REJECTS the metadata write → **every** `timer_*` and `scratchpad_append` op for that agent fails with `-32602 metadata must be at most 16384 bytes when serialized`. - `flower-ops-2` = 14952 bytes → just under the cap → ops's timers happen to work (by luck, ~1.4KB from breaking). - Worker agents spawned with a bare name + `send_input` kickoff = 143 bytes → timers fully work. Deterministic on packet size → it **recurs on every orchestrator spawn/reset** (observed across the 40→43→45 reset chain). NOT orphaned-timer accumulation; a Solo restart does NOT durably fix it. Impact: no Solo self-poll timers, no `daemon_poke` delivery, no Solo scratchpad writes for any daemon whose packet ≥ 16KB (all orchestrators today; ops/refine too once v7 charters push them over). The fleet currently limps on a background-shell self-poll workaround. ## The single seam (all daemon spawns funnel here) `App\Services\Daemons\SpawnDaemonBridge::spawn()` (`app/Services/Daemons/SpawnDaemonBridge.php:166-171`): ```php $soloProcess = $this->solo->spawnAgent( projectId: (string) $plan['solo_project']['id'], agentToolId: (string) $plan['agent_tool_id'], name: (string) $plan['agent_name'], agentArgs: [(string) $plan['packet']], // <-- entire packet becomes the launch command ); ``` Reset-successor spawns reuse this via `DaemonResetService::spawnSuccessorAttempt()` → `$this->spawn->spawn(...)` (`DaemonResetService.php:365`), so fixing this ONE seam covers initial spawns AND resets. ## Fix — deliver the packet out-of-band; keep the launch command short Change the spawn so the launch command passed to Solo is a **short bootstrap** (a few hundred bytes) and the full packet reaches the daemon another way. **Recommended (proven — it's exactly how dispatched workers stay at 143 bytes):** - Write `$plan['packet']` to a **file** (unique per spawn, e.g. keyed by actor_ref + a non-time token, under a temp/known dir the daemon can read by absolute path) and pass a short agentArg such as: `Read <absolute-packet-path> and follow it as your daemon boot packet.` The daemon's first action reads the file and proceeds exactly as today. - **Keep the packet CONTENT byte-identical — only the DELIVERY changes.** Charter, reset-handshake, grounding steps all still reach the daemon unchanged. - Handle cleanup sensibly (leaving the small file for provenance is fine; or remove after boot). - Acceptable alternative if cleaner in this codebase: deliver via `send_input` after PTY readiness. But the file approach avoids PTY-readiness races — prefer it. - Note: scripts must not rely on `Date.now()`-style nondeterminism only if that matters for tests; use a stable unique token (actor_ref + daemon id) for the filename, not a raw timestamp, so tests are deterministic. Do NOT regress: the daemon must still boot, check in, and (for resets) run the successor handshake exactly as before. ## Acceptance - After the change, a freshly-spawned **orchestrator** (the largest packet) yields `agents.metadata.command` well under 16384 bytes, and `timer_list` / `timer_set` / `scratchpad_append` SUCCEED for it. - Worker verification: a unit/feature test that inspects the `SoloClient::spawnAgent` call (mock the client) and asserts `agentArgs` is the SHORT bootstrap string, NOT the full packet, AND that the packet content is written to the delivery file (content preserved). Assert the bootstrap length is comfortably < 1KB. - Live end-to-end verification (spawn a real daemon on MAIN, check `processes.command` length + run `timer_list`) is the **orchestrator's** job post-merge — call it out in your brief_append. - Rendered packet content unchanged (same charter/boot steps); only delivery differs. - `SpawnPacketServiceTest` / any `SpawnDaemonBridge` test updated for the new delivery; `MEILISEARCH_KEY=LARAVEL-HERD ANTHROPIC_API_KEY= ~/bin/php artisan test` GREEN; `~/bin/php ./vendor/bin/pint` on changed files. - `Brief: #271` trailer on every commit. Use `~/bin/php` (PHP 8.4). ## Touch points `app/Services/Daemons/SpawnDaemonBridge.php` (the seam), `app/Services/Daemons/SpawnPacketService.php` (if a delivery helper fits there), `DaemonResetService.php` (confirm reset path inherits the fix — it should, via the shared bridge), the SoloClient (`app/Services/Solo/*` — confirm `spawnAgent` arg handling / how agentArgs become the command). Also check but likely out of scope: `DirectDispatchService.php:211` `spawnAgent` (workers already stay small — confirm unaffected). `CompactionPacketService` renders a packet too — if it is delivered as a spawn command anywhere, apply the same treatment; if it's delivered via send_input already, no change. Authority: operator-directed **TOP PRIORITY** (Mike, 2026-07-05). Dispatch to Codex. This restores Solo timers/pokes/scratchpads fleet-wide once merged + all daemons cycle onto the fixed spawn path.

    agent · flower-orchestrator
  12. link added 11h ago
    agent · flower-orchestrator
  13. note added 11h ago

    Operator-directed TOP PRIORITY (Mike, 2026-07-05): brief + dispatch to Codex + merge. Root cause confirmed live in bug #144. Full spec set via brief_update_spec.

    agent · flower-orchestrator
  14. participant joined 11h ago
    system · flower-orchestrator

epic · dependencies

Relationships

epic parent

depends on

No dependencies — dispatchable once planned.

agents · waves

Participants

  • flower-orchestrator participant · active
  • flower-271-worker participant · active
  • system:commit-trailer participant · active

trace · graph

Links

  • Commit #4075 execution
  • Feedback #144 seed

scope

Projects

  • flower · primary

dogfood · read-only

Agent’s-eye view

The literal recall_brief payload an agent gets — same service path as the MCP tool.