flower
/
All briefs
complete feedback flower
from feedback #91 · Graceful Horizon reload gotcha: `php artisan horizon...

Feedback #91: Graceful Horizon reload gotcha: `php artisan horizon:terminate` no-ops ("No processes to terminate") in the Solo-launched setup — reliable reload is SIGTERM to the master pid. Add a `flower:horizon-reload` helper / hando

canonical · plan

Spec

markdown

hand-off · dispatch

Dispatch

Auto-dispatch

when it reaches planned

Design-loop

design pass before build

This brief is complete — dispatch is closed.

#85 done fresh flower · flower/191-horizon-reload
agent: claude
You are being dispatched from flower Brief #191: Feedback #91: Graceful Horizon reload gotcha: `php artisan horizon:terminate` no-ops ("No processes to terminate") in the Solo-launched setup — reliable reload is SIGTERM to the master pid. Add a `flower:horizon-reload` helper / hando

Recall pointer:
- Use recall_brief with id 191 for the full folder if you need provenance.

Target:
- project: flower (/Users/mikeferrara/Documents/code/flower)
- branch: flower/191-horizon-reload
- worktree: not specified
- kind: fresh

Current brief spec:
(no spec yet)

This is a direct request, not a fully-specced plan. If it's clear, resolve it. If you hit a blocking ambiguity, call brief_ask (or brief_append) with your questions and flip the brief to `refining` before proceeding — don't guess.

Recent/key trace events:
[1] participant_joined operator:mike: (no body)
[2] note_added operator:mike: Feedback #91
Authority: OPERATOR APPROVAL REQUIRED
Funnel: A - operator approved brief
Gate: this brief must not be dispatchable until the operator approves it.
Kind: idea
Source: flower-orchestrator

Summary:
Graceful Horizon reload gotcha: `php artisan horizon:terminate` no-ops ("No processes to terminate") in the Solo-launched setup — reliable reload is SIGTERM to the master pid. Add a `flower:horizon-reload` helper / handoff note.

Detail:
While deploying the #189 embed-OOM fix I hit this: after merging job code, the documented reload `php artisan horizon:terminate` printed 'No processes to terminate' and did NOT cycle the workers — yet `php artisan horizon:status` reported 'Horizon is running' at the same moment. Root cause: Horizon's TerminateCommand filters masters by `MasterSupervisor::basename()` (hostname); the Solo-launched `php artisan horizon` master (Solo proc 967, pid 49074) registered under a hostname that the CLI invocation's basename() doesn't match, so terminate skips it. horizon:status doesn't apply that filter, hence the contradiction. Reliable graceful reload that works here: SIGTERM directly to the `php artisan horizon` master pid (resolve via `pgrep -f 'artisan horizon$'`), which Horizon traps for graceful shutdown; Solo then auto-restarts the command with the new code (verified: 49074→92827, code live, failed_jobs cleared, no new OOMs). Idea: add a small `flower:horizon-reload` artisan command (resolve master pid → SIGTERM → wait for Solo restart → confirm) so future job-code deploys have a one-shot reliable reload, and/or document this in HANDOFF.md next to the existing horizon:terminate note (which currently implies terminate works). This bites every job/pipeline deploy (e.g. the FLOWER-K summarize fixes, this embed fix).

Context JSON:
{
    "brief": 189,
    "signal": 62,
    "command": "php artisan horizon:terminate",
    "observed": "No processes to terminate (no-op) while horizon:status shows running",
    "root_cause": "MasterSupervisor::basename() hostname filter mismatch for Solo-launched master",
    "reliable_workaround": "SIGTERM to php artisan horizon master pid"
}
[3] link_added operator:mike: (no body)
[4] note_added operator:mike: Operator approved this feedback-born brief for dispatch.
[5] status_change operator:mike: (no body)

Recommended linked context:
{
    "todos": [],
    "scratchpads": []
}

Execution notes:
- Treat the brief as the source of truth.
- Keep work scoped to this dispatch request.
- Use brief_append / brief_update_status when reporting material progress; as your final dispatched-worker step, call brief_dispatch_complete with dispatch_request_id (or brief_id) and actor_ref.
- Codex workers should verify mutating Flower tools with tool_search query `brief_append brief_dispatch_complete flower_feedback` (limit 20) when tool availability is in doubt; report raw SEE/LOAD vs NOT visible instead of silently using local fallbacks.
- Add a git commit trailer `Brief: #191` to every commit for this brief so flower can exact-link commits back to the brief.

provenance · append-only

Trace

live
or paste a screenshot uploading…
  1. link added 1d ago
    agent · system:commit-trailer
  2. link added 1d ago
    agent · system:commit-trailer
  3. participant joined 1d ago
    system · system:commit-trailer
  4. merged 1d ago

    Merged to master by flower-orchestrator (daemon 29). Merge commit 3327fda (worker commit 8ae49e5, branch flower/191-horizon-reload). Adds app/Services/Horizon/HorizonReloader.php + app/Console/Commands/HorizonReload.php (flower:horizon-reload: SIGTERM to master pid via pgrep -f 'artisan horizon$', --force/--hard = SIGKILL, clear no-op when none running) + 8 fully-mocked tests + CLAUDE.md canonical-reload note. Adversarial safety review PASS: no test path ever calls real posix_kill (FakeHorizonReloader records signals; service tests use Process::fake + preventStrayProcesses). Full suite green on MAIN: 955 tests / 953 passed / 2 skipped / 0 failed. Additive-only (new files); no reload needed to land. This is now the canonical Horizon reload for future merges.

    agent · flower-orchestrator
  5. status change 1d ago
    agent · flower-191-horizon-reload-worker
  6. dispatched 1d ago

    Dispatch request #85 marked done.

    agent · flower-191-horizon-reload-worker
  7. note added 1d ago

    Built `flower:horizon-reload` (dispatch #85). Branch `flower/191-horizon-reload` off master; commit 8ae49e5. What changed: - NEW app/Services/Horizon/HorizonReloader.php — findMasterPid() resolves the master via `pgrep -f 'artisan horizon$'` through the Process facade (parses pids, returns the lowest when >1 match, null on no match / pgrep exit 1); sendSignal() wraps posix_kill. Both public + injectable for testing. SIGTERM=15 / SIGKILL=9 constants (no ext-pcntl dependency). - NEW app/Console/Commands/HorizonReload.php — `flower:horizon-reload`. Default graceful SIGTERM; `--force`/`--hard` → SIGKILL. Prints the exact pid signaled ("Sent SIGTERM to Horizon master pid N — in-flight jobs will finish, then Solo auto-restarts…") or a clear "No Horizon master running … nothing to reload" no-op (SUCCESS). Failed signal → FAILURE. Auto-registered (verified in `artisan list`). - NEW tests/Feature/HorizonReloadCommandTest.php — 8 tests. Command tests inject a FakeHorizonReloader that RECORDS signal requests and NEVER calls posix_kill (safety pin honored: no real process signaled). Service tests use Process::fake() + preventStrayProcesses() so pgrep can never fall through to the real system. Covers: SIGTERM default, --force/--hard→SIGKILL, no-op when no master, failed-signal→FAILURE, pgrep parse, lowest-of-multiple, null-on-no-match. - CLAUDE.md runtime note updated: reload guidance now points at `flower:horizon-reload` and records the horizon:terminate basename()-hostname-filter gotcha (was previously implying terminate works). Root cause (per Feedback #91 / brief #127): Horizon TerminateCommand filters masters by MasterSupervisor::basename() (hostname); the Solo-launched master registers under a hostname the CLI basename() doesn't match, so terminate skips it while horizon:status still shows running. Gates: `MEILISEARCH_KEY=LARAVEL-HERD ~/bin/php artisan test` → 955 tests, 954 passed / 1 skipped / 0 failed. Pint clean on all changed files. Did NOT run the command or signal any real process; no daemons started; isolated worktree. Ready for orchestrator review + merge — I am NOT merging.

    agent · flower-191-horizon-reload-worker
  8. participant joined 1d ago
    system · flower-191-horizon-reload-worker
  9. dispatched 1d ago

    Dispatch request #85 queued for flower.

    agent · flower-orchestrator
  10. status change 1d ago
    agent · flower-orchestrator
  11. participant joined 1d ago
    system · flower-orchestrator
  12. status change 1d ago
    agent · operator:mike
  13. note added 1d ago

    Operator approved this feedback-born brief for dispatch.

    operator · operator:mike
  14. link added 1d ago
    agent · operator:mike
  15. note added 1d ago

    Feedback #91 Authority: OPERATOR APPROVAL REQUIRED Funnel: A - operator approved brief Gate: this brief must not be dispatchable until the operator approves it. Kind: idea Source: flower-orchestrator Summary: Graceful Horizon reload gotcha: `php artisan horizon:terminate` no-ops ("No processes to terminate") in the Solo-launched setup — reliable reload is SIGTERM to the master pid. Add a `flower:horizon-reload` helper / handoff note. Detail: While deploying the #189 embed-OOM fix I hit this: after merging job code, the documented reload `php artisan horizon:terminate` printed 'No processes to terminate' and did NOT cycle the workers — yet `php artisan horizon:status` reported 'Horizon is running' at the same moment. Root cause: Horizon's TerminateCommand filters masters by `MasterSupervisor::basename()` (hostname); the Solo-launched `php artisan horizon` master (Solo proc 967, pid 49074) registered under a hostname that the CLI invocation's basename() doesn't match, so terminate skips it. horizon:status doesn't apply that filter, hence the contradiction. Reliable graceful reload that works here: SIGTERM directly to the `php artisan horizon` master pid (resolve via `pgrep -f 'artisan horizon$'`), which Horizon traps for graceful shutdown; Solo then auto-restarts the command with the new code (verified: 49074→92827, code live, failed_jobs cleared, no new OOMs). Idea: add a small `flower:horizon-reload` artisan command (resolve master pid → SIGTERM → wait for Solo restart → confirm) so future job-code deploys have a one-shot reliable reload, and/or document this in HANDOFF.md next to the existing horizon:terminate note (which currently implies terminate works). This bites every job/pipeline deploy (e.g. the FLOWER-K summarize fixes, this embed fix). Context JSON: { "brief": 189, "signal": 62, "command": "php artisan horizon:terminate", "observed": "No processes to terminate (no-op) while horizon:status shows running", "root_cause": "MasterSupervisor::basename() hostname filter mismatch for Solo-launched master", "reliable_workaround": "SIGTERM to php artisan horizon master pid" }

    agent · operator:mike
  16. participant joined 1d ago
    system · operator:mike

epic · dependencies

Relationships

epic parent

depends on

No dependencies — dispatchable once planned.

agents · waves

Participants

  • operator:mike participant · active
  • flower-orchestrator participant · active
  • flower-191-horizon-reload-worker participant · active
  • system:commit-trailer participant · active

trace · graph

Links

  • Commit #3977 execution
  • Commit #3979 execution
  • Feedback #91 seed

scope

Projects

  • flower · primary

dogfood · read-only

Agent’s-eye view

The literal recall_brief payload an agent gets — same service path as the MCP tool.