flower
/
All feedback
Idea addressed #91 routed · brief

Graceful Horizon reload gotcha: `php artisan horizon:terminate` no-ops ("No processes to terminate") in the Solo-launched setup — reliable reload is SIGTERM to the master pid. Add a `flower:horizon-reload` helper / handoff note.

flower-orchestrator · submitted 1 day ago

detail

What they reported

While deploying the #189 embed-OOM fix I hit this: after merging job code, the documented reload `php artisan horizon:terminate` printed 'No processes to terminate' and did NOT cycle the workers — yet `php artisan horizon:status` reported 'Horizon is running' at the same moment. Root cause: Horizon's TerminateCommand filters masters by `MasterSupervisor::basename()` (hostname); the Solo-launched `php artisan horizon` master (Solo proc 967, pid 49074) registered under a hostname that the CLI invocation's basename() doesn't match, so terminate skips it. horizon:status doesn't apply that filter, hence the contradiction. Reliable graceful reload that works here: SIGTERM directly to the `php artisan horizon` master pid (resolve via `pgrep -f 'artisan horizon$'`), which Horizon traps for graceful shutdown; Solo then auto-restarts the command with the new code (verified: 49074→92827, code live, failed_jobs cleared, no new OOMs). Idea: add a small `flower:horizon-reload` artisan command (resolve master pid → SIGTERM → wait for Solo restart → confirm) so future job-code deploys have a one-shot reliable reload, and/or document this in HANDOFF.md next to the existing horizon:terminate note (which currently implies terminate works). This bites every job/pipeline deploy (e.g. the FLOWER-K summarize fixes, this embed fix).

context

Structured context

{
    "brief": 189,
    "routed": {
        "target": "brief",
        "brief_id": 191,
        "authority": "operator_approval",
        "routed_at": "2026-07-04T05:09:09+00:00",
        "routed_by": "operator:mike",
        "project_id": 16,
        "approved_at": "2026-07-04T05:09:13+00:00",
        "approved_by": "operator:mike",
        "approval_status": "approved",
        "default_project_id": 16,
        "needs_operator_approval": false
    },
    "signal": 62,
    "command": "php artisan horizon:terminate",
    "observed": "No processes to terminate (no-op) while horizon:status shows running",
    "root_cause": "MasterSupervisor::basename() hostname filter mismatch for Solo-launched master",
    "promotion_ledger": [
        {
            "at": "2026-07-04T05:09:09+00:00",
            "action": "brief_created",
            "target": "brief",
            "brief_id": 191,
            "actor_ref": "operator:mike",
            "cycle_key": "2026070405"
        },
        {
            "at": "2026-07-04T05:09:13+00:00",
            "action": "approved",
            "target": "brief",
            "brief_id": 191,
            "actor_ref": "operator:mike"
        },
        {
            "at": "2026-07-04T08:18:59+00:00",
            "action": "source_brief_completed",
            "target": "brief",
            "brief_id": 191,
            "actor_ref": "flower-191-horizon-reload-worker"
        }
    ],
    "reliable_workaround": "SIGTERM to php artisan horizon master pid"
}

state · operator override

Lifecycle

created
1d ago
triaged
1d ago
resolved
1d ago
resolved by
flower-191-horizon-reload-worker

resolution
Brief #191: Feedback #91: Graceful Horizon reload gotcha: `php artisan horizon:terminate` no-ops ("No processes to terminate") in the Solo-launched setup — reliable reload is SIGTERM to the master pid. Add a `flo...

Delete permanently?