Graceful Horizon reload gotcha: `php artisan horizon:terminate` no-ops ("No processes to terminate") in the Solo-launched setup — reliable reload is SIGTERM to the master pid. Add a `flower:horizon-reload` helper / handoff note.
flower-orchestrator · submitted 1 day ago
detail
What they reported
While deploying the #189 embed-OOM fix I hit this: after merging job code, the documented reload `php artisan horizon:terminate` printed 'No processes to terminate' and did NOT cycle the workers — yet `php artisan horizon:status` reported 'Horizon is running' at the same moment. Root cause: Horizon's TerminateCommand filters masters by `MasterSupervisor::basename()` (hostname); the Solo-launched `php artisan horizon` master (Solo proc 967, pid 49074) registered under a hostname that the CLI invocation's basename() doesn't match, so terminate skips it. horizon:status doesn't apply that filter, hence the contradiction. Reliable graceful reload that works here: SIGTERM directly to the `php artisan horizon` master pid (resolve via `pgrep -f 'artisan horizon$'`), which Horizon traps for graceful shutdown; Solo then auto-restarts the command with the new code (verified: 49074→92827, code live, failed_jobs cleared, no new OOMs). Idea: add a small `flower:horizon-reload` artisan command (resolve master pid → SIGTERM → wait for Solo restart → confirm) so future job-code deploys have a one-shot reliable reload, and/or document this in HANDOFF.md next to the existing horizon:terminate note (which currently implies terminate works). This bites every job/pipeline deploy (e.g. the FLOWER-K summarize fixes, this embed fix).
context
Structured context
{
"brief": 189,
"routed": {
"target": "brief",
"brief_id": 191,
"authority": "operator_approval",
"routed_at": "2026-07-04T05:09:09+00:00",
"routed_by": "operator:mike",
"project_id": 16,
"approved_at": "2026-07-04T05:09:13+00:00",
"approved_by": "operator:mike",
"approval_status": "approved",
"default_project_id": 16,
"needs_operator_approval": false
},
"signal": 62,
"command": "php artisan horizon:terminate",
"observed": "No processes to terminate (no-op) while horizon:status shows running",
"root_cause": "MasterSupervisor::basename() hostname filter mismatch for Solo-launched master",
"promotion_ledger": [
{
"at": "2026-07-04T05:09:09+00:00",
"action": "brief_created",
"target": "brief",
"brief_id": 191,
"actor_ref": "operator:mike",
"cycle_key": "2026070405"
},
{
"at": "2026-07-04T05:09:13+00:00",
"action": "approved",
"target": "brief",
"brief_id": 191,
"actor_ref": "operator:mike"
},
{
"at": "2026-07-04T08:18:59+00:00",
"action": "source_brief_completed",
"target": "brief",
"brief_id": 191,
"actor_ref": "flower-191-horizon-reload-worker"
}
],
"reliable_workaround": "SIGTERM to php artisan horizon master pid"
}promoted · work
Linked brief
state · operator override
Lifecycle
- created
- 1d ago
- triaged
- 1d ago
- resolved
- 1d ago
- resolved by
- flower-191-horizon-reload-worker
resolution
Brief #191: Feedback #91: Graceful Horizon reload gotcha: `php artisan horizon:terminate` no-ops ("No processes to terminate") in the Solo-launched setup — reliable reload is SIGTERM to the master pid. Add a `flo...