review · segments

Fix peaky chat activity chart: send-time bucketing, dedicated ingest queue, sche

claude 421 events 2 segments main

segment 1 of 2

Investigate cause of peaky chat activity chart

Done

Sent two Explore agents to locate the chat chart display code and the message ingest timestamp handling. Discovered the chart buckets per-second counts on chat_messages.created_at (row insert time), not the Twitch send time (tmi-sent-ts). Also found that public metrics broadcasting fires as a side effect of MetricsService::increment(), throttled per-worker singleton, causing redundant broadcasts and synchronous blocking in the ingest path. The peaky chart is caused by Horizon job completion timing, not real chat cadence.

outcome

Identified that the chart uses insert time (created_at) and the public broadcast is coupled to message ingest via a per-worker in-memory throttle.

next steps

—

key decisions

—

open questions

—

1 week ago → 1 week ago

segment 2 of 2

Implement fixes: send-time bucketing, dedicated ingest queue, steady broadcast tick

Done

Implemented three changes per agreement: (1) ConsumeTwitchMessageJob now buckets the per-second counter using tmi-sent-ts (Twitch send time), falling back to created_at when absent. (2) The job is routed to a new dedicated 'ingest' queue (configurable via INGEST_QUEUE env) with a Horizon supervisor sup-ingest (prod min 8 / max 16) to avoid autoscale yoyo; sup-default min lowered from 8 to 3. (3) Removed public-broadcast side effect from MetricsService::increment(); made broadcastPublicMetrics() public; created new metrics:broadcast-public command scheduled everySecond; cached the visible-game list (60s) in compilePublicMetrics. All files lint clean, config resolves, command registers, Pint formatted, and the command runs (exit 0). Committed as a3945a2, excluding pre-existing hunger-games WIP.

outcome

Commit a3945a2 with all three fixes; tmi-cluster daemon restart needed for queue routing to take effect.

next steps

Restart tmi-cluster daemon on production to activate the ingest queue routing.
Monitor chat activity chart behavior post-deploy; verify bars are smoother and reflect actual Twitch chat cadence.
Consider adding a separate daemon if everySecond scheduler overhead becomes a concern.

key decisions

Bucket per-second counts on Twitch send time (tmi-sent-ts) when available, fall back to insert time.
Dedicated 'ingest' queue for ConsumeTwitchMessageJob, configured via config tarkovai.ingest_queue.
Horizon supervisor sup-ingest with prod minProcesses=8, maxProcesses=16, balanceMaxShift=4, balanceCooldown=8.
Lower sup-default prod minProcesses from 8 to 3 after moving chat ingest off it.
Replace per-message public-broadcast side effect with a steady 1s scheduler tick (metrics:broadcast-public), avoiding redundant broadcasts and removing synchronous blocking from ingest.
Cache the visible-game list in compilePublicMetrics (60s TTL) to avoid per-second DB query.
Commit only the chat-ingest changes, leaving the hunger-games WIP unstaged.

open questions

—

1 week ago → 1 week ago