Build conductor service watcher, janitor, and thin API

next steps

—

key decisions

• The canonical plan at /Users/mikeferrara/Documents/code/_conductor-plan.md is the authoritative source for the build
• All work follows the plan's §2 (ground truth), §2.5 (three planes + naming), §4 (stream contract), §5 (service spec), §6 + §6.1 (client package + storage layer)

open questions

—

next steps

• Read the key reference files: GpuEmbeddingRuntimeManager, GpuEmbeddingRuntimeAdapter, GpuRuntimeStatus, DTOs, store, models, config, and CLI commands

key decisions

• The Scaffold phase created the conductor app with minimal structure—empty config, placeholder routes, no Runtime directory
• The reference code in lounge (app/Services/Embeddings/GpuRuntime/) contains all the patterns to port: manager, adapters, DTOs, instance store, model classes

open questions

• Whether the port/service-config agent has already created the App\Runtime\* classes in the conductor app

next steps

• Write WatchCommand with a pure, testable decide() method
• Write JanitorCommand with XAUTOCLAIM-based PEL reclaim
• Write ConductorApiController and ConductorTokenAuth middleware
• Write routes/api.php
• Write WatchPolicyTest covering all decision scenarios

key decisions

• The watch policy will use availabilityPasses(GpuRuntimeStatus) as: configured === true && state === 'available' (capacity exists and provider is configured)
• The GHCR pre-flight is part of the adapter's availability() — the watcher just trusts the adapter's verdict
• The watcher must resolve its Redis connection name from config('conductor.connection', 'conductor') since the 'conductor' redis connection may not be registered yet
• The middleware will be applied by FQCN (ConductorTokenAuth::class) rather than an alias, avoiding dependency on bootstrap/app.php edits
• Config values will have sensible defaults baked in so code works even before the service-config agent populates them

open questions

• Whether the 'conductor' redis connection will eventually be registered in database.php by the service-config agent
• Whether the apiPrefix: 'api' in bootstrap/app.php will be removed or if healthz should be at /api/healthz

next steps

• Write the JanitorCommand
• Write the API controller and middleware
• Write the routes file
• Write the unit test

key decisions

• WatchState and WatchDecision value objects are defined in the same file (WatchCommand.php) to respect the create-only-assigned-files rule while keeping the decision logic testable
• decide() is a pure, public method taking WatchState and returning WatchDecision — no I/O, fully unit-testable
• The watch polls every ~15s by default, configurable via config('conductor.watch.poll_interval_seconds')
• The spawn policy requires TWO consecutive polls over threshold to damp flapping
• Cooldown after teardown is configurable (default 5m) to prevent rapid respawn cycles
• Long-running alerts fire at 24h and re-alert at most daily — never kill the runtime
• GpuRuntimeEvent records are created for spawn/teardown/long-running-alert actions

open questions

—

next steps

• Write the API controller
• Write the middleware
• Write the routes
• Write the tests

key decisions

• XAUTOCLAIM is used via rawCommand to bypass Laravel's Redis key prefixing, using the physical stream name
• The janitor re-XADDs reclaimed entries as fresh copies (new ID) then XACK+XDELs the originals
• Dead-letter entries (delivery count > 3) go to dead:{stream} with a quarantine marker record
• A GpuRuntimeEvent is recorded for each poison pill quarantined
• The janitor is safe to run on a schedule and is a no-op if stream/group doesn't exist

open questions

—

next steps

• Write the API controller
• Write the routes file
• Write the tests

key decisions

• The middleware uses hash_equals() for timing-safe comparison
• Returns 503 (Service Unavailable) when no token is configured — fail-closed policy
• Returns 401 on missing/mismatched token
• Applies to all /api/* routes except /healthz

open questions

—

next steps

• Write the routes file
• Write the tests

key decisions

• status() assembles per-stream depth/PEL, active instances with their status, and recent events
• ensure() checks for existing active instances first, then walks provider order calling availability() and start() on the first viable adapter
• stop() and destroy() resolve the GpuRuntimeInstance by id, extract its provider, and call the corresponding adapter method
• healthz() returns a simple JSON liveness check

open questions

—

next steps

• Write the unit test
• Run php -l on all created files
• Run the test suite

key decisions

• Routes are registered with middleware applied by FQCN (ConductorTokenAuth::class) rather than an alias
• healthz is placed at /api/healthz due to the apiPrefix: 'api' in bootstrap/app.php — a root-level /healthz would require changes to Scaffold-owned files
• The middleware group covers all runtime control endpoints but not healthz

open questions

—

next steps

—

key decisions

• The test extends PHPUnit\Framework\TestCase directly (not Laravel's TestCase) since decide() is a pure function needing no container boot
• The test covers all 9 decision paths: spawn (2 consecutive polls over threshold), hold (1 poll over threshold, depth below threshold), teardown (idle streams, PEL empty, after idle window), no-teardown (PEL not empty, nothing running, before idle window), long-running alert, daily re-alert suppression, respawn cooldown, cooldown elapsed, max instances
• An additional availabilityPasses() test uses a fake adapter to verify the availability gate against real GpuRuntimeStatus DTOs (4 sub-cases: available+configured, no_capacity, unconfigured, unhealthy)

open questions

—