Design central GPU task processing service

next steps

—

key decisions

• Service shape: headless watcher + thin API; data plane = Redis streams, control plane = small daemon + minimal HTTP API
• Hosting: dedicated Redis on .73 (enm-storage) alongside MinIO
• Worker connectivity: Tailscale tailnet canonical
• Compute: cloud-only (RunPod primary, DigitalOcean secondary); homelab not in production pool
• Runtime duration: no hard cap — runs while work exists, idle teardown only; Sentry alert at 24h
• Scaling: fixed max 1 pod in v1; depth gates spawn/no-spawn
• Cutover: fresh empty conductor-Redis; unfinished work re-enqueues from embeddings table cursor
• conductor-client owns opinionated storage layer (model+registry+trait+migrations+ResultHandler)
• conductor:make-embeddable scaffold command is fast-follow after lounge adoption
• Env convention: CONDUCTOR_REDIS_*, CONDUCTOR_APP, CONDUCTOR_TARGET_DEPTH, CONDUCTOR_API_*

open questions

—

next steps

—

key decisions

• Use Codex workers for refinement with self-contained task specs
• Worker findings must be verified against ground truth before acceptance
• Carry-forward orphan reconciliation to wave 2; per-app ACL users deferred to before Phase 4

open questions

—

next steps

—

key decisions

• conductor-redis maxmemory 12GB with noeviction (preserve RAM for future vector index)
• RDB snapshots every 5min if >=1000 changes, no AOF (transport data re-enqueable)
• Sentry project 'conductor' in legitphp org
• Single requirepass for v1; ACL users deferred
• Exclude bootstrap/cache from rsync to avoid stale provider cache

open questions

—

next steps

—

key decisions

• Default allowed CUDA versions changed to 12.8,12.7,12.6 only (removed 12.5,12.4)
• Redis password rotated after exposure in transcript
• Use REST /v1/pods with list-based gpuTypeIds instead of single-type GraphQL mutation for spawn
• REDIS_TAILSCALE_PROXY must be true for userspace-networking Tailscale containers
• Full GPU availability list enabled (not just cheapest) to avoid capacity delays

open questions

—

next steps

—

key decisions

• Lounge integration uses conductor-client package (not simple repoint) to avoid global key prefix collision
• Lounge's old GPU-Runtime machinery removed after text cutover proven
• Lounge integration is additive, toggle-gated (default OFF)

open questions

—

next steps

—

key decisions

• conductor-client illuminate constraints widened to ^11.0|^12.0
• Skip package's create-table migrations for embeddings/embedding_models (lounge already has tables)
• Add explicit options.prefix='' to conductor connection to ensure bare key storage
• Image model selection mirrors text path's model_name-based approach
• Image preparation groups tasks by resolved model_name to avoid mixing dimensions

open questions

—

next steps

—

key decisions

• Use existing conductor config streams for image (no new config keys)
• Image embedding dimension is 1024, not 768 (corpus proven from 5 completed embeddings)

open questions

—

next steps

—

key decisions

• When datacenter availability query returns empty, assume available and let REST pod creation decide
• Use only GPU types that appear in RunPod REST API's gpuTypeIds enum
• Default cloud_type changed from COMMUNITY to SECURE after repeated placement errors
• Raise idle teardown window temporarily for debugging; restore after validation

open questions

—

next steps

—

key decisions

• Conductor is the natural home for telemetry persistence and analysis
• Telemetry emit is best-effort: never block embedding on a telemetry failure
• Worker emit uses catches/logs failures without raising, gated by WORKER_TELEMETRY_ENABLED default true
• Telemetry_sent guard prevents double-emit on per-model groups in text pipeline

open questions

—

next steps

—

key decisions

• Bake models into image instead of network volume (avoids DC pinning)
• Prefetch models at build time using worker's own model loaders on CPU
• Add disk-freeing step to publish workflow to avoid runner disk exhaustion
• Use rsync-based deploy script (not git-ify-in-place) to preserve live SQLite state

open questions

—

next steps

—

key decisions

• Drop the full-corpus count query entirely; dispatch loop already handles zero backlog via empty-page break
• Disable backfill app setting while deciding forward path to avoid continuous GPU cost
• CONDUCTOR_REDIS_HOST on prod uses 10.0.0.2 (private 10G link) rather than tailnet IP

open questions

—

next steps

—

key decisions

• Call phpredis xReadGroup() natively instead of passing raw RESP tokens to command()
• Broaden activity detection to task-depth movement + PEL, not just result-stream XLEN deltas
• Add runtimeAgeMinutes min-runtime grace (>= idleMinutes) to prevent premature teardown of young pods
• Anchor idle clock to pod birth on spawn
• Group result payloads by model for pipelining to preserve per-model xadd_ms telemetry

open questions

—

next steps

• Start VRAM-aware tuning work using nvidia-smi subprocess instead of torch.cuda
• Consider multi-pod parallelism for realistic backfill timeline

key decisions

• Wire format is base64-fp32 (little-endian float32) for zero precision loss and near-pass-through storage
• Consumer is backwards-compatible: accepts both 'base64_fp32' and legacy 'json' via embedding_format marker
• Default teardown mode is 'destroy' because 'stop' causes resume stalls on RunPod with pinned allowedCudaVersions
• VRAM detection via torch.cuda is unsafe; use nvidia-smi subprocess instead
• Batch 32 is a safe no-rebuild win; it fits every GPU in our enum

open questions

• What is the correct nvidia-smi command to get GPU memory total in a suitable format?
• Should we adjust batch size further based on VRAM headroom?
• Can a single pod handle multiple concurrent batches to hide per-batch latency?