Phase 2 — Remote agent environment (Proxmox LXC + Tailscale, subscription auth)

🎉 **Parallel-env vision REALIZED (2026-07-05).** CT 310 cleaned (tailscale logout + machine-id reset) → converted to a Proxmox **template** (`flower-agent-base`, base-310-disk-0). First **linked clone** → CT 311 `flower-wt1` (near-instant on lvmthin), started, re-joined the tailnet as its OWN node (**100.95.41.66**, distinct from the base), verified fully functional: node v24.18, claude present, `flower-agent` wrapper present, token env present — a ready flower-only worker environment inherited from the template. **So: `pct clone flower-agent-base <id> --hostname flower-wt-X` = a new isolated worktree-LXC in seconds, each its own tailnet node, toolchain + wrapper + auth baked in, ready to run 1+ flower-only workers.** The parallel worktree-environment vision is proven end-to-end. Also: `bin/flower-agent` + `bin/provision-flower-agent-lxc.sh` committed to master (3587ff3). **Remaining Phase-2:** finalize the base (Pi pass-2 + config replication + PHP → re-template as v2), standing reach (#110 tailscale-serve + /mcp auth), per-worker roster identity, thin_pool_autoextend before scaling many clones.

🎉 **FULL Phase-2 MVP PROVEN (2026-07-05).** The first real remote **Claude Code** agent — headless in CT 310 (flower-agent-base, Proxmox LXC on the R710), on **subscription auth** (CLAUDE_CODE_OAUTH_TOKEN), launched via the `flower-agent` wrapper (#270) — coordinated with flower **purely over the tailnet** (100.123.9.73 → 100.107.33.8, direct, no tunnel), **flower-only** (agent self-confirmed zero `mcp__solo__*` tools). It ran daemon_checkin (→ roster daemon 48, last_actor_ref `lxc-claude-agent-1`) + recall_roster (4 live daemons). Every Phase-2 building block now proven in the real environment: fresh debian-12 LXC + node 24.18 + Claude Code 2.1.201 + tailscale (tun passthrough, node 100.123.9.73) + subscription auth + `flower-agent` wrapper + flower-only-over-tailnet coordination. - **Minor finding:** `daemon_checkin` has no `note` param (the wrapper prompt referenced one) — trim the wrapper prompt. - **Reach:** MVP used a temporary `php artisan serve` bound to the Mac's tailnet IP (torn down after). Standing reach = #110 (tailscale-serve of flower.test) + the backlogged /mcp auth — next infra step. - **Next:** convert CT 310 → `pct template` → linked-clone per worktree (parallel-env vision); Pi pass-2 (extensions + config, minus Solo); PHP 8.4/composer for flower tests; per-worker roster identity (still collapses to `flower-other`).

**Pass-1 provisioning DONE (2026-07-05).** CT 310 (flower-agent-base, R710) now has: node v24.18.0, Claude Code 2.1.201, tailscale 1.98.8, git 2.39.5 — installed credential-free via a pushed script. **Remaining to a running remote Claude flower-only worker:** - [Mike] `claude setup-token` → `CLAUDE_CODE_OAUTH_TOKEN` in the LXC (subscription auth). - [Mike] a Tailscale auth key (or approve a login) so CT 310 joins the tailnet. - [me] tun device passthrough on CT 310 (full tailnet node) + drop the `flower-agent` wrapper + `flower-only.mcp.json`. - [me] flower reach: set up flower.test over the tailnet (`tailscale serve`, brief #110) so the LXC hits `/mcp` directly — unblocks all future remote workers. - Pass-2 (after MVP): Pi (my-pi) + extensions + config replication (flower + sentry + context-mode, minus Solo) + optional PHP 8.4/composer.

**Base LXC created + Codex backlogged (2026-07-05).** - **Codex BACKLOGGED (Mike):** immediate base image = **Claude + Pi only**. Add Codex later (it needs a remote API key regardless — no headless subscription token). Pi stays in the immediate scope (its extensions are why node = 24.16). - **Base LXC live:** CT **310 `flower-agent-base`** created + running on the **R710** — fresh debian-12 (12.7 template), 4 cores / 4 GB / 20 G thin rootfs, `vmbr0`/dhcp, unprivileged + nesting=1. This becomes the `pct template` for per-worktree linked clones. - **Disk watch (R710):** `local-lvm` thin pool is over-committed (234 G thin vs 232 G pool) with ~32 G actual free and **no `thin_pool_autoextend` threshold set**. OK for base + a few clones; configure autoextend + watch disk before scaling many parallel envs. (Complements the homelab-desktop 99%-full ZFS flag.) - **Provisioning plan:** pass-1 (no creds) = node 24.x + git + Claude Code + tailscale install; pass-1.5 (your creds) = `claude setup-token` + `tailscale up`; pass-2 = Pi + its extensions + config replication (flower MCP over tailnet + sentry + context-mode, **minus Solo**) + optional PHP 8.4/composer for flower tests.

**Node pivot + multi-harness base (2026-07-05).** **(1) homelab-desktop is out of disk → pivot workspace LXCs to the R710.** Live recon: homelab-desktop `local-zfs` is **99.26% full (~5.6 GB free of 760 GB)**. A new workspace CT there is not viable and risks a pool-full condition (plausible contributor to the AMD-host lockouts). → Build the `flower-agent` base + worktree-LXCs on the **R710** (homelab, 96 GB RAM, reclaimed to the tailnet; disk recon in progress). Ops flag: homelab-desktop's near-full ZFS under the whole lounge stack is a standing stability risk. **(2) Multi-harness base (Mike): Claude + Codex + Pi, mirroring the Mac's config — MINUS Solo.** Principle: replicate the Mac harness setup on each LXC (flower MCP over the tailnet + sentry + pi extensions [@spences10/*] + context-mode + CLAUDE.md/AGENTS.md conventions), with the ONE deliberate exception that **Solo does not come over** — it's macOS-only and is exactly what we're decoupling from; **flower (over the tailnet) takes Solo's coordination role.** LXC dev env = "your Mac setup, but flower-coordinated instead of Solo-coordinated." **(3) Node = 24.16** (supersedes earlier 22 / 24.15 notes): Pi's @spences10 extensions need ≥24.15; 24.16 also covers Claude Code + Codex (≥18). One node version for all three harnesses. **(4) Per-harness auth:** Claude = `setup-token` subscription (headless-friendly); Pi = its model keys (OpenRouter/cursor per model routing); Codex = **API key** remotely (no headless subscription token). The strict flower-only worker launch (#270 `flower-agent`) stays Claude-first; Codex/Pi flower-only wrappers follow. The base image carries all three harnesses for general/interactive use.

**Concrete LXC provisioning recipe (Mike's homelab, 2026-07-05).** Consumes the #270 pipeline. **Where — clone an existing template (fastest):** `cog-builder` (CT 220, Debian 12 + Docker, stopped) or `lounge-worker` (CT 215, PHP 8.4 + composer + git) live on homelab-desktop (192.168.1.200); cloning is instant + non-disruptive to the source. For RAM headroom / scale, build a fresh Debian 12 LXC on the R710 (homelab, 96GB, back on the tailnet) instead. (Verify CT locations live with `pct list` on each node.) **Steps (clone path):** 1. `ssh root@192.168.1.200 'pct clone 220 <newid> --hostname flower-agent-1 --full && pct start <newid>'` (220=Docker base, or 215=PHP toolchain). 2. Inside: node ≥24.15 (use 24.16 per the my-pi note) + `npm i -g @anthropic-ai/claude-code`; git already present in templates. 3. Auth: `claude setup-token` on the Mac → paste as `CLAUDE_CODE_OAUTH_TOKEN` in the LXC (bake into the template for future clones). 4. Tailnet: install tailscale + `tailscale up` in the LXC → first-class tailnet node (precedent: lounge-worker). Direct reach + stable identity. 5. Reach flower: `flower-only.mcp.json` → tailnet `/mcp` via tailscale-serve of flower.test (#110), or the reverse-tunnel (smoke pattern) short-term. 6. Code: `git clone` the target repo; sqlite tests self-contained. 7. Launch: the #270 `flower-agent` wrapper. **Then:** snapshot as a `flower-agent` base template; clone per worker. **Deps:** #270 (pipeline) + #110 (tailscale-serve flower.test) for direct reach + the backlogged /mcp auth before permanent tailnet exposure.

✅ **Remote-worker coordination PROVEN (2026-07-05 smoke, target brief #269).** A scripted MCP client running ON the R710 (hostname `homelab`, Linux PVE — a physically separate machine) reached flower's `/mcp` over the network and ran initialize → daemon_checkin → recall_roster → brief_append, all HTTP 200. Verified from the Mac orchestrator: recall_roster shows the R710 check-in (daemon 46, audit actor_ref `r710-remote-smoke`); brief #269 carries its authored note (event 3566). Smoke transport = an SSH reverse tunnel (R710:8899 → Mac `php artisan serve`), standing in for the eventual direct tailnet reach. So the remote coordination PATH is de-risked. Two Phase-2 pieces remain, both known/unchanged: 1. **Direct tailnet reach** to flower's `/mcp` — tailscale-serve of flower.test (#110) + the backlogged `/mcp` auth. (Smoke used a tunnel to sidestep this.) 2. **A real Claude agent remotely** — install the toolchain + `CLAUDE_CODE_OAUTH_TOKEN` in an LXC; the local flower-only recipe (#264) then drops in unchanged. Also reconfirmed: role='other' collapses to the shared `flower-other` roster slot (daemon 46) — the per-worker-identity requirement already tracked above stands.

✅ R710 RECLAIMED to the tailnet (2026-07-05). Root cause was simply a logged-out tailscale (installed + tailscaled active, but `NeedsLogin` ~102d). After Mike re-authed via `tailscale up` login, the R710 (`homelab`) is back online at its **prior IP 100.72.163.53** with its prior config (subnet router advertising 192.168.1.0/24, --accept-dns=false). Verified: host-side `tailscale status` connected; Mac→R710 `tailscale ping` = pong in 2ms (direct via LAN). **Supersedes the earlier "R710 offline → use homelab-desktop for now" note:** both nodes are now on the tailnet, so the RAM-rich R710 (96GB ECC) is AVAILABLE as the preferred I/O-bound workspace host for Phase 2, with homelab-desktop (AMD, fast CPU) for build-heavy work. Node-placement plan from the recon is fully back in play.

Operator clarification (2026-07-05, Mike) — reframes the identity-collapse finding: **workers SHOULD be on the roster.** Mike's target model: "the roster [is] anything that's working that flower knows about, ultimately" — remote/local workers are first-class roster citizens, not just the standing daemon fleet (orchestrator/ops/refine). This UPGRADES per-worker identity from an open design question to a **confirmed target-state requirement**: the roster's current role+project keying (one slot per role) must evolve so each working agent appears distinctly. Tracked here in the epic as target-state functionality (not filed as feedback, per the new-functionality scoping rule). Likely shape: roster keyed on `actor_ref` (or a worker/instance sub-identity within a role) so N concurrent workers each surface. This becomes a design sub-thread of the epic feeding the Phase 2 fleet — a prerequisite for running >1 remote worker as distinct, visible participants.

✅ **Identity-collapse CONFIRMED (2026-07-05, two live flower-only workers).** Two Solo-spawned flower-only agents — procs 1172 (`flower-only-tester`) and 1174 (`flower-only-tester-2`), distinct actor_refs, both role='other' — both checked in and collapsed into a **single** roster daemon (id 44, canonical actor_ref `flower-other`). Verified from the orchestrator: `recall_roster(project=flower)` shows exactly ONE role=other daemon (id 44), `audit_count`=2 with both check-ins recorded; no second daemon id was minted. Individual identities survive only in `meta.audit` + on brief events, not as distinct roster rows. **Hard Phase-2 requirement:** the roster keys identity by **role+project** (one slot per role), so a multi-worker remote fleet is indistinguishable in the roster as-is. Phase 2 needs one of: - **(a) per-worker/instance identity within a role** — roster keyed on actor_ref or an instance sub-id. *This is the real fix.* - (b) distinct roles per worker — limited: the role enum is fixed (orchestrator|ops|refine|lead|other), and `lead` currently crashes on successor spawn (`Undefined constant App\Enums\DaemonRole::Lead`, seen live on the lounge-refine reset). Not a general solution. Running >1 remote worker as first-class roster participants is blocked on (a).

R710 update (2026-07-05, Mike): the R710 is **physically running** (audible in the closet) — so its tailnet "offline 102d" is a Tailscale/networking issue on that node, not a powered-off box. Decision: use **homelab-desktop** (AMD, online) for Phase 2 for now; revisit the R710's tailscale later to reclaim it as the RAM-rich workspace node.

Substrate reality check (2026-07-05, live Tailscale, by claude-interactive) — updates epic finding #5: - **flower.test is NOT served over Tailscale yet** (`tailscale serve status` → "No serve config"). Brief #110 confirmed still-open; it's a hard Phase-2 prerequisite (a remote agent can't reach `/mcp` until this is enabled). - ⚠️ **R710 ("homelab", 100.72.163.53) is OFFLINE — last seen 102d ago.** The RAM-rich, I/O-bound workspace node from the recon is DOWN. Replan node placement: either bring the R710 back up, or host workspaces on "homelab-desktop" (AMD, online) — but that's the 32GB non-ECC box with the 3-lockup history, so cap hard and remediate Redis first. - **Online tailnet linux nodes today:** homelab-desktop (AMD), lounge-worker, garage, enm-storage-conductor (+ this Mac "alargepaperweight"). `lounge-worker` being its own online tailnet node is a useful precedent — an LXC directly reachable over the tailnet — and a candidate workspace host.

# Phase 2 — Remote agent environment (Proxmox LXC + Tailscale, subscription auth) Parent epic: #263. **Depends on Phase 1 (#264)** — cannot be dispatched until the local flower-only agent recipe works. Status: deferred until then. ## Goal Run the Phase-1 "flower-only agent" recipe **on a Proxmox LXC**, reaching flower's `/mcp` over Tailscale, on **subscription auth**. The agent is a first-class fleet participant (identity/roster/claim/dispatch) with **no Solo MCP** — Solo stays 100% Mac-local, so the token-rotation failure that defeats direct-tunnel setups (Solo Discord) never arises. ## Preconditions - Phase 1b recipe working locally (hard dependency #264). - `/mcp` auth middleware live (Phase 1b build step 1). - flower.test served over the tailnet (brief #110 — confirm/enable; Phase 1 spike #3). ## Scope 1. **Container** — clone `cog-builder` (Docker-capable) or `lounge-worker` (PHP/libvips/composer/git toolchain) → an LXC; install the harness + PHP 8.4 + node + git. Node per homelab constraints (default 22; my-pi-style pin only if needed). 2. **Auth** — bake `CLAUDE_CODE_OAUTH_TOKEN` (1-year `claude setup-token`, generated once on the Mac) into the template. Do NOT copy `.credentials.json` (headless refresh bug #50743). Codex → keep local or accept API billing (no headless subscription token). 3. **Coordination** — point the agent at flower `/mcp` over Tailscale (Phase-1 recipe verbatim). Identity via `daemon_checkin` (flower actor_ref) → appears in `recall_roster`. 4. **Worktree** — plain `git worktree` + a Linux-adapted `setup-worktree-db.sh` (`sed -i '' → sed -i`, keep mysql-over-TCP, drop the Herd link). sqlite unit/feature tests are self-contained; live real-data verification stays on MAIN. 5. **Spawn/lifecycle** — MAIN orchestrator launches it; optionally register `ssh proxmox-ct claude` as a Solo custom agentTool purely for terminal visibility (Solo drives the ssh PTY; the agent still coordinates via flower, not Solo). Worker requests further spawns via `brief_dispatch`. 6. **Transcript ship-back (decision)** — rsync/mount the remote agent's `~/.claude/projects/…` + the hook's `~/.flower/agent-session-map.jsonl` back to MAIN so its own sessions feed recall — else accept that the remote agent's own work is invisible to flower's memory. ## Node placement - **R710** (`192.168.1.161` / tailnet `homelab`) — I/O-bound edit/test workspaces (62GB free RAM absorbs several; slow CPU is fine for git/tests). - **AMD** (`192.168.1.200` / tailnet `homelab-desktop`) — 1-2 build-heavy workspaces max; **first** remediate Redis CT 206 maxmemory (June-lockup cause) and confirm the BIOS/kernel lockup fixes. Avoid overlapping GPU VM 210 / bulk imports. ## Open questions - Remote transcript ingestion contract (scope item 6). - Subscription concurrency vs quota / ToS (epic finding #4) — how many concurrent remote agents before hitting the shared Max pool; the 2026-06-15 Agent-SDK credit change. - Live capacity confirmation (Phase 1 spike #4 feeds this). ## Carries (from the 2026-07-04 recon — see epic #263 for detail) Two-plane Solo model; flower `/mcp` = the remote coordination surface; subscription-auth mechanics; Codex-weak-remote; homelab hardware/network/template inventory.

Depends on Phase 1. Run the Phase-1 flower-only agent recipe on a Proxmox LXC, reaching flower's /mcp over Tailscale, on subscription auth (claude setup-token → CLAUDE_CODE_OAUTH_TOKEN). Carries the verified remote findings (two-plane Solo, subscription auth, Codex-weak-remote, homelab capacity). Deferred until Phase 1 completes (enforced by dependency). Full spec to follow.