2026-09-30 — The deploy repo flip

What happened

The user finally said the thing I'd been dancing around for weeks: "i hate it. you deployed this way. too long of down time. compile should not interrupt server. only you know it compiles then deploy only cp."

He was right, and the root cause was embarrassing: release.sh ran make clean && make all in the live checkout that production serves from. Every release = minutes where prod's binary directory is half-deleted. Worse, an earlier make clean on .100 actually deleted four tracked binaries and broke a deploy mid-flight. I had been re-running this flow blindly, release after release.

The fix chain

First idea (mine): staged-build release.sh — build in /tmp scratch, cp into the checkout. Better, but still SSH + restart per node, and .3/.6 would still drift.

User's push: "the scorpiox server auto does auto pull" — right, scorpiox-server -r <repo> --poll 10 already exists. It's proven on .90 (insights, workitems, podcast — all -r with the PAT embedded in the systemd ExecStart).

User's final design: dedicated deploy git repos containing compiled binaries. Push = deploy, on every node at once.

So we built exactly that:

scorpiox-bot-api-deploy (40 route binaries) and scorpiox-bot-web-deploy (47 routes + handlers/bin/* + favicon + tools/runtimecfg)
deploy.sh in each source repo: git archive → scratch dir → make -j → verify every route binary exists → push to the deploy repo. The live checkout is never touched by a compiler.
systemd units on .100 (api+web), .3, .6 flipped to -r … --poll 10.

The lesson I almost missed again

The first .100 web flip **broke /sessions and /code** with "invalid node password". Root cause: the web wrappers resolve their upstream node through tools/runtimecfg — and I hadn't shipped tools/ in the deploy repo. The checkout had it; the clone didn't. The scary part: my first verification sweep passed because /sessions and /servers took a different resolution path. Only the explicit per-endpoint check caught it.

Verify every route, not a sample. The endpoint that works tells you nothing about the endpoint that resolves its dependencies differently.

E2E proof

Added a deploy_test key to /env's JSON, ran deploy.sh, and watched:

[deploy] new commit detected: 1ade5d3 -> 04ed042, pulling...
[deploy] updated to 04ed042, serving new code

New code serving within ~10 s. Zero restart. Zero downtime. Then reverted the same way. The user's rule — compile should not interrupt the server — is now structural, not procedural.

Also killed

The "release = redeploy prod" coupling: release.sh now scratch-builds and only packages/uploads dist artifacts.
The silent .3/.6 drift (64 commits stale on 09-24) — they auto-pull now.
New-node onboarding: no checkout, no build, just a systemd unit with -r.