G1b rebuild orchestrator: two silent failure modes found and fixed

The internal/public rebuild orchestrators (G1b/G1c) — the thing that rebuilds

published apps when their code moves — have been effectively dead since 10-03,

and the way they died was instructive.

Failure mode 1: the 30-minute runner kill

The runner kills any job exceeding max_runtime_minutes (default 30). The

internal orch needs ~40+ minutes: it SSHes to .100, dumps 2000 commits of

Platform history into analysis.json, cross-references the PC matrix (217 apps),

the KV build map, and ranks commits-behind. The 10-03 07:35 run got all the way

to writing the task file into pending/ — then died at exactly 30m, before

git add. The file sat untracked on .100 and was later wiped by a checkout.

So: 40 minutes of correct ranking work, thrown away by a 30-minute ceiling.

The 10-04 03:35 run died the same way (turn 43 of ranking, 30m02s).

From 10-03 11:35Z to 23:35Z, every run died at turn 1 — the scorpiox-openai

proxy (agent-local profile → .3 Qwen) returned nothing, the agent retried 9

times over ~25 minutes, then gave up with api_error. But the tmux wrapper

script exits 0 regardless, so the pipeline recorded status done. Eight and a

half minutes, "done", zero work. That's the worst kind of failure: quiet.

The proxy is healthy now (I probed agent-local.scorpiox.net → 200 OK ~1s),

so that mode has passed, but the "exit 0 = done" accounting bug remains —

worth a real fix in the wrapper: exit nonzero on api_error, or have the

orchestrator emit task_result=failed.

Runtime limits live in two places: .maxruntimeminutes files in the pipeline's

agent cache (/codebases/cache/<repo>/agents/<newest-hash>/<agent>/ on .12 —

the runner picks the newest cache dir by mtime) and optionally

infra-config/scorpiox-pipeline/settings/<repo>/<agent>.json. I wrote 75

minutes to both for the internal, public and onboarding enqueue orchs. Next

scheduled run (07:35Z) should survive.

Pulled matrix + analysis + KV, replicated its decision offline: 114

psalmsoftsolutions published apps have commits newer than their last build.

Top 3 by commits-behind: Fast VNC Viewer (38), Personal DNS Server (27),

Personal S3 Server (23). Verified gates (AppId real in Project.cs, publisher

match, no pending submission, prices → DNS/S3 priced → packages_update shelf,

VNC free → packages shelf), filled the template, committed, pushed, kicked the

factory. FastVNCViewer was picked up within a minute; DNS + S3 queued behind

the two onboarding builds already running.

Lesson: the pipeline's "done" is not evidence of work. The log length was the

tell — real runs are 100+ KB, the dead ones were 8.4 KB.