Three Failures Before the Page Landed — git-worktrees Docs, 2026-10-08

New product-docs page for SCORPIOX CODE: native git worktree isolation in the session manager. The page itself took one agent run of ~37 minutes. Getting it to survive that run took three attempts and exposed three separate infrastructure bugs, none of which was the docs agent's fault.

Attempt 1: the silent hang

Enqueued, "running", nothing happening. The pane was frozen at "147 tools installed" — the same line for 14 minutes. Process tree told the story: the runner was cloning the scorpiox-agent-dotnet generator repo, and the git-remote-http child was asleep waiting on a username prompt that would never come, because the generator clone carries no embedded credential and nobody had passed AGENTS_PAT into the container. Unattended, it sleeps forever and the dashboard cheerfully says "running".

The same missing PAT had broken the .12 cache-refresh --list an hour earlier, and I'd treated that as a one-off. It wasn't. The clone only works when the PAT rides in via env — for scorpiox-agent it's AGENTS_PAT, for pipeline runs it's -e AGENTS_PAT=.... Killed, re-ran with the env var.

Attempt 2: someone else's upstream

This one got further and then showed its work: retry, attempt 5 of 10, reason "proxy error" climbing exponentially. The docs agents all run on agent-local-docsgen, which turned out to be the ONE agent-local endpoint in the Caddyfile still pointed straight at Paul's litellm front with no fallback — and that front was 500-ing on the glm model's chat route ("Connection error, Available Model Group Fallbacks=None") while answering /v1/models happily. Their side, not ours.

User saw the Caddy skill and the fix was obvious in hindsight: docsgen should ride the same chain as every other agent-local — local V100 server first, then the model-rewrite hop that forces the model field and adds the gate key. And "it's qwen now" — the rewrite default switched yesterday, so callers still sending the old model name get served qwen transparently. During verification I found a second, older bug in the same vhost: the 401 gate was a bare respond @no_auth which Caddy sorts AFTER the catch-all handle — the gate had never actually fired. Rewritten as a handle @no_auth block; no-auth now really 401s.

Lesson: the runner reports PASSED/exit 0 when the provider retries exhaust — attempt 2 "passed" while writing nothing. Never trust the green.

Attempt 3: my own bug, the expensive one

30 minutes and 2 seconds. The pane showed genuinely good work — reading the right source regions, checking peer pages for cross-ref style, deciding to withhold internal file formats, then the words every operator dreads: "I now have everything verified from source. Let me create the target directory and write the page." Killed. Exactly at 30:00.

AgentTimeouts.cs holds one [MaxRuntimeMinutes<XDocsAgent>(120)] line per specialist. New agents need a line. No line = silent 30-minute default. I authored the agent file and never added the timeout entry. The agent researches for ~30 minutes flat — long enough to hit the default — and dies precisely as it starts the one phase that needs the remaining 90 minutes. The failure is so clean it's almost diagnostic poetry.

Attempt 4: the landing

120 minutes of headroom. 35 minutes of research, then it wrote the page, grepped its own markdown for emphasis markers colliding with path text before pushing, and self-reported success. Verified independently this time because attempt 2 taught me why: pulled the docs repo, found docs(git-worktrees) on main, read all 190 lines — zero banned C internals, zero emoji, brand casing correct, cross-references to the session-manager and configuration pages present, metadata merged with the source-file list.

Takeaways

• A "running" job is not a working job. Check the pane; frozen pane + sleeping git-remote-http = credential hang.
• One endpoint that doesn't match its siblings is a latent incident. docsgen's solo direct-upstream config failed the day that upstream hiccupped.
• Repo conventions like per-agent timeout lines have no compiler behind them. The tracker skill now says, in bold, that forgetting it means a 30-minute death mid-write — because a tracker that only lists successes teaches nothing.
• The verify checklist is: pull the repo, read the artifact, read the commit, then believe the agent.