The 8 KB wall — 2026-10-11
Every worktree chat on bot.scorpiox.net went blank at once: "No messages yet. Send a prompt to start." The messages were right there on disk — rdrsave had 20 event files, starting with "hi pico".
Nothing had been deployed. The cause was growth. bot-api asks scorpiox-tmux --list --output-json where each session's worktree is, and read the answer into char buf[8192]. With 47 sessions running the answer was 9,620 bytes. The JSON got cut mid-object, the parse failed quietly, and the API fell back to the main repo's mirror copy of each session — which has no events/ folder. Zero messages, HTTP 200, no error anywhere.
What found it fast:
- Headless Chromium first: zero page errors, so not the web layer.
curl :8150/hashes?id=rdrsave→total:0, while the files existed. API layer.straceon the handler binary showed exactly whicheventspath it probed — the mirror.- Piping the tmux list through
head -c 8191into a JSON parser reproduced the failure character-for-character.
Fix: read the whole stream (sx_read_all), and when the API does fall back to scanning, trust meta.cwd — the folder the agent actually runs in — over the mirror. Then, while there, raised the session-root scan cap from 600 to 4000 (we were at 307) so the next wall isn't sitting just past the horizon.
Lesson: a fixed buffer on a list that grows with usage is a time bomb with no error message. The failure mode wasn't a crash — it was a silent fallback to a plausible-looking wrong answer. Those are the worst kind, because everything reports success.