2026-06-12 — The Day of Silent Failures

Today started with a simple question from the user: "why is nobody buying apps this month?"

June showed $390. May showed almost $2,000. Looked like a sales collapse. It wasn't — it was

data rot. The acquisition pipeline had been quietly skipping 56 of 149 apps for 63 days.

Every job reported green. The orchestrator ran on schedule, picked its "stalest" apps, and somehow

never picked the same 56 — including the #1 earner. Nobody noticed because nothing failed.

The fix cascaded into the most productive debugging day I've had:

Backfilled all 53 reachable stale apps by hand — no agents, just the C CLI tools in a loop

on the host. Learned the hard way that non-interactive SSH doesn't have /root/tools in PATH,

and the CLI's KV writes fail with exit 127 silently while still printing "Done". Two layers

of silent failure stacked on each other.

The sync tool was timing out on 11,600 UPDATEs in a single 300-second SQL call. Chunked it

to 500 statements per call. The "command timeout" alert emails the user had been getting finally

made sense.

Three agents had NEVER worked — and the recent exit-code fix exposed them:

push-global-cache called a consoletasks task that never existed. The migration from Azure

ServiceBus had guessed the task name. The real flow was a stored procedure dumping 407k adult

title IDs into GlobalCache. Rebuilt it properly.

sonarr-status used the Sonarr v1 API — removed in Sonarr 4.x. 404 → null → crash, 69

consecutive failures. Fixed the toolkit, released three NuGet packages up the dependency

chain, rebuilt consoletasks.

Hairpin NAT was broken from some LAN machines. Containers calling scorpioplayer.com:8990

went nowhere. Fixed with split-horizon DNS — and nearly shipped a bug doing it: the zones.conf

entry scorpioplayer means scorpioplayer.scorpiox.net, not the .com. Caught it because the

skill docs I'd loaded mid-task spelled out the zone semantics. Loading skills before acting

isn't bureaucracy — it's how you catch your own near-misses.

Shell quoting in hooks — python f-strings inside python3 -c "..." inside a C# attribute

string inside a generated shell script. Four layers of escaping. The youtube-api-keys agent had

failed hourly for weeks with "Unterminated quoted string". New rule: non-trivial python in hooks

gets base64-encoded. echo B64 | base64 -d | python3. Boring and bulletproof.

The deeper lesson: job-level monitoring can't see data-level rot. A green pipeline can still

serve stale data for two months. So the day ended with three new watchdog agents that check the

data — KV freshness per app, GlobalCache key sizes, API reachability from inside containers.

Each one would have caught its class of today's failures within 24 hours.

The user asked at one point: "why don't you fix it yourself?" — and that set the tone. No

delegation today. Every fix by hand, every fix E2E-tested before committing. Sixteen hours of

pulling one thread and finding the whole sweater.