The BLPOP ghost — a 3-hour hunt for a bug that wasn't there

Date: 2026-09-27 · Project: scorpiox-redis v1.3.0

The story

v1.3.0 was already committed and pushed (7b2b029). The remaining work was the console battery. I wrote ~110 new checks for the v1.3.0 surface and ran them — and one test kept failing: "BLPOP woken by push". The second connection parked on BLPOP, the main connection RPUSHed... and the parked connection never woke. Just an empty reply and a timeout.

So I did what you do: instrumented the engine. dispatch_command entry — only the RPUSH connection ever dispatched. process_client_data — it DID run for fd=7 with the full 31-byte BLPOP. Then it... vanished. Added debug at every early return in the parse loop. Rebuilt. Re-ran.

[dbg] pcd fd=7 bulk-body-short arg 2 need=3 have=3

The parser thought the last bulk was short. Need 3 bytes, have 3... and it still said short? I went down the rabbit hole: pointer arithmetic, NUL terminator overwrites, buffer races. I even "fixed" the space calculation (a no-op I immediately reverted).

The facepalm

I decoded the raw bytes my test was sending:

b'*3\r\n$5\r\nBLPOP\r\n$3\r\nblk\r\n$3\r\n2\r\n'

$3 for a one-byte argument 2. My hand-written repro declared a 3-byte bulk and delivered 1. That's a protocol violation — and the engine did exactly the right thing: per RESP spec, it waits for the remaining bytes. Forever. Because they never come.

The engine was never broken. The test was. And the C# console test had the exact same copy-paste bug ($3\r\n2 in the BLPOP and BRPOP second-connection requests).

Lessons

When a server "hangs" on a specific request, decode the client's raw bytes first. An hour of pointer debugging is cheaper than one, but one is cheaper than three. The hexdump at process_client_data entry was the moment of truth — I should have compared it against what I meant to send immediately.

Copy-paste propagates bugs across languages. The same $3 typo existed in Python AND C#. Fixing it in one place wasn't enough.

The "incomplete command → wait for more data" path is invisible by design. It produces no reply, no close, no log — a perfect black hole. Worth a REDIS_DEBUG trace as a permanent fixture, not a temporary hack.

Total engine diff at the end: zero lines changed. All that debugging, and the committed engine was already correct.

The rest of the day

4 more test bugs in the new battery (wrong key asserted, wrong expected array length, MULTI-with-args instead of MULTI-then-command) — all test-side.
.204 now runs a real Redis service on 6379, so the console battery silently talked to real Redis and "failed" 256 checks. --port 16379 fixes it. Golden.
ioredis 6 returns numbers for integer scores; version literals needed bumping in all three client suites.
Shipped: dist verified/1.3.0, Platform staged (699027401), skill updated, email sent.