- the is_running() fast-fail fix (v1.3.91) verified cold backend-http-ready
23.5s -> 3.86s and warm 5.0s -> 3.32s on the signed build, no 9Router regression
- strip the [perf] bg entry logs from mcp/skill/settings/service/9router now that
the stall is diagnosed; keep Apps.py per-lifespan timing but only print a
lifespan over 50ms plus the total (cheap regression tripwire)
- document the root cause + fix + before/after in docs/perf/winv2/README.md
- faulthandler on the signed cold build caught the asyncio loop frozen in
socket.create_connection inside is_running() (process.py): a synchronous
httpx.get to "localhost:20128" called ~5x on the boot path before 9Router is up
- on Windows a dead-port connect to "localhost" stalls ~7s each (tries ::1 first,
loopback refusal is slow), freezing the loop ~18s so uvicorn could not answer
the health probe -> cold backend-http-ready was ~23s
- fix: probe 127.0.0.1 with a 0.3s TCP timeout first (measured 306ms vs ~7s), only
HTTP-confirm when the port is open; 9Router binds 0.0.0.0 so reachability is
unchanged. drop the faulthandler diagnostic from main.py
- v1.3.89 cold proved two stacked stalls: ~5s in is_running() (sync httpx) and a
bigger ~13s BEFORE any background task runs, a silent event-loop freeze
- add faulthandler.dump_traceback_later(7s, repeat) to a temp file so a dump lands
inside the 13s window and names the exact synchronous call the loop is stuck in
- add an entry log to settings._boot_router_then_sync (first startup bg task)
- diagnostic only; reverted once the stall is pinned
- instrumented cold v1.3.88 proved the lifespans are 141ms even cold; the ~18s
cold gap is a backgrounded create_task blocking the event loop AFTER lifespan
startup but BEFORE uvicorn reports ready (what the health probe waits on)
- add [perf] entry/segment logs to the post-startup background tasks: mcp refresh,
skill refresh, 9router ensure (+ prelude bisection), and svc._post/_post_or_spool
- logging only, no behavior change; next cold log names the exact blocking call
- audited all 16 lifespans + the service client in source: every body is trivial
(yield / makedirs / early-return migrate / fire-and-forget svc.sync)
- the no-op debug() shipped and verified live but cold stayed 21.5s, so the
DEBUGLETON scan was not the cold driver; residual ~16s is cold first-run paging
- next cold build with the new [perf] markers pins it or confirms distributed I/O
- debug(sub_app.name) is a no-op in packaged builds, so the packaged backend.log
had zero per-SubApp markers and a cold stall could only be guessed at
- wrap each enter_async_context with time.perf_counter + a flushed [perf] print,
plus a lifespans-total line, so a cold launch names the exact slow lifespan
- logging only, no functional change; loop logic validated warm
- first debug() call instantiated Debugleton -> recursive os.scandir project scan on the boot path (~17s cold, ~80ms warm)
- early-return when OPENSWARM_PACKAGED=1; dev keeps the full debugger; debug() returns None so callers are unaffected
- validated: with the flag debug() no-ops (no scan); debugger is pip-installed from repo/debugger at build so this ships
ciregenz