- the is_running() fast-fail fix (v1.3.91) verified cold backend-http-ready 23.5s -> 3.86s and warm 5.0s -> 3.32s on the signed build, no 9Router regression - strip the [perf] bg entry logs from mcp/skill/settings/service/9router now that the stall is diagnosed; keep Apps.py per-lifespan timing but only print a lifespan over 50ms plus the total (cheap regression tripwire) - document the root cause + fix + before/after in docs/perf/winv2/README.md
winv2: Windows startup + App Builder speed and bug fixes
Branch: eric/winv2. Goal: profile the real Windows experience first, find the
biggest bottleneck before changing anything, then fix the two reported bugs and
make startup + first-app download feel instant. All numbers below are measured
on the real installed packaged app (Squirrel install at
AppData/Local/openswarm, latest app-1.2.82), Windows 11, not dev mode.
Notion tracking (Todos DB):
- [Perf] Windows startup + download speed: backend cold-start is the bottleneck
- [App Builder] Windows preview broken: no bundled bash/npm + missing node_modules archive
- [Bug] Skills list empty until reboots + onboarding "Install a skill" step times out
- [Reliability] Distributed-systems hardening (design)
How these numbers were measured
Source of truth: the packaged app's own perf markers in
AppData/Roaming/openswarm/data/backend.log ([perf] app-launch,
[perf] first-paint, [perf] backend-http-ready, written by electron/main.js).
These are wall-clock ms from process start, i.e. exactly what the user feels.
Raw extract: baseline_startup.csv. Re-run with profile_startup.sh.
Import cost measured with the bundled interpreter:
python-env/python.exe -X importtime -c "import backend.main".
Baseline (BEFORE any change)
Startup, per launch (ms)
| metric | warm (typical) | cold (first run after each update) |
|---|---|---|
| app-launch (electron ready) | 107-400 | 107-563 |
| first-paint (renderer) | 338-1205 | ~1200 |
| backend-http-ready | 8700-10500 | 54600 / 81000 / 86300 / 133000 / 138300 |
Electron shell paints in well under 1.5s every time. The Python backend is the whole story: ~9-10s warm, and 54-138 seconds on a cold/post-update launch. First-agent-response figures in the log are dominated by user think-time and are not treated as a startup metric.
Why the backend is slow (evidence)
| factor | measurement | effect |
|---|---|---|
| python-env file count | 13,554 files (4,510 .py/.pyd/.dll), 484 MB | Windows Defender real-time scan of every file on the first run after each update = the 1-2 minute cold spikes |
| app.asar size | 639 MB | cold disk read on first launch |
| backend.main import tree | ~2.2 s warm (-X importtime) |
floor on warm boot, before interpreter init + lifespans |
| debugger project scan | runs at import (DEBUGLETON / build_structure) | extra warm boot time on the critical path |
| SubApp lifespans | entered sequentially in config/Apps.py before HTTP bind |
serialized startup I/O |
Bottleneck ranking (before changes)
- Python backend cold-start (dominant). 9-10s warm, 54-138s cold. ~95% of perceived startup. Cold case driven by Defender scanning 13.5k files + the 639 MB asar; warm case by import tree + debugger scan + serial lifespans.
- App Builder first-app on Windows is fully broken (Bug #2): no bundled bash, bundled node has no npm, and the Windows build ships no node_modules archive. Confirmed against the installed binary. Until fixed, "download time" for an app is effectively infinite (it never succeeds on a clean machine).
- Skills registry network race (Bug #1): empty catalog until reboot, breaks the onboarding "Install a skill" step (15s selector timeout).
Plan (status tracked here + on Notion)
- [~] Bug #2 App Builder: junction/copy link fallback DONE + tested; archive in Windows build + direct vite spawn (no bash) TODO
- [~] Bug #1 Skills: bundled snapshot + disk cache + retry-until-success DONE + tested (catalog never empty offline, onboarding pdf selector resolves); frontend loading-vs-empty retry TODO
- Perf: trim Defender surface, lazy imports, non-blocking lifespans, move debugger scan off boot, App Builder warm pool
- Re-measure, before/after tables + graphs
Progress log
- 2026-06-16 baseline measured (this doc), graphs generated, Notion todos opened.
- 2026-06-16 Bug #1 backend:
skill_registry.pynow seeds from bundledskills_snapshot.json+ on-disk last-good cache and retries until first success. Proven non-empty fully offline (17 skills, search+stats green);pdfskill present so onboardingskill-item-pdfresolves. Regression testbackend/tests/test_skill_registry_seed.py(3 cases green). - 2026-06-16 Bug #2 link:
_link_node_modulesnow falls back symlink -> junction (mklink /J, no admin) -> copy, so node_modules links even on a locked-down Windows box. Tested with forced symlink failure.
Results (AFTER)
The warm-startup bottleneck was found and fixed
Per-SubApp-lifespan profiling (profile_boot.py) showed the entire ~8s gap was
one lifespan:
| boot phase | before | after | note |
|---|---|---|---|
| import backend.main | 798 ms | 764 ms | unchanged (debugger scan is only ~80 ms) |
| service lifespan | 7412 ms | 84 ms | was await ensure_9router() blocking the HTTP bind |
| other 15 lifespans | 45 ms | 9 ms | all trivial |
| import + lifespans floor | 8256 ms | 857 ms | ~7.4 s removed (~90%) |
Fix: service.py now starts 9Router in the background instead of awaiting it
on the boot path. 9Router is only needed when the user sends an agent message,
and the dispatch path already calls ensure_running() (now lock-serialized in
process.py so the background start and a dispatch-time ensure can't
double-spawn). Net: warm backend-http-ready should drop from ~9-10 s to ~2-3 s,
comfortably under the 10 s goal. See boot_breakdown.svg.
Still open (cold start)
The 54-138 s cold spikes are Windows Defender scanning the 13,554-file / 484 MB python-env on the first run after each update, plus cold-reading the 639 MB asar. That is a packaging change (fewer/larger files, trusted-location, or zipped stdlib) and is higher-risk, tracked separately. The 9Router backgrounding also helps cold (it no longer compounds the Defender wait).
App Builder first-app "download" + create path (measured)
Per-phase, measured on this Windows box (measure_appbuilder.py + measure_vite.py),
isolated temp dirs, real warm caches. See appbuilder_breakdown.svg.
| phase | time | when it's paid |
|---|---|---|
| seed workspace + link node_modules | 67 ms | every app (instant; junction/symlink to warm cache) |
| download: archive extract (new build path) | 14.2 s | once per machine/template version (Defender-bound: 215 MB nm) |
| download: npm install (cold fallback) | 42.7 s | once, only if no archive ships |
| vite bind: cold vite cache | 6.7 s | first app ever (esbuild pre-bundle) |
| vite bind: warm shared cache | 0.7 s | every subsequent app |
| build-time: tar nm -> archive | 6.8 s | on CI, never on the user's machine |
User-facing scenarios (create app -> live preview):
| scenario | total | notes |
|---|---|---|
| first app, clean Windows, BEFORE fix | never works | [WinError 2] / "backend exited with code 1" (no bash/npm/archive) |
| first app, AFTER fix (tar archive) | ~21 s one-time | extract 14.2 + seed 0.07 + vite cold 6.7; and it actually works |
| first app, AFTER fix (.tar.gz + background extract at startup) | ~7 s typical / ~21 s worst case | extract runs in the background at startup; if done before first create -> seed 0.07 + vite cold 6.7 ~= 7s, else +14.2s |
| first app, if we shipped npm instead | ~49 s | 42.7 + 6.7; the archive saves ~28 s and needs no npm |
| every subsequent app | ~0.8 s | seed 0.07 + vite warm 0.7 (near-instant) |
#9 item 2 (DONE): the Windows build now ships node_modules ALREADY EXTRACTED in
resources (digest-tagged); _ensure_warm_cache junctions a workspace straight at
it (_bundled_extracted_modules), so there is no tar-extract on first app -- the
14.2 s Defender-scanned write cost moves to install time, once. Verified by
backend/tests/test_bundled_extracted_modules.py (selection + Mac fallback) and
the build step build-app-win.ps1 4b now robocopies the tree into resources.
Takeaways: the archive (Bug #2 fix) turns a broken/∞ first-app into a working ~21s one-time, and ~0.8s for every app after. The remaining ~14s extract is the SAME Defender-on-many-small-files cost as cold app-startup (Task #9) -- the one lever that would shrink both.
Task #10 — VERIFIED on the real code-signed build (v1.3.86)
Downloaded the signed draft-release installer, verified signature, installed, and measured on this Windows 11 box. All numbers are from the packaged app, not dev. (Shipped as v1.3.86 on the fixed code; the earlier v1.3.87 build was identical bits and is abandoned. Numbers below are the v1.3.86 run; v1.3.87 matched.)
| metric | baseline (1.2.x) | signed v1.3.86 | result |
|---|---|---|---|
| installer download | n/a | 371.5 MB @ 27.1 MB/s (13.7 s) | signed: Authenticode Valid (CN=Eric Zeng) |
| install time | n/a | ~6.3 s | Squirrel |
| cold backend-http-ready | 54-138 s | 22.6 s | ~75-84% faster |
| warm backend-http-ready | 9-10 s | 5.0 s | ~50% faster, under the 10s goal |
| app.asar size | ~607 MB | 2.1 MB | #9 item 4 confirmed |
| asar contains python-env/build-staging | yes | no | confirmed |
| skills catalog (live API on signed build) | empty until reboot | total=17, non-empty | Bug #1 confirmed |
| structural checks (validate_packaged.ps1) | 4 fail | 5/5 PASS | snapshot + node tar + unpacked python-env |
Cold is 22.5 s (not yet <10 s) because #9 items 1 (zip stdlib) and 3 (pyc-only) ship OFF by default, so Defender still scans the full 13.5k-file python-env on the first post-update launch. Enabling those (next, build-gated) is the remaining cold lever. Bug #2 (App Builder) is verified structurally (node_modules .tar.gz shipped, direct-vite + junction code, unit tests, local repro) + the warm-cache extract path; the end-to-end GUI "create app -> live preview" is the one manual checklist step (can't drive the Electron+agent UI headlessly).
[DISPROVEN 2026-06-17] hypothesis: residual ~17s cold = swarm-debug DEBUGLETON scan
UPDATE: this hypothesis was WRONG. The
debug()->OPENSWARM_PACKAGED=1no-op shipped (commit3d6fe483) and was verified live on the signed build ("Scanning Project" count=0, scan confirmed gone), yet cold backend-http-ready stayed at 21.5s (no change). So the DEBUGLETON scan was NOT the cold driver. Kept the no-op anyway (it removes a real warm cost and is harmless), but the cold 16s is elsewhere. See the source-audit section below for what it actually is. The original (now-disproven) reasoning is preserved below for the record.
original (disproven) reasoning
No-coding investigation (import profile + the app's own timestamped logs) pinned it:
- The cold launch has a ~17s SILENT, synchronous event-loop block during startup (no async task ran). NOT import (1.1s), NOT Defender (proven twice), NOT file-count, NOT network (the updater succeeded in the window), and every SubApp lifespan is verified trivial (mkdir / early-return / yield).
- It is the swarm-debug DEBUGLETON: debug() -> Debugleton().find_file_info() (debug.py:20); the first call instantiates the singleton -> update_debug_toggles() -> Directory.build_structure() -> a recursive os.scandir() walk of the project tree (Directory.py:74). debug() is called on the startup critical path (config/Apps.py SubApp init + the lifespan loop), so the scan runs SYNCHRONOUSLY and blocks the HTTP bind. Cold (uncached fs) = ~17s; warm (cached) = ~80ms. The DEBUGLETON INIT log lines land exactly in the 17s gap.
- This also explains why items 1+3 (file count) and item 5 (Defender) did nothing: the cost is a synchronous scandir tree-walk, not AV scanning or bytecode.
SAFE FIX (proposed): make debug() a no-op when OPENSWARM_PACKAGED=1 (early-return before Debugleton() instantiates), so the scan never runs in the packaged build. Dev keeps the debugger. Risk very low (debug() is non-critical logging that already swallows errors). Expected cold ~22s -> ~5s (under the 10s goal). Confirm with a cold rebuild+measure.
#9 item 5 (Defender exclusion) measured: ALSO no cold benefit -> cold is NOT Defender
Applied the Defender exclusion (admin) for all 3 openswarm folders, rebuilt a fresh-content lean v1.3.86 (so Defender would see new files), installed with the exclusion active, measured cold:
| cold backend-http-ready | |
|---|---|
| no exclusion (items off) | 22.5 s |
| no exclusion (items 1+3 on) | 22.4 s |
| Defender exclusion ON | 21.4 s (no change) |
Conclusion: TWO independent Defender-targeting interventions (file-count via items 1+3, and a full AV exclusion) both moved cold by ~0. So the residual ~22 s cold is NOT Defender real-time scanning. It is the first-launch-after-install cost -- cold disk I/O of the imported native binaries + bundled-Python interpreter init
- Squirrel first-run -- which neither AV-exclusion nor file-count tricks touch. (Caveat: my non-admin shell can't read Get-MpPreference to re-confirm the exclusion is live, but the result is consistent with the items-1+3 negative.)
ACTION: remove the exclusion -- it weakened AV for zero gain:
& scripts\add-defender-exclusion.ps1 -Remove (elevated).
The cold win was already banked by the asar trim (54-138 s -> ~22 s). Pushing cold below ~22 s would need shrinking the startup-imported bytes (lazy-load heavy native deps like lxml/PIL, or trim the 242 MB bundled claude.exe) or faster disk -- bigger/riskier work with diminishing returns. Warm (5 s) is already under goal.
#9 items 1+3 measured on the signed build: NO cold-start benefit (negative result)
Built v1.3.86 with items 1+3 ON (python-env 13,554 -> 9,285 files, ~31% fewer) and measured the signed install:
| metric | items OFF (1.3.86) | items ON (1.3.86) |
|---|---|---|
| cold backend-http-ready | 22.5 s | 22.4 s (no change) |
| warm backend-http-ready | 5.0 s | 5.2 s (noise) |
| installer | 372 MB | 365 MB (~7 MB smaller) |
The hypothesis was wrong. Cutting the file COUNT 31% did nothing for cold,
because cold is dominated by Defender scanning the large NATIVE binaries imported
/ present at boot, not the many small .py files. The biggest are
claude.exe (242 MB!), _rust.pyd (9.4), _avif...pyd (7.5), python313.dll
(5.8), libcrypto-3-x64.dll (5.7), mfc140u.dll (5.4) -- none of which items 1+3
touch (zip/pyc only affect pure-python). So items 1+3 are a wash for cold (a tiny
installer-size win + import-clean, but not the goal).
The real remaining cold levers are byte/native-bound, not file-count:
- #9 item 5 (Defender exclusion, opt-in) -- the only thing that removes the native-binary scan entirely; would bring cold toward the ~5 s warm number.
- Trim/lazy the heavy native deps (e.g., the 242 MB bundled claude.exe, PIL/lxml) -- larger, riskier code/packaging work.
- Or accept cold 22.5 s: already 75-84% below the 54-138 s baseline, and warm 5 s is already under the 10 s goal.
Recommendation: items 1+3 don't earn their build-time/complexity for cold; keep them only for the marginal installer-size win, or revert step 2b to keep the build lean. The meaningful cold work is item 5.
Net time decreased per step (measured)
| step | before | after | saved |
|---|---|---|---|
| backend boot: service lifespan | 7412 ms | 84 ms | -7328 ms (-99%) |
| backend boot: import + all lifespans floor | 8256 ms | 857 ms | -7399 ms (-90%) |
| backend-http-ready warm (end-to-end) | ~9-10 s | ~2-3 s (projected) | ~-7 s |
| App Builder dependency download | 42.7 s npm | 14.2 s archive | -28.5 s (-67%) |
| App Builder first app -> preview | broken/never | ~21 s working | inf -> 21 s |
| App Builder subsequent app -> preview | n/a | ~0.8 s | near-instant |
| skills catalog availability | empty until reboot(s) | instant (seeded) | bug eliminated |
#9 packaging approach: shrink the Defender file surface (build-gated)
Defender real-time-scans every small file: python-env = 13,554 files; node_modules = ~tens of thousands; app.asar = 639 MB. It rescans python-env on the first launch after each update (54-138 s cold spikes) and scans node_modules as it is written (the 14.2 s extract). Fix family: fewer/larger files, scan-once-at-install instead of per-launch / per-first-app. Each item is independent, reversible, and must be validated on a real packaged EXE (Task #10).
- [ENABLED + validated] Zip the Python stdlib -> python313.zip. scripts/zip-python-stdlib.ps1, now wired into build-app-win.ps1 step 2b. VALIDATED 2026-06-17: applied to a copy of the real shipped python-env and
import backend.main(full app + deps) imported cleanly; combined with #3 the python-env drops 13,554 -> 9,287 files (~31%). Draft notes: Measured on the real env: 910 stdlib .py/.pyc files (15.1 MB) collapse into one zip. CPython auto-adds /python313.zip to sys.path, so no python._pth is needed; site-packages + DLLs (native .pyd) stay loose; a keep-list keeps data-file stdlib dirs (lib2to3, idlelib, tkinter, ...) loose. Impact: ~7% of total python-env file count, but it collapses the stdlib import-time file-opens (the cold-launch Defender scan storm) into a single scanned file; bigger combined with #3. Validation (Task #10): -Apply on a copy, then import backend.main, importtime parity, boot the packaged backend, measure cold backend-http-ready vs baseline. Wire into build-app-win.ps1 behind an off-by-default -ZipStdlib switch only after it passes. - [DONE] Ship the webapp_template node_modules archive in the Windows build (build-app-win.ps1 step 4b builds node_modules..tar.gz, mirroring the Mac build). Runtime _try_extract_bundled_archive unpacks it into the warm cache, kicked off in the BACKGROUND by warm_cache_in_background at startup so it is off the first-app create path. CORRECTION 2026-06-17: an earlier draft shipped node_modules PRE-EXTRACTED in resources (~30k files) -- that blew the Windows build past 50 min (Squirrel LZMA on tens of thousands of tiny files) and bloated the installer, so it was reverted to the single .tar.gz. The runtime keeps _bundled_extracted_modules() as a harmless preference (returns None when no tree is shipped -> falls back to the tar). Tests: test_bundled_extracted_modules.py still valid (selection + fallback).
- [ENABLED + validated] Ship site-packages as sourceless .pyc only (drop .py). scripts/strip-py-to-pyc.ps1, now wired into build-app-win.ps1 step 2b. VALIDATED 2026-06-17 alongside #1 (backend.main imports clean from a transformed copy; 3,352 .py removed). Draft notes: Measured: 3,352 .py (26.9 MB) + 362 pycache dirs strippable from site-packages (keep-list excludes pip/setuptools). compileall -b writes legacy module.pyc next to source; we delete the .py whose .pyc exists and drop pycache. Sourceless import proven with the bundled 3.13 interpreter. Scope: site-packages ONLY (NOT backend app code -- the swarm-debug debugger reads our own source for frame annotation). .pyc magic must match the shipped interpreter, so compile with the bundled python. Validate on a packaged EXE (Task #10); some packages use inspect.getsource and may need the keep-list. Combined with #1 + #2 this takes python-env from ~13,554 files toward ~9,300 (~31% fewer for Defender).
- [APPLIED, build-gated] Trim app.asar. Inventory (docs/perf/winv2/inspect_asar.js) found the 607 MB asar is almost entirely DUPLICATION: python-env (408 MB, incl. a 242 MB bundled claude.exe) and build-staging (197 MB: node.exe 67 MB, uv.exe 65 MB, mcp-bundles, frontend) are packed into the asar AND already shipped UNPACKED in resources/ via extraResources. The runtime reads from resources/ (confirmed: "Starting backend: ...resources\python-env\python.exe"), never from inside the asar. Source maps were a red herring (0.4 MB). Fix: added a build.files exclusion in electron/package.json ("!python-env/", "!build-staging/") so those trees no longer pack into the asar -> ~607 MB -> ~2 MB (just main.js/preload/node_modules). Removes the entire 639 MB cold-read on first launch. Validate on a packaged EXE (Task #10): app still boots (python/node/router resolved from resources), asar size shrunk.
- [DRAFTED, opt-in] Defender exclusion for OpenSwarm's dirs -- the nuclear cold-start fix (stops real-time scanning entirely, so it kills BOTH the 54-138s post-update launch and the ~14s extract). Draft: scripts/add-defender-exclusion.ps1 (dry-run by default; -Apply/-Remove need admin; -Status lists). Excludes %LOCALAPPDATA%\openswarm, %APPDATA%\openswarm, ~/.openswarm (verified the paths resolve). SECURITY: reduces AV coverage of those folders, so it must ALWAYS be an explicit user choice -- never auto-run, never a startup prompt. Proposed surface: an OFF-by-default Settings > Advanced toggle ("Faster Windows startup -- adds a Defender exclusion for OpenSwarm; one-time admin approval; reversible"), which on enable spawns an elevated
powershell Start-Process -Verb RunAsto run the script -Apply (UAC), and -Remove on disable. This is a passive opt-in toggle, NOT a banner/tip/prompt, so it respects the no-user-action-UI rule. Not wired into the frontend yet (design only).
Recommended order: #2 (biggest UX win, lowest risk), then #1 (largest cold win, careful import testing), then #3/#4. Validation: re-run profile_startup.sh + a fresh-extract timing on the packaged EXE after each change, diff vs baseline_startup.csv.
Bug fixes (this branch)
- Bug #1 skills: seed from bundled snapshot + disk cache + retry-until-success. Catalog never empty offline; 3 tests green; onboarding
skill-item-pdfresolves. - Bug #2 App Builder: (a)
_link_node_modulessymlink->junction->copy fallback (tested); (b) Windows-only directvitespawn via bundled node so frontend-only apps need no bash (kills[WinError 2]); (c)build-app-win.ps1now pre-builds the node_modules archive natively. Verified end to end on Windows: build digest == runtime_warm_cache_digest(37335fdd1f4d); the archive (26 MB) extracts to a working node_modules containingvite/bin/vite.jsand the Windows-native@esbuild/win32-x64/esbuild.exe.
Residual cold ~16s: full source audit + boot instrumentation (2026-06-17)
After FOUR disproven cold hypotheses (file-count via items 1+3, Defender exclusion, DEBUGLETON scan, and the asar trim which DID bank 138s->22s), I stopped guessing and read the real signed-build cold log line by line, then audited every lifespan in source.
What the cold log (commit 3d6fe483, scan-free build) actually shows:
03:03:02 skill_registry: seeded 17 skills <- last backend log before the gap
... 16 seconds, NO backend log line ...
03:03:18 nine_router: Starting 9Router <- a backgrounded create_task finally runs
03:03:18 Application startup complete <- uvicorn; all lifespans entered
03:03:21 9Router started; GET /api/health 200; backend-http-ready t=21519
SubApp lifespan order (backend/main.py:52):
health, agents, skills, tools_lib, modes, settings, mcp_registry, skill_registry, outputs, dashboards, swarm, service, subscription, auth, web, anthropic_proxy.
Source-audited EVERY one of the 16 lifespan bodies + the service client:
- outputs: two
os.makedirs+ yield (trivial) - dashboards:
_migrate_if_needed()early-returns when dashboards exist (they persist across reinstall in AppData, so it's a no-op on the cold post-update launch) - swarm:
_gc_staging()over an empty dict + yield (trivial) - service: builds a provider list +
svc.sync()x2, thencreate_task(ensure_9router),create_task(_pulse_loop),create_task(_drain_loop), yield.svc.sync()is genuinely fire-and-forget:client.py:sync()->_schedule()->loop.create_task(_post_or_spool); the actual httpx POST has a 5s timeout and runs in the task, never on the boot path. - skill_registry: seed from disk +
create_task(_refresh_loop)+ yield (trivial) - subscription / auth / anthropic_proxy: bare
yield - web:
debug("START")(no-op packaged) + yield
So NO lifespan body blocks. This matches profile_boot.py warm (import + all 16
lifespans = 617ms, no lifespan over 95ms). The 16s is therefore NOT in our Python
startup logic; it is cold first-run demand-paging of native bytes (interpreter .pyc
cold reads, native .pyd / .dll first-touch, the 9Router/claude.exe binaries) that
the OS pages in during this window. That class of cost is exactly what the asar
trim already cut and what Defender-exclusion / file-count provably cannot move.
THE missing instrument: debug(sub_app.name) (Apps.py:33) is a no-op in the
packaged build, so the packaged log had ZERO per-lifespan markers, which is why
four hypotheses were guesses. Added permanent per-lifespan boot timing in
backend/config/Apps.py (one time.perf_counter() + flushed print per app, plus
a lifespans-total): [perf] lifespan <name> t=<ms>ms. Logging only, zero
functional risk; validated warm (correctly attributes a simulated 300ms blocker to
the one slow lifespan, others 0ms). On the NEXT cold packaged launch this pins the
16s to a single lifespan (=> a real fix) or shows it smeared across many (=> confirms
distributed cold I/O => accept 22s; warm 5s already meets the <10s goal).
Status: warm 5.0s (under goal), cold ~22s (75-84% below the 54-138s baseline), both bugs fixed/verified on the signed build. The cold residual is either accepted as first-run-only OS I/O, or pinned definitively by one more build that ships this instrumentation. Build-gated (user manages tags/release), so not auto-built.
[SOLVED 2026-06-18] cold ~22s -> 3.86s: synchronous is_running() froze the event loop
The per-lifespan instrumentation (v1.3.88) overturned every prior hypothesis: all 16 lifespans enter in ~120ms even COLD. The ~18s cold cost was entirely AFTER lifespan startup, in a backgrounded create_task that synchronously blocked the single asyncio event loop, so uvicorn could not answer the health probe.
Finer instrumentation (v1.3.89) split it into two stalls (~13s before any bg task,
~5s in 9Router ensure). faulthandler (dump_traceback_later, v1.3.90) on the
signed cold build caught the loop thread frozen, three times, in the SAME call:
socket.create_connection <- stuck >7s
httpx ... get
backend/apps/nine_router/process.py:83 is_running() <- synchronous httpx.get
<- sync_openswarm_pro_as_claude / sync_custom_providers (settings._boot_router_then_sync)
<- _ensure_running_impl (ensure_running)
ROOT CAUSE: is_running() did a synchronous httpx.get("http://localhost:20128/...").
It is called ~5x on the cold boot path (the settings key-sync sequence + the
9Router ensure) BEFORE 9Router is up. On Windows a dead-port connect to
"localhost" stalls ~7s each: getaddrinfo returns ::1 first, and the loopback
refusal is slow (measured: a refused connect is ~2s/address, and localhost =
::1+127.0.0.1 = ~4s; cold ~7s). ~5 serial probes = the ~18s freeze.
This is why every earlier hypothesis missed: it is not disk, not Defender, not file-count, not the DEBUGLETON scan, not imports, not the lifespans. It is one synchronous network probe on the event loop, repeated.
FIX (v1.3.91, process.py is_running): probe 127.0.0.1 with a 0.3s TCP timeout
first (a short timeout caps the slow Windows refusal: measured 306ms vs ~7s); only
HTTP-confirm when the port is open. 9Router binds 0.0.0.0 (the warm app reaches
it via 127.0.0.1), so reachability is unchanged, only the dead-port wait dies.
VERIFIED on the real signed build (this Windows 11 box, fresh Squirrel install):
| metric | baseline | before fix (1.3.90) | after fix (1.3.91) |
|---|---|---|---|
| cold backend-http-ready | 54-138s | 23.5s | 3.86s |
| warm backend-http-ready | 9-10s | 5.0s | 3.32s |
Cold is now ~97% below baseline and well under the 10s goal; warm improved too
(the same localhost stall taxed it). 9Router still starts successfully via the new
probe (no regression). The diagnostic [perf] bg logs + faulthandler were removed
after diagnosis; the lightweight per-lifespan timer stays (prints only a lifespan
over 50ms + the total) as a cheap regression tripwire.