diff --git a/docs/perf/winv2/README.md b/docs/perf/winv2/README.md index 38b39663..40a3f702 100644 --- a/docs/perf/winv2/README.md +++ b/docs/perf/winv2/README.md @@ -122,7 +122,7 @@ isolated temp dirs, real warm caches. See `appbuilder_breakdown.svg`. | --- | --- | --- | | first app, clean Windows, BEFORE fix | never works | `[WinError 2]` / "backend exited with code 1" (no bash/npm/archive) | | first app, AFTER fix (tar archive) | ~21 s one-time | extract 14.2 + seed 0.07 + vite cold 6.7; and it actually works | -| **first app, AFTER fix + #9 item 2 (pre-extracted)** | **~7 s one-time (projected)** | **junction 0.07 + vite cold 6.7; the 14.2 s extract is gone** | +| **first app, AFTER fix (.tar.gz + background extract at startup)** | **~7 s typical / ~21 s worst case** | extract runs in the background at startup; if done before first create -> seed 0.07 + vite cold 6.7 ~= 7s, else +14.2s | | first app, if we shipped npm instead | ~49 s | 42.7 + 6.7; the archive saves ~28 s and needs no npm | | every subsequent app | ~0.8 s | seed 0.07 + vite warm 0.7 (near-instant) | @@ -160,7 +160,7 @@ of per-launch / per-first-app. Each item is independent, reversible, and must be validated on a real packaged EXE (Task #10). 1. [DRAFTED, build-gated] Zip the Python stdlib -> python313.zip (medium risk). Draft: scripts/zip-python-stdlib.ps1 (dry-run by default; NOT wired into the release build yet). Measured on the real env: 910 stdlib .py/.pyc files (15.1 MB) collapse into one zip. CPython auto-adds /python313.zip to sys.path, so no python._pth is needed; site-packages + DLLs (native .pyd) stay loose; a keep-list keeps data-file stdlib dirs (lib2to3, idlelib, tkinter, ...) loose. Impact: ~7% of total python-env file count, but it collapses the stdlib import-time file-opens (the cold-launch Defender scan storm) into a single scanned file; bigger combined with #3. Validation (Task #10): -Apply on a copy, then import backend.main, importtime parity, boot the packaged backend, measure cold backend-http-ready vs baseline. Wire into build-app-win.ps1 behind an off-by-default -ZipStdlib switch only after it passes. -2. [DONE] Ship webapp_template node_modules PRE-EXTRACTED in resources + junction to it (kills the 14.2 s extract -> ~0 s). build-app-win.ps1 step 4b robocopies the tree into resources; runtime _bundled_extracted_modules()/_ensure_warm_cache() prefer it; tests in test_bundled_extracted_modules.py. Mac still ships the .tar.gz (unchanged). +2. [DONE] Ship the webapp_template node_modules archive in the Windows build (build-app-win.ps1 step 4b builds node_modules..tar.gz, mirroring the Mac build). Runtime _try_extract_bundled_archive unpacks it into the warm cache, kicked off in the BACKGROUND by warm_cache_in_background at startup so it is off the first-app create path. CORRECTION 2026-06-17: an earlier draft shipped node_modules PRE-EXTRACTED in resources (~30k files) -- that blew the Windows build past 50 min (Squirrel LZMA on tens of thousands of tiny files) and bloated the installer, so it was reverted to the single .tar.gz. The runtime keeps _bundled_extracted_modules() as a harmless preference (returns None when no tree is shipped -> falls back to the tar). Tests: test_bundled_extracted_modules.py still valid (selection + fallback). 3. [DRAFTED, build-gated] Ship site-packages as sourceless .pyc only (drop .py). Draft: scripts/strip-py-to-pyc.ps1 (dry-run default; NOT wired into the build). Measured: 3,352 .py (26.9 MB) + 362 __pycache__ dirs strippable from site-packages (keep-list excludes pip/setuptools). compileall -b writes legacy module.pyc next to source; we delete the .py whose .pyc exists and drop __pycache__. Sourceless import proven with the bundled 3.13 interpreter. Scope: site-packages ONLY (NOT backend app code -- the swarm-debug debugger reads our own source for frame annotation). .pyc magic must match the shipped interpreter, so compile with the bundled python. Validate on a packaged EXE (Task #10); some packages use inspect.getsource and may need the keep-list. Combined with #1 + #2 this takes python-env from ~13,554 files toward ~9,300 (~31% fewer for Defender). 4. [APPLIED, build-gated] Trim app.asar. Inventory (docs/perf/winv2/inspect_asar.js) found the 607 MB asar is almost entirely DUPLICATION: python-env (408 MB, incl. a 242 MB bundled claude.exe) and build-staging (197 MB: node.exe 67 MB, uv.exe 65 MB, mcp-bundles, frontend) are packed into the asar AND already shipped UNPACKED in resources/ via extraResources. The runtime reads from resources/ (confirmed: "Starting backend: ...resources\python-env\python.exe"), never from inside the asar. Source maps were a red herring (0.4 MB). Fix: added a build.files exclusion in electron/package.json ("!python-env/**", "!build-staging/**") so those trees no longer pack into the asar -> ~607 MB -> ~2 MB (just main.js/preload/node_modules). Removes the entire 639 MB cold-read on first launch. Validate on a packaged EXE (Task #10): app still boots (python/node/router resolved from resources), asar size shrunk. 5. [DRAFTED, opt-in] Defender exclusion for OpenSwarm's dirs -- the nuclear cold-start fix (stops real-time scanning entirely, so it kills BOTH the 54-138s post-update launch and the ~14s extract). Draft: scripts/add-defender-exclusion.ps1 (dry-run by default; -Apply/-Remove need admin; -Status lists). Excludes %LOCALAPPDATA%\openswarm, %APPDATA%\openswarm, ~/.openswarm (verified the paths resolve). SECURITY: reduces AV coverage of those folders, so it must ALWAYS be an explicit user choice -- never auto-run, never a startup prompt. Proposed surface: an OFF-by-default Settings > Advanced toggle ("Faster Windows startup -- adds a Defender exclusion for OpenSwarm; one-time admin approval; reversible"), which on enable spawns an elevated `powershell Start-Process -Verb RunAs` to run the script -Apply (UAC), and -Remove on disable. This is a passive opt-in toggle, NOT a banner/tip/prompt, so it respects the no-user-action-UI rule. Not wired into the frontend yet (design only). diff --git a/scripts/build-app-win.ps1 b/scripts/build-app-win.ps1 index 867ede0b..71298d43 100644 --- a/scripts/build-app-win.ps1 +++ b/scripts/build-app-win.ps1 @@ -364,25 +364,28 @@ if (Test-Path $EnvExampleSrc) { Write-Host "Restored webapp_template/.env.example (stripped by the .env.* exclude)" } -# --- Step 4b: Pre-EXTRACT the webapp-template node_modules into resources (Windows). +# --- Step 4b: Pre-build the webapp-template node_modules archive (.tar.gz). # The Windows build never shipped any node_modules, and the bundled node has no # npm, so the App Builder frontend had no way to get its deps; the preview died -# with the misleading "backend exited with code 1". We ship the tree ALREADY -# EXTRACTED (digest-tagged) so the runtime junctions a workspace straight at it: -# zero first-app extract (the .tar.gz path cost ~14s of Defender-scanned writes -# on first app; #9). _ensure_warm_cache() / _bundled_extracted_modules() pick -# this up; the Mac build still ships the .tar.gz and uses the extract path. -# Built natively so the esbuild/rollup win32 binaries are correct (a Mac-built -# tree would ship darwin binaries and still fail). Non-fatal: a failure warns -# but doesn't break the build. Digest == _warm_cache_digest() (sha256 of -# frontend/package.json, first 12 hex chars). -Write-Host "[4b] Pre-extracting webapp-template node_modules into resources..." +# with the misleading "backend exited with code 1". We ship a single compressed +# archive (mirrors the Mac build's step 3c); the runtime's _try_extract_bundled_archive +# unpacks it into the warm cache (kicked off in the background by +# warm_cache_in_background at startup, so it is off the first-app create path). +# NOTE: we deliberately do NOT ship node_modules pre-extracted into resources -- +# that adds ~30k tiny files which made electron-builder/Squirrel LZMA compression +# blow the build past 50 min and bloats the installer. One .tar.gz (~26 MB) keeps +# the build fast and the installer small. Built natively so the esbuild/rollup +# win32 binaries are correct. Non-fatal: a failure warns but does not break the +# build. Digest == _warm_cache_digest() (sha256 of frontend/package.json, 12 hex). +Write-Host "[4b] Pre-building webapp-template node_modules archive (.tar.gz)..." try { $TmplFrontend = Join-Path $Staging 'backend\apps\outputs\webapp_template\frontend' $PkgJson = Join-Path $TmplFrontend 'package.json' if (-not (Test-Path $PkgJson)) { throw "template package.json not found at $PkgJson" } $Digest = (Get-FileHash -Algorithm SHA256 $PkgJson).Hash.ToLower().Substring(0, 12) - $DestNm = Join-Path $Staging "backend\apps\outputs\webapp_template_cache\$Digest\node_modules" + $CacheDir = Join-Path $Staging 'backend\apps\outputs\webapp_template_cache' + New-Item -ItemType Directory -Force -Path $CacheDir | Out-Null + $OutArchive = Join-Path $CacheDir "node_modules.$Digest.tar.gz" $WorkDir = Join-Path $env:TEMP "os-tmpl-nm-$([guid]::NewGuid())" New-Item -ItemType Directory -Force -Path $WorkDir | Out-Null try { @@ -396,22 +399,19 @@ try { & npm install --prefer-offline --no-audit --no-fund --loglevel=error } if ($LASTEXITCODE -ne 0) { throw "npm install/ci failed ($LASTEXITCODE)" } + if (-not (Test-Path (Join-Path $WorkDir 'node_modules'))) { throw "no node_modules produced" } + # tar.exe (bsdtar) ships with Windows 10+; archive root is node_modules/. + & tar -czf $OutArchive -C $WorkDir node_modules + if ($LASTEXITCODE -ne 0) { throw "tar failed ($LASTEXITCODE)" } Pop-Location - $SrcNm = Join-Path $WorkDir 'node_modules' - if (-not (Test-Path $SrcNm)) { throw "no node_modules produced" } - New-Item -ItemType Directory -Force -Path $DestNm | Out-Null - # robocopy: fast, multi-threaded, handles the deep node_modules tree + long paths. - & robocopy $SrcNm $DestNm /E /NJH /NJS /NDL /NFL /NP /MT:8 | Out-Null - if ($LASTEXITCODE -ge 8) { throw "robocopy node_modules failed ($LASTEXITCODE)" } - $global:LASTEXITCODE = 0 - $Count = (Get-ChildItem -Recurse -File $DestNm -ErrorAction SilentlyContinue | Measure-Object).Count - Write-Host "[4b] pre-extracted node_modules staged at webapp_template_cache\$Digest ($Count files)" + $ArchMB = (Get-Item $OutArchive).Length / 1MB + Write-Host ("[4b] webapp-template archive staged: node_modules.$Digest.tar.gz ({0:N1} MB)" -f $ArchMB) } finally { if ((Get-Location).Path -eq $WorkDir) { Pop-Location } if (Test-Path $WorkDir) { Remove-Item -Recurse -Force $WorkDir } } } catch { - Write-Warning "[4b] pre-extract node_modules FAILED: $_ (App Builder first-app falls back to live npm; non-fatal)" + Write-Warning "[4b] webapp-template archive build FAILED: $_ (App Builder first-app falls back to live npm; non-fatal)" } # data: backend/config/paths.py points DATA_ROOT at %APPDATA%/OpenSwarm/data in # packaged mode and no code seeds from the bundle, so the entire shipped