Stage tasks hold a slot for their whole tool run, so step hand-overs and
finalize queued behind them and background jobs took scan slots.
- worker-scans consumes scans (run_scan_stage only), scalable with --scale
- worker-control consumes scan_control: launch, hand-over, resume, finalize
- worker-default consumes default and critical
- reaper reads both scan queues; readiness requires a consumer on every queue
- connection budget counts every worker; max_connections 250
Repo-wide sweep of every user-facing string: frontend components, routes,
config and stores; API details and route docs; shared definitions and
models; stage config help; report prose; MCP, toolbox, interest and
connector text; notification and worker messages.
Descriptions that explained, reassured, justified or restated their title
are deleted. Confirm dialogs name what is removed and stop. Failed actions
read "<Noun> not <verbed>", successes "<Noun> <verbed>". No reader address,
no product-as-subject, no em-dashes or parentheticals in prose.
Mirrors updated on both sides: EPSS band help, compare refusal reasons,
interest labels. Four tests asserting on old wording follow the new text.
The rules are condensed into CLAUDE.md under Writing.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SK5o8xR96cbSnAfjYSTPSe
Scan page froze on a duplicate each key; every tab body now sits in a
svelte:boundary. Composite discovered_at indexes on the five result
tables, leads counted in one FILTER pass, dashboard and scan-service
first-seen counts bounded to the scans in the window, ANALYZE at
finalize, shm_size on the db service, SQL_ECHO instead of DEBUG, an
entrypoint that serves with workers outside reload.
Settled-scope aggregates and search pages are cached by target
revision; triage, dismissal, interest evaluation, notes, certificate
rechecks and re-ranking bump it. Requests per page cut: single-flight
GETs, one schemas endpoint, batched whois correlations, a counts
endpoint for the dashboard, deduped store loads, trimmed coverage
payloads.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NWRq8utu7KXsVBVU3gYNi7
The comment rule in CLAUDE.md is one line maximum, never narration: no
multi-line blocks, no module-header explainers, no sentences that explain or
justify the code to the reader. The tree had drifted a long way from it, and
most of the drift is mine.
Removed:
232 lines of multi-line comment blocks
203 single-line comments that argued instead of stating
154 comments joined by so/because/otherwise/rather-than
143 narrating comments in the frontend
112 test docstrings (the test name already says what it asserts)
25 test module docstrings and 25 decorative section dividers
Collapsed:
324 multi-line docstrings to one factual line
271 one-line docstrings that had a justifying clause bolted on
Kept: functional directives, section dividers, short field labels, alembic
revision headers, and the eight FastAPI route docstrings that render in /docs.
Where a fact was load-bearing it is in CLAUDE.md, which is where it belongs.
284 python tests and 34 frontend tests pass, ruff and ruff-format clean across
ten packages, svelte-check 0 errors, prettier and eslint clean, app.main imports.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013dKUV96WkQQETZBMoAG98C
There was no backup path at all, and the connection arithmetic did not add up.
scripts/backup.sh dumps the four things an instance actually is: the database, the
scan media on disk, the docker volumes holding uploaded wordlists, nuclei templates,
report themes and fonts, and the settings a restore needs. SECRET_KEY travels with
it deliberately — API keys, proxy credentials and notification secrets are Fernet
encrypted with a key derived from it, so an archive without it restores a database
full of unreadable secrets. The header says so, and backups/ and *.tar are
gitignored, because an archive is a credential.
Restore is the half that is usually never tried, so it was: a 305 MB archive of this
instance was restored over it and came back at 9 scans, 59,858 subdomains, 535,000
endpoints, 29 findings and 7 targets — identical — with the migration head intact,
the API answering and an encrypted API key still decrypting. It stops the api and
both workers first so nothing writes underneath it, and asks for the database name
before it destroys anything.
The pools could ask for more connections than the server would give. The api takes
10 + 20, each celery prefork child takes 2 + 3, and beat takes another pool: at the
concurrency of 8 this box runs that is 75 against a max_connections of 100, and at
the 12 the compose file now defaults to it is 95 — five short of the ceiling, which
fails as a checkout timeout under load and reads like a slow query.
max_connections is now set from compose with the arithmetic beside it, and the api
checks it at startup: pool_demand() against max_connections less the superuser
reserve, warned once rather than discovered during an incident. The inputs moved to
shared settings because the sum spans both processes and neither could see the
other's numbers.
Worth recording: .env on this box pins CELERY_SCAN_CONCURRENCY=8, so raising the
compose default to 12 changes new installs and not this one. It also still carries
CELERY_DEFAULT_CONCURRENCY, which nothing has read since it was renamed.
Three tests: the demand counts every pool, the shipped defaults fit the shipped
server, and an oversized pool warns rather than failing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013dKUV96WkQQETZBMoAG98C
The SSE route authenticated through CurrentUser, whose session dependency is
only released when the response ends — so every open stream left one connection
idle in transaction for as long as the tab stayed open. The engine was created
with no pool arguments at all, so the ceiling was SQLAlchemy's default 5+10:
fifteen tabs and every DB endpoint blocked for pool_timeout and then 500'd.
Reproduced exactly, and both pinned backends' last query was get_current_user's
SELECT users.
StreamUser opens and closes its own short session, so a stream holds nothing
pooled — 200 concurrent streams now report checked_out 1.
The pool is configured rather than defaulted (10+20, pre-ping, 30m recycle,
fail-fast timeout) and carries an application_name, because a blank one is why
this needed a live reproduction to diagnose. idle_in_transaction_session_timeout
is the backstop: any future leak from any path is reclaimed by Postgres instead
of accumulating, and pre-ping makes the next checkout succeed on a fresh socket.
A stream also no longer outlives its token. It re-checks expiry and revocation
on each heartbeat — JWT plus Redis, no DB — and emits `event: unauthorized`, so
logout actually ends the feed and the client refreshes at once instead of
burning three failed reconnects.
Worker pools are sized per prefork child (8 x 15 was over max_connections) and
the inherited engine is disposed on fork so a child never shares a parent socket.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015biqVXTng75nnfKXMqm61H
- Implemented TargetEnrichmentStage for enriching targets with DNS, WHOIS, and BGP context.
- Added VhostStage for virtual host discovery using host-header fuzzing.
- Introduced WafDetectStage for fingerprinting web application firewalls.
- Created configuration files for each new stage with relevant parameters.
- Updated Dockerfile to copy stages instead of engines.
- Enhanced existing tools and services for better integration with new stages.
- Refactored task management in Celery to accommodate new stages and improve error handling.
Yogesh Ojha