Every stage config field is a decision; rate, concurrency, timeout and
retries come from a per-tool intensity profile the stage reads as its
transport. 173 fields become 83: 41 transport values, 40 constants and
the dead ones leave the models, and the builder groups by capability
with support stages as checkboxes and advanced fields behind one
disclosure. Aggressive now changes what a tool is handed.
Every project carries a built-in Default engine, seeded on project
creation and on first list, undeletable: stage defaults, subfinder as
the only passive source, vulnerability scan on with nuclei.
From the tracing review: tool args can no longer restate a flag the
stage set, the session check obeys the probe scheme, excluded scope
reaches every stage that reads addresses, hosts or paths, the sitemap
and source-map providers throttle at the transport rate, the blind tier
obeys its switch, the fuzzing lanes split the budget, and the removed
private-address and evidence switches are back as advanced fields.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Pq4xYJbgW5euUa9sCWBKFw
A full static review of the 3.0 tree, run as two fan-out passes with every
finding verified against the running stack rather than taken on inspection.
Correctness and security
- instance settings encrypted the mask over the stored AI key: `!= MASK` never
matched `••••••••abcd`, so saving settings destroyed the key and `test_ai`
posted the mask as a bearer token
- the MFA token was replayable for its full five-minute life; the gate now
checks revocation and consumes the token after a successful exchange
- logout revoked nothing for a bearer client while answering "Logged out"
- `POST /api-keys/{id}/test` returned the plaintext key inside httpx's error
string, which embeds the whole URL
- an interest rule could be edited or deleted through another project's id
- redis ran the celery broker under `allkeys-lru`, free to evict a queued task
or a chord counter; now `noeviction`
- nuclei's stored command carried the OAST token and every auth header
- a finalize step could poison the session and leave a scan RUNNING for ever
- `is:issue` raised TypeError on every search
- offset paging repeated and skipped rows in four dimensions
- a cancelled run kept a stale `scans.*_found` rollup
- every PDF report raised `KeyError: 'software'`
- `version_key` truncated a wide numeric run and inverted release ordering
- psycopg raises a NUL as a bare ValueError, above StatementError, so every
savepoint guard was blind to the case it was written for
Performance
- endpoints.host had no trigram index: 35.9ms seq scan -> 0.389ms
- free text compiled to one OR, not the prescribed UNION ALL: 132ms -> 41ms
- `/http-assets/summary` and `/ips/summary` materialised whole tables
- the cross-link `ip` kind ran an estate query per page and drew nothing
Frontend
- `$effect` loaders that read the state they write, an unbounded request loop
on a failing endpoint
- error states that rendered as empty states
- 2FA backup codes were unusable while the copy said they worked
- a "Scan history" dialog that always reported none
- ~400 duplicated lines across five filter bars, now one component
Build and tooling
- every scanner installed at @latest; all 13 pinned to the measured versions
- CI built `web/`, gone since 2.x, with tags that cannot be valid
- the prettier pre-commit hook discarded its filenames and always passed
- the test suite destroyed itself when run twice
Also: subdomains from public code, via a pinned github-subdomains source and a
GitHub API key that subfinder receives too; sublist3r and oneforall removed
after measuring one unique host in 89 across four targets.
Verified: 853 tests, ruff clean over eleven packages, svelte-check 0 errors,
the count-is-a-promise gate at 131 checks and 0 mismatches, and 143 GET routes
with no 5xx. Four migrations, each with an exercised downgrade.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DVkdx5ynZ6aof5kLFczmhX
global_threads was multiplied by the scan context's thread multiplier and
then discarded; every stage carries its own threads field with its own
default, so the engine-level number had nothing to apply to.
global_http_crawl was stored, exported, imported and never read at all.
Both reached the engine form, the YAML, the launch preview and the run
comparison, where "Crawl responses" was a material facet: two runs that
differed on it were reported as able to find different things.
Gone from the table, the three schemas, ResolvedScanConfig, AdHocEngine,
the preview, both Compare facets and every frontend site, with a
migration to drop the columns. An engine exported before this still
imports: _RETIRED_KEYS accepts the two keys and ignores them, while an
unknown key is refused as before.
_MAX_ENGINE_THREADS goes with the validator that used it. It was declared
twice in the same file.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01X5roHYDYzSPRNaTkn8hSkJ
The comment rule in CLAUDE.md is one line maximum, never narration: no
multi-line blocks, no module-header explainers, no sentences that explain or
justify the code to the reader. The tree had drifted a long way from it, and
most of the drift is mine.
Removed:
232 lines of multi-line comment blocks
203 single-line comments that argued instead of stating
154 comments joined by so/because/otherwise/rather-than
143 narrating comments in the frontend
112 test docstrings (the test name already says what it asserts)
25 test module docstrings and 25 decorative section dividers
Collapsed:
324 multi-line docstrings to one factual line
271 one-line docstrings that had a justifying clause bolted on
Kept: functional directives, section dividers, short field labels, alembic
revision headers, and the eight FastAPI route docstrings that render in /docs.
Where a fact was load-bearing it is in CLAUDE.md, which is where it belongs.
284 python tests and 34 frontend tests pass, ruff and ruff-format clean across
ten packages, svelte-check 0 errors, prettier and eslint clean, app.main imports.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013dKUV96WkQQETZBMoAG98C
SQLModel emitted index=True beside primary_key=True on every id field, so 41
tables carried ix_<table>_id alongside <table>_pkey: the same single-column btree
twice. 12 MB of duplicate index, and every insert maintaining both — on the hot
path of a scan writing 345,152 endpoints.
The planner had been using them (72,160 scans across the set) simply because
they were there and equivalent; with them gone it uses the primary key, verified
on subdomains: Index Scan using subdomains_pkey.
The migration finds them rather than naming them, because which tables exist
depends on how far an install has migrated and a name already gone must not fail
an upgrade. It drops only a non-unique, single-column btree on id belonging to a
table that has a primary key — no other index can match that shape. There is no
downgrade: the primary key serves every lookup they served.
The 33 models are corrected too, so the next hand-written migration is not
comparing against a schema that would want them back.
Counted honestly: 41, not the 102 an earlier pass reported — that number included
the 61 primary keys themselves.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013dKUV96WkQQETZBMoAG98C
a5d835df made ip_enrichment catalog_hidden. That removed it from the
builder entirely, which is worse: a stage that always runs and that
every address surface reads should not be invisible. An operator has
to know it exists.
It is now `always_on` — served in the catalog with the flag, rendered
as a row with an "Always on" badge and a checked, disabled switch.
asset_seed keeps catalog_hidden: it is the focused-rescan mechanism and
is meaningless in an authored engine.
Neither flag is presentational. Hiding the control was never the
mechanism, because the resolver still honoured a "no" from two other
directions: a launch could send overrides {ip_enrichment: {enabled:
false}} (verified: preview reported "Disabled in engine"), and an
engine document saved before the stage was hidden kept disabling it.
merge_engine_context now takes a hidden stage's defaults outright and
forces enabled=True on an always_on one. validate_overrides and
_validate_stages strip the key rather than reject, so a legacy document
and an imported YAML both keep saving — rejecting is what broke the
Full Sweep and Passive Recon presets in 4600a4c8. preset_stages skips
both, so even Blank cannot author a "no". asset_seed is unaffected by
the force: ScanService enables it on resolved.stages after the merge.
Also fixes the copy, which is what prompted this. The description read
"Sweep up every address the scan found and complete what only the
finished run can know" — the only one of twenty not written as a plain
statement of what the stage produces. Five more in the same pass:
vhost.enabled used jargon and contradicted its own stage description,
endpoint_probe.skip_static said nothing concrete,
dns_batch_concurrency carried a measurement anecdote in a tooltip, and
three stages inherited the placeholder "Run this stage during a scan."
which is the text YAML autocomplete shows for the toggle.
Verified: catalog 18 -> 19 with always_on true; POST/PATCH/import of a
document disabling it all save with the key stripped; preview reports
will_run with the engine document and an override both saying no; a
real scan launched with that override resolved {"enabled": true} and
finished ip_enrichment success {"ips": 2, "new": 0, "enriched": 2},
with the stored overrides recording only the two that applied; a
focused rescan still seeds (asset_seed success). Row checked in light
and dark; the YAML lint warns only on an explicit enabled: false.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01N2QTNoW3jEi3pX7uMLXBkk
Naming an engine or a scan context with more than 200 characters reached
Postgres and came back as a 500 — the column is bounded, the request
model was not. A name of spaces was accepted outright by engines,
contexts, schedules and projects, and a project created that way got an
empty slug, which is the segment its own URL is built from.
Proxies and notification channels already ran their names through
clean_name; every other named entity now does the same, on update as
well as on create, where none of them did. A rejected name says whether
it was empty or too long instead of failing at the database.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K6LcH8uUddjJbKrahf34ys
The wordlist field was a free-text absolute path, so the only lists anyone could
actually use were the two baked into the image. Changing that meant a bind mount or a
docker cp — a sysadmin action, not a product one — which in practice meant every
install guessed from the same list. Bruteforce quality is almost entirely wordlist
quality, so the knob was there and nobody could reach it.
Wordlists are now a library, on the same shape as the nuclei check library. Shipped
lists stay in the image and are indexed on read so the picker never starts empty;
uploads land under a `wordlists` volume shared by api and worker. A config field holds
a slug, never a path — `stages/config.py:wordlist(kind)` declares one the way
`threads()` and `rate()` declare theirs, and carries the kind the picker filters on so
a DNS stage is never offered a list of URL paths. `lookup()` still accepts the
absolute paths older engines stored, so an existing engine keeps working.
The UI is the Arsenal's second tab, which is why the Arsenal is now "what your scans
run with" rather than only scanners and their checks: upload, filter by what the words
are, preview the first 200 in the order a scan will read them, delete an upload. Every
list shows the name an engine refers to it by, because that is the thing you type into
a YAML document.
`resolve_path()` refuses anything resolving outside its origin's root and
`delete_custom()` refuses a shipped list — the guard the template library learned the
hard way. A stage that cannot find its list says so and reports PARTIAL rather than
quietly guessing from a different one; that is how the shared volume being absent was
caught during the end-to-end run.
Verified end to end: a seven-word list uploaded through the API, selected by slug,
every passive source disabled — the scan returned six hosts, all attributed to
bruteforce.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DPueZXQMbjLXMokXeA8LFP
A scan no longer needs a saved engine. The launch dialog has two
configuration modes: Scan Engine (default, runs a saved engine exactly
as configured, shown as a pipeline and its YAML) and Custom Scan (pick
the capabilities to run; dependent and supporting stages are included
automatically). Targets are created on launch from typed values, the
scan context is a first-class field, and "Run again" restores the
engine or the selection that ran.
Backend
- scans.engine_id is nullable; an absent engine resolves through
AdHocEngine and the run is named after its capabilities (plan_label)
- ScanCreate/ScanBatchCreate accept target_value(s) (create or reuse via
TargetService.ensure_targets), a per-run intensity, and preview works
for a target that does not exist yet
- every stage declares consumes/produces (AssetKind), group (StageGroup)
and role (capability | support); the registry validates them and the
catalog serves them with groups and seed_produces
- stage_effects walks levels so a producer only feeds later levels and
marks an unfed consumer skipped_no_input
Frontend
- components/scans/launch/: launch-dialog, launch-state, target-picker,
plan-picker (segmented Scan Engine | Custom Scan, pipeline + YAML
Config views, capability pills with full per-stage settings),
stage-pill, stage-config-popover, launch-summary + what-runs,
launch-context-field, launch-context-form, save-engine-dialog
- utilities/launch-plan.ts resolves implied producers and following
support stages; the last engine and last custom selection persist
- quick-scan-fields offers presets and engines in one select; the
command palette offers "Scan <value>"; scan list action is "Run again"
- effect-panel drops its explanatory header; NO_CONTEXT_LABEL is
"No scan context"; launch-modal and execution-preview are removed
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014erb6Jbm6v9sxPgYFBxB7E
The unit of exposure is now a service — (ip, port) — not a bare open port.
Pipeline (EXPANSION): L0 produce hosts, L1 cdn_check + passive_ports,
L2 port_scan, L3 http_probe, L4 screenshot/waf_detect/service_fingerprint.
- cdn_check applies to every target type and records cdn_type, so a domain
scan finally knows which addresses are CDN-fronted. cdn/waf is an edge you
do not own (edge ports only); cloud is your own machine (scanned in full).
- port_scan decides and records a per-address scan_policy (full/web/skip)
honouring the scan context's excluded_ips, and offers port profiles,
exclusions, scan type, a tarpit threshold and an address budget.
- passive_ports reads Shodan internetdb through naabu -passive: keyless,
touches_target=False, so a passive-intensity scan still gets exposure data.
- http_probe seeds its host x port matrix from open ports and folds every
live response back onto the port (is_http, tls, product).
- service_fingerprint names the software behind non-web ports with a pure
Python banner prober (tools/banner/), including a SOCKS5 CONNECT so it
honours the context proxy. naabu -sV is unimplemented.
- shared/definitions/ports.py is the one vocabulary; ip_inventory and
port_inventory are the only writers of ip_addresses/ports.
- open_ports_found is measured from the table (DERIVED_COUNTS): four stages
write ports, so summing stage results double-counted.
- naabu failures no longer report zero open ports as success.
Surfaces: a Services tab (SERVICE_QUERY grammar, 16 fields, 18 counted
leads, 9 group dimensions, detail sheet) built on the shared results shell,
and an outlier-first Exposure panel on the Overview. Every count is a
promise — leads, groups and panel pivots were verified to equal the row
count they land on.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BnC2xMMzV6WQnpDp5zcXoU
- Updated the engine management UI in `+page.svelte` to improve user experience with new controls for filtering stages, toggling side panels, and managing YAML configurations.
- Introduced new components such as `EngineSummaryBar` and `Resizable` for better layout handling.
- Added clipboard functionality for copying YAML configurations and formatting options.
- Enhanced the backend models in `scan_engine.py` to include a new `PreviewResolved` class for better handling of scan context and parameters.
- Updated presets in `presets.py` to clarify descriptions and improve user understanding of stage settings.
- Implemented TargetEnrichmentStage for enriching targets with DNS, WHOIS, and BGP context.
- Added VhostStage for virtual host discovery using host-header fuzzing.
- Introduced WafDetectStage for fingerprinting web application firewalls.
- Created configuration files for each new stage with relevant parameters.
- Updated Dockerfile to copy stages instead of engines.
- Enhanced existing tools and services for better integration with new stages.
- Refactored task management in Celery to accommodate new stages and improve error handling.
- Add aggregate functions for scan status and counts in orchestrator.
- Create ScanEventPublisher for emitting scan-related events.
- Introduce ScanActivityService for managing scan activities.
- Implement ScanCommandRecorder for logging command executions.
- Develop canvas building and finalization logic for scan tasks.
- Refactor scan task to utilize orchestrator functions for running stages and finalizing scans.
- Enhance scan notification system to dispatch relevant events.
- Integrate command recording into CLI tool execution for better tracking.
- Updated the default route for automation to redirect to engines instead of scans.
- Removed the scans page and replaced it with a new scans overview page that lists scan history for projects.
- Introduced a detailed scan view page to display individual scan results and subdomain discoveries.
- Added new enums for scan statuses and subdomain sources to improve code clarity and maintainability.
- Implemented a new scan task in Celery to handle scan execution and logging.
- Enhanced the database models for scans and subdomains to support new features and improve data integrity.
- Updated the worker Dockerfile to include necessary scanning tools and dependencies.
- Added utility functions for domain matching and pattern compilation to streamline scan operations.
