Stages pass custom arguments to nuclei, httpx, naabu, ffuf, wafw00f and
cdncheck — every one of them calls tool_args for its own binary — but
none was listed as a tool that takes them, and the validator dropped an
unlisted tool without a word. Setting nuclei arguments returned 200 and
stored nothing. The six are now listed, so what the stages already read
is what a user can set, and a tool that genuinely takes no arguments is
refused by name instead of silently ignored.
A failing tool reported only its exit code when it wrote its complaint
to stdout rather than stderr, which naabu does for a bad flag: the scan
said "naabu exited with code 2" while the reason sat one table away.
A short stdout is now used when stderr is empty, so the scan says
"naabu exited with code 2: flag provided but not defined: -source-port".
Every shipped example argument was checked against the binary it names.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K6LcH8uUddjJbKrahf34ys
Measured end to end against three targets on the default plan.
Execution order is now declared, not hand-numbered. `Stage.depends_on` names the
stages one may not start before and the registry derives `level` from it by
longest-path layering, rejecting an unknown name or a cycle. Eleven hand-assigned
barriers collapse to seven real ones: target enrichment no longer gates subdomain
discovery, and the whole post-probe fan-out (origin probe, service fingerprint,
WAF detection, URL discovery, vulnerability scan) is one parallel level instead of
five. The frontend plan solver groups by the same derived level.
Four tool invocations were wrong in ways that cost whole scans:
- naabu reads `-timeout` as a go duration, so a bare `-timeout 3000` pinned every
invocation at a fixed ~135s whatever the host and port count. `-timeout 3s`
returns the identical open-port set: 138s -> 5s on three addresses, and on a
1,610-address estate 7,428s (two 3,600s timeouts, stage failed) -> 240s.
- urlfinder has no `-t`; passing one made it exit 2, so the archive URL source had
never returned a row.
- amass v4 prints a relationship graph, not a host list, so its output was parsed
as hostnames and discarded — zero rows in every scan on record while costing the
full 360s tool budget. Parsed properly now, and out of the default source set.
- httpx's stream probe was capped at a 900s total, silently truncating a large HTTP
probe and reporting it as success. It is idle-bounded now, the way dnsx already
was, and reports PARTIAL when it is stopped.
Celery's redis visibility timeout (3600s default) sat below the 8h task limit, so any
stage running over an hour was delivered twice and the whole rest of the scan ran
again — one run duplicated its entire post-port-scan pipeline, another looped for ten
hours. It now sits above the hard limit.
Work inside stages that was serial and did not need to be: resolver batches
(dns_batch_concurrency, default 3), URL discovery providers, robots/sitemap fetches,
and wafw00f (which has no threads of its own) now run concurrently, with DB writes
kept on the stage thread. wafw00f, screenshots and the WAF stage report what they did
not cover instead of reporting a short run as complete. The cancellation flag is
polled at most every two seconds instead of once per resolved name.
Also: WAF detection and screenshots no longer load whole http_asset rows (stored
response bodies included) to write one column; tool output decodes with
errors="replace" so one non-UTF-8 byte cannot discard a run; the worker runs at
CELERY_SCAN_CONCURRENCY (8) because the widest level now holds five stages.
Measured: vulnweb.com 656s -> 218s with identical results. hamropatro.com completes
in 2,273s where it previously never finished. gov.np's port scan 7,428s -> 240s.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DPueZXQMbjLXMokXeA8LFP
dnsx drops 20–60% of answers under load and exits 0, so a subdomain stage
could store half its hosts unresolved and still read Success. Resolution now
runs in shuffled batches (dns_batch_size) with the thread count clamped to the
range the resolver actually keeps up with; a batch answering far below the
median of its peers is retried, and one that stops producing is abandoned after
dns_idle_timeout instead of the whole run waiting on a total ceiling. What the
resolver managed — batches, retries, stalls — is reported, never assumed.
StageResult gains warnings + partial. The stage runner records a new PARTIAL
activity status carrying the notes, logs the stage outcome as a warning and
emits it live. The run ribbon paints a partial stage amber with a triangle, the
stage dialog shows the note in a warning box, and activityRan() is the one "did
it run" test so progress and summaries count partial as done. reverse_dns
reports the same way when dnsx times out or is missing.
stream_json yields a StreamOutcome (return code, timed_out, record count,
stderr) instead of a bare iterator, kills on idle as well as on total time, and
a timed-out run keeps the records that landed instead of discarding them.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012YK61xuAJGMhq1uniLhHX5
- Implemented TargetEnrichmentStage for enriching targets with DNS, WHOIS, and BGP context.
- Added VhostStage for virtual host discovery using host-header fuzzing.
- Introduced WafDetectStage for fingerprinting web application firewalls.
- Created configuration files for each new stage with relevant parameters.
- Updated Dockerfile to copy stages instead of engines.
- Enhanced existing tools and services for better integration with new stages.
- Refactored task management in Celery to accommodate new stages and improve error handling.
- Introduced new utilities for scan insights in `scan-insights.ts`, including functions for certificate state, days until expiry, and query compilation.
- Added interfaces for web asset queries, subdomain filters, and insights structures.
- Enhanced `scan-status.ts` with new constants for pagination and search debounce.
- Updated the scan results page to utilize the new insights and remove unused subdomain and asset APIs.
- Modified the `HttpAsset` model to include raw request/response data and body previews.
- Created new models for scan correlation and insights, including surface stats and attention items.
- Implemented streaming JSON parsing in the HTTPX client for memory efficiency.
- Enhanced the executor to support streaming JSON output from subprocesses.
Rebuild the scan detail view into a tabbed results page: Overview
(tech/status/web-server distributions, TLS health, top networks and
services), Web assets (rich httpx table with screenshots, status,
tech, CDN/WAF/TLS-expired badges + a full per-asset detail drawer),
Subdomains (denormalized HTTP summary), Infrastructure (IP-centric
cards correlating open ports and the hosts resolving to each IP), and
Pipeline. Assets correlate by value within the scan (subdomain.name /
http_asset.host / ip), so one IP card surfaces every co-located host.
API: add per-scan http-assets, ports and ips list+summary endpoints
plus services mirroring the subdomains pattern, and serve screenshots
through an authenticated /media/screenshot route; mount scan_media
into the api container so it can read worker-written captures.
Fix two serializers that silently dropped fields: ScanRead omitted
http_assets_found (scan detail and list 500'd), and the subdomain
serializer never emitted the denormalized HTTP summary it stores.
Drop leftover docstrings and banner comments from the httpx client,
parser and field-map.
The host:port -> live HTTP asset convergence. After discovery, a scan now
runs expansion:
- port_scan: naabu (cdn-gated, well-known service names) -> Port
- cdn_check: cdncheck flags CDN/WAF/cloud IPs (ip family)
- http_probe: httpx probes the host:port matrix; one pass enriches each
service with status/title/tech/server/TLS/cdn/jarm/favicon/hash ->
HttpAsset, with asn backfilled from the resolved IP
httpx -tls-grab carries cert data inline (no separate tlsx pass); nmap is
dropped — naabu for ports, httpx for HTTP service id.
Also: every engine moved to its own folder; net_options() threads
proxy/headers/user-agent into every tool; rengine-tools no longer installs
top-level modules that shadowed libs (httpx); tool binaries resolve from the
go bin ahead of venv console scripts. New Port + HttpAsset models + migration
and the http_assets headline rollup.
- Add aggregate functions for scan status and counts in orchestrator.
- Create ScanEventPublisher for emitting scan-related events.
- Introduce ScanActivityService for managing scan activities.
- Implement ScanCommandRecorder for logging command executions.
- Develop canvas building and finalization logic for scan tasks.
- Refactor scan task to utilize orchestrator functions for running stages and finalizing scans.
- Enhance scan notification system to dispatch relevant events.
- Integrate command recording into CLI tool execution for better tracking.
- Added `clean_name` function to `validation.py` for name validation with max length.
- Simplified docstrings in `dnsx` client, parser, and service files for clarity.
- Enhanced error handling in `ViewDNSService` to prevent disabling keys on rate limits.
- Improved WHOIS service to handle concurrent inserts gracefully with integrity checks.
- Updated `Settings` class in worker config to inherit from `BaseAppSettings`.
- Added new dependencies in `pyproject.toml` for `apprise` and `cryptography`.
