A nested chord canvas put the whole remaining pipeline into every
stage message, duplicated per sibling per level: 33 MB per message,
39 of them filled a 1 GB Redis, and the worker was OOM-killed
deserializing them. Every later launch then failed with "command not
allowed when used memory > 'maxmemory'".
shared/services/orchestrator/canvas.py now builds one step at a time;
a step's chord body is app.tasks.scan.run_scan_step(steps, index+1),
which dispatches the next step once the current one settles. Messages
are ~1 KB now; tests/pipeline/test_canvas_shape.py caps them.
Also, from an end-to-end review of the pipeline:
- The reaper re-queues a PENDING scan with no launch anywhere in the
queue, so a lost or failed send_task no longer strands it forever.
api create() now dispatches off the event loop and records a failed
dispatch the way create_batch already did.
- Cancel no longer revokes queued stage tasks. A cancelled run used to
walk every remaining step recording one aborted row per stage, and
whether finalize ran depended on a race with the revoke broadcast.
The step body now finalizes a cancelled run directly, and the api
also dispatches finalize so a run with no live canvas still settles.
- The reaper only skips scans that actually have a message queued,
read off the scans queue, instead of skipping every scan whenever
the queue holds anything.
- Every beat tick carries expires, so a worker that comes back after
being down does not replay hours of stacked ticks.
- worker-default and worker-beat restart on their own.
- _run_process kills the tool in a finally; a soft time limit raised
inside proc.wait used to leave the tool running past the stage.
- on_worker_ready crashed on a missing Consumer attribute, so the
worker's IP-range and threat-feed warm-up never ran.
- Worker logging now goes through the root logger; it was configured
on a "rengine.worker" logger nothing writes to, so INFO logs (task
starts, step hand-overs) never appeared in the container output.
- _normalize_url no longer raises on an unparseable value, which had
been killing the whole bounty feed sync on one bad scope entry.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0181JfVNatLVWnDbtGWRWPov
A target can follow the check library: once a day, when the library gains
checks, one focused run tests every web asset the target's completed scans
found, with those checks only. Per-target watermark, one sweep lock, a
`daily.run` chain that registers future once-a-day jobs, two notifications,
a What's new group, a "New" badge on the library, a monitored-target signal
and a "New checks runs" scan filter.
The pipeline review behind it, in three passes:
Planner and scanner. The deep sweep is its own tier, outside ROOT_TIERS and
after replay, so a cut costs the tail and cluster members are still
confirmed; the one-request tier runs first, sized on its clustered cost;
blind core ranks CISA then VulnCheck then severity then cost; the planned
tiers come from the selection; skipped hosts are marked alone; library gaps
are one row per run; the sweep cap is 8 requests.
Output. Payload-free fingerprints for fuzzing, out-of-band and dalfox
findings; suppression follows the replay edge; the digest baseline is every
earlier scan; host and port read through split_host_port; interaction obeys
store_evidence and redaction; the rollup excludes replays.
Tools. nuclei's exit code, abort and budget are read; a tool arg cannot
restate a stage flag under its short spelling; one refused row costs one
row; a headless block decides the protocol.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B58eEM68a5qDFTQXPE5aoW
Every stage config field is a decision; rate, concurrency, timeout and
retries come from a per-tool intensity profile the stage reads as its
transport. 173 fields become 83: 41 transport values, 40 constants and
the dead ones leave the models, and the builder groups by capability
with support stages as checkboxes and advanced fields behind one
disclosure. Aggressive now changes what a tool is handed.
Every project carries a built-in Default engine, seeded on project
creation and on first list, undeletable: stage defaults, subfinder as
the only passive source, vulnerability scan on with nuclei.
From the tracing review: tool args can no longer restate a flag the
stage set, the session check obeys the probe scheme, excluded scope
reaches every stage that reads addresses, hosts or paths, the sitemap
and source-map providers throttle at the transport rate, the blind tier
obeys its switch, the fuzzing lanes split the budget, and the removed
private-address and evidence switches are back as advanced fields.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Pq4xYJbgW5euUa9sCWBKFw
A full static review of the 3.0 tree, run as two fan-out passes with every
finding verified against the running stack rather than taken on inspection.
Correctness and security
- instance settings encrypted the mask over the stored AI key: `!= MASK` never
matched `••••••••abcd`, so saving settings destroyed the key and `test_ai`
posted the mask as a bearer token
- the MFA token was replayable for its full five-minute life; the gate now
checks revocation and consumes the token after a successful exchange
- logout revoked nothing for a bearer client while answering "Logged out"
- `POST /api-keys/{id}/test` returned the plaintext key inside httpx's error
string, which embeds the whole URL
- an interest rule could be edited or deleted through another project's id
- redis ran the celery broker under `allkeys-lru`, free to evict a queued task
or a chord counter; now `noeviction`
- nuclei's stored command carried the OAST token and every auth header
- a finalize step could poison the session and leave a scan RUNNING for ever
- `is:issue` raised TypeError on every search
- offset paging repeated and skipped rows in four dimensions
- a cancelled run kept a stale `scans.*_found` rollup
- every PDF report raised `KeyError: 'software'`
- `version_key` truncated a wide numeric run and inverted release ordering
- psycopg raises a NUL as a bare ValueError, above StatementError, so every
savepoint guard was blind to the case it was written for
Performance
- endpoints.host had no trigram index: 35.9ms seq scan -> 0.389ms
- free text compiled to one OR, not the prescribed UNION ALL: 132ms -> 41ms
- `/http-assets/summary` and `/ips/summary` materialised whole tables
- the cross-link `ip` kind ran an estate query per page and drew nothing
Frontend
- `$effect` loaders that read the state they write, an unbounded request loop
on a failing endpoint
- error states that rendered as empty states
- 2FA backup codes were unusable while the copy said they worked
- a "Scan history" dialog that always reported none
- ~400 duplicated lines across five filter bars, now one component
Build and tooling
- every scanner installed at @latest; all 13 pinned to the measured versions
- CI built `web/`, gone since 2.x, with tags that cannot be valid
- the prettier pre-commit hook discarded its filenames and always passed
- the test suite destroyed itself when run twice
Also: subdomains from public code, via a pinned github-subdomains source and a
GitHub API key that subfinder receives too; sublist3r and oneforall removed
after measuring one unique host in 89 across four targets.
Verified: 853 tests, ruff clean over eleven packages, svelte-check 0 errors,
the count-is-a-promise gate at 131 checks and 0 mismatches, and 143 GET routes
with no 5xx. Four migrations, each with an exercised downgrade.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DVkdx5ynZ6aof5kLFczmhX
aborting_on binds one stage's abort check for the whole worker process,
but only stream_json consulted it. CLIToolRunner.run was a blocking
subprocess.run, so nine tool paths ran to their own budget after a cancel
or a pause: naabu under host_discovery for up to an hour, ffuf per IP for
thirty minutes, wafw00f, urlfinder, alterx, cdncheck, dnsx and every
subdomain provider for theirs. revoke_scan_tasks deliberately does not
terminate, so nothing else was going to stop them.
run() now waits in poll-sized slices and raises StageAbortedError, which
moves to tools/runner/abort.py so a tool raises the same class the
orchestrator already catches. A finished tool still returns immediately:
the wait is proc.wait(timeout=slice), not a sleep.
Killing the direct child was not enough either. A tool that forks leaves
a grandchild holding the pipes, which keeps hitting the target and blocks
the reader threads until their join times out — twelve seconds to stop a
sleep in the test that found it. Every tool now starts in its own process
group and is stopped with killpg, on both paths.
stdin moves to its own thread beside the readers so a tool that never
reads it cannot deadlock the write.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01X5roHYDYzSPRNaTkn8hSkJ
Pause revoked with terminate=True and SIGKILL. A SIGKILLed celery child
runs no finally and no watchdog thread, so its subprocess.Popen tool was
reparented to init and kept hitting the target with its budget and idle
timer dead — an unbounded scan against a host the operator believes is
paused. Revoke now drops the queued tasks only; a task already running
stops through its own abort check, which kills the tool and reaps it.
tools/runner/abort.py binds the stage's check for the whole process, so a
tool stops even where the call site passed no should_stop. The blocking
run() path has no abort poll yet and still ends at the tool's own
timeout.
Resume was answered by the worker, so the api replied "paused" and the
page waited on a celery round trip — with the worker busy the control
looked dead. The api now takes the row lock, folds the pause, sets
RUNNING and bumps the epoch itself, then dispatches a task that only
builds the canvas. Pause and resume answer in ~20ms with the true status
and the stores hand the row back, so the page flips without a refetch.
run_scan read run_epoch at execution time, so a launch still queued when
a scan was paused and resumed adopted the new epoch and dispatched a
second canvas beside the resumed one: two subdomain_discovery processes
on one scan. It carries its epoch like every other signature now, and
superseded() is the one definition all four guards share.
Also: cancel kept the open pause out of paused_seconds, so a run
cancelled after sitting paused reported the pause as run time; pause and
cancel take the row lock, so five concurrent calls give one winner
instead of duplicate ledger entries; a watch probe refuses to pause,
since its host waits on a terminal status that pause never reaches; the
hero, ribbon and empty state get a paused arm rather than falling into
the settled one and claiming a delta for a run that has not looked yet;
isOpenStatus mirrors SCAN_OPEN_STATUSES so Compare, the timeline and bulk
cancel agree with the api; and the pause confirm is gone — it removes
nothing, so the cost is stated on the control instead.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Rg2cefwCvz9f77A2SHEHDb
Repo-wide sweep of every user-facing string: frontend components, routes,
config and stores; API details and route docs; shared definitions and
models; stage config help; report prose; MCP, toolbox, interest and
connector text; notification and worker messages.
Descriptions that explained, reassured, justified or restated their title
are deleted. Confirm dialogs name what is removed and stop. Failed actions
read "<Noun> not <verbed>", successes "<Noun> <verbed>". No reader address,
no product-as-subject, no em-dashes or parentheticals in prose.
Mirrors updated on both sides: EPSS band help, compare refusal reasons,
interest labels. Four tests asserting on old wording follow the new text.
The rules are condensed into CLAUDE.md under Writing.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SK5o8xR96cbSnAfjYSTPSe
The comment rule in CLAUDE.md is one line maximum, never narration: no
multi-line blocks, no module-header explainers, no sentences that explain or
justify the code to the reader. The tree had drifted a long way from it, and
most of the drift is mine.
Removed:
232 lines of multi-line comment blocks
203 single-line comments that argued instead of stating
154 comments joined by so/because/otherwise/rather-than
143 narrating comments in the frontend
112 test docstrings (the test name already says what it asserts)
25 test module docstrings and 25 decorative section dividers
Collapsed:
324 multi-line docstrings to one factual line
271 one-line docstrings that had a justifying clause bolted on
Kept: functional directives, section dividers, short field labels, alembic
revision headers, and the eight FastAPI route docstrings that render in /docs.
Where a fact was load-bearing it is in CLAUDE.md, which is where it belongs.
284 python tests and 34 frontend tests pass, ruff and ruff-format clean across
ten packages, svelte-check 0 errors, prettier and eslint clean, app.main imports.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013dKUV96WkQQETZBMoAG98C
Nine bugs found by a cloud review scoped to the celery engine and the stage
orchestration (46 files, 7,790 lines).
Stuck scans:
- run_stage did load_resolved(), the Stage construction and should_run() OUTSIDE
the try/except, after committing a RUNNING activity. A raise there — a stage
config model that moved under a queued scan, a constructor failure — left the
row RUNNING for ever, so aggregate_status kept returning RUNNING and finalize
deferred; the link_error calls the same finalize, so it never rescued it. The
setup now runs inside the same handler, and the post-run bookkeeping is
wrapped too: recording a result must never be what fails a finished stage.
- _fail_stage crashed on a missing activity row (a cascade-deleted scan), and
its own except-branch called finish() with the same None and raised again.
Stage failures that were not failures:
- the abort poll opened a session on a path stages hit in tight loops and let
any database error propagate, so a pool blip failed a running stage. It now
fails open and keeps the last answer, like the Redis guards.
Wrong accounts of a run:
- finalize picked the scan's error with `next(a for a in activities if a.error)`,
ignoring status over an unordered fetch — so a FAILED scan reported "superseded
by stage retry" or a PARTIAL warning instead of the failure. Only a FAILED row
may explain a failure.
- _undispatched compared the activity ROW count to the stage count, so one
retried stage (which leaves a superseded row plus a new one) cancelled out one
genuinely undispatched stage and the scan reported COMPLETED with a stage
missing. It compares stage names now.
- endpoint_probe counted static endpoints excluded by skip_static as
budget-skipped, so the coverage warning claimed a cap that was never hit.
Work done twice, and work undone:
- reap_stalled resumed the whole level, re-running stages whose activity was
already SUCCESS: a finished hour-long stage ran again because a level-mate
stalled, and if the second run failed the whole scan failed. build_canvas
takes the set of finished stages and omits them (an emptied level is dropped);
the done=None path is byte-identical to before.
- run_scan read the scan without a lock and committed status=RUNNING plus a
fresh celery_task_ids over whatever was there, so an API cancel landing in
that window was silently overwritten and the ids it needed to revoke were
lost. finalize already row-locks; run_scan now does too.
- _write_temp_file called os.close(fd) after os.fdopen had already closed it,
masking the real write error with EBADF and, in the threaded stages, risking
a close of another thread's reused descriptor.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SAeBy7TBUn3Fd1ckyQp9r6
Stages pass custom arguments to nuclei, httpx, naabu, ffuf, wafw00f and
cdncheck — every one of them calls tool_args for its own binary — but
none was listed as a tool that takes them, and the validator dropped an
unlisted tool without a word. Setting nuclei arguments returned 200 and
stored nothing. The six are now listed, so what the stages already read
is what a user can set, and a tool that genuinely takes no arguments is
refused by name instead of silently ignored.
A failing tool reported only its exit code when it wrote its complaint
to stdout rather than stderr, which naabu does for a bad flag: the scan
said "naabu exited with code 2" while the reason sat one table away.
A short stdout is now used when stderr is empty, so the scan says
"naabu exited with code 2: flag provided but not defined: -source-port".
Every shipped example argument was checked against the binary it names.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K6LcH8uUddjJbKrahf34ys
Measured end to end against three targets on the default plan.
Execution order is now declared, not hand-numbered. `Stage.depends_on` names the
stages one may not start before and the registry derives `level` from it by
longest-path layering, rejecting an unknown name or a cycle. Eleven hand-assigned
barriers collapse to seven real ones: target enrichment no longer gates subdomain
discovery, and the whole post-probe fan-out (origin probe, service fingerprint,
WAF detection, URL discovery, vulnerability scan) is one parallel level instead of
five. The frontend plan solver groups by the same derived level.
Four tool invocations were wrong in ways that cost whole scans:
- naabu reads `-timeout` as a go duration, so a bare `-timeout 3000` pinned every
invocation at a fixed ~135s whatever the host and port count. `-timeout 3s`
returns the identical open-port set: 138s -> 5s on three addresses, and on a
1,610-address estate 7,428s (two 3,600s timeouts, stage failed) -> 240s.
- urlfinder has no `-t`; passing one made it exit 2, so the archive URL source had
never returned a row.
- amass v4 prints a relationship graph, not a host list, so its output was parsed
as hostnames and discarded — zero rows in every scan on record while costing the
full 360s tool budget. Parsed properly now, and out of the default source set.
- httpx's stream probe was capped at a 900s total, silently truncating a large HTTP
probe and reporting it as success. It is idle-bounded now, the way dnsx already
was, and reports PARTIAL when it is stopped.
Celery's redis visibility timeout (3600s default) sat below the 8h task limit, so any
stage running over an hour was delivered twice and the whole rest of the scan ran
again — one run duplicated its entire post-port-scan pipeline, another looped for ten
hours. It now sits above the hard limit.
Work inside stages that was serial and did not need to be: resolver batches
(dns_batch_concurrency, default 3), URL discovery providers, robots/sitemap fetches,
and wafw00f (which has no threads of its own) now run concurrently, with DB writes
kept on the stage thread. wafw00f, screenshots and the WAF stage report what they did
not cover instead of reporting a short run as complete. The cancellation flag is
polled at most every two seconds instead of once per resolved name.
Also: WAF detection and screenshots no longer load whole http_asset rows (stored
response bodies included) to write one column; tool output decodes with
errors="replace" so one non-UTF-8 byte cannot discard a run; the worker runs at
CELERY_SCAN_CONCURRENCY (8) because the widest level now holds five stages.
Measured: vulnweb.com 656s -> 218s with identical results. hamropatro.com completes
in 2,273s where it previously never finished. gov.np's port scan 7,428s -> 240s.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DPueZXQMbjLXMokXeA8LFP
dnsx drops 20–60% of answers under load and exits 0, so a subdomain stage
could store half its hosts unresolved and still read Success. Resolution now
runs in shuffled batches (dns_batch_size) with the thread count clamped to the
range the resolver actually keeps up with; a batch answering far below the
median of its peers is retried, and one that stops producing is abandoned after
dns_idle_timeout instead of the whole run waiting on a total ceiling. What the
resolver managed — batches, retries, stalls — is reported, never assumed.
StageResult gains warnings + partial. The stage runner records a new PARTIAL
activity status carrying the notes, logs the stage outcome as a warning and
emits it live. The run ribbon paints a partial stage amber with a triangle, the
stage dialog shows the note in a warning box, and activityRan() is the one "did
it run" test so progress and summaries count partial as done. reverse_dns
reports the same way when dnsx times out or is missing.
stream_json yields a StreamOutcome (return code, timed_out, record count,
stderr) instead of a bare iterator, kills on idle as well as on total time, and
a timed-out run keeps the records that landed instead of discarding them.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012YK61xuAJGMhq1uniLhHX5
- Implemented TargetEnrichmentStage for enriching targets with DNS, WHOIS, and BGP context.
- Added VhostStage for virtual host discovery using host-header fuzzing.
- Introduced WafDetectStage for fingerprinting web application firewalls.
- Created configuration files for each new stage with relevant parameters.
- Updated Dockerfile to copy stages instead of engines.
- Enhanced existing tools and services for better integration with new stages.
- Refactored task management in Celery to accommodate new stages and improve error handling.
- Introduced new utilities for scan insights in `scan-insights.ts`, including functions for certificate state, days until expiry, and query compilation.
- Added interfaces for web asset queries, subdomain filters, and insights structures.
- Enhanced `scan-status.ts` with new constants for pagination and search debounce.
- Updated the scan results page to utilize the new insights and remove unused subdomain and asset APIs.
- Modified the `HttpAsset` model to include raw request/response data and body previews.
- Created new models for scan correlation and insights, including surface stats and attention items.
- Implemented streaming JSON parsing in the HTTPX client for memory efficiency.
- Enhanced the executor to support streaming JSON output from subprocesses.
Rebuild the scan detail view into a tabbed results page: Overview
(tech/status/web-server distributions, TLS health, top networks and
services), Web assets (rich httpx table with screenshots, status,
tech, CDN/WAF/TLS-expired badges + a full per-asset detail drawer),
Subdomains (denormalized HTTP summary), Infrastructure (IP-centric
cards correlating open ports and the hosts resolving to each IP), and
Pipeline. Assets correlate by value within the scan (subdomain.name /
http_asset.host / ip), so one IP card surfaces every co-located host.
API: add per-scan http-assets, ports and ips list+summary endpoints
plus services mirroring the subdomains pattern, and serve screenshots
through an authenticated /media/screenshot route; mount scan_media
into the api container so it can read worker-written captures.
Fix two serializers that silently dropped fields: ScanRead omitted
http_assets_found (scan detail and list 500'd), and the subdomain
serializer never emitted the denormalized HTTP summary it stores.
Drop leftover docstrings and banner comments from the httpx client,
parser and field-map.
The host:port -> live HTTP asset convergence. After discovery, a scan now
runs expansion:
- port_scan: naabu (cdn-gated, well-known service names) -> Port
- cdn_check: cdncheck flags CDN/WAF/cloud IPs (ip family)
- http_probe: httpx probes the host:port matrix; one pass enriches each
service with status/title/tech/server/TLS/cdn/jarm/favicon/hash ->
HttpAsset, with asn backfilled from the resolved IP
httpx -tls-grab carries cert data inline (no separate tlsx pass); nmap is
dropped — naabu for ports, httpx for HTTP service id.
Also: every engine moved to its own folder; net_options() threads
proxy/headers/user-agent into every tool; rengine-tools no longer installs
top-level modules that shadowed libs (httpx); tool binaries resolve from the
go bin ahead of venv console scripts. New Port + HttpAsset models + migration
and the http_assets headline rollup.
- Add aggregate functions for scan status and counts in orchestrator.
- Create ScanEventPublisher for emitting scan-related events.
- Introduce ScanActivityService for managing scan activities.
- Implement ScanCommandRecorder for logging command executions.
- Develop canvas building and finalization logic for scan tasks.
- Refactor scan task to utilize orchestrator functions for running stages and finalizing scans.
- Enhance scan notification system to dispatch relevant events.
- Integrate command recording into CLI tool execution for better tracking.
- Added `clean_name` function to `validation.py` for name validation with max length.
- Simplified docstrings in `dnsx` client, parser, and service files for clarity.
- Enhanced error handling in `ViewDNSService` to prevent disabling keys on rate limits.
- Improved WHOIS service to handle concurrent inserts gracefully with integrity checks.
- Updated `Settings` class in worker config to inherit from `BaseAppSettings`.
- Added new dependencies in `pyproject.toml` for `apprise` and `cryptography`.
