Yogesh OjhaandClaude Opus 5 05be5301d4 perf(scans): the pipeline runs its real dependency graph, and its tools run right ...

Measured end to end against three targets on the default plan.

Execution order is now declared, not hand-numbered. `Stage.depends_on` names the
stages one may not start before and the registry derives `level` from it by
longest-path layering, rejecting an unknown name or a cycle. Eleven hand-assigned
barriers collapse to seven real ones: target enrichment no longer gates subdomain
discovery, and the whole post-probe fan-out (origin probe, service fingerprint,
WAF detection, URL discovery, vulnerability scan) is one parallel level instead of
five. The frontend plan solver groups by the same derived level.

Four tool invocations were wrong in ways that cost whole scans:

  - naabu reads `-timeout` as a go duration, so a bare `-timeout 3000` pinned every
    invocation at a fixed ~135s whatever the host and port count. `-timeout 3s`
    returns the identical open-port set: 138s -> 5s on three addresses, and on a
    1,610-address estate 7,428s (two 3,600s timeouts, stage failed) -> 240s.
  - urlfinder has no `-t`; passing one made it exit 2, so the archive URL source had
    never returned a row.
  - amass v4 prints a relationship graph, not a host list, so its output was parsed
    as hostnames and discarded — zero rows in every scan on record while costing the
    full 360s tool budget. Parsed properly now, and out of the default source set.
  - httpx's stream probe was capped at a 900s total, silently truncating a large HTTP
    probe and reporting it as success. It is idle-bounded now, the way dnsx already
    was, and reports PARTIAL when it is stopped.

Celery's redis visibility timeout (3600s default) sat below the 8h task limit, so any
stage running over an hour was delivered twice and the whole rest of the scan ran
again — one run duplicated its entire post-port-scan pipeline, another looped for ten
hours. It now sits above the hard limit.

Work inside stages that was serial and did not need to be: resolver batches
(dns_batch_concurrency, default 3), URL discovery providers, robots/sitemap fetches,
and wafw00f (which has no threads of its own) now run concurrently, with DB writes
kept on the stage thread. wafw00f, screenshots and the WAF stage report what they did
not cover instead of reporting a short run as complete. The cancellation flag is
polled at most every two seconds instead of once per resolved name.

Also: WAF detection and screenshots no longer load whole http_asset rows (stored
response bodies included) to write one column; tool output decodes with
errors="replace" so one non-UTF-8 byte cannot discard a run; the worker runs at
CELERY_SCAN_CONCURRENCY (8) because the widest level now holds five stages.

Measured: vulnweb.com 656s -> 218s with identical results. hamropatro.com completes
in 2,273s where it previously never finished. gov.np's port scan 7,428s -> 240s.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DPueZXQMbjLXMokXeA8LFP

2026-09-06 03:53:53 +05:30

..

__init__.py

feat(scans): expansion phase — cdn check, port scan, http probe

2026-06-26 16:17:52 +05:30

client.py

perf(scans): the pipeline runs its real dependency graph, and its tools run right

2026-09-06 03:53:53 +05:30

parser.py

feat: add scan insights and correlation utilities

2026-06-26 16:25:30 +05:30