EndpointSource.FUZZ had a label, a rank and a SourceKind and no producer.
stages/content_discovery/ is it: ffuf against every live site with a ranked
wordlist, level 5 beside url_discovery, a capability a person chooses, and
enabled=False by default so an existing engine does not silently gain a run
that sends 25,000 requests.
tools/data/content.txt ships with the image (6,000 words, SecLists raft small,
directories interleaved before files at each rank, regenerated by
scripts/fetch_content_wordlist.py and committed like the port registry so an
offline box still has one). The order is the budget, exactly as the subdomain
list's is.
One ffuf process per site, never several sites through a second wordlist
keyword. The clusterbomb form looks like the efficient shape and silently
defeats the only filter that matters: measured on one soft-404 host,
`-w hosts:HOSTW -w words:FUZZ -u HOSTW/FUZZ -ac -ach` returned 278 of 300
guessed paths, while the same host asked in its own process returned 0. ffuf
cannot calibrate against a host it does not know until the payload is
substituted. Sites run over a small thread pool and only the stage thread
writes, the url_discovery rule.
A second belt sits behind calibration because no calibration is perfect: a site
answering to 25% or more of a ranked list is answering to anything, so its whole
run is dropped and the sites are named in a warning rather than quietly
filtered. -maxtime is the per-site budget, -sf gives up on a host returning 403
to everything, and a run that hit its budget, a site that was dropped and a site
that could not be reached are each counted and reported.
Rows store is_probed=True with the status ffuf observed — it made the request,
so the status is an observation, not an inference. `input.FUZZ` is base64 in
-json mode (plain in -of json, which is why vhost() is fine), so the word is
read off `url`.
Verified live: 3 sites x 300 words in 11.7s, 295 raw hits became 17 real ones —
/admin, /login, /media, /modules and /index.php among them.
29 tests, each proven to fail against 10 mutations.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013dKUV96WkQQETZBMoAG98C
Discovery was passive-only — subfinder, ctfr, crtname, assetfinder and certificate
SANs — so anything absent from a public source was absent from the scan, and every
later stage inherited the miss. It now runs two more sources after the passive merge,
carrying SubdomainSource.BRUTEFORCE and PERMUTATION through the same scoping and
merge every other provider uses.
Bruteforce is `dnsx -d <apex> -w <list>` over a wordlist shipped in the repo:
4,860 labels ranked by real-world frequency, regenerated by
scripts/fetch_subdomain_wordlist.py and committed like the port registry, so an
install with no egress still has one. The word budget reads from the top of that
ranking, which is why a smaller budget is simply the first N lines. Permutation is
alterx over the names already found, seeded apex-first because a name one label from
the apex has more siblings worth guessing than a five-label one.
Two things this needed to be worth running at all:
- **A budget, not an idle watchdog.** A guessed name that does not exist produces
no output, so the stall detector that protects the resolver killed both runs at
exactly 90s the first time. Guessing is bounded by total time instead, floored at
5 names/s against a measured 9, and says so when it stops early.
- **Wildcard rejection.** The wildcard probe moved ahead of expansion: on a wildcard
zone every guess answers, and storing 5,000 fake names would be worse than not
guessing. Answers that resolve only to the wildcard address are dropped and
counted in the source's note.
Both are skipped at passive intensity and say why — the stage still touches no
target, but a wordlist run does query the target's own nameservers. Bruteforce
defaults on at 1,000 words (110s measured, 2 hits on vulnweb.com, both attributed
correctly alongside their passive sources); permutations default off, because every
variant is another DNS query.
ProviderResult gained `note` so a source can come up short without being reported as
failed. alterx reads seeds from stdin — `-l <file>` answers "no input found".
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DPueZXQMbjLXMokXeA8LFP
Rounds out expansion (host discovery, virtual hosts, screenshots, WAF):
- host_discovery: naabu liveness sweep marks IpAddress.is_alive for
ip_range/asn; port_scan then deep-scans only live hosts
- vhost: ffuf Host-header bruteforce (domain, opt-in) -> vhost-only
Subdomains that http_probe then picks up
- screenshot: httpx headless chromium -> HttpAsset.screenshot_path under
a scan_media volume
- waf_detect: wafw00f fingerprints WAFs -> HttpAsset.waf
All config-driven (ExpansionConfig) with proxy / user-agent / headers +
per-tool custom args threaded through. Adds ffuf, chromium and wafw00f to
the worker image and a scan_media volume.
