mirror of
https://github.com/yogeshojha/rengine.git
synced 2026-10-20 15:21:30 +02:00
Yogesh OjhaandClaude Opus 5
824f2c8ec3
Discovery was passive-only — subfinder, ctfr, crtname, assetfinder and certificate
SANs — so anything absent from a public source was absent from the scan, and every
later stage inherited the miss. It now runs two more sources after the passive merge,
carrying SubdomainSource.BRUTEFORCE and PERMUTATION through the same scoping and
merge every other provider uses.
Bruteforce is `dnsx -d <apex> -w <list>` over a wordlist shipped in the repo:
4,860 labels ranked by real-world frequency, regenerated by
scripts/fetch_subdomain_wordlist.py and committed like the port registry, so an
install with no egress still has one. The word budget reads from the top of that
ranking, which is why a smaller budget is simply the first N lines. Permutation is
alterx over the names already found, seeded apex-first because a name one label from
the apex has more siblings worth guessing than a five-label one.
Two things this needed to be worth running at all:
- **A budget, not an idle watchdog.** A guessed name that does not exist produces
no output, so the stall detector that protects the resolver killed both runs at
exactly 90s the first time. Guessing is bounded by total time instead, floored at
5 names/s against a measured 9, and says so when it stops early.
- **Wildcard rejection.** The wildcard probe moved ahead of expansion: on a wildcard
zone every guess answers, and storing 5,000 fake names would be worse than not
guessing. Answers that resolve only to the wildcard address are dropped and
counted in the source's note.
Both are skipped at passive intensity and say why — the stage still touches no
target, but a wordlist run does query the target's own nameservers. Bruteforce
defaults on at 1,000 words (110s measured, 2 hits on vulnweb.com, both attributed
correctly alongside their passive sources); permutations default off, because every
variant is another DNS query.
ProviderResult gained `note` so a source can come up short without being reported as
failed. alterx reads seeds from stdin — `-l <file>` answers "no input found".
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DPueZXQMbjLXMokXeA8LFP
64 lines
2.0 KiB
Python
64 lines
2.0 KiB
Python
#!/usr/bin/env python3
|
|
"""Regenerate tools/data/subdomains.txt from SecLists' DNS discovery list.
|
|
|
|
The list is MIT-licensed and slow-moving, so the generated file is committed rather
|
|
than fetched at build time — a scan on a box with no egress still gets a wordlist.
|
|
Run this when you want a newer cut:
|
|
|
|
python3 scripts/fetch_subdomain_wordlist.py [--size 5000]
|
|
|
|
Names are ranked by how often they appeared in a real DNS corpus, so the order is the
|
|
budget: the resolver walks the file top to bottom and a smaller wordlist is simply the
|
|
first N lines. Keep it that way.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import argparse
|
|
import re
|
|
import sys
|
|
import urllib.request
|
|
from pathlib import Path
|
|
|
|
SOURCE = (
|
|
"https://raw.githubusercontent.com/danielmiessler/SecLists/master/"
|
|
"Discovery/DNS/subdomains-top1million-{size}.txt"
|
|
)
|
|
SIZES = (5000, 20000, 110000)
|
|
OUTPUT = Path(__file__).resolve().parent.parent / "tools" / "data" / "subdomains.txt"
|
|
|
|
# one DNS label: what dnsx will prefix onto the apex, nothing else
|
|
_LABEL = re.compile(r"^[a-z0-9_](?:[a-z0-9_-]{0,61}[a-z0-9_])?$")
|
|
|
|
|
|
def fetch(size: int) -> list[str]:
|
|
with urllib.request.urlopen(SOURCE.format(size=size), timeout=60) as response: # noqa: S310
|
|
body = response.read().decode("utf-8", errors="replace")
|
|
seen: set[str] = set()
|
|
labels: list[str] = []
|
|
for raw in body.splitlines():
|
|
label = raw.strip().lower()
|
|
if not label or label in seen or not _LABEL.match(label):
|
|
continue
|
|
seen.add(label)
|
|
labels.append(label)
|
|
return labels
|
|
|
|
|
|
def main() -> int:
|
|
parser = argparse.ArgumentParser()
|
|
parser.add_argument("--size", type=int, default=5000, choices=SIZES)
|
|
args = parser.parse_args()
|
|
|
|
labels = fetch(args.size)
|
|
if len(labels) < args.size // 2:
|
|
print(f"only {len(labels)} usable labels, refusing to write", file=sys.stderr)
|
|
return 1
|
|
OUTPUT.write_text("\n".join(labels) + "\n")
|
|
print(f"wrote {OUTPUT} ({len(labels)} labels)")
|
|
return 0
|
|
|
|
|
|
if __name__ == "__main__":
|
|
raise SystemExit(main())
|