diff --git a/README.md b/README.md index 78c7678b..f6e5f8ae 100644 --- a/README.md +++ b/README.md @@ -7,7 +7,7 @@ theHarvester gathers open-source intelligence about a domain or organization from search engines, certificate transparency logs, DNS datasets, code repositories, threat-intelligence platforms, and other public sources. -It is built for the early reconnaissance stage of authorized security assessments. Use it only on targets you own or have explicit permission to test. +Use theHarvester during the early reconnaissance stage of an authorized security assessment. Run it only against targets you own or have explicit permission to test. ## What it does @@ -25,11 +25,11 @@ Providers control their own availability, quotas, and response formats, so indiv ### Discovery routes and enrichment -![theHarvester discovery routes and enrichment](docs/images/run-evidence-architecture.svg) +[![theHarvester discovery routes and enrichment](docs/images/run-evidence-architecture.svg)](docs/images/run-evidence-architecture.svg) ### HarvestView run desk -![HarvestView run desk architecture](docs/images/harvestview-architecture.svg) +[![HarvestView run desk architecture](docs/images/harvestview-architecture.svg)](docs/images/harvestview-architecture.svg) ## Quick start @@ -76,7 +76,7 @@ uv run theHarvester -d example.com -b emails,ips,urls --no-hosts -f non-host-res `--no-hosts` skips hostname-only sources and omits hostname results while keeping other result types. It cannot be combined with actions that depend on hostnames. HarvestView and the REST API expose the same option as `no_hosts`. -Save a durable JSONL report: +Save results as JSONL: ```bash uv run theHarvester -d example.com -b crtsh,certspotter -f report @@ -114,13 +114,13 @@ export THEHARVESTER_API_KEY='replace-with-a-long-random-value' uv run harvestview ``` -Open [HarvestView](http://127.0.0.1:5000/) to submit and inspect finite runs. The browser receives a derived HttpOnly session cookie and never stores the API key. See the [installation guide](docs/wiki/Installation.md) for local assets, screenshots, and isolated deployments. +Open [HarvestView](http://127.0.0.1:5000/) to start runs and inspect their results. The browser receives a derived HttpOnly session cookie and never stores the API key. See the [installation guide](docs/wiki/Installation.md) for local assets, screenshots, and isolated deployments. Open [Swagger](http://127.0.0.1:5000/docs) or [ReDoc](http://127.0.0.1:5000/redoc) for the automation contract. ### Docker Compose -The Compose service runs as an unprivileged user, stores runs in a named volume, reads the operator key from a file secret, and publishes only to host loopback: +The Compose service runs as an unprivileged user and binds only to host loopback. It stores runs in a named volume and reads the operator key from a file secret: ```bash install -d -m 0700 .secrets @@ -149,87 +149,89 @@ docker compose down HarvestView can start screenshot and DNS brute-force runs from a hostname result. Each action creates its own run and leaves the original evidence unchanged. -API clients send `THEHARVESTER_API_KEY` in the `X-API-Key` header. Provider credentials stay in server-side configuration. Keep the service on localhost unless you add TLS and network access controls. The [REST API guide](docs/wiki/Rest-API.md) documents requests, imports, exports, and authentication. +API clients send `THEHARVESTER_API_KEY` in the `X-API-Key` header. Provider API settings stay on the server. Keep the service on localhost unless you add TLS and network access controls. The [REST API guide](docs/wiki/Rest-API.md) documents requests, imports, exports, and authentication. ## Discovery sources -Select sources by name or by any result route listed below. `-b all` runs the P0 sources. P1 and P2 sources require explicit selection. Credentials marked optional can provide additional access but are not required. +Select sources by name or by a result route listed below. `-b all` runs the P0 sources. P1 and P2 sources require explicit selection. + +Result types in this table always appear in this order: `subdomains`, `emails`, `ips`, `asns`, `urls`, `people`, `breaches`. A result followed by `only` means the source contributes no other result type. The `API key` column refers to provider settings in `api-keys.yaml`; some providers require more than one value. `Optional` means the source can run without a key. The `shodan` source contributes subdomains. Shodan host enrichment through `-s` or `--shodan` is a separate action and is not a source result route.
-View the source and result matrix +View all 58 discovery sources -| Source | Result routes | Activity | Credentials | +| Source | Returns | Activity | API key | | --- | --- | :---: | :---: | | [`apis-guru`](https://apis.guru/) | subdomains, emails, urls | P0 | No | -| [`arquivo`](https://arquivo.pt/) | subdomains | P0 | No | +| [`arquivo`](https://arquivo.pt/) | subdomains only | P0 | No | | [`baidu`](https://www.baidu.com/) | subdomains, emails | P0 | No | | [`bevigil`](https://bevigil.com/osint-api) | subdomains, urls | P0 | Required | +| [`brave`](https://brave.com/search/api/) | subdomains, emails | P0 | Required | | [`bufferoverun`](https://tls.bufferover.run/) | subdomains, ips | P0 | Required | | [`builtwith`](https://builtwith.com/) | subdomains, urls | P0 | Required | -| [`brave`](https://brave.com/search/api/) | subdomains, emails | P0 | Required | | [`censys`](https://search.censys.io/) | subdomains, emails | P0 | Required | -| [`certspotter`](https://sslmate.com/certspotter/) | subdomains | P0 | No | -| [`commoncrawl`](https://commoncrawl.org/) | subdomains | P0 | No | +| [`certspotter`](https://sslmate.com/certspotter/) | subdomains only | P0 | No | +| [`commoncrawl`](https://commoncrawl.org/) | subdomains only | P0 | No | | [`criminalip`](https://www.criminalip.io/) | subdomains, ips, asns | P2 | Required | -| [`crt-name`](https://crt.name/) | subdomains | P0 | No | -| [`crtsh`](https://crt.sh/) | subdomains | P0 | No | +| [`crt-name`](https://crt.name/) | subdomains only | P0 | No | +| [`crtsh`](https://crt.sh/) | subdomains only | P0 | No | | [`dehashed`](https://dehashed.com/) | emails, ips | P0 | Required | -| [`dnsdb`](https://docs.domaintools.com/api/dnsdb/) | subdomains | P0 | Required | +| [`dnsdb`](https://docs.domaintools.com/api/dnsdb/) | subdomains only | P0 | Required | | [`dnsdumpster`](https://dnsdumpster.com/) | subdomains, ips | P0 | Required | | [`duckduckgo`](https://duckduckgo.com/) | subdomains, emails | P0 | No | -| [`dymo`](https://docs.tpeoficial.com/docs/dymo-api/private/data-verifier) | subdomains | P0 | Required | +| [`dymo`](https://docs.tpeoficial.com/docs/dymo-api/private/data-verifier) | subdomains only | P0 | Required | | [`fofa`](https://en.fofa.info/) | subdomains, ips | P0 | Required | -| [`fullhunt`](https://fullhunt.io/) | subdomains | P0 | Required | +| [`fullhunt`](https://fullhunt.io/) | subdomains only | P0 | Required | | [`github-code`](https://github.com/) | subdomains, emails | P0 | Required | | [`gitlab`](https://gitlab.com/) | subdomains, emails, urls | P0 | No | | [`hackertarget`](https://hackertarget.com/) | subdomains, ips | P0 | Optional | -| [`haveibeenpwned`](https://haveibeenpwned.com/) | breaches | P0 | No | +| [`haveibeenpwned`](https://haveibeenpwned.com/) | breaches only | P0 | No | | [`hibpverified`](https://haveibeenpwned.com/API/v3#BreachedDomain) | emails, breaches | P0 | Required | | [`hudsonrock`](https://www.hudsonrock.com/) | subdomains, emails, ips | P0 | No | | [`hunter`](https://hunter.io/) | subdomains, emails | P0 | Required | -| [`hunterhow`](https://hunter.how/) | subdomains | P0 | Required | +| [`hunterhow`](https://hunter.how/) | subdomains only | P0 | Required | | [`intelx`](https://intelx.io/) | subdomains, emails, urls | P0 | Required | -| [`leakix`](https://leakix.net/) | subdomains | P0 | Required | +| [`leakix`](https://leakix.net/) | subdomains only | P0 | Required | | [`leaklookup`](https://leak-lookup.com/) | emails, breaches | P0 | Required | | [`mojeek`](https://www.mojeek.com/services/search/web-search-api/) | subdomains, emails | P0 | Optional | -| [`netlas`](https://netlas.io/) | subdomains | P0 | Required | +| [`netlas`](https://netlas.io/) | subdomains only | P0 | Required | | [`onyphe`](https://www.onyphe.io/) | subdomains, ips, asns | P0 | Required | | [`otx`](https://otx.alienvault.com/) | subdomains, ips | P0 | No | | [`pentesttools`](https://pentest-tools.com/) | subdomains, ips | P1 | Required | -| [`projectdiscovery`](https://chaos.projectdiscovery.io/) | subdomains | P0 | Required | +| [`projectdiscovery`](https://chaos.projectdiscovery.io/) | subdomains only | P0 | Required | | [`rapiddns`](https://rapiddns.io/) | subdomains, ips | P0 | No | -| [`robtex`](https://www.robtex.com/) | ips | P0 | No | +| [`robtex`](https://www.robtex.com/) | ips only | P0 | No | | [`rocketreach`](https://rocketreach.co/) | emails, urls | P0 | Required | | [`securityscorecard`](https://securityscorecard.com/) | subdomains, ips | P0 | Required | | [`securityTrails`](https://securitytrails.com/) | subdomains, ips | P0 | Required | | [`sherlockeye`](https://sherlockeye.io/) | subdomains, emails, ips | P0 | Required | -| [`shodan`](https://www.shodan.io/) | subdomains | P1 | Required | +| [`shodan`](https://www.shodan.io/) | subdomains only | P1 | Required | +| [`shodanct`](https://ctl.shodan.io/) | subdomains only | P0 | No | | [`shodanInternetDB`](https://internetdb.shodan.io/) | subdomains, ips | P1 | No | -| [`shodanct`](https://ctl.shodan.io/) | subdomains | P0 | No | -| [`sourcegraph`](https://sourcegraph.com/search) | subdomains | P0 | No | -| [`subdomaincenter`](https://www.subdomain.center/) | subdomains | P0 | No | -| [`subdomainfinderc99`](https://subdomainfinder.c99.nl/) | subdomains | P1 | No | -| [`thc`](https://ip.thc.org/) | subdomains | P0 | No | +| [`sourcegraph`](https://sourcegraph.com/search) | subdomains only | P0 | No | +| [`subdomaincenter`](https://www.subdomain.center/) | subdomains only | P0 | No | +| [`subdomainfinderc99`](https://subdomainfinder.c99.nl/) | subdomains only | P1 | No | +| [`thc`](https://ip.thc.org/) | subdomains only | P0 | No | | [`tomba`](https://tomba.io/) | subdomains, emails | P0 | Required | | [`urlscan`](https://urlscan.io/) | subdomains, ips, asns, urls | P0 | No | -| [`virustotal`](https://www.virustotal.com/) | subdomains | P0 | Required | -| [`waybackarchive`](https://web.archive.org/) | subdomains | P0 | No | -| [`whoisxml`](https://subdomains.whoisxmlapi.com/) | subdomains | P0 | Required | +| [`virustotal`](https://www.virustotal.com/) | subdomains only | P0 | Required | +| [`waybackarchive`](https://web.archive.org/) | subdomains only | P0 | No | +| [`whoisxml`](https://subdomains.whoisxmlapi.com/) | subdomains only | P0 | Required | | [`windvane`](https://windvane.lichoin.com/) | subdomains, emails, ips | P0 | Optional | | [`yahoo`](https://www.yahoo.com/) | subdomains, emails | P0 | No | | [`zoomeye`](https://www.zoomeye.ai/) | subdomains, emails, ips, asns, urls | P0 | Required |
-Each source name links to its provider's site or documentation, where current plans, quotas, and terms are published. See [Configuration and API keys](docs/wiki/Configuration-and-API-Keys.md) for credential names and setup. Contributors can add a provider through the [module guide](docs/wiki/How-to-add-a-new-module.md); the source catalog remains the executable inventory. +Each source name links to its provider's site or documentation for current plans, quotas, and terms. See [Configuration and API keys](docs/wiki/Configuration-and-API-Keys.md) for the required fields and setup instructions. Contributors can add a provider through the [module guide](docs/wiki/How-to-add-a-new-module.md). The CLI and API read their source inventory from the source catalog. ## Configuration On first use, theHarvester creates default configuration files under `~/.theHarvester/`. It also reads system configuration from `/etc/theHarvester/` and `/usr/local/etc/theHarvester/`. -- `api-keys.yaml` stores provider credentials. +- `api-keys.yaml` stores provider API keys and related values such as organization IDs. - `proxies.yaml` configures HTTP and SOCKS5 proxies used with `-p`. - The `shodan` source and `-s` / `--shodan` enrichment use Shodan's Host REST API. When `-p` is enabled, both send those requests through `proxies.yaml`. - `routeviews.key` is optional and enables authenticated RouteViews access for PeeringDB-verified users. @@ -246,9 +248,16 @@ Treat collected OSINT as potentially sensitive. Keep report files, screenshots, JSONL is the primary format for automation and one-run interchange. The first line describes the run. Each remaining line is one sorted, deduplicated finding with its source and action provenance. +This example contains six findings from two sources. The `counts` object summarizes the result lines that follow it. + ```jsonl -{"action_executions":[],"artifacts":[],"completed_at":"2026-08-07T12:01:00Z","counts":{"hostname":1},"evidence_status":"complete","result_count":1,"run_id":"123e4567-e89b-12d3-a456-426614174000","source_executions":[],"started_at":"2026-08-07T12:00:00Z","target":"example.com","type":"summary"} -{"sources":[],"type":"hostname","value":"api.example.com"} +{"action_executions":[],"artifacts":[],"completed_at":"2026-08-17T12:01:00Z","counts":{"asn":1,"breach":1,"email":1,"hostname":1,"ip":1,"url":1},"evidence_status":"complete","result_count":6,"run_id":"123e4567-e89b-12d3-a456-426614174000","source_executions":[{"duration_ms":127.4,"error_type":null,"result_count":1,"source":"haveibeenpwned","status":"completed","stop_reason":null},{"duration_ms":482.3,"error_type":null,"result_count":5,"source":"zoomeye","status":"completed","stop_reason":null}],"started_at":"2026-08-17T12:00:00Z","target":"example.com","type":"summary"} +{"sources":["zoomeye"],"type":"asn","value":"AS64500"} +{"sources":["haveibeenpwned"],"type":"breach","value":"Example breach"} +{"sources":["zoomeye"],"type":"email","value":"security@example.com"} +{"sources":["zoomeye"],"type":"hostname","value":"api.example.com"} +{"sources":["zoomeye"],"type":"ip","value":"192.0.2.10"} +{"sources":["zoomeye"],"type":"url","value":"https://api.example.com/login"} ``` Extract common result types with `jq`: @@ -282,7 +291,7 @@ The `subdomains` capability produces `hostname` records because a result can be ### SQLite, JSON, and XML -CLI and API runs use the same SQLite evidence model. JSONL moves one run at a time. The API can import or export completed runs in bulk as a portable SQLite database while leaving queue and worker state behind. Screenshot files are managed separately from their metadata. +CLI and API runs use the same SQLite evidence model. JSONL moves one run at a time. SQLite import and export handle completed runs in bulk but exclude queue and worker state. Screenshot files remain separate from their metadata. JSON and XML are compatibility reports grouped by result type. They do not include the full provenance, lifecycle outcomes, or structured action evidence available in JSONL, SQLite, the API, and HarvestView. diff --git a/docs/images/harvestview-architecture.svg b/docs/images/harvestview-architecture.svg index a048f334..566eaf20 100644 --- a/docs/images/harvestview-architecture.svg +++ b/docs/images/harvestview-architecture.svg @@ -1,109 +1,158 @@ - - HarvestView run desk architecture - HarvestView validates local browser requests through the authenticated API, records one durable run, executes it in an isolated worker, and returns normalized evidence to the route workbench. - - - - - - - + + HarvestView run desk architecture + A loopback-only HarvestView workflow sends a reviewed plan through the authenticated REST API and durable run control to an isolated worker, preserves normalized evidence separately from lifecycle state, supports inspection and child actions, and imports or exports JSONL and SQLite evidence. + + + + + + - - HARVESTVIEW · LOCAL OPERATOR WORKFLOW - One run desk, one durable lifecycle - The browser reviews scope; the API validates; an isolated worker executes; evidence remains available after terminal state. + + HARVESTVIEW · LOCAL OPERATOR WORKFLOW + One run desk, one durable evidence path + The browser reviews scope; the API validates; an isolated worker executes; completed evidence remains portable. - - - LOCAL HOST · 127.0.0.1 + + + LOCALHOST · 127.0.0.1:5000 - - - - - - + + + + + + + - - - BROWSER - Create or import - new run · JSONL · SQLite - source and action review - P0 · P1 · P2 authorization + + + BROWSER + Plan and authorize + target · sources + actions · P0/P1/P2 + review before submit - - - API GATE - Authenticated REST API - derived HttpOnly browser session - X-API-Key for API clients - target · scope · options · limits + + + API GATE + Authenticated REST API + HttpOnly · X-API-Key + target · scope · limits + create · list + detail · cancel - - - CONTROL - Durable run control - queued → running → terminal - cancel request · source status - one record per finite run + + + EVIDENCE + Canonical evidence + results · provenance + source/action outcomes + artifacts · status + complete · partial · failed - - - WORKER - Isolated run worker - claim · checkpoint · cancel - bounded output · child cleanup + + + WORKBENCH + Inspect and act + history · assessment + route tabs · search + filters · copy · logs + screenshots · child runs - - - ENGINE - theHarvester engine - cataloged sources · explicit actions - DNS · Shodan · RouteViews · vhost + + + CONTROL + Durable run lifecycle + queued · running + cancelling · cancelled + completed · failed + leases · deadlines - - - EVIDENCE - Evidence & artifacts - CompletedResult · route records - SQLite · screenshots · logs - partial evidence remains inspectable + + + WORKER + Isolated run worker + claim · execute + checkpoint · cancel + 58 sources + 10 explicit actions - - - WORKBENCH - Inspect and export - history · assessment · route tabs - screenshots · source outcomes · logs - review actions · JSONL / SQLite export + + + INTERCHANGE + Import and export + JSONL · one run + SQLite · completed set + screenshot files separate - - - control and lifecycle handoff - - normalized evidence handoff - - durable evidence boundary - provider credentials remain server-side - + Provider credentials stay server-side. Lifecycle state never replaces terminal evidence status. + + + + run control + + evidence handoff + + import path + diff --git a/docs/images/run-evidence-architecture.svg b/docs/images/run-evidence-architecture.svg index 61da2c73..d4c0e6cf 100644 --- a/docs/images/run-evidence-architecture.svg +++ b/docs/images/run-evidence-architecture.svg @@ -1,137 +1,173 @@ - - Discovery routes and enrichment - theHarvester source selection and adapters feed passive, DNS, and direct actions before normalization into identity, contact, network, and action evidence routes. - - - - - - - + + theHarvester discovery and evidence architecture + Seven result capabilities and 58 cataloged discovery sources join ten explicit P0, P1, and P2 actions in one normalized evidence contract, which routes identity, network, and action findings to terminal, JSONL, SQLite, REST, HarvestView, screenshots, JSON, and XML outputs. + + + + + + - - DISCOVERY · ENRICHMENT · EVIDENCE - Sources become attributable evidence - Catalog policy selects the work; one normalized result preserves routes, producers, outcomes, and partial evidence. + + DISCOVERY · ENRICHMENT · ATTRIBUTABLE EVIDENCE + From source selection to durable evidence + Every selected source and action reports its own outcome before results are deduplicated, attributed, and exported. - - - COLLECT - - - ACTIONS - - - NORMALIZE & ROUTE + + + SELECT + EXECUTE + + + EVIDENCE ROUTES - - - - - - - - - + + + + + + + + - - - CATALOG - Source selection - subdomains · emails · IPs - ASNs · URLs · people - breaches · explicit names - activity · keys · routes + + + SCOPE + Scope and source + selection + 7 result capabilities + names · all · no-hosts - - - SOURCES - Discovery adapters - search · certificates · DNS - code · archives · threat intel - normalized observations - truthful source outcomes + + + SOURCES + 58 discovery adapters + search · CT · DNS data + code · archives + threat intelligence + activity · keys · outcomes - - - P0 - Passive enrichment - Shodan host detail - RouteViews routes · origins · RPKI + + + NORMALIZE + CompletedResult evidence contract + dedupe · provenance · source and action outcomes + complete · partial · failed evidence · artifacts - - - P1 - DNS interaction - resolve · reverse lookup - brute force · recursive discovery + + + P0 + Provider + enrichment + shodan + routeviews - - - P2 - Direct interaction - vhost · screenshots · takeover - API path discovery + + + P1 + DNS interaction + dns-resolve + dns-lookup + dns-brute + dns-recursive - - - NORMALIZED - CompletedResult - dedupe · provenance · outcomes - artifacts · evidence status - partial evidence stays attributable + + + P2 + Direct action + api-scan + screenshot + takeover + vhost - - - IDENTITY - Identity & contact routes - hostname / subdomain - email · URL - person · person-link · breach - vhost observations stay on hostname + + + IDENTITY + Identity and contact + subdomains · emails · urls + people · person links · breaches - - - NETWORK - Network & action routes - IP · ASN · prefix - Shodan host · takeover - scope extension - external relationship · other - JSONL · SQLite · REST · HarvestView + + + NETWORK + Network and action evidence + ips · asns · prefixes + Shodan hosts · takeover + recursive DNS · vhost evidence + scope extension · external relations - - - data handoff - - normalized evidence - - P1 · DNS interaction - - P2 · direct interaction - P0 sources and enrichments remain passive provider requests - \ No newline at end of file + + + OUTPUT + Durable outputs + Terminal · JSONL · SQLite · REST + HarvestView · screenshots · JSON/XML + + + + evidence flow + + P0 provider + + P1 DNS + + P2 direct + diff --git a/docs/images/vhost-classifier.svg b/docs/images/vhost-classifier.svg new file mode 100644 index 00000000..801121f1 --- /dev/null +++ b/docs/images/vhost-classifier.svg @@ -0,0 +1,120 @@ + + + Virtual-host response classifier + The classifier rejects unusable or unstable comparisons, treats baseline matches as default, accepts non-body differences as distinct, and requires body-only differences to repeat before retaining evidence. + + + + + + + + + + VHOST RESPONSE CLASSIFIER + A body-only difference must repeat + The candidate must differ from both the literal-IP context and a stable unknown-host control before evidence is retained. + + + + + + + + + + + + + + NO + + YES + + YES + + NO + + NO · DIRECT + + YES · BODY + + YES + + NO + + + + Candidate + baseline fingerprints + phase · status · location + body size · body digest + + + + All responses usable + and controls stable? + + + + Indeterminate + retain no finding + + + + Candidate matches + context or control? + + + + Default + retain no finding + + + + Only body signals differ + from either baseline? + + + + Repeat candidate request + + + + Usable body difference + repeats exactly? + + + + + DISTINCT + Retain hostname + evidence + status · location + phase + repeated body + difference + + + + distinct = stored + default / indeterminate = not stored + \ No newline at end of file diff --git a/docs/images/vhost-sweep-overview.svg b/docs/images/vhost-sweep-overview.svg new file mode 100644 index 00000000..66ef8c29 --- /dev/null +++ b/docs/images/vhost-sweep-overview.svg @@ -0,0 +1,80 @@ + + + Bounded virtual-host sweep + Candidate hostnames and literal IP endpoints pass through exact-scope validation and one shared request and runtime budget before context, control, and candidate responses are sent to the classifier. + + + + + + + + + + VIRTUAL HOST DISCOVERY · P2 DIRECT INTERACTION + The sweep spends one bounded budget + Names stay inside the exact target boundary; endpoints stay literal IPs; every baseline, candidate, and confirmation counts. + + + + + + ALLOCATES PROBES + + + + + CANDIDATES + In-scope hostname pool + harvested names · reverse DNS · explicit names + normalize, deduplicate, enforce exact target scope + + + + + ENDPOINTS + Literal-IP endpoint pool + harvested IPs or one operator override + HTTPS :443 across the set, then HTTP :80 + + + + + BOUNDS + Candidate–endpoint sweep planner + shared request cap · runtime deadline + per-request timeout · bounded concurrency + + + + + ONE ENDPOINT + Comparable response set + literal-IP context · 3 shape-matched controls + candidate with aligned TLS SNI + HTTP Host + + + + planning handoff + + shared safety budget + names are not resolved · redirects are not followed + \ No newline at end of file diff --git a/docs/wiki/Configuration-and-API-Keys.md b/docs/wiki/Configuration-and-API-Keys.md index 32f68ace..87b7d5b4 100644 --- a/docs/wiki/Configuration-and-API-Keys.md +++ b/docs/wiki/Configuration-and-API-Keys.md @@ -13,7 +13,7 @@ If no file exists, theHarvester creates the default template under `~/.theHarves Run theHarvester once to create the user configuration, then edit: ```bash -${EDITOR:-vi} ~/.theHarvester/api-keys.yaml +vi ~/.theHarvester/api-keys.yaml chmod 600 ~/.theHarvester/api-keys.yaml ``` @@ -43,11 +43,13 @@ Do not commit populated configuration files. Prefer provider credentials scoped The [README source matrix](https://github.com/laramies/theHarvester/blob/dev/README.md#discovery-sources) lists each source's result routes, activity class, and credential requirement. Every source name links to its provider's site or documentation for current plans, quotas, and terms. The executable source catalog is the authoritative inventory. -`censys.token` is a Censys Platform Personal Access Token. Set `organization_id` when searches should use an entitled organization. This source uses the Global Search API, which is unavailable to Free accounts because they are limited to asset lookups. Search API ID and secret fields are not accepted. +### Provider notes -`hibpverified` queries [HIBP's authenticated verified-domain endpoint](https://haveibeenpwned.com/API/v3#BreachedDomain). It is selected by its name, the `breaches` capability, and `all`. Without a configured HIBP API key it is skipped like other unavailable keyed sources. Live use requires a user-owned paid HIBP API key and a user-owned domain verified in that account. The keyless `haveibeenpwned` source queries only the public breach catalogue. - -`routeviews.key` is optional. RouteViews provides authenticated API keys to verified PeeringDB users. `--routeviews` uses the authenticated endpoint and documented 10-request-per-second allowance when the key is configured; otherwise it uses guest access at one request per second. If RouteViews rejects a configured key, the action fails without retrying as a guest; remove the key to select guest access. RouteViews does not document this as a paid subscription. +| Provider | Configuration and behavior | +| --- | --- | +| Censys | `censys.token` is a Censys Platform Personal Access Token. Set `organization_id` to search through an entitled organization. The source uses the Global Search API, which Free accounts cannot access because they are limited to asset lookups. Search API ID and secret fields are not accepted. | +| HIBP verified domains | `hibpverified` queries [HIBP's authenticated verified-domain endpoint](https://haveibeenpwned.com/API/v3#BreachedDomain). Select it by name, through the `breaches` capability, or with `all`. Without a configured key, it is skipped like other unavailable keyed sources. Live use requires a user-owned paid HIBP API key and a domain verified in that account. The keyless `haveibeenpwned` source queries only the public breach catalogue. | +| RouteViews | `routeviews.key` is optional. A configured key selects the authenticated endpoint for PeeringDB-verified users and its documented 10-request-per-second allowance. Without a key, the action uses guest access at one request per second. If RouteViews rejects a configured key, the action fails instead of retrying as a guest. Remove the key to select guest access. RouteViews does not document this as a paid subscription. | ## Proxies @@ -62,6 +64,8 @@ socks5: Enable configured proxies with `-p`: +Network activity: provider-facing passive lookup through a configured proxy. + ```bash uv run theHarvester -d example.com -b crtsh -p ``` diff --git a/docs/wiki/Home.md b/docs/wiki/Home.md index fe445d57..44c836db 100644 --- a/docs/wiki/Home.md +++ b/docs/wiki/Home.md @@ -9,17 +9,19 @@ DNS brute force, DNS resolution, virtual host discovery, takeover checks, screen ## Start here 1. [Install theHarvester](Installation). -2. Follow the [Quick Start](Quick-Start) for a small passive run. -3. Read [Responsible Use and Scope](Responsible-Use-and-Scope) before enabling active features. +2. Read [Responsible Use and Scope](Responsible-Use-and-Scope) before choosing a target or feature. +3. Follow the [Quick Start](Quick-Start) for a small passive run. 4. Add credentials through [Configuration and API Keys](Configuration-and-API-Keys) when a selected provider requires them. 5. Learn where findings are stored in [Results and Local Data](Results-and-Local-Data). ## Choose an interface -- **Command line:** best for interactive reconnaissance and report generation. -- **HarvestView:** best for creating and inspecting durable local enumeration runs in a browser. See [REST API](Rest-API). -- **REST API:** best for authenticated local integrations and Swagger/ReDoc documentation. See [REST API](Rest-API). -- **Docker Compose:** packages HarvestView and the REST API, not the normal interactive CLI. +| Interface | Use it for | +| --- | --- | +| Command line | Interactive reconnaissance and report generation. | +| HarvestView | Creating and inspecting durable local runs in a browser. See [REST API](Rest-API). | +| REST API | Authenticated local integrations and the generated Swagger/ReDoc reference. See [REST API](Rest-API). | +| Docker Compose | Running HarvestView and the REST API. It does not provide the normal interactive CLI. | The repository [README](https://github.com/laramies/theHarvester) owns the current feature summary and source/result matrix. The live `theHarvester -h` output owns the complete CLI reference. diff --git a/docs/wiki/How-to-add-a-new-module.md b/docs/wiki/How-to-add-a-new-module.md index 1965d89c..3ba28b65 100644 --- a/docs/wiki/How-to-add-a-new-module.md +++ b/docs/wiki/How-to-add-a-new-module.md @@ -25,7 +25,17 @@ An adapter normally provides: Do not return fields the provider did not supply. Normalize and deduplicate before returning results. -Return `None` when the provider conversation completed normally, including a valid zero-result response. Return an immutable `SourceExecutionReport` with a stable provider-specific reason for another terminal condition: `completed` for a successful early stop such as reaching the requested result limit, `failed` for provider or transport failure, `rate-limited` for a terminal rate limit, or `partial` when the provider confirms incomplete coverage. Adapters must not define mutable `execution_status` or `stop_reason` fields. The source runner checks for either field before execution and again before evidence collection. It owns finalization, promotes incomplete reports with retained normalized evidence to `partial`, and records a normal zero-result completion as `completed` with `no-results`. +Return one of these values from `process()`: + +| Return | Meaning | +| --- | --- | +| `None` | The provider conversation completed normally, including a valid zero-result response. | +| `SourceExecutionReport('completed', reason)` | The source stopped successfully before its natural end, such as after reaching the requested result limit. | +| `SourceExecutionReport('failed', reason)` | A provider or transport failure ended the source. | +| `SourceExecutionReport('rate-limited', reason)` | A terminal rate limit ended the source. | +| `SourceExecutionReport('partial', reason)` | The provider confirmed incomplete coverage. | + +Use a stable provider-specific reason. Do not define mutable `execution_status` or `stop_reason` fields. The source runner rejects either field before execution and before evidence collection. It owns finalization, promotes incomplete reports with retained normalized evidence to `partial`, and records a normal zero-result completion as `completed` with `no-results`. ### Own the provider conversation diff --git a/docs/wiki/Operator-Workflows.md b/docs/wiki/Operator-Workflows.md index eb3ef17d..00ff94da 100644 --- a/docs/wiki/Operator-Workflows.md +++ b/docs/wiki/Operator-Workflows.md @@ -2,8 +2,14 @@ Start with the smallest source set and least active behavior that can answer the engagement question. Replace `example.com` only with an authorized target. +Sources and explicit actions contribute different result routes and different levels of network activity. This map shows how they meet in one normalized evidence model: + +![theHarvester discovery routes and enrichment](https://raw.githubusercontent.com/laramies/theHarvester/dev/docs/images/run-evidence-architecture.svg) + ## Passive subdomain discovery +Network activity: provider-facing passive lookups. + ```bash uv run theHarvester -d example.com -b crtsh,certspotter,commoncrawl ``` @@ -12,6 +18,8 @@ Use the [README source matrix](https://github.com/laramies/theHarvester/blob/dev ## Save results for automation +Network activity: provider-facing discovery plus local report writes. + ```bash uv run theHarvester -d example.com -b crtsh,certspotter -f report ``` @@ -20,6 +28,8 @@ Use `report.jsonl` for automation, provenance, and run interchange. The same com ## DNS resolution +Network activity: provider-facing discovery followed by resolver-facing DNS queries. + ```bash AUTHORIZED_DOMAIN='replace-with-a-domain-you-control' uv run theHarvester -d "$AUTHORIZED_DOMAIN" -b crtsh,certspotter -r @@ -40,6 +50,8 @@ Reverse DNS (`-n`) uses an independent run-wide job set. It deduplicates address Configure the Shodan key, then enrich resolved hosts: +Network activity: provider-facing discovery, resolver-facing DNS, and Shodan API requests. + ```bash AUTHORIZED_DOMAIN='replace-with-a-domain-you-control' uv run theHarvester -d "$AUTHORIZED_DOMAIN" -b crtsh -r -s @@ -51,6 +63,8 @@ Shodan host enrichment runs after discovery and is separate from the `shodan` so Use only an owned or explicitly authorized target: +Network activity: resolver-facing DNS queries for generated candidate names. + ```bash AUTHORIZED_DOMAIN='replace-with-a-domain-you-control' uv run theHarvester -d "$AUTHORIZED_DOMAIN" -c @@ -60,6 +74,8 @@ DNS brute force actively tests candidate names. Do not run it against `example.c ## Takeover checks +Network activity: provider-facing discovery followed by target-facing checks. + ```bash AUTHORIZED_DOMAIN='replace-with-a-domain-you-control' uv run theHarvester -d "$AUTHORIZED_DOMAIN" -b crtsh,certspotter -t @@ -67,10 +83,36 @@ uv run theHarvester -d "$AUTHORIZED_DOMAIN" -b crtsh,certspotter -t Treat matches as leads requiring manual confirmation. Do not claim a takeover from a fingerprint match alone. +## RouteViews pivots + +Network activity: provider-facing RouteViews requests. + +Set an ASN that is part of the authorized assessment scope: + +```bash +AUTHORIZED_ASN='replace-with-an-authorized-asn' +uv run theHarvester -d "$AUTHORIZED_ASN" --routeviews -f report +``` + +The target may also be an authorized IP or CIDR. The action writes external routing relationships and RPKI observations to `report.jsonl`. These records can guide analysis, but they do not establish ownership, authorization, or reachability. RouteViews is never enabled by `-b all`; see [Responsible Use and Scope](Responsible-Use-and-Scope#routeviews) for its fixed limits. + +## Virtual host discovery + +Network activity: provider-facing discovery followed by target-facing requests to harvested IP endpoints. + +```bash +AUTHORIZED_DOMAIN='replace-with-a-domain-you-control' +uv run theHarvester -d "$AUTHORIZED_DOMAIN" -b rapiddns --vhost -f report +``` + +The action keeps only confirmed hostnames and records endpoint observations in JSONL. Read [Virtual Host Discovery](Virtual-Host-Discovery) before changing its endpoint, candidate, request, runtime, or TLS controls. + ## Screenshots Install Chromium first, then choose an output directory: +Network activity: provider-facing discovery, resolver-facing DNS, and target-facing browser requests. + ```bash uv run playwright install chromium AUTHORIZED_DOMAIN='replace-with-a-domain-you-control' @@ -81,6 +123,8 @@ Screenshots actively open discovered web services and may retain sensitive page ## API-path scanning +Network activity: target-facing HTTP requests. + ```bash AUTHORIZED_DOMAIN='replace-with-a-domain-you-control' uv run theHarvester -d "$AUTHORIZED_DOMAIN" -a @@ -92,6 +136,8 @@ Provide a custom path wordlist with `-w FILE`. This sends requests directly to t When a combined run fails, rerun only the affected source with a conservative result limit: +Network activity: provider-facing lookup to the named source. + ```bash uv run theHarvester -d example.com -b source-name -l 10 ``` diff --git a/docs/wiki/Quick-Start.md b/docs/wiki/Quick-Start.md index 8dc78e00..119d699c 100644 --- a/docs/wiki/Quick-Start.md +++ b/docs/wiki/Quick-Start.md @@ -1,11 +1,13 @@ # Quick start -These examples use the IANA-reserved `example.com` domain as inert test data. Replace it only with a target that is within your authorized scope. +These examples use the IANA-reserved `example.com` domain to show command syntax. Passive providers still receive the target string. Replace it only with a target that is within your authorized scope. ## Run a small passive query From a source checkout: +Network activity: provider-facing passive lookups. + ```bash uv run theHarvester -d example.com -b crtsh,certspotter ``` @@ -16,20 +18,24 @@ From Kali or another installed package, omit `uv run`: theHarvester -d example.com -b crtsh,certspotter ``` -This queries two passive certificate sources and prints consolidated findings. Passive does not mean private: the selected providers receive the target string. +This queries two passive certificate sources and prints consolidated findings. An empty result can mean that the providers found nothing; it does not by itself prove that the run failed. ## Save a report +Network activity: provider-facing passive lookups plus local report writes. + ```bash uv run theHarvester -d example.com -b crtsh,certspotter -f report ``` -This writes `report.jsonl` for automation and interchange. It also writes `report.json` and `report.xml` compatibility reports. See [Results and Local Data](Results-and-Local-Data). +This writes `report.jsonl` for automation and interchange. It also writes `report.json` and `report.xml` compatibility reports. The JSONL summary records source outcomes and evidence status, which distinguish a normal empty result from incomplete or failed work. See [Results and Local Data](Results-and-Local-Data). ## Resolve discovered hosts DNS resolution creates additional network activity. Use it only within scope: +Network activity: provider-facing discovery followed by resolver-facing DNS queries. + ```bash AUTHORIZED_DOMAIN='replace-with-a-domain-you-control' uv run theHarvester -d "$AUTHORIZED_DOMAIN" -b crtsh,certspotter -r @@ -44,10 +50,6 @@ uv run theHarvester -d "$AUTHORIZED_DOMAIN" -b crtsh -r resolvers.txt ## Choose sources deliberately -The [README source matrix](https://github.com/laramies/theHarvester/blob/dev/README.md#discovery-sources) shows the result types and credential requirements for every current source. - -Do not start with `-b all`. It contacts many independent services and can consume quotas. It also increases runtime and makes provider failures harder to isolate. - -Choose a small group of sources that provides the result types you need. - -Use `theHarvester -h` for the current option and source list. +1. Use the [README source matrix](https://github.com/laramies/theHarvester/blob/dev/README.md#discovery-sources) to find sources that return the result types you need. +2. Start with a small source set. `-b all` contacts many independent services, can consume quotas, and makes provider failures harder to isolate. +3. Use `theHarvester -h` for the current option and source list. diff --git a/docs/wiki/Responsible-Use-and-Scope.md b/docs/wiki/Responsible-Use-and-Scope.md index cf01c5df..92505980 100644 --- a/docs/wiki/Responsible-Use-and-Scope.md +++ b/docs/wiki/Responsible-Use-and-Scope.md @@ -10,23 +10,36 @@ Select only the providers needed for the task. Do not treat a provider key, bug- ## Features that add network activity -The following options require additional care: +| Option | Network path | What it does | +| --- | --- | --- | +| `-r`, `--dns-resolve` | Resolver-facing | Resolves discovered names to A, AAAA, and CNAME records. | +| `-n`, `--dns-lookup` | Resolver-facing | Performs reverse DNS across discovered `/24` ranges. | +| `-c`, `--dns-brute` | Resolver-facing | Tries candidate subdomains against DNS. | +| `-t`, `--take-over` | Target-facing | Checks discovered hosts for takeover indicators. | +| `-s`, `--shodan` | Provider-facing | Enriches discovered hosts through Shodan. | +| `--routeviews` | Provider-facing | Queries RouteViews for external routing relationships. | +| `--vhost`, `--vhost-*` | Target-facing | Probes literal IP endpoints with candidate SNI and HTTP `Host` values. | +| `--screenshot DIR` | Target-facing | Opens discovered web services in a browser. | +| `-a`, `--api-scan` | Target-facing | Requests common API paths from the target. | -| Option | Behavior | -| --- | --- | -| `-r`, `--dns-resolve` | Deduplicates discovered names, then resolves A, AAAA, and CNAME records once per name through one run-wide phase with at most 20 active hostname jobs and per-query resolver timeouts; it has no default query-count or phase-runtime ceiling. | -| `-n`, `--dns-lookup` | Deduplicates addresses across discovered `/24` ranges, then performs reverse DNS through a separate run-wide phase with at most 20 active PTR jobs and per-query resolver timeouts; it has no default request-count or phase-runtime ceiling. | -| `-c`, `--dns-brute` | Tries candidate subdomains against DNS. | -| `-t`, `--take-over` | Checks discovered hosts for takeover indicators. | -| `-s`, `--shodan` | Enriches discovered hosts through Shodan. | -| `--routeviews` | Sends discovered IPs with sourced ASN attribution, or an explicitly targeted ASN, IP, or CIDR, to RouteViews and records external routing relationships. | -| `--vhost`, `--vhost-*` | Probes literal IP endpoints with candidate SNI and HTTP `Host` values. | -| `--screenshot DIR` | Opens discovered web services in a browser. | -| `-a`, `--api-scan` | Requests common API paths from the target. | +### DNS actions + +Hostname resolution deduplicates discovered names, then queries A, AAAA, and CNAME once per name. One run can have up to 20 active hostname jobs. Queries have resolver timeouts, but the phase has no default query-count or runtime ceiling. + +Reverse DNS deduplicates addresses across overlapping `/24` ranges. It uses a separate run-wide phase with up to 20 active PTR jobs and per-query timeouts, but no default request-count or runtime ceiling. Use `--dns-resolvers IPS_OR_FILE` to select resolver addresses for DNS brute force, reverse lookup, or recursive DNS without enabling hostname resolution. `--dns-resolve [IPS_OR_FILE]` selects resolvers and enables hostname resolution. -`--routeviews` is a separately selected P0 provider action. It is never enabled by `-b all` and ignores `-l`; one run is internally bounded to 300 sequential requests and 300 seconds. With no configured key it uses the documented guest rate of one request per second. A configured `routeviews.key` selects PeeringDB-verified authenticated access and the documented 10-request-per-second allowance. When selected for a domain run, it automatically queries harvested IPs backed by sourced IP-to-ASN attribution. It also accepts an ASN, IP, or CIDR supplied as the run target. Harvested IPs without that attribution are not sent, and bare ASN findings are not expanded into complete prefix inventories. IP lookups retain only the most-specific returned prefix, including all origins for a multi-origin prefix. The action does not recursively query returned prefixes and never promotes returned CIDRs into DNS or direct-action scope. Cloud and CDN routes can be useful relationship evidence, but route origins and RPKI states do not establish ownership or authorization. +### RouteViews + +`--routeviews` is a separately selected P0 provider action. `-b all` never enables it, and `-l` does not change its fixed limit of 300 sequential requests or 300 seconds. + +- Guest access runs at the documented rate of one request per second. A configured `routeviews.key` selects PeeringDB-verified authenticated access and the documented 10-request-per-second allowance. +- A domain run queries only harvested IPs backed by sourced IP-to-ASN attribution. Harvested IPs without that attribution are not sent, and bare ASN findings are not expanded into complete prefix inventories. +- An explicit run target may be an ASN, IP, or CIDR. IP lookups retain only the most-specific returned prefix, including every origin for a multi-origin prefix. +- Returned prefixes are not queried recursively or promoted into DNS or direct-action scope. + +Treat cloud and CDN routes as relationship evidence. Route origins and RPKI states do not establish ownership, authorization, or reachability. Use an owned or explicitly authorized domain for active examples. Do not substitute universities, public companies, bounty targets, or reserved example domains for recurring active scans. diff --git a/docs/wiki/Rest-API.md b/docs/wiki/Rest-API.md index 27d85052..6393858f 100644 --- a/docs/wiki/Rest-API.md +++ b/docs/wiki/Rest-API.md @@ -2,10 +2,14 @@ `harvestview` serves the web application at `/` and one versioned API for local automation. +![HarvestView run desk architecture](https://raw.githubusercontent.com/laramies/theHarvester/dev/docs/images/harvestview-architecture.svg) + ## Start the service Set a long random API key before startup: +Network activity: local-only with the default loopback binding. + ```bash export THEHARVESTER_API_KEY='replace-with-a-long-random-value' uv run harvestview @@ -58,6 +62,8 @@ Provider credentials remain in theHarvester's server-side configuration. Request Source names and capability selectors share the `sources` array. Multiple capabilities select the union of matching sources and do not filter fields returned by those sources. +Network activity: the API request is local, but the queued run contacts the selected providers and any enabled action targets. + ```bash run_id="$(curl -s http://127.0.0.1:5000/api/v1/runs \ -X POST \ @@ -77,13 +83,26 @@ curl -s "http://127.0.0.1:5000/api/v1/runs/$run_id" \ | jq '{status, evidence_status, results, source_executions, action_executions, artifacts}' ``` -Run submission is asynchronous. Lifecycle status is `queued`, `running`, `cancelling`, `cancelled`, `completed`, or `failed`. Terminal evidence status is reported separately as `complete`, `partial`, or `failed` when evidence exists. +Run submission is asynchronous. Read the two status fields separately: + +| Field | Values | Meaning | +| --- | --- | --- | +| `status` | `queued`, `running`, `cancelling`, `cancelled`, `completed`, `failed` | Worker lifecycle state. | +| `evidence_status` | `complete`, `partial`, `failed` | Quality of terminal evidence when it exists. | `source_workers` is the same positive concurrency used by CLI `-j` or `--source-workers` and HarvestView. It defaults to three, is reduced when fewer sources are selected, and never skips sources or limits their results. P1 DNS and P2 direct options are fields on the same run request. The OpenAPI schema shows their current defaults, limits, and descriptions. The server uses the operator-selected target and does not impose a public-only egress policy. -RouteViews is the explicit P0 `routeviews` action. When selected for a domain run, it automatically enriches harvested IPs that have sourced IP-to-ASN attribution. It also accepts an AS-prefixed ASN or IP address supplied as the run target. Harvested IPs without that attribution are not sent, and bare ASN findings are not expanded into complete prefix inventories. For IP pivots, only the most-specific matching prefix is retained, including every origin when that prefix is multi-origin. An explicit ASN target still requests its complete prefix inventory. The fixed internal budget is 300 sequential requests and 300 seconds; `limit` does not change it. A server-side `routeviews.key` is used automatically for PeeringDB-verified authenticated access at the documented 10-request-per-second allowance; otherwise the action uses guest access at one request per second. Provider credentials cannot be supplied in a run request. Returned prefixes remain external relationships and are never scheduled as DNS or P2 targets. The CLI also accepts a literal CIDR target. +### Query RouteViews + +RouteViews is the explicit P0 `routeviews` action. + +- A domain run enriches only harvested IPs that have sourced IP-to-ASN attribution. Harvested IPs without that attribution are not sent, and bare ASN findings are not expanded into complete prefix inventories. +- A run may target an AS-prefixed ASN or IP address. The CLI also accepts a literal CIDR. An IP pivot keeps the most-specific matching prefix and every origin for a multi-origin prefix. An explicit ASN target requests its complete prefix inventory. +- The fixed budget is 300 sequential requests and 300 seconds. `limit` does not change it. +- A server-side `routeviews.key` selects PeeringDB-verified authenticated access at the documented 10-request-per-second allowance. Without a key, the action uses guest access at one request per second. Requests cannot supply provider credentials. +- Returned prefixes remain external relationships. They are never scheduled as DNS or P2 targets. ```json { @@ -97,13 +116,15 @@ RouteViews is the explicit P0 `routeviews` action. When selected for a domain ru Screenshots and DNS brute force can run directly against an authorized hostname without repeating discovery. Submit an empty `sources` array and select one action: +Network activity: target-facing for screenshots and resolver-facing for DNS brute force. The API request itself is local. + ```bash curl -s http://127.0.0.1:5000/api/v1/runs \ -X POST \ -H "X-API-Key: $THEHARVESTER_API_KEY" \ -H 'Content-Type: application/json' \ -d '{ - "target": "subdomain.example.com", + "target": "replace-with-an-authorized-hostname", "sources": [], "screenshot": true }' \ @@ -116,7 +137,7 @@ The action catalog and run request use the same names. For example, set `takeove ```json { - "target": "api.example.com", + "target": "replace-with-an-authorized-hostname", "sources": [], "api_scan": true, "api_scan_paths": ["/api/v2", "/health"] @@ -127,6 +148,8 @@ Every custom API scan entry must be a URL path beginning with `/`. The API does Virtual host discovery is the `vhost` action. An action-only run must provide both a literal-IP endpoint and at least one in-scope hostname candidate: +Network activity: target-facing literal-IP requests with the candidate in SNI and HTTP `Host`. + ```json { "target": "authorized.example", @@ -142,7 +165,7 @@ HarvestView's subdomain action buttons call this route and create a separate run ## Import and export -Import records existing evidence and never contacts the target. For one run, send the same JSONL written by `theHarvester -f NAME`: +These operations are local-only. Import records existing evidence without contacting providers or targets. For one run, send the same JSONL written by `theHarvester -f NAME`: ```bash curl -s "http://127.0.0.1:5000/api/v1/runs/import?filename=report.jsonl" \ @@ -168,7 +191,13 @@ curl -s "http://127.0.0.1:5000/api/v1/runs/import-database?filename=stash.sqlite | jq ``` -The server checks the SQLite header, integrity, schema, and each completed run before copying it. Original run IDs are preserved. Exact duplicates are skipped, while a reused ID with different evidence is rejected. Close the source process or checkpoint its WAL before uploading the database. Screenshot metadata is imported, but screenshot files must be copied separately. The default upload ceiling is 1 GiB and can be changed with `THEHARVESTER_MAX_DATABASE_IMPORT_BYTES`. +Before importing SQLite: + +- Close the source process or checkpoint its WAL. +- Expect the server to check the SQLite header, integrity, schema, and each completed run before copying it. +- Original run IDs are preserved. Exact duplicates are skipped; a reused ID with different evidence is rejected. +- Screenshot metadata is imported, but screenshot files must be copied separately. +- The default upload ceiling is 1 GiB. Change it with `THEHARVESTER_MAX_DATABASE_IMPORT_BYTES`. Export every completed run as a consistent database that can be imported elsewhere: @@ -178,7 +207,7 @@ curl -s "http://127.0.0.1:5000/api/v1/runs/export-database" \ -o theharvester-completed-runs.sqlite ``` -The export is rebuilt from canonical completed evidence, so it excludes queue state, cancellation state, worker leases, and legacy observations. It includes screenshot metadata but not screenshot files. The server checkpoints and closes the temporary database before download; no manual WAL handling is required. +The export contains canonical completed evidence and screenshot metadata. It excludes queue state, cancellation state, worker leases, and legacy observations. Screenshot files are also excluded. The server checkpoints and closes the temporary database before download, so no manual WAL handling is required. Export one normalized result set in the same streamable format: diff --git a/docs/wiki/Results-and-Local-Data.md b/docs/wiki/Results-and-Local-Data.md index e8f36aec..9844660b 100644 --- a/docs/wiki/Results-and-Local-Data.md +++ b/docs/wiki/Results-and-Local-Data.md @@ -1,6 +1,14 @@ # Results and local data -theHarvester can print findings, write reports, retain selected records in SQLite, save screenshots, and expose durable run records through the API. These outputs have different schemas and sensitivity. +Choose the output that matches the next task: + +| Output | Use it for | Important limit | +| --- | --- | --- | +| Terminal | Interactive review | Not a stable automation interface. | +| JSONL | Automation, one-run interchange, and provenance | One summary record followed by normalized findings. | +| SQLite | Local history and bulk transfer of completed runs | Contains sensitive evidence across runs. | +| JSON or XML | Compatibility with older consumers | Does not preserve per-item source attribution. | +| REST API | Lifecycle state, normalized results, and local integrations | Requires API authentication. | ## Terminal output @@ -10,6 +18,8 @@ The CLI groups findings by result type. It can also print separate enrichment, s Use `-f NAME` to write a durable run report: +Network activity: provider-facing passive discovery plus local report writes. + ```bash uv run theHarvester -d example.com -b crtsh,certspotter -f report ``` @@ -46,28 +56,28 @@ The database persists across runs. Account for it in engagement cleanup and rete Completed CLI executions store one normalized terminal record keyed by run UUID. API executions use the same database by default and may override its path with `THEHARVESTER_RUN_DB`. Lifecycle rows keep queue, cancellation, and worker state separate from terminal evidence. Imported JSONL is stored without executing discovery, and source attribution is rebuilt from each finding's `sources` array. A SQLite import copies every completed run after validating the database and keeps the original run IDs. -The normalized persistence model can represent active-action provenance and artifact metadata through six core tables: +Six tables hold completed evidence: -- `runs`: one finite enumeration run; -- `executions`: each passive source or active action represented by the model; -- `results`: deduplicated hostnames, IPs, emails, URLs, and structured outputs; -- `result_origins`: which execution produced each result; -- `asn_attributions`: sourced organization labels linking an ASN result to the exact hostname or IP result supplied by the same execution; and +- `runs`: one finite enumeration run. +- `executions`: each passive source or active action represented by the model. +- `results`: deduplicated hostnames, IPs, emails, URLs, and structured outputs. +- `result_origins`: the execution that produced each result. +- `asn_attributions`: sourced organization labels linking an ASN result to the exact hostname or IP supplied by the same execution. - `artifacts`: files such as screenshots, linked to their creating action and subject result. -Virtual-host evidence stays inside this model. `results` holds one `hostname` row, `result_origins` links it to the `vhost` action execution, and the result's `details_json` contains the canonical endpoint observation array. If one hostname is distinct on several IP endpoints, it remains one result with several observations. +Virtual-host evidence uses the same model. `results` holds one `hostname` row, `result_origins` links it to the `vhost` action execution, and `details_json` contains the endpoint observations. A hostname that is distinct on several IP endpoints remains one result with several observations. Runtime collection records passive source executions plus DNS, takeover, Shodan, and API endpoint scan executions and origins. Screenshot actions attach file metadata to their captured hostname or URL without creating screenshot findings. -RouteViews creates `prefix` results with `scope: external-relationship` and `routeviews` action provenance. Native observations distinguish one ASN-prefix origin claim, one collector/peer BGP route, and one RPKI validation state. They are routing evidence, not registration, ownership, authorization, reachability, or expanded target scope. +RouteViews creates `prefix` results with `scope: external-relationship` and `routeviews` action provenance. Its observations distinguish ASN-prefix origin claims, collector/peer BGP routes, and RPKI validation states. Treat them as routing evidence, not registration, ownership, authorization, reachability, or expanded target scope. -URLScan, ONYPHE, and Shodan can attach a provider organization label to an ASN. SQLite stores each relationship in `asn_attributions`; JSONL, the API, CLI output, and HarvestView project the same typed observation. Labels remain time-bound provider evidence, so missing or conflicting values are retained rather than replaced by one ASN owner property. Shodan's documented `org` field is used for the organization label; its separate `isp` field remains part of the existing Shodan payload and is not treated as equivalent. +URLScan, ONYPHE, and Shodan can attach a provider organization label to an ASN. SQLite stores each relationship in `asn_attributions`; JSONL, the API, CLI output, and HarvestView expose the same typed observation. These labels are time-bound provider evidence. Missing or conflicting values remain separate instead of being replaced by one ASN owner property. Shodan's documented `org` field supplies the organization label; its `isp` field remains part of the Shodan payload and is not treated as equivalent. Every discovered URL is stored as the `url` result kind. Its source or action origins identify whether it came from BuiltWith, GitLab, RocketReach, API scanning, or another producer; provider-specific URL kinds are not stored. Hostname and IP evidence use the `hostname` and `ip` result kinds in SQLite, JSONL, the API, and HarvestView. A hostname may be the authorized target itself or a subordinate name, so the result kind does not claim that every value is a subdomain. -Two operational tables support the API without changing those six evidence concepts: `run_records` stores queue and lifecycle state, and `run_worker_leases` prevents two local workers from claiming the same queue. Runless rows are stored in `legacy_observations`. SQLite upgrades supported schemas automatically during normal initialization. +Two operational tables support the API: `run_records` stores queue and lifecycle state, and `run_worker_leases` prevents two local workers from claiming the same queue. Runless rows are stored in `legacy_observations`. SQLite upgrades supported schemas during normal initialization. ## Screenshots diff --git a/docs/wiki/Troubleshooting.md b/docs/wiki/Troubleshooting.md index c92ec1f9..e69e9650 100644 --- a/docs/wiki/Troubleshooting.md +++ b/docs/wiki/Troubleshooting.md @@ -34,20 +34,25 @@ The first existing file wins. Check the [README source matrix](https://github.com/laramies/theHarvester/blob/dev/README.md#discovery-sources). Configure credentials for that provider or choose a keyless source. `-q` suppresses missing-key notices; it does not make keyed providers work without credentials. -## Provider errors, timeouts, or empty results +## A provider fails, times out, or returns no results -Rerun one source with a small limit: +Rerun one source with a small limit and save its summary: + +Network activity: provider-facing lookup plus local report writes. ```bash -uv run theHarvester -d example.com -b source-name -l 10 +uv run theHarvester -d example.com -b source-name -l 10 -f diagnostic +jq -c 'select(.type == "summary") | {evidence_status, source_executions}' diagnostic.jsonl ``` -Then check: +Read the source outcome before treating an empty result as a failure. A completed `no-results` outcome means the provider conversation finished normally. A `partial`, `failed`, or `rate-limited` outcome names a separate coverage problem. + +If the source did not complete normally, check: - provider status and current API documentation; - credential validity and subscription access; - provider rate limits or temporary blocking of shared CI/cloud addresses; -- whether the source legitimately has no results for the target. +- whether the provider documents the target or query type as supported. Do not post credentials, private targets, account details, or raw provider responses in a public issue. @@ -62,6 +67,8 @@ uv run theHarvester -d "$AUTHORIZED_DOMAIN" -b crtsh -r resolvers.txt If theHarvester reports invalid resolvers, remove hostnames, comments, blank values, or `host:port` entries; the resolver list accepts IP addresses only. Check local firewall and DNS policy before substituting public resolvers. +The input is accepted when the invalid-resolver message is gone. A valid resolver list does not guarantee that a name has DNS records. + ## Screenshots and Chromium Install the browser used by Playwright: @@ -87,6 +94,8 @@ Then open [http://127.0.0.1:5000/docs](http://127.0.0.1:5000/docs). - `429`: a reverse proxy or remote provider applied its own rate limit. `harvestview` has no built-in request limiter. - `503` when creating a run: the execution worker is disabled or unavailable. +After correcting the cause, retry `GET /api/v1/sources`. A successful authenticated response returns the source catalog. + ## Docker ```bash diff --git a/docs/wiki/Virtual-Host-Discovery.md b/docs/wiki/Virtual-Host-Discovery.md index b8c2f309..16693957 100644 --- a/docs/wiki/Virtual-Host-Discovery.md +++ b/docs/wiki/Virtual-Host-Discovery.md @@ -4,6 +4,37 @@ A web server can host several sites on one IP address. The site selected by the This is a P2 direct action. It sends requests to harvested IP addresses or to one literal-IP endpoint supplied by the operator. Use it only when the target, addresses, ports, and technique are covered by the assessment authorization. +## Run a bounded sweep + +Network activity: provider-facing discovery followed by target-facing requests to harvested IP endpoints. + +Harvest hostnames and IP addresses, then run the sweep with its bounded defaults: + +```bash +AUTHORIZED_DOMAIN='replace-with-a-domain-you-control' + +uv run theHarvester \ + -d "$AUTHORIZED_DOMAIN" \ + -b rapiddns \ + --vhost \ + -f report +``` + +`rapiddns` is used here because it can return both hostnames and literal IP addresses. If the selected sources return only hostnames, enable a DNS action that contributes IP evidence or supply `--vhost-endpoint`. + +Repeat `--vhost-candidate` to add names that are already authorized but were not returned by the selected sources: + +```bash +uv run theHarvester \ + -d "$AUTHORIZED_DOMAIN" \ + -b rapiddns \ + --vhost \ + --vhost-candidate "admin.${AUTHORIZED_DOMAIN}" \ + --vhost-candidate "preview.${AUTHORIZED_DOMAIN}" +``` + +Virtual host discovery uses direct transport and rejects `--proxies`. HTTPS certificate verification is enabled by default. `--vhost-insecure` disables verification and records `tls_verified: false` in the evidence. Use it only when the engagement requires unverified TLS and the endpoint is authorized. + ## What gets probed `--vhost` uses the run's collected evidence: @@ -21,6 +52,8 @@ The shared request cap may stop the sweep before it reaches every endpoint. When An explicit endpoint replaces the harvested endpoint set: +Network activity: target-facing requests to the literal IP endpoint. + ```bash AUTHORIZED_DOMAIN='replace-with-a-domain-you-control' AUTHORIZED_IP='replace-with-an-authorized-ip' @@ -42,28 +75,13 @@ For each endpoint, theHarvester makes a literal-IP context request and at least HTTPS candidate and control requests send the same hostname in TLS SNI and the HTTP `Host` header. HTTP requests send the hostname only in `Host`. Redirects are recorded as evidence but are not followed. -```mermaid -flowchart TD - A[Collect hostnames and literal IPs] --> B[Keep candidates inside the exact target scope] - A --> C[Build HTTPS endpoints, then HTTP endpoints] - B --> D[Select the next endpoint and candidates that fit the shared cap] - C --> D - D --> E[Request the literal-IP context] - D --> F[Request three unknown controls for each candidate shape] - D --> G[Request the candidate with aligned SNI and Host] - E --> H{Are both baselines usable and controls stable?} - F --> H - G --> H - H -->|No| I[Indeterminate] - H -->|Yes| J{Does the candidate match either baseline?} - J -->|Yes| K[Default] - J -->|No, status or redirect differs| L[Distinct] - J -->|Only the body differs| M[Repeat the candidate request] - M --> N{Does the body difference repeat?} - N -->|Yes| L - N -->|No| I - L --> O[Store structured virtual host evidence] -``` +The sweep first turns harvested evidence and any operator override into one bounded set of comparable requests: + +![Bounded virtual-host sweep](https://raw.githubusercontent.com/laramies/theHarvester/dev/docs/images/vhost-sweep-overview.svg) + +The response set then enters a separate classifier. A status, redirect, or request-phase difference can be accepted immediately. A body-only difference must repeat before it becomes a finding: + +![Virtual-host response classifier](https://raw.githubusercontent.com/laramies/theHarvester/dev/docs/images/vhost-classifier.svg) Before comparison, the classifier replaces an exact reflection of the current authority in the response body or `Location` header. A generic error page that merely repeats the requested hostname should not become a discovery. @@ -123,36 +141,7 @@ Its `stop_reason` explains the outcome in more detail: Cancellation requested by the operator is different from a runtime limit. It propagates through the worker lifecycle and closes active connections. -## Basic CLI use - -Harvest hostnames and IP addresses, then run the bounded sweep: - -```bash -AUTHORIZED_DOMAIN='replace-with-a-domain-you-control' - -uv run theHarvester \ - -d "$AUTHORIZED_DOMAIN" \ - -b rapiddns \ - --vhost \ - -f report -``` - -`rapiddns` is used here because it can return both hostnames and literal IP addresses. If the selected sources return only hostnames, enable a DNS action that contributes IP evidence or supply `--vhost-endpoint`. - -Repeat `--vhost-candidate` to add names that are already authorized but were not returned by the selected sources: - -```bash -uv run theHarvester \ - -d "$AUTHORIZED_DOMAIN" \ - -b rapiddns \ - --vhost \ - --vhost-candidate "admin.${AUTHORIZED_DOMAIN}" \ - --vhost-candidate "preview.${AUTHORIZED_DOMAIN}" -``` - -Virtual host discovery uses direct transport. It rejects `--proxies`. HTTPS certificate verification is enabled by default. `--vhost-insecure` disables verification and records `tls_verified: false` in the evidence. Use that option only when the engagement requires it and the endpoint is authorized. - -### Reading terminal output +## Read terminal output The summary reports confirmed endpoint observations and coverage: @@ -170,6 +159,8 @@ HarvestView exposes a virtual host discovery checkbox plus optional endpoint and The same run can be submitted to `POST /api/v1/runs`: +Network activity: the API request is local, but the queued run performs provider-facing discovery and target-facing virtual host requests. + ```json { "target": "authorized.example", @@ -188,6 +179,9 @@ JSONL is unversioned. After the summary line, it stores one canonical finding fo If a hostname is distinct on two endpoints, both endpoint records stay under that one hostname finding: +
+Full JSONL finding example + ```json { "type": "hostname", @@ -255,6 +249,8 @@ If a hostname is distinct on two endpoints, both endpoint records stay under tha } ``` +
+ The `context_*` fields contain the literal-IP response, while `control_*` contains the stable unknown-host response. A distinct candidate must differ from both. `confirmation_body_sha256` is present only when a repeated response confirmed a body-only difference. These fields let a reviewer verify the recorded `distinct_signals` without retaining every synthetic control request. The source list is empty because virtual host discovery is an action, not a passive source. SQLite uses the same shape without inventing another evidence concept: `results` holds one `hostname` row, `result_origins` links it to the `vhost` execution, and that result's details hold the endpoint observation array. JSONL export and API run details expose the array as native JSON. `vhost` is an action name, not a result kind. diff --git a/docs/wiki/_Footer.md b/docs/wiki/_Footer.md index 6e4cdb19..4c8de9b0 100644 --- a/docs/wiki/_Footer.md +++ b/docs/wiki/_Footer.md @@ -1 +1,3 @@ [Repository](https://github.com/laramies/theHarvester) · [Releases](https://github.com/laramies/theHarvester/releases) · [Issues](https://github.com/laramies/theHarvester/issues) · [Contributing](https://github.com/laramies/theHarvester/blob/dev/CONTRIBUTING.md) · [Security](https://github.com/laramies/theHarvester/blob/dev/SECURITY.md) · [License](https://github.com/laramies/theHarvester/blob/dev/README/COPYING) + +Reviewed wiki changes belong in [`docs/wiki/`](https://github.com/laramies/theHarvester/tree/dev/docs/wiki). The live GitHub wiki is the published copy. diff --git a/docs/wiki/_Sidebar.md b/docs/wiki/_Sidebar.md index a844157c..a699d1bc 100644 --- a/docs/wiki/_Sidebar.md +++ b/docs/wiki/_Sidebar.md @@ -3,12 +3,12 @@ **Get started** - [Installation](Installation) -- [Quick Start](Quick-Start) +- [Configuration and API Keys](Configuration-and-API-Keys) - [Responsible Use and Scope](Responsible-Use-and-Scope) +- [Quick Start](Quick-Start) **Operate** -- [Configuration and API Keys](Configuration-and-API-Keys) - [Operator Workflows](Operator-Workflows) - [Virtual Host Discovery](Virtual-Host-Discovery) - [Results and Local Data](Results-and-Local-Data) diff --git a/tests/test_readme.py b/tests/test_readme.py index d311936b..cadc1cf3 100644 --- a/tests/test_readme.py +++ b/tests/test_readme.py @@ -6,9 +6,11 @@ from pathlib import Path import yaml -from theHarvester.lib.source_catalog import SOURCE_SPECS +from theHarvester.lib.completed_result import parse_result_jsonl +from theHarvester.lib.source_catalog import ACTION_ACTIVITIES, RESULT_CAPABILITIES, SOURCE_SPECS OPTIONAL_API_KEY_SOURCES = {'hackertarget', 'mojeek', 'windvane'} +SOURCE_ROUTE_ORDER = ('subdomains', 'emails', 'ips', 'asns', 'urls', 'people', 'breaches') API_KEY_SOURCE_ALIASES = { 'github': {'github-code'}, 'pentestTools': {'pentesttools'}, @@ -98,7 +100,7 @@ def _declared_source_contracts() -> dict[str, set[str]]: def _source_matrix(readme: str) -> str: - return readme.split('View the source and result matrix', 1)[1].split('', 1)[0] + return readme.split('View all 58 discovery sources', 1)[1].split('', 1)[0] def _documented_source_rows(readme: str) -> dict[str, list[str]]: @@ -121,7 +123,10 @@ def _documented_source_links(readme: str) -> list[tuple[str, str]]: def _documented_source_contracts(readme: str) -> dict[str, set[str]]: - return {source: {route.strip() for route in cells[0].split(',')} for source, cells in _documented_source_rows(readme).items()} + return { + source: {route.strip().removesuffix(' only') for route in cells[0].split(',')} + for source, cells in _documented_source_rows(readme).items() + } def _documented_source_activities(readme: str) -> dict[str, str]: @@ -142,7 +147,8 @@ def test_readme_matches_declared_source_contracts() -> None: documented = _documented_source_contracts(readme) declared = _declared_source_contracts() - assert '| Source | Result routes | Activity | Credentials |' in readme + assert _source_matrix(readme).count('| Source | Returns | Activity | API key |') == 1 + assert 'Credentials |' not in _source_matrix(readme) assert len(declared) == 58 assert len(documented) == 58 assert documented == declared @@ -153,6 +159,23 @@ def test_readme_matches_declared_source_contracts() -> None: assert {'securitytrails', 'shodaninternetdb'}.isdisjoint(documented) +def test_readme_source_matrix_is_one_table_and_ordered() -> None: + readme = Path('README.md').read_text() + matrix = _source_matrix(readme) + rows = _documented_source_rows(readme) + names = list(rows) + + assert '#### Subdomain-only sources' not in matrix + assert '#### Sources that return other results' not in matrix + assert names == sorted(names, key=str.casefold) + + route_rank = {route: index for index, route in enumerate(SOURCE_ROUTE_ORDER)} + for source, (routes, *_) in rows.items(): + documented_order = [route.strip().removesuffix(' only') for route in routes.split(',')] + assert documented_order == sorted(documented_order, key=route_rank.__getitem__) + assert routes.endswith(' only') == (len(SOURCE_SPECS[source].capabilities) == 1) + + def test_readme_api_key_markers_match_configuration() -> None: readme = Path('README.md').read_text() requirements = _documented_api_key_requirements(readme) @@ -182,6 +205,16 @@ def test_wiki_navigation_and_readme_links_resolve() -> None: assert all(Path(target).is_file() for target in readme_wiki_links) +def test_wiki_examples_use_copy_safe_operator_language() -> None: + quick_start = Path('docs/wiki/Quick-Start.md').read_text() + configuration = Path('docs/wiki/Configuration-and-API-Keys.md').read_text() + + assert 'inert test data' not in quick_start + assert 'Passive providers still receive the target string.' in quick_start + assert '${EDITOR:-vi}' not in configuration + assert 'vi ~/.theHarvester/api-keys.yaml' in configuration + + def test_readme_architecture_diagrams_are_local_and_accessible() -> None: readme = Path('README.md').read_text() diagrams = ( @@ -189,24 +222,78 @@ def test_readme_architecture_diagrams_are_local_and_accessible() -> None: 'theHarvester discovery routes and enrichment', Path('docs/images/run-evidence-architecture.svg'), 'run-evidence-architecture', - ('subdomains · emails · IPs', 'Shodan host detail', 'RouteViews routes', 'vhost · screenshots · takeover'), + ('58 discovery adapters', 'CompletedResult evidence contract', 'Terminal · JSONL · SQLite · REST'), ), ( 'HarvestView run desk architecture', Path('docs/images/harvestview-architecture.svg'), 'harvestview-architecture', - ('Authenticated REST API', 'queued → running → terminal', 'Isolated run worker', 'JSONL / SQLite export'), + ('Authenticated REST API', 'queued · running', 'Isolated run worker', 'JSONL · one run'), ), ) for alt, svg, slug, expected_text in diagrams: svg_text = svg.read_text() - assert f'![{alt}]({svg})' in readme + assert f'[![{alt}]({svg})]({svg})' in readme + assert 'role="img"' in svg_text + assert f'' in svg_text + assert f'<desc id="{slug}-desc">' in svg_text + assert 'viewBox="0 0 960 640"' in svg_text + assert '@media (prefers-color-scheme: light)' in svg_text + assert "font: 600 16px 'Geist'" in svg_text + assert all(text in svg_text for text in expected_text) + + run_diagram = diagrams[0][1].read_text() + harvestview_diagram = diagrams[1][1].read_text() + assert all(capability in run_diagram for capability in RESULT_CAPABILITIES) + assert all(action in run_diagram for action in ACTION_ACTIVITIES) + assert f'{len(SOURCE_SPECS)} discovery adapters' in run_diagram + assert f'{len(ACTION_ACTIVITIES)} explicit actions' in harvestview_diagram + + +def test_wiki_diagrams_are_local_accessible_and_used_deliberately() -> None: + raw_root = 'https://raw.githubusercontent.com/laramies/theHarvester/dev/' + diagrams = ( + ( + Path('docs/wiki/Virtual-Host-Discovery.md'), + 'Bounded virtual-host sweep', + Path('docs/images/vhost-sweep-overview.svg'), + 'vhost-sweep-overview', + ('exact target scope', 'Literal-IP endpoint pool', 'shared request cap', '3 shape-matched controls'), + ), + ( + Path('docs/wiki/Virtual-Host-Discovery.md'), + 'Virtual-host response classifier', + Path('docs/images/vhost-classifier.svg'), + 'vhost-classifier', + ('All responses usable', 'Candidate matches', 'Repeat candidate request', 'Retain hostname'), + ), + ( + Path('docs/wiki/Operator-Workflows.md'), + 'theHarvester discovery routes and enrichment', + Path('docs/images/run-evidence-architecture.svg'), + 'run-evidence-architecture', + (), + ), + ( + Path('docs/wiki/Rest-API.md'), + 'HarvestView run desk architecture', + Path('docs/images/harvestview-architecture.svg'), + 'harvestview-architecture', + (), + ), + ) + + for page, alt, svg, slug, expected_text in diagrams: + svg_text = svg.read_text() + assert f'![{alt}]({raw_root}{svg})' in page.read_text() assert 'role="img"' in svg_text assert f'<title id="{slug}-title">' in svg_text assert f'<desc id="{slug}-desc">' in svg_text assert all(text in svg_text for text in expected_text) + assert '```mermaid' not in Path('docs/wiki/Virtual-Host-Discovery.md').read_text() + def test_virtual_host_wiki_examples_match_the_structured_result_contract() -> None: page = Path('docs/wiki/Virtual-Host-Discovery.md').read_text() @@ -271,8 +358,16 @@ def test_operator_docs_cover_portable_database_export() -> None: def test_readme_explains_jsonl_record_and_structured_evidence_parsing() -> None: readme = Path('README.md').read_text() + example = re.search(r'```jsonl\n(.*?)\n```', readme, flags=re.DOTALL) + assert example is not None + summary, findings = parse_result_jsonl(example.group(1)) - assert '{"sources":[],"type":"hostname","value":"api.example.com"}' in readme + finding_types = ('asn', 'breach', 'email', 'hostname', 'ip', 'url') + assert summary['counts'] == {result_type: 1 for result_type in finding_types} + assert summary['result_count'] == len(findings) == 6 + assert tuple(finding['type'] for finding in findings) == finding_types + assert all(finding['sources'] for finding in findings) + assert {execution['source'] for execution in summary['source_executions']} == {'haveibeenpwned', 'zoomeye'} for result_kind in ('hostname', 'ip', 'asn', 'email', 'url', 'person', 'breach'): assert f'select(.type == "{result_kind}")' in readme assert 'select(.type == "person") | .value | fromjson' in readme