mirror of
https://github.com/laramies/theHarvester.git
synced 2026-08-17 19:35:40 +02:00
Merge pull request #2569 from laramies/dev
release: sync dev into master
This commit is contained in:
@@ -7,7 +7,7 @@
|
|||||||
|
|
||||||
theHarvester gathers open-source intelligence about a domain or organization from search engines, certificate transparency logs, DNS datasets, code repositories, threat-intelligence platforms, and other public sources.
|
theHarvester gathers open-source intelligence about a domain or organization from search engines, certificate transparency logs, DNS datasets, code repositories, threat-intelligence platforms, and other public sources.
|
||||||
|
|
||||||
It is built for the early reconnaissance stage of authorized security assessments. Use it only on targets you own or have explicit permission to test.
|
Use theHarvester during the early reconnaissance stage of an authorized security assessment. Run it only against targets you own or have explicit permission to test.
|
||||||
|
|
||||||
## What it does
|
## What it does
|
||||||
|
|
||||||
@@ -25,11 +25,11 @@ Providers control their own availability, quotas, and response formats, so indiv
|
|||||||
|
|
||||||
### Discovery routes and enrichment
|
### Discovery routes and enrichment
|
||||||
|
|
||||||

|
[](docs/images/run-evidence-architecture.svg)
|
||||||
|
|
||||||
### HarvestView run desk
|
### HarvestView run desk
|
||||||
|
|
||||||

|
[](docs/images/harvestview-architecture.svg)
|
||||||
|
|
||||||
## Quick start
|
## Quick start
|
||||||
|
|
||||||
@@ -76,7 +76,7 @@ uv run theHarvester -d example.com -b emails,ips,urls --no-hosts -f non-host-res
|
|||||||
|
|
||||||
`--no-hosts` skips hostname-only sources and omits hostname results while keeping other result types. It cannot be combined with actions that depend on hostnames. HarvestView and the REST API expose the same option as `no_hosts`.
|
`--no-hosts` skips hostname-only sources and omits hostname results while keeping other result types. It cannot be combined with actions that depend on hostnames. HarvestView and the REST API expose the same option as `no_hosts`.
|
||||||
|
|
||||||
Save a durable JSONL report:
|
Save results as JSONL:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
uv run theHarvester -d example.com -b crtsh,certspotter -f report
|
uv run theHarvester -d example.com -b crtsh,certspotter -f report
|
||||||
@@ -114,13 +114,13 @@ export THEHARVESTER_API_KEY='replace-with-a-long-random-value'
|
|||||||
uv run harvestview
|
uv run harvestview
|
||||||
```
|
```
|
||||||
|
|
||||||
Open [HarvestView](http://127.0.0.1:5000/) to submit and inspect finite runs. The browser receives a derived HttpOnly session cookie and never stores the API key. See the [installation guide](docs/wiki/Installation.md) for local assets, screenshots, and isolated deployments.
|
Open [HarvestView](http://127.0.0.1:5000/) to start runs and inspect their results. The browser receives a derived HttpOnly session cookie and never stores the API key. See the [installation guide](docs/wiki/Installation.md) for local assets, screenshots, and isolated deployments.
|
||||||
|
|
||||||
Open [Swagger](http://127.0.0.1:5000/docs) or [ReDoc](http://127.0.0.1:5000/redoc) for the automation contract.
|
Open [Swagger](http://127.0.0.1:5000/docs) or [ReDoc](http://127.0.0.1:5000/redoc) for the automation contract.
|
||||||
|
|
||||||
### Docker Compose
|
### Docker Compose
|
||||||
|
|
||||||
The Compose service runs as an unprivileged user, stores runs in a named volume, reads the operator key from a file secret, and publishes only to host loopback:
|
The Compose service runs as an unprivileged user and binds only to host loopback. It stores runs in a named volume and reads the operator key from a file secret:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
install -d -m 0700 .secrets
|
install -d -m 0700 .secrets
|
||||||
@@ -149,87 +149,89 @@ docker compose down
|
|||||||
|
|
||||||
HarvestView can start screenshot and DNS brute-force runs from a hostname result. Each action creates its own run and leaves the original evidence unchanged.
|
HarvestView can start screenshot and DNS brute-force runs from a hostname result. Each action creates its own run and leaves the original evidence unchanged.
|
||||||
|
|
||||||
API clients send `THEHARVESTER_API_KEY` in the `X-API-Key` header. Provider credentials stay in server-side configuration. Keep the service on localhost unless you add TLS and network access controls. The [REST API guide](docs/wiki/Rest-API.md) documents requests, imports, exports, and authentication.
|
API clients send `THEHARVESTER_API_KEY` in the `X-API-Key` header. Provider API settings stay on the server. Keep the service on localhost unless you add TLS and network access controls. The [REST API guide](docs/wiki/Rest-API.md) documents requests, imports, exports, and authentication.
|
||||||
|
|
||||||
## Discovery sources
|
## Discovery sources
|
||||||
|
|
||||||
Select sources by name or by any result route listed below. `-b all` runs the P0 sources. P1 and P2 sources require explicit selection. Credentials marked optional can provide additional access but are not required.
|
Select sources by name or by a result route listed below. `-b all` runs the P0 sources. P1 and P2 sources require explicit selection.
|
||||||
|
|
||||||
|
Result types in this table always appear in this order: `subdomains`, `emails`, `ips`, `asns`, `urls`, `people`, `breaches`. A result followed by `only` means the source contributes no other result type. The `API key` column refers to provider settings in `api-keys.yaml`; some providers require more than one value. `Optional` means the source can run without a key.
|
||||||
|
|
||||||
The `shodan` source contributes subdomains. Shodan host enrichment through `-s` or `--shodan` is a separate action and is not a source result route.
|
The `shodan` source contributes subdomains. Shodan host enrichment through `-s` or `--shodan` is a separate action and is not a source result route.
|
||||||
|
|
||||||
<details>
|
<details>
|
||||||
<summary><strong>View the source and result matrix</strong></summary>
|
<summary><strong>View all 58 discovery sources</strong></summary>
|
||||||
|
|
||||||
| Source | Result routes | Activity | Credentials |
|
| Source | Returns | Activity | API key |
|
||||||
| --- | --- | :---: | :---: |
|
| --- | --- | :---: | :---: |
|
||||||
| [`apis-guru`](https://apis.guru/) | subdomains, emails, urls | P0 | No |
|
| [`apis-guru`](https://apis.guru/) | subdomains, emails, urls | P0 | No |
|
||||||
| [`arquivo`](https://arquivo.pt/) | subdomains | P0 | No |
|
| [`arquivo`](https://arquivo.pt/) | subdomains only | P0 | No |
|
||||||
| [`baidu`](https://www.baidu.com/) | subdomains, emails | P0 | No |
|
| [`baidu`](https://www.baidu.com/) | subdomains, emails | P0 | No |
|
||||||
| [`bevigil`](https://bevigil.com/osint-api) | subdomains, urls | P0 | Required |
|
| [`bevigil`](https://bevigil.com/osint-api) | subdomains, urls | P0 | Required |
|
||||||
|
| [`brave`](https://brave.com/search/api/) | subdomains, emails | P0 | Required |
|
||||||
| [`bufferoverun`](https://tls.bufferover.run/) | subdomains, ips | P0 | Required |
|
| [`bufferoverun`](https://tls.bufferover.run/) | subdomains, ips | P0 | Required |
|
||||||
| [`builtwith`](https://builtwith.com/) | subdomains, urls | P0 | Required |
|
| [`builtwith`](https://builtwith.com/) | subdomains, urls | P0 | Required |
|
||||||
| [`brave`](https://brave.com/search/api/) | subdomains, emails | P0 | Required |
|
|
||||||
| [`censys`](https://search.censys.io/) | subdomains, emails | P0 | Required |
|
| [`censys`](https://search.censys.io/) | subdomains, emails | P0 | Required |
|
||||||
| [`certspotter`](https://sslmate.com/certspotter/) | subdomains | P0 | No |
|
| [`certspotter`](https://sslmate.com/certspotter/) | subdomains only | P0 | No |
|
||||||
| [`commoncrawl`](https://commoncrawl.org/) | subdomains | P0 | No |
|
| [`commoncrawl`](https://commoncrawl.org/) | subdomains only | P0 | No |
|
||||||
| [`criminalip`](https://www.criminalip.io/) | subdomains, ips, asns | P2 | Required |
|
| [`criminalip`](https://www.criminalip.io/) | subdomains, ips, asns | P2 | Required |
|
||||||
| [`crt-name`](https://crt.name/) | subdomains | P0 | No |
|
| [`crt-name`](https://crt.name/) | subdomains only | P0 | No |
|
||||||
| [`crtsh`](https://crt.sh/) | subdomains | P0 | No |
|
| [`crtsh`](https://crt.sh/) | subdomains only | P0 | No |
|
||||||
| [`dehashed`](https://dehashed.com/) | emails, ips | P0 | Required |
|
| [`dehashed`](https://dehashed.com/) | emails, ips | P0 | Required |
|
||||||
| [`dnsdb`](https://docs.domaintools.com/api/dnsdb/) | subdomains | P0 | Required |
|
| [`dnsdb`](https://docs.domaintools.com/api/dnsdb/) | subdomains only | P0 | Required |
|
||||||
| [`dnsdumpster`](https://dnsdumpster.com/) | subdomains, ips | P0 | Required |
|
| [`dnsdumpster`](https://dnsdumpster.com/) | subdomains, ips | P0 | Required |
|
||||||
| [`duckduckgo`](https://duckduckgo.com/) | subdomains, emails | P0 | No |
|
| [`duckduckgo`](https://duckduckgo.com/) | subdomains, emails | P0 | No |
|
||||||
| [`dymo`](https://docs.tpeoficial.com/docs/dymo-api/private/data-verifier) | subdomains | P0 | Required |
|
| [`dymo`](https://docs.tpeoficial.com/docs/dymo-api/private/data-verifier) | subdomains only | P0 | Required |
|
||||||
| [`fofa`](https://en.fofa.info/) | subdomains, ips | P0 | Required |
|
| [`fofa`](https://en.fofa.info/) | subdomains, ips | P0 | Required |
|
||||||
| [`fullhunt`](https://fullhunt.io/) | subdomains | P0 | Required |
|
| [`fullhunt`](https://fullhunt.io/) | subdomains only | P0 | Required |
|
||||||
| [`github-code`](https://github.com/) | subdomains, emails | P0 | Required |
|
| [`github-code`](https://github.com/) | subdomains, emails | P0 | Required |
|
||||||
| [`gitlab`](https://gitlab.com/) | subdomains, emails, urls | P0 | No |
|
| [`gitlab`](https://gitlab.com/) | subdomains, emails, urls | P0 | No |
|
||||||
| [`hackertarget`](https://hackertarget.com/) | subdomains, ips | P0 | Optional |
|
| [`hackertarget`](https://hackertarget.com/) | subdomains, ips | P0 | Optional |
|
||||||
| [`haveibeenpwned`](https://haveibeenpwned.com/) | breaches | P0 | No |
|
| [`haveibeenpwned`](https://haveibeenpwned.com/) | breaches only | P0 | No |
|
||||||
| [`hibpverified`](https://haveibeenpwned.com/API/v3#BreachedDomain) | emails, breaches | P0 | Required |
|
| [`hibpverified`](https://haveibeenpwned.com/API/v3#BreachedDomain) | emails, breaches | P0 | Required |
|
||||||
| [`hudsonrock`](https://www.hudsonrock.com/) | subdomains, emails, ips | P0 | No |
|
| [`hudsonrock`](https://www.hudsonrock.com/) | subdomains, emails, ips | P0 | No |
|
||||||
| [`hunter`](https://hunter.io/) | subdomains, emails | P0 | Required |
|
| [`hunter`](https://hunter.io/) | subdomains, emails | P0 | Required |
|
||||||
| [`hunterhow`](https://hunter.how/) | subdomains | P0 | Required |
|
| [`hunterhow`](https://hunter.how/) | subdomains only | P0 | Required |
|
||||||
| [`intelx`](https://intelx.io/) | subdomains, emails, urls | P0 | Required |
|
| [`intelx`](https://intelx.io/) | subdomains, emails, urls | P0 | Required |
|
||||||
| [`leakix`](https://leakix.net/) | subdomains | P0 | Required |
|
| [`leakix`](https://leakix.net/) | subdomains only | P0 | Required |
|
||||||
| [`leaklookup`](https://leak-lookup.com/) | emails, breaches | P0 | Required |
|
| [`leaklookup`](https://leak-lookup.com/) | emails, breaches | P0 | Required |
|
||||||
| [`mojeek`](https://www.mojeek.com/services/search/web-search-api/) | subdomains, emails | P0 | Optional |
|
| [`mojeek`](https://www.mojeek.com/services/search/web-search-api/) | subdomains, emails | P0 | Optional |
|
||||||
| [`netlas`](https://netlas.io/) | subdomains | P0 | Required |
|
| [`netlas`](https://netlas.io/) | subdomains only | P0 | Required |
|
||||||
| [`onyphe`](https://www.onyphe.io/) | subdomains, ips, asns | P0 | Required |
|
| [`onyphe`](https://www.onyphe.io/) | subdomains, ips, asns | P0 | Required |
|
||||||
| [`otx`](https://otx.alienvault.com/) | subdomains, ips | P0 | No |
|
| [`otx`](https://otx.alienvault.com/) | subdomains, ips | P0 | No |
|
||||||
| [`pentesttools`](https://pentest-tools.com/) | subdomains, ips | P1 | Required |
|
| [`pentesttools`](https://pentest-tools.com/) | subdomains, ips | P1 | Required |
|
||||||
| [`projectdiscovery`](https://chaos.projectdiscovery.io/) | subdomains | P0 | Required |
|
| [`projectdiscovery`](https://chaos.projectdiscovery.io/) | subdomains only | P0 | Required |
|
||||||
| [`rapiddns`](https://rapiddns.io/) | subdomains, ips | P0 | No |
|
| [`rapiddns`](https://rapiddns.io/) | subdomains, ips | P0 | No |
|
||||||
| [`robtex`](https://www.robtex.com/) | ips | P0 | No |
|
| [`robtex`](https://www.robtex.com/) | ips only | P0 | No |
|
||||||
| [`rocketreach`](https://rocketreach.co/) | emails, urls | P0 | Required |
|
| [`rocketreach`](https://rocketreach.co/) | emails, urls | P0 | Required |
|
||||||
| [`securityscorecard`](https://securityscorecard.com/) | subdomains, ips | P0 | Required |
|
| [`securityscorecard`](https://securityscorecard.com/) | subdomains, ips | P0 | Required |
|
||||||
| [`securityTrails`](https://securitytrails.com/) | subdomains, ips | P0 | Required |
|
| [`securityTrails`](https://securitytrails.com/) | subdomains, ips | P0 | Required |
|
||||||
| [`sherlockeye`](https://sherlockeye.io/) | subdomains, emails, ips | P0 | Required |
|
| [`sherlockeye`](https://sherlockeye.io/) | subdomains, emails, ips | P0 | Required |
|
||||||
| [`shodan`](https://www.shodan.io/) | subdomains | P1 | Required |
|
| [`shodan`](https://www.shodan.io/) | subdomains only | P1 | Required |
|
||||||
|
| [`shodanct`](https://ctl.shodan.io/) | subdomains only | P0 | No |
|
||||||
| [`shodanInternetDB`](https://internetdb.shodan.io/) | subdomains, ips | P1 | No |
|
| [`shodanInternetDB`](https://internetdb.shodan.io/) | subdomains, ips | P1 | No |
|
||||||
| [`shodanct`](https://ctl.shodan.io/) | subdomains | P0 | No |
|
| [`sourcegraph`](https://sourcegraph.com/search) | subdomains only | P0 | No |
|
||||||
| [`sourcegraph`](https://sourcegraph.com/search) | subdomains | P0 | No |
|
| [`subdomaincenter`](https://www.subdomain.center/) | subdomains only | P0 | No |
|
||||||
| [`subdomaincenter`](https://www.subdomain.center/) | subdomains | P0 | No |
|
| [`subdomainfinderc99`](https://subdomainfinder.c99.nl/) | subdomains only | P1 | No |
|
||||||
| [`subdomainfinderc99`](https://subdomainfinder.c99.nl/) | subdomains | P1 | No |
|
| [`thc`](https://ip.thc.org/) | subdomains only | P0 | No |
|
||||||
| [`thc`](https://ip.thc.org/) | subdomains | P0 | No |
|
|
||||||
| [`tomba`](https://tomba.io/) | subdomains, emails | P0 | Required |
|
| [`tomba`](https://tomba.io/) | subdomains, emails | P0 | Required |
|
||||||
| [`urlscan`](https://urlscan.io/) | subdomains, ips, asns, urls | P0 | No |
|
| [`urlscan`](https://urlscan.io/) | subdomains, ips, asns, urls | P0 | No |
|
||||||
| [`virustotal`](https://www.virustotal.com/) | subdomains | P0 | Required |
|
| [`virustotal`](https://www.virustotal.com/) | subdomains only | P0 | Required |
|
||||||
| [`waybackarchive`](https://web.archive.org/) | subdomains | P0 | No |
|
| [`waybackarchive`](https://web.archive.org/) | subdomains only | P0 | No |
|
||||||
| [`whoisxml`](https://subdomains.whoisxmlapi.com/) | subdomains | P0 | Required |
|
| [`whoisxml`](https://subdomains.whoisxmlapi.com/) | subdomains only | P0 | Required |
|
||||||
| [`windvane`](https://windvane.lichoin.com/) | subdomains, emails, ips | P0 | Optional |
|
| [`windvane`](https://windvane.lichoin.com/) | subdomains, emails, ips | P0 | Optional |
|
||||||
| [`yahoo`](https://www.yahoo.com/) | subdomains, emails | P0 | No |
|
| [`yahoo`](https://www.yahoo.com/) | subdomains, emails | P0 | No |
|
||||||
| [`zoomeye`](https://www.zoomeye.ai/) | subdomains, emails, ips, asns, urls | P0 | Required |
|
| [`zoomeye`](https://www.zoomeye.ai/) | subdomains, emails, ips, asns, urls | P0 | Required |
|
||||||
|
|
||||||
</details>
|
</details>
|
||||||
|
|
||||||
Each source name links to its provider's site or documentation, where current plans, quotas, and terms are published. See [Configuration and API keys](docs/wiki/Configuration-and-API-Keys.md) for credential names and setup. Contributors can add a provider through the [module guide](docs/wiki/How-to-add-a-new-module.md); the source catalog remains the executable inventory.
|
Each source name links to its provider's site or documentation for current plans, quotas, and terms. See [Configuration and API keys](docs/wiki/Configuration-and-API-Keys.md) for the required fields and setup instructions. Contributors can add a provider through the [module guide](docs/wiki/How-to-add-a-new-module.md). The CLI and API read their source inventory from the source catalog.
|
||||||
|
|
||||||
## Configuration
|
## Configuration
|
||||||
|
|
||||||
On first use, theHarvester creates default configuration files under `~/.theHarvester/`. It also reads system configuration from `/etc/theHarvester/` and `/usr/local/etc/theHarvester/`.
|
On first use, theHarvester creates default configuration files under `~/.theHarvester/`. It also reads system configuration from `/etc/theHarvester/` and `/usr/local/etc/theHarvester/`.
|
||||||
|
|
||||||
- `api-keys.yaml` stores provider credentials.
|
- `api-keys.yaml` stores provider API keys and related values such as organization IDs.
|
||||||
- `proxies.yaml` configures HTTP and SOCKS5 proxies used with `-p`.
|
- `proxies.yaml` configures HTTP and SOCKS5 proxies used with `-p`.
|
||||||
- The `shodan` source and `-s` / `--shodan` enrichment use Shodan's Host REST API. When `-p` is enabled, both send those requests through `proxies.yaml`.
|
- The `shodan` source and `-s` / `--shodan` enrichment use Shodan's Host REST API. When `-p` is enabled, both send those requests through `proxies.yaml`.
|
||||||
- `routeviews.key` is optional and enables authenticated RouteViews access for PeeringDB-verified users.
|
- `routeviews.key` is optional and enables authenticated RouteViews access for PeeringDB-verified users.
|
||||||
@@ -246,9 +248,16 @@ Treat collected OSINT as potentially sensitive. Keep report files, screenshots,
|
|||||||
|
|
||||||
JSONL is the primary format for automation and one-run interchange. The first line describes the run. Each remaining line is one sorted, deduplicated finding with its source and action provenance.
|
JSONL is the primary format for automation and one-run interchange. The first line describes the run. Each remaining line is one sorted, deduplicated finding with its source and action provenance.
|
||||||
|
|
||||||
|
This example contains six findings from two sources. The `counts` object summarizes the result lines that follow it.
|
||||||
|
|
||||||
```jsonl
|
```jsonl
|
||||||
{"action_executions":[],"artifacts":[],"completed_at":"2026-08-07T12:01:00Z","counts":{"hostname":1},"evidence_status":"complete","result_count":1,"run_id":"123e4567-e89b-12d3-a456-426614174000","source_executions":[],"started_at":"2026-08-07T12:00:00Z","target":"example.com","type":"summary"}
|
{"action_executions":[],"artifacts":[],"completed_at":"2026-08-17T12:01:00Z","counts":{"asn":1,"breach":1,"email":1,"hostname":1,"ip":1,"url":1},"evidence_status":"complete","result_count":6,"run_id":"123e4567-e89b-12d3-a456-426614174000","source_executions":[{"duration_ms":127.4,"error_type":null,"result_count":1,"source":"haveibeenpwned","status":"completed","stop_reason":null},{"duration_ms":482.3,"error_type":null,"result_count":5,"source":"zoomeye","status":"completed","stop_reason":null}],"started_at":"2026-08-17T12:00:00Z","target":"example.com","type":"summary"}
|
||||||
{"sources":[],"type":"hostname","value":"api.example.com"}
|
{"sources":["zoomeye"],"type":"asn","value":"AS64500"}
|
||||||
|
{"sources":["haveibeenpwned"],"type":"breach","value":"Example breach"}
|
||||||
|
{"sources":["zoomeye"],"type":"email","value":"security@example.com"}
|
||||||
|
{"sources":["zoomeye"],"type":"hostname","value":"api.example.com"}
|
||||||
|
{"sources":["zoomeye"],"type":"ip","value":"192.0.2.10"}
|
||||||
|
{"sources":["zoomeye"],"type":"url","value":"https://api.example.com/login"}
|
||||||
```
|
```
|
||||||
|
|
||||||
Extract common result types with `jq`:
|
Extract common result types with `jq`:
|
||||||
@@ -282,7 +291,7 @@ The `subdomains` capability produces `hostname` records because a result can be
|
|||||||
|
|
||||||
### SQLite, JSON, and XML
|
### SQLite, JSON, and XML
|
||||||
|
|
||||||
CLI and API runs use the same SQLite evidence model. JSONL moves one run at a time. The API can import or export completed runs in bulk as a portable SQLite database while leaving queue and worker state behind. Screenshot files are managed separately from their metadata.
|
CLI and API runs use the same SQLite evidence model. JSONL moves one run at a time. SQLite import and export handle completed runs in bulk but exclude queue and worker state. Screenshot files remain separate from their metadata.
|
||||||
|
|
||||||
JSON and XML are compatibility reports grouped by result type. They do not include the full provenance, lifecycle outcomes, or structured action evidence available in JSONL, SQLite, the API, and HarvestView.
|
JSON and XML are compatibility reports grouped by result type. They do not include the full provenance, lifecycle outcomes, or structured action evidence available in JSONL, SQLite, the API, and HarvestView.
|
||||||
|
|
||||||
|
|||||||
@@ -1,109 +1,158 @@
|
|||||||
<?xml version="1.0" encoding="UTF-8"?>
|
<?xml version="1.0" encoding="UTF-8"?>
|
||||||
<svg viewBox="0 0 1280 720" xmlns="http://www.w3.org/2000/svg" role="img" aria-labelledby="harvestview-architecture-title harvestview-architecture-desc">
|
<svg viewBox="0 0 960 640" xmlns="http://www.w3.org/2000/svg" role="img" aria-labelledby="harvestview-architecture-title harvestview-architecture-desc">
|
||||||
<title id="harvestview-architecture-title">HarvestView run desk architecture</title>
|
<title id="harvestview-architecture-title">HarvestView run desk architecture</title>
|
||||||
<desc id="harvestview-architecture-desc">HarvestView validates local browser requests through the authenticated API, records one durable run, executes it in an isolated worker, and returns normalized evidence to the route workbench.</desc>
|
<desc id="harvestview-architecture-desc">A loopback-only HarvestView workflow sends a reviewed plan through the authenticated REST API and durable run control to an isolated worker, preserves normalized evidence separately from lifecycle state, supports inspection and child actions, and imports or exports JSONL and SQLite evidence.</desc>
|
||||||
<defs>
|
<defs>
|
||||||
<style>@import url('https://fonts.googleapis.com/css2?family=Instrument+Serif:ital@0;1&family=Geist:wght@400;500;600&family=Geist+Mono:wght@400;500;600&display=swap');</style>
|
<style>
|
||||||
<style>
|
@import url('https://fonts.googleapis.com/css2?family=Instrument+Serif:ital@0;1&family=Geist:wght@400;500;600&family=Geist+Mono:wght@400;500;600&display=swap');
|
||||||
.title { font: 400 28px "Instrument Serif", Georgia, serif; fill: #0e1d22; }
|
:root {
|
||||||
.subtitle { font: 400 12px "Geist", system-ui, sans-serif; fill: #4f5e63; }
|
--paper: #2d3142;
|
||||||
.eyebrow { font: 600 8px "Geist Mono", ui-monospace, monospace; letter-spacing: .16em; fill: #4f5e63; }
|
--paper-2: #393e53;
|
||||||
.node-title { font: 600 12px "Geist", system-ui, sans-serif; fill: #0e1d22; }
|
--ink: #f5f5f5;
|
||||||
.node-copy { font: 400 8px "Geist Mono", ui-monospace, monospace; fill: #4f5e63; }
|
--muted: #bfc0c0;
|
||||||
.node-copy-strong { font: 500 8px "Geist Mono", ui-monospace, monospace; fill: #34464c; }
|
--soft: #8e98ac;
|
||||||
.boundary { fill: rgba(14,29,34,.018); stroke: #889ca3; stroke-width: 1; stroke-dasharray: 8 8; }
|
--rule: rgba(245,245,245,.16);
|
||||||
.node { fill: #fdfaf4; stroke: #889ca3; stroke-width: 1; }
|
--rule-strong: rgba(245,245,245,.28);
|
||||||
.node-muted { fill: #ede7dd; stroke: #889ca3; stroke-width: 1; }
|
--boundary-fill: rgba(245,245,245,.025);
|
||||||
.node-focal { fill: #bcede4; stroke: #009685; stroke-width: 1.2; }
|
--accent: #f08a59;
|
||||||
.tag { fill: #f5efe7; stroke: #889ca3; stroke-width: .8; }
|
--accent-tint: rgba(240,138,89,.10);
|
||||||
.tag-teal { fill: #bcede4; stroke: #009685; stroke-width: .8; }
|
--link: #6a95d8;
|
||||||
.tag-text { font: 600 8px "Geist Mono", ui-monospace, monospace; fill: #34464c; text-anchor: middle; }
|
--link-tint: rgba(106,149,216,.10);
|
||||||
.tag-text-teal { fill: #00594c; }
|
}
|
||||||
.connector { fill: none; stroke: #4f5e63; stroke-width: 1.2; }
|
@media (prefers-color-scheme: light) {
|
||||||
.connector-accent { fill: none; stroke: #009685; stroke-width: 1.2; }
|
:root {
|
||||||
.legend { font: 400 8px "Geist Mono", ui-monospace, monospace; fill: #4f5e63; }
|
--paper: #f5f5f5;
|
||||||
</style>
|
--paper-2: #ececec;
|
||||||
<marker id="harvestview-architecture-arrow" markerWidth="8" markerHeight="6" refX="7" refY="3" orient="auto"><polygon points="0 0, 8 3, 0 6" fill="#4f5e63"/></marker>
|
--ink: #2d3142;
|
||||||
<marker id="harvestview-architecture-arrow-accent" markerWidth="8" markerHeight="6" refX="7" refY="3" orient="auto"><polygon points="0 0, 8 3, 0 6" fill="#009685"/></marker>
|
--muted: #4f5d75;
|
||||||
<marker id="harvestview-architecture-arrow-link" markerWidth="8" markerHeight="6" refX="7" refY="3" orient="auto"><polygon points="0 0, 8 3, 0 6" fill="#00548c"/></marker>
|
--soft: #7a8399;
|
||||||
</defs>
|
--rule: rgba(45,49,66,.14);
|
||||||
|
--rule-strong: rgba(45,49,66,.28);
|
||||||
|
--boundary-fill: rgba(45,49,66,.02);
|
||||||
|
--accent: #b84f24;
|
||||||
|
--accent-tint: rgba(235,108,54,.10);
|
||||||
|
--link: #2e5aa8;
|
||||||
|
--link-tint: rgba(46,90,168,.08);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
.bg { fill: var(--paper); }
|
||||||
|
.title { font: 400 28px 'Instrument Serif', Georgia, serif; fill: var(--ink); }
|
||||||
|
.subtitle { font: 400 12px 'Geist', system-ui, sans-serif; fill: var(--muted); }
|
||||||
|
.eyebrow { font: 600 8px 'Geist Mono', ui-monospace, monospace; letter-spacing: .16em; fill: var(--soft); }
|
||||||
|
.boundary { fill: var(--boundary-fill); stroke: var(--rule-strong); stroke-width: 1; stroke-dasharray: 8 8; }
|
||||||
|
.boundary-mask { fill: var(--paper); }
|
||||||
|
.node { fill: var(--paper); stroke: var(--rule-strong); stroke-width: 1; }
|
||||||
|
.node-muted { fill: var(--paper-2); stroke: var(--rule-strong); stroke-width: 1; }
|
||||||
|
.node-link { fill: var(--link-tint); stroke: var(--link); stroke-width: 1; }
|
||||||
|
.node-focal { fill: var(--accent-tint); stroke: var(--accent); stroke-width: 1.2; }
|
||||||
|
.tag { fill: var(--paper-2); stroke: var(--rule-strong); stroke-width: .8; }
|
||||||
|
.tag-link { fill: var(--link-tint); stroke: var(--link); stroke-width: .8; }
|
||||||
|
.tag-accent { fill: var(--accent-tint); stroke: var(--accent); stroke-width: .8; }
|
||||||
|
.tag-text { font: 600 8px 'Geist Mono', ui-monospace, monospace; letter-spacing: .08em; fill: var(--muted); text-anchor: middle; }
|
||||||
|
.tag-text-link { fill: var(--link); }
|
||||||
|
.tag-text-accent { fill: var(--accent); }
|
||||||
|
.node-title { font: 600 16px 'Geist', system-ui, sans-serif; fill: var(--ink); }
|
||||||
|
.node-copy { font: 400 12px 'Geist Mono', ui-monospace, monospace; fill: var(--muted); }
|
||||||
|
.node-copy-strong { font: 500 12px 'Geist Mono', ui-monospace, monospace; fill: var(--ink); }
|
||||||
|
.connector { fill: none; stroke: var(--muted); stroke-width: 1.2; }
|
||||||
|
.connector-link { fill: none; stroke: var(--link); stroke-width: 1.2; }
|
||||||
|
.connector-accent { fill: none; stroke: var(--accent); stroke-width: 1.2; }
|
||||||
|
.dashed { stroke-dasharray: 4 4; }
|
||||||
|
.marker-muted { fill: var(--muted); }
|
||||||
|
.marker-link { fill: var(--link); }
|
||||||
|
.marker-accent { fill: var(--accent); }
|
||||||
|
.legend { font: 500 12px 'Geist Mono', ui-monospace, monospace; fill: var(--muted); }
|
||||||
|
</style>
|
||||||
|
<marker id="harvestview-arrow" markerWidth="8" markerHeight="8" refX="8" refY="4" orient="auto"><polygon points="0 0, 8 4, 0 8" class="marker-muted"/></marker>
|
||||||
|
<marker id="harvestview-arrow-link" markerWidth="8" markerHeight="8" refX="8" refY="4" orient="auto"><polygon points="0 0, 8 4, 0 8" class="marker-link"/></marker>
|
||||||
|
<marker id="harvestview-arrow-accent" markerWidth="8" markerHeight="8" refX="8" refY="4" orient="auto"><polygon points="0 0, 8 4, 0 8" class="marker-accent"/></marker>
|
||||||
|
</defs>
|
||||||
|
|
||||||
<rect width="1280" height="720" fill="#f5efe7"/>
|
<rect width="960" height="640" class="bg"/>
|
||||||
<text x="40" y="52" class="eyebrow">HARVESTVIEW · LOCAL OPERATOR WORKFLOW</text>
|
<text x="40" y="32" class="eyebrow">HARVESTVIEW · LOCAL OPERATOR WORKFLOW</text>
|
||||||
<text x="40" y="88" class="title">One run desk, one durable lifecycle</text>
|
<text x="40" y="64" class="title">One run desk, one durable evidence path</text>
|
||||||
<text x="40" y="112" class="subtitle">The browser reviews scope; the API validates; an isolated worker executes; evidence remains available after terminal state.</text>
|
<text x="40" y="88" class="subtitle">The browser reviews scope; the API validates; an isolated worker executes; completed evidence remains portable.</text>
|
||||||
|
|
||||||
<rect x="40" y="132" width="1200" height="436" rx="8" class="boundary"/>
|
<rect x="24" y="120" width="912" height="408" rx="8" class="boundary"/>
|
||||||
<rect x="56" y="136" width="184" height="16" fill="#f5efe7"/>
|
<rect x="40" y="124" width="204" height="16" rx="4" class="boundary-mask"/>
|
||||||
<text x="60" y="148" class="eyebrow">LOCAL HOST · 127.0.0.1</text>
|
<text x="48" y="136" class="eyebrow">LOCALHOST · 127.0.0.1:5000</text>
|
||||||
|
|
||||||
<line x1="224" y1="292" x2="264" y2="292" class="connector" marker-end="url(#harvestview-architecture-arrow)"/>
|
<line x1="220" y1="224" x2="244" y2="224" class="connector" marker-end="url(#harvestview-arrow)"/>
|
||||||
<line x1="464" y1="292" x2="504" y2="292" class="connector" marker-end="url(#harvestview-architecture-arrow)"/>
|
<line x1="444" y1="192" x2="468" y2="192" class="connector-link dashed" marker-end="url(#harvestview-arrow-link)"/>
|
||||||
<line x1="604" y1="364" x2="604" y2="412" class="connector" marker-end="url(#harvestview-architecture-arrow)"/>
|
<line x1="344" y1="296" x2="344" y2="336" class="connector" marker-end="url(#harvestview-arrow)"/>
|
||||||
<line x1="704" y1="472" x2="744" y2="472" class="connector" marker-end="url(#harvestview-architecture-arrow)"/>
|
<line x1="444" y1="408" x2="468" y2="408" class="connector" marker-end="url(#harvestview-arrow)"/>
|
||||||
<line x1="844" y1="412" x2="844" y2="364" class="connector-accent" marker-end="url(#harvestview-architecture-arrow-accent)"/>
|
<line x1="568" y1="336" x2="568" y2="296" class="connector-accent" marker-end="url(#harvestview-arrow-accent)"/>
|
||||||
<line x1="944" y1="292" x2="984" y2="292" class="connector" marker-end="url(#harvestview-architecture-arrow)"/>
|
<line x1="668" y1="224" x2="692" y2="224" class="connector" marker-end="url(#harvestview-arrow)"/>
|
||||||
|
<line x1="806" y1="296" x2="806" y2="336" class="connector-link" marker-end="url(#harvestview-arrow-link)"/>
|
||||||
|
|
||||||
<rect x="64" y="220" width="160" height="144" rx="8" class="node"/>
|
<rect x="40" y="152" width="180" height="144" rx="8" class="node"/>
|
||||||
<rect x="76" y="232" width="72" height="20" rx="4" class="tag"/>
|
<rect x="52" y="164" width="72" height="20" rx="4" class="tag"/>
|
||||||
<text x="112" y="246" class="tag-text">BROWSER</text>
|
<text x="88" y="180" class="tag-text">BROWSER</text>
|
||||||
<text x="80" y="276" class="node-title">Create or import</text>
|
<text x="56" y="216" class="node-title">Plan and authorize</text>
|
||||||
<text x="80" y="300" class="node-copy-strong">new run · JSONL · SQLite</text>
|
<text x="56" y="240" class="node-copy-strong">target · sources</text>
|
||||||
<text x="80" y="320" class="node-copy-strong">source and action review</text>
|
<text x="56" y="256" class="node-copy-strong">actions · P0/P1/P2</text>
|
||||||
<text x="80" y="344" class="node-copy">P0 · P1 · P2 authorization</text>
|
<text x="56" y="272" class="node-copy">review before submit</text>
|
||||||
|
|
||||||
<rect x="264" y="220" width="200" height="144" rx="8" class="node"/>
|
<rect x="244" y="152" width="200" height="144" rx="8" class="node-link"/>
|
||||||
<rect x="276" y="232" width="96" height="20" rx="4" class="tag"/>
|
<rect x="256" y="164" width="88" height="20" rx="4" class="tag-link"/>
|
||||||
<text x="324" y="246" class="tag-text">API GATE</text>
|
<text x="300" y="180" class="tag-text tag-text-link">API GATE</text>
|
||||||
<text x="280" y="276" class="node-title">Authenticated REST API</text>
|
<text x="260" y="216" class="node-title">Authenticated REST API</text>
|
||||||
<text x="280" y="300" class="node-copy-strong">derived HttpOnly browser session</text>
|
<text x="260" y="240" class="node-copy-strong">HttpOnly · X-API-Key</text>
|
||||||
<text x="280" y="320" class="node-copy-strong">X-API-Key for API clients</text>
|
<text x="260" y="256" class="node-copy-strong">target · scope · limits</text>
|
||||||
<text x="280" y="344" class="node-copy">target · scope · options · limits</text>
|
<text x="260" y="272" class="node-copy">create · list</text>
|
||||||
|
<text x="260" y="288" class="node-copy">detail · cancel</text>
|
||||||
|
|
||||||
<rect x="504" y="220" width="200" height="144" rx="8" class="node-muted"/>
|
<rect x="468" y="152" width="200" height="144" rx="8" class="node-focal"/>
|
||||||
<rect x="516" y="232" width="96" height="20" rx="4" class="tag"/>
|
<rect x="480" y="164" width="88" height="20" rx="4" class="tag-accent"/>
|
||||||
<text x="564" y="246" class="tag-text">CONTROL</text>
|
<text x="524" y="180" class="tag-text tag-text-accent">EVIDENCE</text>
|
||||||
<text x="520" y="276" class="node-title">Durable run control</text>
|
<text x="484" y="216" class="node-title">Canonical evidence</text>
|
||||||
<text x="520" y="300" class="node-copy-strong">queued → running → terminal</text>
|
<text x="484" y="240" class="node-copy-strong">results · provenance</text>
|
||||||
<text x="520" y="320" class="node-copy-strong">cancel request · source status</text>
|
<text x="484" y="256" class="node-copy-strong">source/action outcomes</text>
|
||||||
<text x="520" y="344" class="node-copy">one record per finite run</text>
|
<text x="484" y="272" class="node-copy">artifacts · status</text>
|
||||||
|
<text x="484" y="288" class="node-copy">complete · partial · failed</text>
|
||||||
|
|
||||||
<rect x="504" y="412" width="200" height="120" rx="8" class="node"/>
|
<rect x="692" y="152" width="228" height="144" rx="8" class="node"/>
|
||||||
<rect x="516" y="424" width="80" height="20" rx="4" class="tag"/>
|
<rect x="704" y="164" width="112" height="20" rx="4" class="tag"/>
|
||||||
<text x="556" y="438" class="tag-text">WORKER</text>
|
<text x="760" y="180" class="tag-text">WORKBENCH</text>
|
||||||
<text x="520" y="468" class="node-title">Isolated run worker</text>
|
<text x="708" y="216" class="node-title">Inspect and act</text>
|
||||||
<text x="520" y="492" class="node-copy-strong">claim · checkpoint · cancel</text>
|
<text x="708" y="240" class="node-copy-strong">history · assessment</text>
|
||||||
<text x="520" y="512" class="node-copy">bounded output · child cleanup</text>
|
<text x="708" y="256" class="node-copy-strong">route tabs · search</text>
|
||||||
|
<text x="708" y="272" class="node-copy">filters · copy · logs</text>
|
||||||
|
<text x="708" y="288" class="node-copy">screenshots · child runs</text>
|
||||||
|
|
||||||
<rect x="744" y="412" width="200" height="120" rx="8" class="node"/>
|
<rect x="244" y="336" width="200" height="144" rx="8" class="node-muted"/>
|
||||||
<rect x="756" y="424" width="80" height="20" rx="4" class="tag"/>
|
<rect x="256" y="348" width="88" height="20" rx="4" class="tag"/>
|
||||||
<text x="796" y="438" class="tag-text">ENGINE</text>
|
<text x="300" y="364" class="tag-text">CONTROL</text>
|
||||||
<text x="760" y="468" class="node-title">theHarvester engine</text>
|
<text x="260" y="392" class="node-title">Durable run lifecycle</text>
|
||||||
<text x="760" y="492" class="node-copy-strong">cataloged sources · explicit actions</text>
|
<text x="260" y="420" class="node-copy-strong">queued · running</text>
|
||||||
<text x="760" y="512" class="node-copy">DNS · Shodan · RouteViews · vhost</text>
|
<text x="260" y="436" class="node-copy-strong">cancelling · cancelled</text>
|
||||||
|
<text x="260" y="452" class="node-copy">completed · failed</text>
|
||||||
|
<text x="260" y="468" class="node-copy">leases · deadlines</text>
|
||||||
|
|
||||||
<rect x="744" y="220" width="200" height="144" rx="8" class="node-focal"/>
|
<rect x="468" y="336" width="200" height="144" rx="8" class="node"/>
|
||||||
<rect x="756" y="232" width="96" height="20" rx="4" class="tag-teal"/>
|
<rect x="480" y="348" width="80" height="20" rx="4" class="tag"/>
|
||||||
<text x="804" y="246" class="tag-text tag-text-teal">EVIDENCE</text>
|
<text x="520" y="364" class="tag-text">WORKER</text>
|
||||||
<text x="760" y="276" class="node-title">Evidence & artifacts</text>
|
<text x="484" y="392" class="node-title">Isolated run worker</text>
|
||||||
<text x="760" y="300" class="node-copy-strong">CompletedResult · route records</text>
|
<text x="484" y="420" class="node-copy-strong">claim · execute</text>
|
||||||
<text x="760" y="320" class="node-copy-strong">SQLite · screenshots · logs</text>
|
<text x="484" y="436" class="node-copy-strong">checkpoint · cancel</text>
|
||||||
<text x="760" y="344" class="node-copy">partial evidence remains inspectable</text>
|
<text x="484" y="452" class="node-copy">58 sources</text>
|
||||||
|
<text x="484" y="468" class="node-copy">10 explicit actions</text>
|
||||||
|
|
||||||
<rect x="984" y="220" width="232" height="144" rx="8" class="node"/>
|
<rect x="692" y="336" width="228" height="144" rx="8" class="node-link"/>
|
||||||
<rect x="996" y="232" width="112" height="20" rx="4" class="tag"/>
|
<rect x="704" y="348" width="104" height="20" rx="4" class="tag-link"/>
|
||||||
<text x="1052" y="246" class="tag-text">WORKBENCH</text>
|
<text x="756" y="364" class="tag-text tag-text-link">INTERCHANGE</text>
|
||||||
<text x="1000" y="276" class="node-title">Inspect and export</text>
|
<text x="708" y="392" class="node-title">Import and export</text>
|
||||||
<text x="1000" y="300" class="node-copy-strong">history · assessment · route tabs</text>
|
<text x="708" y="420" class="node-copy-strong">JSONL · one run</text>
|
||||||
<text x="1000" y="320" class="node-copy-strong">screenshots · source outcomes · logs</text>
|
<text x="708" y="436" class="node-copy-strong">SQLite · completed set</text>
|
||||||
<text x="1000" y="340" class="node-copy">review actions · JSONL / SQLite export</text>
|
<text x="708" y="452" class="node-copy">screenshot files separate</text>
|
||||||
|
|
||||||
<line x1="40" y1="636" x2="1240" y2="636" stroke="rgba(14,29,34,.14)" stroke-width=".8"/>
|
<text x="48" y="512" class="node-copy">Provider credentials stay server-side. Lifecycle state never replaces terminal evidence status.</text>
|
||||||
<line x1="48" y1="676" x2="88" y2="676" class="connector" marker-end="url(#harvestview-architecture-arrow)"/>
|
|
||||||
<text x="104" y="680" class="legend">control and lifecycle handoff</text>
|
<line x1="40" y1="584" x2="920" y2="584" stroke="var(--rule)" stroke-width=".8"/>
|
||||||
<line x1="352" y1="676" x2="392" y2="676" class="connector-accent" marker-end="url(#harvestview-architecture-arrow-accent)"/>
|
<line x1="48" y1="616" x2="80" y2="616" class="connector" marker-end="url(#harvestview-arrow)"/>
|
||||||
<text x="408" y="680" class="legend">normalized evidence handoff</text>
|
<text x="92" y="620" class="legend">run control</text>
|
||||||
<rect x="676" y="664" width="20" height="20" rx="4" class="tag-teal"/>
|
<line x1="248" y1="616" x2="280" y2="616" class="connector-accent" marker-end="url(#harvestview-arrow-accent)"/>
|
||||||
<text x="708" y="680" class="legend">durable evidence boundary</text>
|
<text x="292" y="620" class="legend">evidence handoff</text>
|
||||||
<text x="1240" y="680" class="legend" text-anchor="end">provider credentials remain server-side</text>
|
<line x1="500" y1="616" x2="532" y2="616" class="connector-link dashed" marker-end="url(#harvestview-arrow-link)"/>
|
||||||
</svg>
|
<text x="544" y="620" class="legend">import path</text>
|
||||||
|
</svg>
|
||||||
|
|||||||
|
Before Width: | Height: | Size: 8.2 KiB After Width: | Height: | Size: 10 KiB |
@@ -1,137 +1,173 @@
|
|||||||
<?xml version="1.0" encoding="UTF-8"?>
|
<?xml version="1.0" encoding="UTF-8"?>
|
||||||
<svg viewBox="0 0 1280 720" xmlns="http://www.w3.org/2000/svg" role="img" aria-labelledby="run-evidence-architecture-title run-evidence-architecture-desc">
|
<svg viewBox="0 0 960 640" xmlns="http://www.w3.org/2000/svg" role="img" aria-labelledby="run-evidence-architecture-title run-evidence-architecture-desc">
|
||||||
<title id="run-evidence-architecture-title">Discovery routes and enrichment</title>
|
<title id="run-evidence-architecture-title">theHarvester discovery and evidence architecture</title>
|
||||||
<desc id="run-evidence-architecture-desc">theHarvester source selection and adapters feed passive, DNS, and direct actions before normalization into identity, contact, network, and action evidence routes.</desc>
|
<desc id="run-evidence-architecture-desc">Seven result capabilities and 58 cataloged discovery sources join ten explicit P0, P1, and P2 actions in one normalized evidence contract, which routes identity, network, and action findings to terminal, JSONL, SQLite, REST, HarvestView, screenshots, JSON, and XML outputs.</desc>
|
||||||
<defs>
|
<defs>
|
||||||
<style>@import url('https://fonts.googleapis.com/css2?family=Instrument+Serif:ital@0;1&family=Geist:wght@400;500;600&family=Geist+Mono:wght@400;500;600&display=swap');</style>
|
<style>
|
||||||
<style>
|
@import url('https://fonts.googleapis.com/css2?family=Instrument+Serif:ital@0;1&family=Geist:wght@400;500;600&family=Geist+Mono:wght@400;500;600&display=swap');
|
||||||
.title { font: 400 28px "Instrument Serif", Georgia, serif; fill: #0e1d22; }
|
:root {
|
||||||
.subtitle { font: 400 12px "Geist", system-ui, sans-serif; fill: #4f5e63; }
|
--paper: #2d3142;
|
||||||
.eyebrow { font: 600 8px "Geist Mono", ui-monospace, monospace; letter-spacing: .16em; fill: #4f5e63; }
|
--paper-2: #393e53;
|
||||||
.node-title { font: 600 12px "Geist", system-ui, sans-serif; fill: #0e1d22; }
|
--ink: #f5f5f5;
|
||||||
.node-copy { font: 400 8px "Geist Mono", ui-monospace, monospace; fill: #4f5e63; }
|
--muted: #bfc0c0;
|
||||||
.node-copy-strong { font: 500 8px "Geist Mono", ui-monospace, monospace; fill: #34464c; }
|
--soft: #8e98ac;
|
||||||
.zone { fill: rgba(14,29,34,.018); stroke: rgba(14,29,34,.14); stroke-width: .8; }
|
--rule: rgba(245,245,245,.16);
|
||||||
.node { fill: #fdfaf4; stroke: #889ca3; stroke-width: 1; }
|
--rule-strong: rgba(245,245,245,.28);
|
||||||
.node-muted { fill: #ede7dd; stroke: #889ca3; stroke-width: 1; }
|
--zone-fill: rgba(245,245,245,.025);
|
||||||
.node-focal { fill: #bcede4; stroke: #009685; stroke-width: 1.2; }
|
--accent: #f08a59;
|
||||||
.node-p1 { fill: #fde4bb; stroke: #723f00; stroke-width: 1; }
|
--accent-tint: rgba(240,138,89,.10);
|
||||||
.node-p2 { fill: #ffddd7; stroke: #a91515; stroke-width: 1; }
|
--link: #6a95d8;
|
||||||
.tag { fill: #f5efe7; stroke: #889ca3; stroke-width: .8; }
|
--link-tint: rgba(106,149,216,.10);
|
||||||
.tag-teal { fill: #bcede4; stroke: #009685; stroke-width: .8; }
|
}
|
||||||
.tag-p1 { fill: #fde4bb; stroke: #723f00; stroke-width: .8; }
|
@media (prefers-color-scheme: light) {
|
||||||
.tag-p2 { fill: #ffddd7; stroke: #a91515; stroke-width: .8; }
|
:root {
|
||||||
.tag-text { font: 600 8px "Geist Mono", ui-monospace, monospace; fill: #34464c; text-anchor: middle; }
|
--paper: #f5f5f5;
|
||||||
.tag-text-teal { fill: #00594c; }
|
--paper-2: #ececec;
|
||||||
.tag-text-p1 { fill: #723f00; }
|
--ink: #2d3142;
|
||||||
.tag-text-p2 { fill: #a91515; }
|
--muted: #4f5d75;
|
||||||
.connector { fill: none; stroke: #4f5e63; stroke-width: 1.2; }
|
--soft: #7a8399;
|
||||||
.legend { font: 400 8px "Geist Mono", ui-monospace, monospace; fill: #4f5e63; }
|
--rule: rgba(45,49,66,.14);
|
||||||
</style>
|
--rule-strong: rgba(45,49,66,.28);
|
||||||
<marker id="run-evidence-architecture-arrow" markerWidth="8" markerHeight="6" refX="7" refY="3" orient="auto"><polygon points="0 0, 8 3, 0 6" fill="#4f5e63"/></marker>
|
--zone-fill: rgba(45,49,66,.02);
|
||||||
<marker id="run-evidence-architecture-arrow-accent" markerWidth="8" markerHeight="6" refX="7" refY="3" orient="auto"><polygon points="0 0, 8 3, 0 6" fill="#009685"/></marker>
|
--accent: #b84f24;
|
||||||
<marker id="run-evidence-architecture-arrow-link" markerWidth="8" markerHeight="6" refX="7" refY="3" orient="auto"><polygon points="0 0, 8 3, 0 6" fill="#00548c"/></marker>
|
--accent-tint: rgba(235,108,54,.10);
|
||||||
</defs>
|
--link: #2e5aa8;
|
||||||
|
--link-tint: rgba(46,90,168,.08);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
.bg { fill: var(--paper); }
|
||||||
|
.title { font: 400 28px 'Instrument Serif', Georgia, serif; fill: var(--ink); }
|
||||||
|
.subtitle { font: 400 12px 'Geist', system-ui, sans-serif; fill: var(--muted); }
|
||||||
|
.eyebrow { font: 600 8px 'Geist Mono', ui-monospace, monospace; letter-spacing: .16em; fill: var(--soft); }
|
||||||
|
.zone { fill: var(--zone-fill); stroke: var(--rule); stroke-width: .8; }
|
||||||
|
.zone-mask { fill: var(--paper); }
|
||||||
|
.node { fill: var(--paper); stroke: var(--rule-strong); stroke-width: 1; }
|
||||||
|
.node-muted { fill: var(--paper-2); stroke: var(--rule-strong); stroke-width: 1; }
|
||||||
|
.node-link { fill: var(--link-tint); stroke: var(--link); stroke-width: 1; }
|
||||||
|
.node-focal { fill: var(--accent-tint); stroke: var(--accent); stroke-width: 1.2; }
|
||||||
|
.tag { fill: var(--paper-2); stroke: var(--rule-strong); stroke-width: .8; }
|
||||||
|
.tag-link { fill: var(--link-tint); stroke: var(--link); stroke-width: .8; }
|
||||||
|
.tag-accent { fill: var(--accent-tint); stroke: var(--accent); stroke-width: .8; }
|
||||||
|
.tag-text { font: 600 8px 'Geist Mono', ui-monospace, monospace; letter-spacing: .08em; fill: var(--muted); text-anchor: middle; }
|
||||||
|
.tag-text-link { fill: var(--link); }
|
||||||
|
.tag-text-accent { fill: var(--accent); }
|
||||||
|
.node-title { font: 600 16px 'Geist', system-ui, sans-serif; fill: var(--ink); }
|
||||||
|
.node-copy { font: 400 12px 'Geist Mono', ui-monospace, monospace; fill: var(--muted); }
|
||||||
|
.node-copy-strong { font: 500 12px 'Geist Mono', ui-monospace, monospace; fill: var(--ink); }
|
||||||
|
.connector { fill: none; stroke: var(--muted); stroke-width: 1.2; }
|
||||||
|
.connector-link { fill: none; stroke: var(--link); stroke-width: 1.2; }
|
||||||
|
.connector-accent { fill: none; stroke: var(--accent); stroke-width: 1.2; }
|
||||||
|
.marker-muted { fill: var(--muted); }
|
||||||
|
.marker-link { fill: var(--link); }
|
||||||
|
.marker-accent { fill: var(--accent); }
|
||||||
|
.legend { font: 500 12px 'Geist Mono', ui-monospace, monospace; fill: var(--muted); }
|
||||||
|
</style>
|
||||||
|
<marker id="run-evidence-arrow" markerWidth="8" markerHeight="8" refX="8" refY="4" orient="auto"><polygon points="0 0, 8 4, 0 8" class="marker-muted"/></marker>
|
||||||
|
<marker id="run-evidence-arrow-link" markerWidth="8" markerHeight="8" refX="8" refY="4" orient="auto"><polygon points="0 0, 8 4, 0 8" class="marker-link"/></marker>
|
||||||
|
<marker id="run-evidence-arrow-accent" markerWidth="8" markerHeight="8" refX="8" refY="4" orient="auto"><polygon points="0 0, 8 4, 0 8" class="marker-accent"/></marker>
|
||||||
|
</defs>
|
||||||
|
|
||||||
<rect width="1280" height="720" fill="#f5efe7"/>
|
<rect width="960" height="640" class="bg"/>
|
||||||
<text x="40" y="52" class="eyebrow">DISCOVERY · ENRICHMENT · EVIDENCE</text>
|
<text x="40" y="32" class="eyebrow">DISCOVERY · ENRICHMENT · ATTRIBUTABLE EVIDENCE</text>
|
||||||
<text x="40" y="88" class="title">Sources become attributable evidence</text>
|
<text x="40" y="64" class="title">From source selection to durable evidence</text>
|
||||||
<text x="40" y="112" class="subtitle">Catalog policy selects the work; one normalized result preserves routes, producers, outcomes, and partial evidence.</text>
|
<text x="40" y="88" class="subtitle">Every selected source and action reports its own outcome before results are deduplicated, attributed, and exported.</text>
|
||||||
|
|
||||||
<rect x="40" y="132" width="432" height="420" rx="8" class="zone"/>
|
<rect x="40" y="120" width="512" height="448" rx="8" class="zone"/>
|
||||||
<rect x="56" y="136" width="72" height="16" fill="#f5efe7"/>
|
<rect x="52" y="124" width="132" height="16" rx="4" class="zone-mask"/>
|
||||||
<text x="60" y="148" class="eyebrow">COLLECT</text>
|
<text x="60" y="136" class="eyebrow">SELECT + EXECUTE</text>
|
||||||
<rect x="492" y="132" width="284" height="420" rx="8" class="zone"/>
|
<rect x="576" y="120" width="344" height="448" rx="8" class="zone"/>
|
||||||
<rect x="508" y="136" width="72" height="16" fill="#f5efe7"/>
|
<rect x="588" y="124" width="124" height="16" rx="4" class="zone-mask"/>
|
||||||
<text x="512" y="148" class="eyebrow">ACTIONS</text>
|
<text x="596" y="136" class="eyebrow">EVIDENCE ROUTES</text>
|
||||||
<rect x="796" y="132" width="444" height="420" rx="8" class="zone"/>
|
|
||||||
<rect x="812" y="136" width="120" height="16" fill="#f5efe7"/>
|
|
||||||
<text x="816" y="148" class="eyebrow">NORMALIZE & ROUTE</text>
|
|
||||||
|
|
||||||
<line x1="224" y1="304" x2="264" y2="304" class="connector" marker-end="url(#run-evidence-architecture-arrow)"/>
|
<line x1="260" y1="216" x2="332" y2="216" class="connector" marker-end="url(#run-evidence-arrow)"/>
|
||||||
<path d="M464 256 H480 Q488 256 488 248 V220 Q488 212 496 212 H516" class="connector" marker-end="url(#run-evidence-architecture-arrow)"/>
|
<line x1="432" y1="288" x2="432" y2="304" class="connector" marker-end="url(#run-evidence-arrow)"/>
|
||||||
<path d="M464 304 H480 Q488 304 488 312 V324 Q488 332 496 332 H516" class="connector" marker-end="url(#run-evidence-architecture-arrow)"/>
|
<line x1="132" y1="424" x2="132" y2="400" class="connector-link" marker-end="url(#run-evidence-arrow-link)"/>
|
||||||
<path d="M464 352 H472 Q480 352 480 360 V444 Q480 452 488 452 H516" class="connector" marker-end="url(#run-evidence-architecture-arrow)"/>
|
<line x1="296" y1="424" x2="296" y2="400" class="connector" marker-end="url(#run-evidence-arrow)"/>
|
||||||
<path d="M752 212 H768 Q776 212 776 220 V268 Q776 276 784 276 H816" class="connector" marker-end="url(#run-evidence-architecture-arrow)"/>
|
<line x1="460" y1="424" x2="460" y2="400" class="connector" marker-end="url(#run-evidence-arrow)"/>
|
||||||
<path d="M752 332 H768 Q776 332 776 324 V312 Q776 304 784 304 H816" class="connector" marker-end="url(#run-evidence-architecture-arrow)"/>
|
<path d="M532 328 H544 Q552 328 552 320 V216 Q552 208 560 208 H600" class="connector" marker-end="url(#run-evidence-arrow)"/>
|
||||||
<path d="M752 452 H760 Q768 452 768 444 V340 Q768 332 776 332 H816" class="connector" marker-end="url(#run-evidence-architecture-arrow)"/>
|
<path d="M532 352 H584 Q592 352 592 360 H600" class="connector" marker-end="url(#run-evidence-arrow)"/>
|
||||||
<path d="M1016 280 H1020 Q1028 280 1028 272 V252 Q1028 244 1036 244 H1040" class="connector" marker-end="url(#run-evidence-architecture-arrow)"/>
|
<path d="M532 376 H560 Q568 376 568 384 V496 Q568 504 576 504 H600" class="connector" marker-end="url(#run-evidence-arrow)"/>
|
||||||
<path d="M1016 328 H1020 Q1028 328 1028 336 V432 Q1028 440 1036 440 H1040" class="connector" marker-end="url(#run-evidence-architecture-arrow)"/>
|
|
||||||
|
|
||||||
<rect x="64" y="224" width="160" height="160" rx="8" class="node-muted"/>
|
<rect x="60" y="144" width="200" height="144" rx="8" class="node"/>
|
||||||
<rect x="76" y="236" width="64" height="20" rx="4" class="tag"/>
|
<rect x="72" y="156" width="56" height="20" rx="4" class="tag"/>
|
||||||
<text x="108" y="250" class="tag-text">CATALOG</text>
|
<text x="100" y="172" class="tag-text">SCOPE</text>
|
||||||
<text x="80" y="280" class="node-title">Source selection</text>
|
<text x="76" y="208" class="node-title">Scope and source</text>
|
||||||
<text x="80" y="304" class="node-copy-strong">subdomains · emails · IPs</text>
|
<text x="76" y="228" class="node-title">selection</text>
|
||||||
<text x="80" y="324" class="node-copy-strong">ASNs · URLs · people</text>
|
<text x="76" y="252" class="node-copy-strong">7 result capabilities</text>
|
||||||
<text x="80" y="344" class="node-copy-strong">breaches · explicit names</text>
|
<text x="76" y="272" class="node-copy">names · all · no-hosts</text>
|
||||||
<text x="80" y="368" class="node-copy">activity · keys · routes</text>
|
|
||||||
|
|
||||||
<rect x="264" y="224" width="200" height="160" rx="8" class="node"/>
|
<rect x="332" y="144" width="200" height="144" rx="8" class="node"/>
|
||||||
<rect x="276" y="236" width="72" height="20" rx="4" class="tag"/>
|
<rect x="344" y="156" width="72" height="20" rx="4" class="tag"/>
|
||||||
<text x="312" y="250" class="tag-text">SOURCES</text>
|
<text x="380" y="172" class="tag-text">SOURCES</text>
|
||||||
<text x="280" y="280" class="node-title">Discovery adapters</text>
|
<text x="348" y="208" class="node-title">58 discovery adapters</text>
|
||||||
<text x="280" y="304" class="node-copy-strong">search · certificates · DNS</text>
|
<text x="348" y="232" class="node-copy-strong">search · CT · DNS data</text>
|
||||||
<text x="280" y="324" class="node-copy-strong">code · archives · threat intel</text>
|
<text x="348" y="248" class="node-copy-strong">code · archives</text>
|
||||||
<text x="280" y="348" class="node-copy">normalized observations</text>
|
<text x="348" y="264" class="node-copy-strong">threat intelligence</text>
|
||||||
<text x="280" y="368" class="node-copy">truthful source outcomes</text>
|
<text x="348" y="280" class="node-copy">activity · keys · outcomes</text>
|
||||||
|
|
||||||
<rect x="516" y="164" width="236" height="96" rx="8" class="node"/>
|
<rect x="60" y="304" width="472" height="96" rx="8" class="node-focal"/>
|
||||||
<rect x="528" y="176" width="40" height="20" rx="4" class="tag-teal"/>
|
<rect x="72" y="316" width="104" height="20" rx="4" class="tag-accent"/>
|
||||||
<text x="548" y="190" class="tag-text tag-text-teal">P0</text>
|
<text x="124" y="332" class="tag-text tag-text-accent">NORMALIZE</text>
|
||||||
<text x="584" y="192" class="node-title">Passive enrichment</text>
|
<text x="188" y="332" class="node-title">CompletedResult evidence contract</text>
|
||||||
<text x="532" y="220" class="node-copy-strong">Shodan host detail</text>
|
<text x="76" y="364" class="node-copy-strong">dedupe · provenance · source and action outcomes</text>
|
||||||
<text x="532" y="240" class="node-copy-strong">RouteViews routes · origins · RPKI</text>
|
<text x="76" y="384" class="node-copy">complete · partial · failed evidence · artifacts</text>
|
||||||
|
|
||||||
<rect x="516" y="284" width="236" height="96" rx="8" class="node-p1"/>
|
<rect x="60" y="424" width="144" height="128" rx="8" class="node-link"/>
|
||||||
<rect x="528" y="296" width="40" height="20" rx="4" class="tag-p1"/>
|
<rect x="72" y="436" width="40" height="20" rx="4" class="tag-link"/>
|
||||||
<text x="548" y="310" class="tag-text tag-text-p1">P1</text>
|
<text x="92" y="452" class="tag-text tag-text-link">P0</text>
|
||||||
<text x="584" y="312" class="node-title">DNS interaction</text>
|
<text x="76" y="480" class="node-title">Provider</text>
|
||||||
<text x="532" y="340" class="node-copy-strong">resolve · reverse lookup</text>
|
<text x="76" y="500" class="node-title">enrichment</text>
|
||||||
<text x="532" y="360" class="node-copy-strong">brute force · recursive discovery</text>
|
<text x="76" y="524" class="node-copy-strong">shodan</text>
|
||||||
|
<text x="76" y="544" class="node-copy-strong">routeviews</text>
|
||||||
|
|
||||||
<rect x="516" y="404" width="236" height="96" rx="8" class="node-p2"/>
|
<rect x="224" y="424" width="144" height="128" rx="8" class="node-muted"/>
|
||||||
<rect x="528" y="416" width="40" height="20" rx="4" class="tag-p2"/>
|
<rect x="236" y="436" width="40" height="20" rx="4" class="tag"/>
|
||||||
<text x="548" y="430" class="tag-text tag-text-p2">P2</text>
|
<text x="256" y="452" class="tag-text">P1</text>
|
||||||
<text x="584" y="432" class="node-title">Direct interaction</text>
|
<text x="240" y="480" class="node-title">DNS interaction</text>
|
||||||
<text x="532" y="460" class="node-copy-strong">vhost · screenshots · takeover</text>
|
<text x="240" y="500" class="node-copy-strong">dns-resolve</text>
|
||||||
<text x="532" y="480" class="node-copy-strong">API path discovery</text>
|
<text x="240" y="516" class="node-copy-strong">dns-lookup</text>
|
||||||
|
<text x="240" y="532" class="node-copy-strong">dns-brute</text>
|
||||||
|
<text x="240" y="548" class="node-copy-strong">dns-recursive</text>
|
||||||
|
|
||||||
<rect x="816" y="240" width="200" height="128" rx="8" class="node-focal"/>
|
<rect x="388" y="424" width="144" height="128" rx="8" class="node-focal"/>
|
||||||
<rect x="828" y="252" width="92" height="20" rx="4" class="tag-teal"/>
|
<rect x="400" y="436" width="40" height="20" rx="4" class="tag-accent"/>
|
||||||
<text x="874" y="266" class="tag-text tag-text-teal">NORMALIZED</text>
|
<text x="420" y="452" class="tag-text tag-text-accent">P2</text>
|
||||||
<text x="832" y="300" class="node-title">CompletedResult</text>
|
<text x="404" y="480" class="node-title">Direct action</text>
|
||||||
<text x="832" y="324" class="node-copy-strong">dedupe · provenance · outcomes</text>
|
<text x="404" y="500" class="node-copy-strong">api-scan</text>
|
||||||
<text x="832" y="344" class="node-copy-strong">artifacts · evidence status</text>
|
<text x="404" y="516" class="node-copy-strong">screenshot</text>
|
||||||
<text x="832" y="360" class="node-copy">partial evidence stays attributable</text>
|
<text x="404" y="532" class="node-copy-strong">takeover</text>
|
||||||
|
<text x="404" y="548" class="node-copy-strong">vhost</text>
|
||||||
|
|
||||||
<rect x="1040" y="164" width="200" height="160" rx="8" class="node"/>
|
<rect x="600" y="152" width="296" height="112" rx="8" class="node"/>
|
||||||
<rect x="1052" y="176" width="104" height="20" rx="4" class="tag"/>
|
<rect x="612" y="164" width="88" height="20" rx="4" class="tag"/>
|
||||||
<text x="1104" y="190" class="tag-text">IDENTITY</text>
|
<text x="656" y="180" class="tag-text">IDENTITY</text>
|
||||||
<text x="1056" y="220" class="node-title">Identity & contact routes</text>
|
<text x="616" y="208" class="node-title">Identity and contact</text>
|
||||||
<text x="1056" y="244" class="node-copy-strong">hostname / subdomain</text>
|
<text x="616" y="236" class="node-copy-strong">subdomains · emails · urls</text>
|
||||||
<text x="1056" y="264" class="node-copy-strong">email · URL</text>
|
<text x="616" y="256" class="node-copy-strong">people · person links · breaches</text>
|
||||||
<text x="1056" y="284" class="node-copy-strong">person · person-link · breach</text>
|
|
||||||
<text x="1056" y="308" class="node-copy">vhost observations stay on hostname</text>
|
|
||||||
|
|
||||||
<rect x="1040" y="348" width="200" height="184" rx="8" class="node"/>
|
<rect x="600" y="288" width="296" height="144" rx="8" class="node"/>
|
||||||
<rect x="1052" y="360" width="96" height="20" rx="4" class="tag"/>
|
<rect x="612" y="300" width="88" height="20" rx="4" class="tag"/>
|
||||||
<text x="1100" y="374" class="tag-text">NETWORK</text>
|
<text x="656" y="316" class="tag-text">NETWORK</text>
|
||||||
<text x="1056" y="404" class="node-title">Network & action routes</text>
|
<text x="616" y="344" class="node-title">Network and action evidence</text>
|
||||||
<text x="1056" y="428" class="node-copy-strong">IP · ASN · prefix</text>
|
<text x="616" y="372" class="node-copy-strong">ips · asns · prefixes</text>
|
||||||
<text x="1056" y="448" class="node-copy-strong">Shodan host · takeover</text>
|
<text x="616" y="392" class="node-copy-strong">Shodan hosts · takeover</text>
|
||||||
<text x="1056" y="468" class="node-copy-strong">scope extension</text>
|
<text x="616" y="412" class="node-copy-strong">recursive DNS · vhost evidence</text>
|
||||||
<text x="1056" y="488" class="node-copy-strong">external relationship · other</text>
|
<text x="616" y="428" class="node-copy">scope extension · external relations</text>
|
||||||
<text x="1056" y="516" class="node-copy">JSONL · SQLite · REST · HarvestView</text>
|
|
||||||
|
|
||||||
<line x1="40" y1="636" x2="1240" y2="636" stroke="rgba(14,29,34,.14)" stroke-width=".8"/>
|
<rect x="600" y="464" width="296" height="88" rx="8" class="node"/>
|
||||||
<line x1="48" y1="676" x2="88" y2="676" class="connector" marker-end="url(#run-evidence-architecture-arrow)"/>
|
<rect x="612" y="476" width="72" height="20" rx="4" class="tag"/>
|
||||||
<text x="104" y="680" class="legend">data handoff</text>
|
<text x="648" y="492" class="tag-text">OUTPUT</text>
|
||||||
<rect x="248" y="664" width="20" height="20" rx="4" class="tag-teal"/>
|
<text x="700" y="492" class="node-title">Durable outputs</text>
|
||||||
<text x="280" y="680" class="legend">normalized evidence</text>
|
<text x="616" y="520" class="node-copy-strong">Terminal · JSONL · SQLite · REST</text>
|
||||||
<rect x="464" y="664" width="20" height="20" rx="4" class="tag-p1"/>
|
<text x="616" y="540" class="node-copy">HarvestView · screenshots · JSON/XML</text>
|
||||||
<text x="496" y="680" class="legend">P1 · DNS interaction</text>
|
|
||||||
<rect x="704" y="664" width="20" height="20" rx="4" class="tag-p2"/>
|
<line x1="40" y1="584" x2="920" y2="584" stroke="var(--rule)" stroke-width=".8"/>
|
||||||
<text x="736" y="680" class="legend">P2 · direct interaction</text>
|
<line x1="48" y1="616" x2="80" y2="616" class="connector" marker-end="url(#run-evidence-arrow)"/>
|
||||||
<text x="1240" y="680" class="legend" text-anchor="end">P0 sources and enrichments remain passive provider requests</text>
|
<text x="92" y="620" class="legend">evidence flow</text>
|
||||||
</svg>
|
<rect x="240" y="604" width="20" height="20" rx="4" class="tag-link"/>
|
||||||
|
<text x="272" y="620" class="legend">P0 provider</text>
|
||||||
|
<rect x="432" y="604" width="20" height="20" rx="4" class="tag"/>
|
||||||
|
<text x="464" y="620" class="legend">P1 DNS</text>
|
||||||
|
<rect x="584" y="604" width="20" height="20" rx="4" class="tag-accent"/>
|
||||||
|
<text x="616" y="620" class="legend">P2 direct</text>
|
||||||
|
</svg>
|
||||||
|
|||||||
|
Before Width: | Height: | Size: 10 KiB After Width: | Height: | Size: 11 KiB |
@@ -0,0 +1,120 @@
|
|||||||
|
<?xml version="1.0" encoding="UTF-8"?>
|
||||||
|
<svg viewBox="0 0 1280 720" role="img" aria-labelledby="vhost-classifier-title vhost-classifier-desc" xmlns="http://www.w3.org/2000/svg">
|
||||||
|
<title id="vhost-classifier-title">Virtual-host response classifier</title>
|
||||||
|
<desc id="vhost-classifier-desc">The classifier rejects unusable or unstable comparisons, treats baseline matches as default, accepts non-body differences as distinct, and requires body-only differences to repeat before retaining evidence.</desc>
|
||||||
|
<defs>
|
||||||
|
<style>@import url('https://fonts.googleapis.com/css2?family=Instrument+Serif:ital@0;1&family=Geist:wght@400;500;600&family=Geist+Mono:wght@400;500;600&display=swap');</style>
|
||||||
|
<style>
|
||||||
|
.title { font: 400 32px "Instrument Serif", Georgia, serif; fill: #0e1d22; }
|
||||||
|
.subtitle { font: 400 16px "Geist", system-ui, sans-serif; fill: #4f5e63; }
|
||||||
|
.eyebrow { font: 600 8px "Geist Mono", ui-monospace, monospace; letter-spacing: .16em; fill: #4f5e63; }
|
||||||
|
.node-title { font: 600 16px "Geist", system-ui, sans-serif; fill: #0e1d22; }
|
||||||
|
.node-copy { font: 400 12px "Geist Mono", ui-monospace, monospace; fill: #4f5e63; }
|
||||||
|
.node-copy-strong { font: 500 12px "Geist Mono", ui-monospace, monospace; fill: #34464c; }
|
||||||
|
.node { fill: #fdfaf4; stroke: #889ca3; stroke-width: 1; }
|
||||||
|
.node-muted { fill: #ede7dd; stroke: #889ca3; stroke-width: 1; }
|
||||||
|
.node-focal { fill: #bcede4; stroke: #009685; stroke-width: 1.2; }
|
||||||
|
.decision { fill: #fdfaf4; stroke: #0e1d22; stroke-width: 1; }
|
||||||
|
.tag { fill: #f5efe7; stroke: #889ca3; stroke-width: .8; }
|
||||||
|
.tag-teal { fill: #bcede4; stroke: #009685; stroke-width: .8; }
|
||||||
|
.tag-text { font: 600 8px "Geist Mono", ui-monospace, monospace; fill: #34464c; text-anchor: middle; }
|
||||||
|
.tag-text-teal { fill: #00594c; }
|
||||||
|
.connector { fill: none; stroke: #4f5e63; stroke-width: 1.2; }
|
||||||
|
.connector-accent { fill: none; stroke: #009685; stroke-width: 1.2; }
|
||||||
|
.arrow-label { font: 600 12px "Geist Mono", ui-monospace, monospace; fill: #4f5e63; letter-spacing: .08em; text-anchor: middle; }
|
||||||
|
.legend { font: 400 12px "Geist Mono", ui-monospace, monospace; fill: #4f5e63; }
|
||||||
|
</style>
|
||||||
|
<marker id="vhost-classifier-arrow" markerWidth="8" markerHeight="6" refX="7" refY="3" orient="auto"><polygon points="0 0, 8 3, 0 6" fill="#4f5e63"/></marker>
|
||||||
|
<marker id="vhost-classifier-arrow-accent" markerWidth="8" markerHeight="6" refX="7" refY="3" orient="auto"><polygon points="0 0, 8 3, 0 6" fill="#009685"/></marker>
|
||||||
|
<marker id="vhost-classifier-arrow-link" markerWidth="8" markerHeight="6" refX="7" refY="3" orient="auto"><polygon points="0 0, 8 3, 0 6" fill="#00548c"/></marker>
|
||||||
|
</defs>
|
||||||
|
|
||||||
|
<rect width="1280" height="720" fill="#f5efe7"/>
|
||||||
|
<text x="40" y="52" class="eyebrow">VHOST RESPONSE CLASSIFIER</text>
|
||||||
|
<text x="40" y="88" class="title">A body-only difference must repeat</text>
|
||||||
|
<text x="40" y="112" class="subtitle">The candidate must differ from both the literal-IP context and a stable unknown-host control before evidence is retained.</text>
|
||||||
|
|
||||||
|
<line x1="640" y1="200" x2="640" y2="220" class="connector" marker-end="url(#vhost-classifier-arrow)"/>
|
||||||
|
<line x1="456" y1="268" x2="280" y2="268" class="connector" marker-end="url(#vhost-classifier-arrow)"/>
|
||||||
|
<line x1="640" y1="316" x2="640" y2="332" class="connector" marker-end="url(#vhost-classifier-arrow)"/>
|
||||||
|
<line x1="824" y1="380" x2="1000" y2="380" class="connector" marker-end="url(#vhost-classifier-arrow)"/>
|
||||||
|
<line x1="640" y1="428" x2="640" y2="444" class="connector" marker-end="url(#vhost-classifier-arrow)"/>
|
||||||
|
<path d="M824 492 H1000 Q1008 492 1008 484 H1024" class="connector-accent" marker-end="url(#vhost-classifier-arrow-accent)"/>
|
||||||
|
<path d="M640 540 V544 Q640 552 632 552 H464 Q456 552 456 560 V564" class="connector" marker-end="url(#vhost-classifier-arrow)"/>
|
||||||
|
<line x1="576" y1="592" x2="640" y2="592" class="connector" marker-end="url(#vhost-classifier-arrow)"/>
|
||||||
|
<line x1="960" y1="592" x2="1024" y2="592" class="connector-accent" marker-end="url(#vhost-classifier-arrow-accent)"/>
|
||||||
|
<path d="M800 632 V640 Q800 648 792 648 H312 Q304 648 304 640 V316 Q304 308 296 308 H188 Q180 308 180 300" class="connector" marker-end="url(#vhost-classifier-arrow)"/>
|
||||||
|
|
||||||
|
<rect x="340" y="240" width="56" height="20" rx="4" fill="#f5efe7"/>
|
||||||
|
<text x="368" y="256" class="arrow-label">NO</text>
|
||||||
|
<rect x="652" y="308" width="56" height="20" rx="4" fill="#f5efe7"/>
|
||||||
|
<text x="680" y="324" class="arrow-label">YES</text>
|
||||||
|
<rect x="864" y="352" width="56" height="20" rx="4" fill="#f5efe7"/>
|
||||||
|
<text x="892" y="368" class="arrow-label">YES</text>
|
||||||
|
<rect x="652" y="420" width="56" height="20" rx="4" fill="#f5efe7"/>
|
||||||
|
<text x="680" y="436" class="arrow-label">NO</text>
|
||||||
|
<rect x="844" y="460" width="104" height="20" rx="4" fill="#f5efe7"/>
|
||||||
|
<text x="896" y="476" class="arrow-label">NO · DIRECT</text>
|
||||||
|
<rect x="476" y="524" width="104" height="20" rx="4" fill="#f5efe7"/>
|
||||||
|
<text x="528" y="540" class="arrow-label">YES · BODY</text>
|
||||||
|
<rect x="964" y="564" width="56" height="20" rx="4" fill="#f5efe7"/>
|
||||||
|
<text x="992" y="580" class="arrow-label">YES</text>
|
||||||
|
<rect x="476" y="620" width="56" height="20" rx="4" fill="#f5efe7"/>
|
||||||
|
<text x="504" y="636" class="arrow-label">NO</text>
|
||||||
|
|
||||||
|
<rect x="480" y="128" width="320" height="72" rx="8" fill="#f5efe7"/>
|
||||||
|
<rect x="480" y="128" width="320" height="72" rx="8" class="node"/>
|
||||||
|
<text x="640" y="156" class="node-title" text-anchor="middle">Candidate + baseline fingerprints</text>
|
||||||
|
<text x="640" y="180" class="node-copy" text-anchor="middle">phase · status · location</text>
|
||||||
|
<text x="640" y="196" class="node-copy" text-anchor="middle">body size · body digest</text>
|
||||||
|
|
||||||
|
<polygon points="640,220 824,268 640,316 456,268" fill="#f5efe7"/>
|
||||||
|
<polygon points="640,220 824,268 640,316 456,268" class="decision"/>
|
||||||
|
<text x="640" y="264" class="node-title" text-anchor="middle">All responses usable</text>
|
||||||
|
<text x="640" y="284" class="node-title" text-anchor="middle">and controls stable?</text>
|
||||||
|
|
||||||
|
<rect x="80" y="236" width="200" height="64" rx="20" fill="#f5efe7"/>
|
||||||
|
<rect x="80" y="236" width="200" height="64" rx="20" class="node-muted"/>
|
||||||
|
<text x="180" y="260" class="node-title" text-anchor="middle">Indeterminate</text>
|
||||||
|
<text x="180" y="284" class="node-copy" text-anchor="middle">retain no finding</text>
|
||||||
|
|
||||||
|
<polygon points="640,332 824,380 640,428 456,380" fill="#f5efe7"/>
|
||||||
|
<polygon points="640,332 824,380 640,428 456,380" class="decision"/>
|
||||||
|
<text x="640" y="376" class="node-title" text-anchor="middle">Candidate matches</text>
|
||||||
|
<text x="640" y="396" class="node-title" text-anchor="middle">context or control?</text>
|
||||||
|
|
||||||
|
<rect x="1000" y="348" width="200" height="64" rx="20" fill="#f5efe7"/>
|
||||||
|
<rect x="1000" y="348" width="200" height="64" rx="20" class="node-muted"/>
|
||||||
|
<text x="1100" y="372" class="node-title" text-anchor="middle">Default</text>
|
||||||
|
<text x="1100" y="396" class="node-copy" text-anchor="middle">retain no finding</text>
|
||||||
|
|
||||||
|
<polygon points="640,444 824,492 640,540 456,492" fill="#f5efe7"/>
|
||||||
|
<polygon points="640,444 824,492 640,540 456,492" class="decision"/>
|
||||||
|
<text x="640" y="488" class="node-title" text-anchor="middle">Only body signals differ</text>
|
||||||
|
<text x="640" y="508" class="node-title" text-anchor="middle">from either baseline?</text>
|
||||||
|
|
||||||
|
<rect x="336" y="564" width="240" height="56" rx="8" fill="#f5efe7"/>
|
||||||
|
<rect x="336" y="564" width="240" height="56" rx="8" class="node"/>
|
||||||
|
<text x="456" y="596" class="node-title" text-anchor="middle">Repeat candidate request</text>
|
||||||
|
|
||||||
|
<polygon points="800,552 960,592 800,632 640,592" fill="#f5efe7"/>
|
||||||
|
<polygon points="800,552 960,592 800,632 640,592" class="decision"/>
|
||||||
|
<text x="800" y="588" class="node-title" text-anchor="middle">Usable body difference</text>
|
||||||
|
<text x="800" y="608" class="node-title" text-anchor="middle">repeats exactly?</text>
|
||||||
|
|
||||||
|
<rect x="1024" y="444" width="176" height="176" rx="8" fill="#f5efe7"/>
|
||||||
|
<rect x="1024" y="444" width="176" height="176" rx="8" class="node-focal"/>
|
||||||
|
<rect x="1040" y="460" width="80" height="20" rx="4" class="tag-teal"/>
|
||||||
|
<text x="1080" y="474" class="tag-text tag-text-teal">DISTINCT</text>
|
||||||
|
<text x="1040" y="508" class="node-title">Retain hostname</text>
|
||||||
|
<text x="1040" y="528" class="node-title">evidence</text>
|
||||||
|
<text x="1040" y="552" class="node-copy-strong">status · location</text>
|
||||||
|
<text x="1040" y="572" class="node-copy-strong">phase</text>
|
||||||
|
<text x="1040" y="592" class="node-copy-strong">repeated body</text>
|
||||||
|
<text x="1040" y="612" class="node-copy-strong">difference</text>
|
||||||
|
|
||||||
|
<line x1="40" y1="668" x2="1240" y2="668" stroke="rgba(14,29,34,.14)" stroke-width=".8"/>
|
||||||
|
<rect x="48" y="684" width="20" height="20" rx="4" class="tag-teal"/>
|
||||||
|
<text x="80" y="700" class="legend">distinct = stored</text>
|
||||||
|
<text x="1240" y="700" class="legend" text-anchor="end">default / indeterminate = not stored</text>
|
||||||
|
</svg>
|
||||||
|
After Width: | Height: | Size: 9.0 KiB |
@@ -0,0 +1,80 @@
|
|||||||
|
<?xml version="1.0" encoding="UTF-8"?>
|
||||||
|
<svg viewBox="0 0 1280 720" role="img" aria-labelledby="vhost-sweep-overview-title vhost-sweep-overview-desc" xmlns="http://www.w3.org/2000/svg">
|
||||||
|
<title id="vhost-sweep-overview-title">Bounded virtual-host sweep</title>
|
||||||
|
<desc id="vhost-sweep-overview-desc">Candidate hostnames and literal IP endpoints pass through exact-scope validation and one shared request and runtime budget before context, control, and candidate responses are sent to the classifier.</desc>
|
||||||
|
<defs>
|
||||||
|
<style>@import url('https://fonts.googleapis.com/css2?family=Instrument+Serif:ital@0;1&family=Geist:wght@400;500;600&family=Geist+Mono:wght@400;500;600&display=swap');</style>
|
||||||
|
<style>
|
||||||
|
.title { font: 400 32px "Instrument Serif", Georgia, serif; fill: #0e1d22; }
|
||||||
|
.subtitle { font: 400 16px "Geist", system-ui, sans-serif; fill: #4f5e63; }
|
||||||
|
.eyebrow { font: 600 8px "Geist Mono", ui-monospace, monospace; letter-spacing: .16em; fill: #4f5e63; }
|
||||||
|
.node-title { font: 600 16px "Geist", system-ui, sans-serif; fill: #0e1d22; }
|
||||||
|
.node-copy { font: 400 12px "Geist Mono", ui-monospace, monospace; fill: #4f5e63; }
|
||||||
|
.node-copy-strong { font: 500 12px "Geist Mono", ui-monospace, monospace; fill: #34464c; }
|
||||||
|
.node { fill: #fdfaf4; stroke: #889ca3; stroke-width: 1; }
|
||||||
|
.node-muted { fill: #ede7dd; stroke: #889ca3; stroke-width: 1; }
|
||||||
|
.node-focal { fill: #bcede4; stroke: #009685; stroke-width: 1.2; }
|
||||||
|
.tag { fill: #f5efe7; stroke: #889ca3; stroke-width: .8; }
|
||||||
|
.tag-teal { fill: #bcede4; stroke: #009685; stroke-width: .8; }
|
||||||
|
.tag-text { font: 600 8px "Geist Mono", ui-monospace, monospace; fill: #34464c; text-anchor: middle; }
|
||||||
|
.tag-text-teal { fill: #00594c; }
|
||||||
|
.connector { fill: none; stroke: #4f5e63; stroke-width: 1.2; }
|
||||||
|
.connector-accent { fill: none; stroke: #009685; stroke-width: 1.2; }
|
||||||
|
.arrow-label { font: 500 12px "Geist Mono", ui-monospace, monospace; fill: #4f5e63; letter-spacing: .06em; }
|
||||||
|
.legend { font: 400 12px "Geist Mono", ui-monospace, monospace; fill: #4f5e63; }
|
||||||
|
</style>
|
||||||
|
<marker id="vhost-sweep-overview-arrow" markerWidth="8" markerHeight="6" refX="7" refY="3" orient="auto"><polygon points="0 0, 8 3, 0 6" fill="#4f5e63"/></marker>
|
||||||
|
<marker id="vhost-sweep-overview-arrow-accent" markerWidth="8" markerHeight="6" refX="7" refY="3" orient="auto"><polygon points="0 0, 8 3, 0 6" fill="#009685"/></marker>
|
||||||
|
<marker id="vhost-sweep-overview-arrow-link" markerWidth="8" markerHeight="6" refX="7" refY="3" orient="auto"><polygon points="0 0, 8 3, 0 6" fill="#00548c"/></marker>
|
||||||
|
</defs>
|
||||||
|
|
||||||
|
<rect width="1280" height="720" fill="#f5efe7"/>
|
||||||
|
<text x="40" y="52" class="eyebrow">VIRTUAL HOST DISCOVERY · P2 DIRECT INTERACTION</text>
|
||||||
|
<text x="40" y="88" class="title">The sweep spends one bounded budget</text>
|
||||||
|
<text x="40" y="112" class="subtitle">Names stay inside the exact target boundary; endpoints stay literal IPs; every baseline, candidate, and confirmation counts.</text>
|
||||||
|
|
||||||
|
<path d="M280 292 V316 Q280 324 288 324 H552 Q560 324 560 332 V356" class="connector" marker-end="url(#vhost-sweep-overview-arrow)"/>
|
||||||
|
<path d="M1000 292 V316 Q1000 324 992 324 H728 Q720 324 720 332 V356" class="connector" marker-end="url(#vhost-sweep-overview-arrow)"/>
|
||||||
|
<line x1="640" y1="484" x2="640" y2="540" class="connector-accent" marker-end="url(#vhost-sweep-overview-arrow-accent)"/>
|
||||||
|
<rect x="660" y="496" width="152" height="20" rx="4" fill="#f5efe7"/>
|
||||||
|
<text x="668" y="512" class="arrow-label">ALLOCATES PROBES</text>
|
||||||
|
|
||||||
|
<rect x="80" y="164" width="400" height="128" rx="8" fill="#f5efe7"/>
|
||||||
|
<rect x="80" y="164" width="400" height="128" rx="8" class="node"/>
|
||||||
|
<rect x="96" y="180" width="96" height="20" rx="4" class="tag"/>
|
||||||
|
<text x="144" y="194" class="tag-text">CANDIDATES</text>
|
||||||
|
<text x="96" y="228" class="node-title">In-scope hostname pool</text>
|
||||||
|
<text x="96" y="256" class="node-copy-strong">harvested names · reverse DNS · explicit names</text>
|
||||||
|
<text x="96" y="280" class="node-copy">normalize, deduplicate, enforce exact target scope</text>
|
||||||
|
|
||||||
|
<rect x="800" y="164" width="400" height="128" rx="8" fill="#f5efe7"/>
|
||||||
|
<rect x="800" y="164" width="400" height="128" rx="8" class="node"/>
|
||||||
|
<rect x="816" y="180" width="88" height="20" rx="4" class="tag"/>
|
||||||
|
<text x="860" y="194" class="tag-text">ENDPOINTS</text>
|
||||||
|
<text x="816" y="228" class="node-title">Literal-IP endpoint pool</text>
|
||||||
|
<text x="816" y="256" class="node-copy-strong">harvested IPs or one operator override</text>
|
||||||
|
<text x="816" y="280" class="node-copy">HTTPS :443 across the set, then HTTP :80</text>
|
||||||
|
|
||||||
|
<rect x="440" y="356" width="400" height="128" rx="8" fill="#f5efe7"/>
|
||||||
|
<rect x="440" y="356" width="400" height="128" rx="8" class="node-focal"/>
|
||||||
|
<rect x="456" y="372" width="72" height="20" rx="4" class="tag-teal"/>
|
||||||
|
<text x="492" y="386" class="tag-text tag-text-teal">BOUNDS</text>
|
||||||
|
<text x="456" y="420" class="node-title">Candidate–endpoint sweep planner</text>
|
||||||
|
<text x="456" y="448" class="node-copy-strong">shared request cap · runtime deadline</text>
|
||||||
|
<text x="456" y="472" class="node-copy">per-request timeout · bounded concurrency</text>
|
||||||
|
|
||||||
|
<rect x="360" y="540" width="560" height="104" rx="8" fill="#f5efe7"/>
|
||||||
|
<rect x="360" y="540" width="560" height="104" rx="8" class="node-muted"/>
|
||||||
|
<rect x="376" y="552" width="112" height="20" rx="4" class="tag"/>
|
||||||
|
<text x="432" y="566" class="tag-text">ONE ENDPOINT</text>
|
||||||
|
<text x="504" y="568" class="node-title">Comparable response set</text>
|
||||||
|
<text x="376" y="600" class="node-copy-strong">literal-IP context · 3 shape-matched controls</text>
|
||||||
|
<text x="376" y="624" class="node-copy-strong">candidate with aligned TLS SNI + HTTP Host</text>
|
||||||
|
|
||||||
|
<line x1="40" y1="668" x2="1240" y2="668" stroke="rgba(14,29,34,.14)" stroke-width=".8"/>
|
||||||
|
<line x1="48" y1="696" x2="88" y2="696" class="connector" marker-end="url(#vhost-sweep-overview-arrow)"/>
|
||||||
|
<text x="104" y="700" class="legend">planning handoff</text>
|
||||||
|
<line x1="320" y1="696" x2="360" y2="696" class="connector-accent" marker-end="url(#vhost-sweep-overview-arrow-accent)"/>
|
||||||
|
<text x="376" y="700" class="legend">shared safety budget</text>
|
||||||
|
<text x="1240" y="700" class="legend" text-anchor="end">names are not resolved · redirects are not followed</text>
|
||||||
|
</svg>
|
||||||
|
After Width: | Height: | Size: 6.3 KiB |
@@ -13,7 +13,7 @@ If no file exists, theHarvester creates the default template under `~/.theHarves
|
|||||||
Run theHarvester once to create the user configuration, then edit:
|
Run theHarvester once to create the user configuration, then edit:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
${EDITOR:-vi} ~/.theHarvester/api-keys.yaml
|
vi ~/.theHarvester/api-keys.yaml
|
||||||
chmod 600 ~/.theHarvester/api-keys.yaml
|
chmod 600 ~/.theHarvester/api-keys.yaml
|
||||||
```
|
```
|
||||||
|
|
||||||
@@ -43,11 +43,13 @@ Do not commit populated configuration files. Prefer provider credentials scoped
|
|||||||
|
|
||||||
The [README source matrix](https://github.com/laramies/theHarvester/blob/dev/README.md#discovery-sources) lists each source's result routes, activity class, and credential requirement. Every source name links to its provider's site or documentation for current plans, quotas, and terms. The executable source catalog is the authoritative inventory.
|
The [README source matrix](https://github.com/laramies/theHarvester/blob/dev/README.md#discovery-sources) lists each source's result routes, activity class, and credential requirement. Every source name links to its provider's site or documentation for current plans, quotas, and terms. The executable source catalog is the authoritative inventory.
|
||||||
|
|
||||||
`censys.token` is a Censys Platform Personal Access Token. Set `organization_id` when searches should use an entitled organization. This source uses the Global Search API, which is unavailable to Free accounts because they are limited to asset lookups. Search API ID and secret fields are not accepted.
|
### Provider notes
|
||||||
|
|
||||||
`hibpverified` queries [HIBP's authenticated verified-domain endpoint](https://haveibeenpwned.com/API/v3#BreachedDomain). It is selected by its name, the `breaches` capability, and `all`. Without a configured HIBP API key it is skipped like other unavailable keyed sources. Live use requires a user-owned paid HIBP API key and a user-owned domain verified in that account. The keyless `haveibeenpwned` source queries only the public breach catalogue.
|
| Provider | Configuration and behavior |
|
||||||
|
| --- | --- |
|
||||||
`routeviews.key` is optional. RouteViews provides authenticated API keys to verified PeeringDB users. `--routeviews` uses the authenticated endpoint and documented 10-request-per-second allowance when the key is configured; otherwise it uses guest access at one request per second. If RouteViews rejects a configured key, the action fails without retrying as a guest; remove the key to select guest access. RouteViews does not document this as a paid subscription.
|
| Censys | `censys.token` is a Censys Platform Personal Access Token. Set `organization_id` to search through an entitled organization. The source uses the Global Search API, which Free accounts cannot access because they are limited to asset lookups. Search API ID and secret fields are not accepted. |
|
||||||
|
| HIBP verified domains | `hibpverified` queries [HIBP's authenticated verified-domain endpoint](https://haveibeenpwned.com/API/v3#BreachedDomain). Select it by name, through the `breaches` capability, or with `all`. Without a configured key, it is skipped like other unavailable keyed sources. Live use requires a user-owned paid HIBP API key and a domain verified in that account. The keyless `haveibeenpwned` source queries only the public breach catalogue. |
|
||||||
|
| RouteViews | `routeviews.key` is optional. A configured key selects the authenticated endpoint for PeeringDB-verified users and its documented 10-request-per-second allowance. Without a key, the action uses guest access at one request per second. If RouteViews rejects a configured key, the action fails instead of retrying as a guest. Remove the key to select guest access. RouteViews does not document this as a paid subscription. |
|
||||||
|
|
||||||
## Proxies
|
## Proxies
|
||||||
|
|
||||||
@@ -62,6 +64,8 @@ socks5:
|
|||||||
|
|
||||||
Enable configured proxies with `-p`:
|
Enable configured proxies with `-p`:
|
||||||
|
|
||||||
|
Network activity: provider-facing passive lookup through a configured proxy.
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
uv run theHarvester -d example.com -b crtsh -p
|
uv run theHarvester -d example.com -b crtsh -p
|
||||||
```
|
```
|
||||||
|
|||||||
+8
-6
@@ -9,17 +9,19 @@ DNS brute force, DNS resolution, virtual host discovery, takeover checks, screen
|
|||||||
## Start here
|
## Start here
|
||||||
|
|
||||||
1. [Install theHarvester](Installation).
|
1. [Install theHarvester](Installation).
|
||||||
2. Follow the [Quick Start](Quick-Start) for a small passive run.
|
2. Read [Responsible Use and Scope](Responsible-Use-and-Scope) before choosing a target or feature.
|
||||||
3. Read [Responsible Use and Scope](Responsible-Use-and-Scope) before enabling active features.
|
3. Follow the [Quick Start](Quick-Start) for a small passive run.
|
||||||
4. Add credentials through [Configuration and API Keys](Configuration-and-API-Keys) when a selected provider requires them.
|
4. Add credentials through [Configuration and API Keys](Configuration-and-API-Keys) when a selected provider requires them.
|
||||||
5. Learn where findings are stored in [Results and Local Data](Results-and-Local-Data).
|
5. Learn where findings are stored in [Results and Local Data](Results-and-Local-Data).
|
||||||
|
|
||||||
## Choose an interface
|
## Choose an interface
|
||||||
|
|
||||||
- **Command line:** best for interactive reconnaissance and report generation.
|
| Interface | Use it for |
|
||||||
- **HarvestView:** best for creating and inspecting durable local enumeration runs in a browser. See [REST API](Rest-API).
|
| --- | --- |
|
||||||
- **REST API:** best for authenticated local integrations and Swagger/ReDoc documentation. See [REST API](Rest-API).
|
| Command line | Interactive reconnaissance and report generation. |
|
||||||
- **Docker Compose:** packages HarvestView and the REST API, not the normal interactive CLI.
|
| HarvestView | Creating and inspecting durable local runs in a browser. See [REST API](Rest-API). |
|
||||||
|
| REST API | Authenticated local integrations and the generated Swagger/ReDoc reference. See [REST API](Rest-API). |
|
||||||
|
| Docker Compose | Running HarvestView and the REST API. It does not provide the normal interactive CLI. |
|
||||||
|
|
||||||
The repository [README](https://github.com/laramies/theHarvester) owns the current feature summary and source/result matrix. The live `theHarvester -h` output owns the complete CLI reference.
|
The repository [README](https://github.com/laramies/theHarvester) owns the current feature summary and source/result matrix. The live `theHarvester -h` output owns the complete CLI reference.
|
||||||
|
|
||||||
|
|||||||
@@ -25,7 +25,17 @@ An adapter normally provides:
|
|||||||
|
|
||||||
Do not return fields the provider did not supply. Normalize and deduplicate before returning results.
|
Do not return fields the provider did not supply. Normalize and deduplicate before returning results.
|
||||||
|
|
||||||
Return `None` when the provider conversation completed normally, including a valid zero-result response. Return an immutable `SourceExecutionReport` with a stable provider-specific reason for another terminal condition: `completed` for a successful early stop such as reaching the requested result limit, `failed` for provider or transport failure, `rate-limited` for a terminal rate limit, or `partial` when the provider confirms incomplete coverage. Adapters must not define mutable `execution_status` or `stop_reason` fields. The source runner checks for either field before execution and again before evidence collection. It owns finalization, promotes incomplete reports with retained normalized evidence to `partial`, and records a normal zero-result completion as `completed` with `no-results`.
|
Return one of these values from `process()`:
|
||||||
|
|
||||||
|
| Return | Meaning |
|
||||||
|
| --- | --- |
|
||||||
|
| `None` | The provider conversation completed normally, including a valid zero-result response. |
|
||||||
|
| `SourceExecutionReport('completed', reason)` | The source stopped successfully before its natural end, such as after reaching the requested result limit. |
|
||||||
|
| `SourceExecutionReport('failed', reason)` | A provider or transport failure ended the source. |
|
||||||
|
| `SourceExecutionReport('rate-limited', reason)` | A terminal rate limit ended the source. |
|
||||||
|
| `SourceExecutionReport('partial', reason)` | The provider confirmed incomplete coverage. |
|
||||||
|
|
||||||
|
Use a stable provider-specific reason. Do not define mutable `execution_status` or `stop_reason` fields. The source runner rejects either field before execution and before evidence collection. It owns finalization, promotes incomplete reports with retained normalized evidence to `partial`, and records a normal zero-result completion as `completed` with `no-results`.
|
||||||
|
|
||||||
### Own the provider conversation
|
### Own the provider conversation
|
||||||
|
|
||||||
|
|||||||
@@ -2,8 +2,14 @@
|
|||||||
|
|
||||||
Start with the smallest source set and least active behavior that can answer the engagement question. Replace `example.com` only with an authorized target.
|
Start with the smallest source set and least active behavior that can answer the engagement question. Replace `example.com` only with an authorized target.
|
||||||
|
|
||||||
|
Sources and explicit actions contribute different result routes and different levels of network activity. This map shows how they meet in one normalized evidence model:
|
||||||
|
|
||||||
|

|
||||||
|
|
||||||
## Passive subdomain discovery
|
## Passive subdomain discovery
|
||||||
|
|
||||||
|
Network activity: provider-facing passive lookups.
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
uv run theHarvester -d example.com -b crtsh,certspotter,commoncrawl
|
uv run theHarvester -d example.com -b crtsh,certspotter,commoncrawl
|
||||||
```
|
```
|
||||||
@@ -12,6 +18,8 @@ Use the [README source matrix](https://github.com/laramies/theHarvester/blob/dev
|
|||||||
|
|
||||||
## Save results for automation
|
## Save results for automation
|
||||||
|
|
||||||
|
Network activity: provider-facing discovery plus local report writes.
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
uv run theHarvester -d example.com -b crtsh,certspotter -f report
|
uv run theHarvester -d example.com -b crtsh,certspotter -f report
|
||||||
```
|
```
|
||||||
@@ -20,6 +28,8 @@ Use `report.jsonl` for automation, provenance, and run interchange. The same com
|
|||||||
|
|
||||||
## DNS resolution
|
## DNS resolution
|
||||||
|
|
||||||
|
Network activity: provider-facing discovery followed by resolver-facing DNS queries.
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
AUTHORIZED_DOMAIN='replace-with-a-domain-you-control'
|
AUTHORIZED_DOMAIN='replace-with-a-domain-you-control'
|
||||||
uv run theHarvester -d "$AUTHORIZED_DOMAIN" -b crtsh,certspotter -r
|
uv run theHarvester -d "$AUTHORIZED_DOMAIN" -b crtsh,certspotter -r
|
||||||
@@ -40,6 +50,8 @@ Reverse DNS (`-n`) uses an independent run-wide job set. It deduplicates address
|
|||||||
|
|
||||||
Configure the Shodan key, then enrich resolved hosts:
|
Configure the Shodan key, then enrich resolved hosts:
|
||||||
|
|
||||||
|
Network activity: provider-facing discovery, resolver-facing DNS, and Shodan API requests.
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
AUTHORIZED_DOMAIN='replace-with-a-domain-you-control'
|
AUTHORIZED_DOMAIN='replace-with-a-domain-you-control'
|
||||||
uv run theHarvester -d "$AUTHORIZED_DOMAIN" -b crtsh -r -s
|
uv run theHarvester -d "$AUTHORIZED_DOMAIN" -b crtsh -r -s
|
||||||
@@ -51,6 +63,8 @@ Shodan host enrichment runs after discovery and is separate from the `shodan` so
|
|||||||
|
|
||||||
Use only an owned or explicitly authorized target:
|
Use only an owned or explicitly authorized target:
|
||||||
|
|
||||||
|
Network activity: resolver-facing DNS queries for generated candidate names.
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
AUTHORIZED_DOMAIN='replace-with-a-domain-you-control'
|
AUTHORIZED_DOMAIN='replace-with-a-domain-you-control'
|
||||||
uv run theHarvester -d "$AUTHORIZED_DOMAIN" -c
|
uv run theHarvester -d "$AUTHORIZED_DOMAIN" -c
|
||||||
@@ -60,6 +74,8 @@ DNS brute force actively tests candidate names. Do not run it against `example.c
|
|||||||
|
|
||||||
## Takeover checks
|
## Takeover checks
|
||||||
|
|
||||||
|
Network activity: provider-facing discovery followed by target-facing checks.
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
AUTHORIZED_DOMAIN='replace-with-a-domain-you-control'
|
AUTHORIZED_DOMAIN='replace-with-a-domain-you-control'
|
||||||
uv run theHarvester -d "$AUTHORIZED_DOMAIN" -b crtsh,certspotter -t
|
uv run theHarvester -d "$AUTHORIZED_DOMAIN" -b crtsh,certspotter -t
|
||||||
@@ -67,10 +83,36 @@ uv run theHarvester -d "$AUTHORIZED_DOMAIN" -b crtsh,certspotter -t
|
|||||||
|
|
||||||
Treat matches as leads requiring manual confirmation. Do not claim a takeover from a fingerprint match alone.
|
Treat matches as leads requiring manual confirmation. Do not claim a takeover from a fingerprint match alone.
|
||||||
|
|
||||||
|
## RouteViews pivots
|
||||||
|
|
||||||
|
Network activity: provider-facing RouteViews requests.
|
||||||
|
|
||||||
|
Set an ASN that is part of the authorized assessment scope:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
AUTHORIZED_ASN='replace-with-an-authorized-asn'
|
||||||
|
uv run theHarvester -d "$AUTHORIZED_ASN" --routeviews -f report
|
||||||
|
```
|
||||||
|
|
||||||
|
The target may also be an authorized IP or CIDR. The action writes external routing relationships and RPKI observations to `report.jsonl`. These records can guide analysis, but they do not establish ownership, authorization, or reachability. RouteViews is never enabled by `-b all`; see [Responsible Use and Scope](Responsible-Use-and-Scope#routeviews) for its fixed limits.
|
||||||
|
|
||||||
|
## Virtual host discovery
|
||||||
|
|
||||||
|
Network activity: provider-facing discovery followed by target-facing requests to harvested IP endpoints.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
AUTHORIZED_DOMAIN='replace-with-a-domain-you-control'
|
||||||
|
uv run theHarvester -d "$AUTHORIZED_DOMAIN" -b rapiddns --vhost -f report
|
||||||
|
```
|
||||||
|
|
||||||
|
The action keeps only confirmed hostnames and records endpoint observations in JSONL. Read [Virtual Host Discovery](Virtual-Host-Discovery) before changing its endpoint, candidate, request, runtime, or TLS controls.
|
||||||
|
|
||||||
## Screenshots
|
## Screenshots
|
||||||
|
|
||||||
Install Chromium first, then choose an output directory:
|
Install Chromium first, then choose an output directory:
|
||||||
|
|
||||||
|
Network activity: provider-facing discovery, resolver-facing DNS, and target-facing browser requests.
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
uv run playwright install chromium
|
uv run playwright install chromium
|
||||||
AUTHORIZED_DOMAIN='replace-with-a-domain-you-control'
|
AUTHORIZED_DOMAIN='replace-with-a-domain-you-control'
|
||||||
@@ -81,6 +123,8 @@ Screenshots actively open discovered web services and may retain sensitive page
|
|||||||
|
|
||||||
## API-path scanning
|
## API-path scanning
|
||||||
|
|
||||||
|
Network activity: target-facing HTTP requests.
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
AUTHORIZED_DOMAIN='replace-with-a-domain-you-control'
|
AUTHORIZED_DOMAIN='replace-with-a-domain-you-control'
|
||||||
uv run theHarvester -d "$AUTHORIZED_DOMAIN" -a
|
uv run theHarvester -d "$AUTHORIZED_DOMAIN" -a
|
||||||
@@ -92,6 +136,8 @@ Provide a custom path wordlist with `-w FILE`. This sends requests directly to t
|
|||||||
|
|
||||||
When a combined run fails, rerun only the affected source with a conservative result limit:
|
When a combined run fails, rerun only the affected source with a conservative result limit:
|
||||||
|
|
||||||
|
Network activity: provider-facing lookup to the named source.
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
uv run theHarvester -d example.com -b source-name -l 10
|
uv run theHarvester -d example.com -b source-name -l 10
|
||||||
```
|
```
|
||||||
|
|||||||
+12
-10
@@ -1,11 +1,13 @@
|
|||||||
# Quick start
|
# Quick start
|
||||||
|
|
||||||
These examples use the IANA-reserved `example.com` domain as inert test data. Replace it only with a target that is within your authorized scope.
|
These examples use the IANA-reserved `example.com` domain to show command syntax. Passive providers still receive the target string. Replace it only with a target that is within your authorized scope.
|
||||||
|
|
||||||
## Run a small passive query
|
## Run a small passive query
|
||||||
|
|
||||||
From a source checkout:
|
From a source checkout:
|
||||||
|
|
||||||
|
Network activity: provider-facing passive lookups.
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
uv run theHarvester -d example.com -b crtsh,certspotter
|
uv run theHarvester -d example.com -b crtsh,certspotter
|
||||||
```
|
```
|
||||||
@@ -16,20 +18,24 @@ From Kali or another installed package, omit `uv run`:
|
|||||||
theHarvester -d example.com -b crtsh,certspotter
|
theHarvester -d example.com -b crtsh,certspotter
|
||||||
```
|
```
|
||||||
|
|
||||||
This queries two passive certificate sources and prints consolidated findings. Passive does not mean private: the selected providers receive the target string.
|
This queries two passive certificate sources and prints consolidated findings. An empty result can mean that the providers found nothing; it does not by itself prove that the run failed.
|
||||||
|
|
||||||
## Save a report
|
## Save a report
|
||||||
|
|
||||||
|
Network activity: provider-facing passive lookups plus local report writes.
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
uv run theHarvester -d example.com -b crtsh,certspotter -f report
|
uv run theHarvester -d example.com -b crtsh,certspotter -f report
|
||||||
```
|
```
|
||||||
|
|
||||||
This writes `report.jsonl` for automation and interchange. It also writes `report.json` and `report.xml` compatibility reports. See [Results and Local Data](Results-and-Local-Data).
|
This writes `report.jsonl` for automation and interchange. It also writes `report.json` and `report.xml` compatibility reports. The JSONL summary records source outcomes and evidence status, which distinguish a normal empty result from incomplete or failed work. See [Results and Local Data](Results-and-Local-Data).
|
||||||
|
|
||||||
## Resolve discovered hosts
|
## Resolve discovered hosts
|
||||||
|
|
||||||
DNS resolution creates additional network activity. Use it only within scope:
|
DNS resolution creates additional network activity. Use it only within scope:
|
||||||
|
|
||||||
|
Network activity: provider-facing discovery followed by resolver-facing DNS queries.
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
AUTHORIZED_DOMAIN='replace-with-a-domain-you-control'
|
AUTHORIZED_DOMAIN='replace-with-a-domain-you-control'
|
||||||
uv run theHarvester -d "$AUTHORIZED_DOMAIN" -b crtsh,certspotter -r
|
uv run theHarvester -d "$AUTHORIZED_DOMAIN" -b crtsh,certspotter -r
|
||||||
@@ -44,10 +50,6 @@ uv run theHarvester -d "$AUTHORIZED_DOMAIN" -b crtsh -r resolvers.txt
|
|||||||
|
|
||||||
## Choose sources deliberately
|
## Choose sources deliberately
|
||||||
|
|
||||||
The [README source matrix](https://github.com/laramies/theHarvester/blob/dev/README.md#discovery-sources) shows the result types and credential requirements for every current source.
|
1. Use the [README source matrix](https://github.com/laramies/theHarvester/blob/dev/README.md#discovery-sources) to find sources that return the result types you need.
|
||||||
|
2. Start with a small source set. `-b all` contacts many independent services, can consume quotas, and makes provider failures harder to isolate.
|
||||||
Do not start with `-b all`. It contacts many independent services and can consume quotas. It also increases runtime and makes provider failures harder to isolate.
|
3. Use `theHarvester -h` for the current option and source list.
|
||||||
|
|
||||||
Choose a small group of sources that provides the result types you need.
|
|
||||||
|
|
||||||
Use `theHarvester -h` for the current option and source list.
|
|
||||||
|
|||||||
@@ -10,23 +10,36 @@ Select only the providers needed for the task. Do not treat a provider key, bug-
|
|||||||
|
|
||||||
## Features that add network activity
|
## Features that add network activity
|
||||||
|
|
||||||
The following options require additional care:
|
| Option | Network path | What it does |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| `-r`, `--dns-resolve` | Resolver-facing | Resolves discovered names to A, AAAA, and CNAME records. |
|
||||||
|
| `-n`, `--dns-lookup` | Resolver-facing | Performs reverse DNS across discovered `/24` ranges. |
|
||||||
|
| `-c`, `--dns-brute` | Resolver-facing | Tries candidate subdomains against DNS. |
|
||||||
|
| `-t`, `--take-over` | Target-facing | Checks discovered hosts for takeover indicators. |
|
||||||
|
| `-s`, `--shodan` | Provider-facing | Enriches discovered hosts through Shodan. |
|
||||||
|
| `--routeviews` | Provider-facing | Queries RouteViews for external routing relationships. |
|
||||||
|
| `--vhost`, `--vhost-*` | Target-facing | Probes literal IP endpoints with candidate SNI and HTTP `Host` values. |
|
||||||
|
| `--screenshot DIR` | Target-facing | Opens discovered web services in a browser. |
|
||||||
|
| `-a`, `--api-scan` | Target-facing | Requests common API paths from the target. |
|
||||||
|
|
||||||
| Option | Behavior |
|
### DNS actions
|
||||||
| --- | --- |
|
|
||||||
| `-r`, `--dns-resolve` | Deduplicates discovered names, then resolves A, AAAA, and CNAME records once per name through one run-wide phase with at most 20 active hostname jobs and per-query resolver timeouts; it has no default query-count or phase-runtime ceiling. |
|
Hostname resolution deduplicates discovered names, then queries A, AAAA, and CNAME once per name. One run can have up to 20 active hostname jobs. Queries have resolver timeouts, but the phase has no default query-count or runtime ceiling.
|
||||||
| `-n`, `--dns-lookup` | Deduplicates addresses across discovered `/24` ranges, then performs reverse DNS through a separate run-wide phase with at most 20 active PTR jobs and per-query resolver timeouts; it has no default request-count or phase-runtime ceiling. |
|
|
||||||
| `-c`, `--dns-brute` | Tries candidate subdomains against DNS. |
|
Reverse DNS deduplicates addresses across overlapping `/24` ranges. It uses a separate run-wide phase with up to 20 active PTR jobs and per-query timeouts, but no default request-count or runtime ceiling.
|
||||||
| `-t`, `--take-over` | Checks discovered hosts for takeover indicators. |
|
|
||||||
| `-s`, `--shodan` | Enriches discovered hosts through Shodan. |
|
|
||||||
| `--routeviews` | Sends discovered IPs with sourced ASN attribution, or an explicitly targeted ASN, IP, or CIDR, to RouteViews and records external routing relationships. |
|
|
||||||
| `--vhost`, `--vhost-*` | Probes literal IP endpoints with candidate SNI and HTTP `Host` values. |
|
|
||||||
| `--screenshot DIR` | Opens discovered web services in a browser. |
|
|
||||||
| `-a`, `--api-scan` | Requests common API paths from the target. |
|
|
||||||
|
|
||||||
Use `--dns-resolvers IPS_OR_FILE` to select resolver addresses for DNS brute force, reverse lookup, or recursive DNS without enabling hostname resolution. `--dns-resolve [IPS_OR_FILE]` selects resolvers and enables hostname resolution.
|
Use `--dns-resolvers IPS_OR_FILE` to select resolver addresses for DNS brute force, reverse lookup, or recursive DNS without enabling hostname resolution. `--dns-resolve [IPS_OR_FILE]` selects resolvers and enables hostname resolution.
|
||||||
|
|
||||||
`--routeviews` is a separately selected P0 provider action. It is never enabled by `-b all` and ignores `-l`; one run is internally bounded to 300 sequential requests and 300 seconds. With no configured key it uses the documented guest rate of one request per second. A configured `routeviews.key` selects PeeringDB-verified authenticated access and the documented 10-request-per-second allowance. When selected for a domain run, it automatically queries harvested IPs backed by sourced IP-to-ASN attribution. It also accepts an ASN, IP, or CIDR supplied as the run target. Harvested IPs without that attribution are not sent, and bare ASN findings are not expanded into complete prefix inventories. IP lookups retain only the most-specific returned prefix, including all origins for a multi-origin prefix. The action does not recursively query returned prefixes and never promotes returned CIDRs into DNS or direct-action scope. Cloud and CDN routes can be useful relationship evidence, but route origins and RPKI states do not establish ownership or authorization.
|
### RouteViews
|
||||||
|
|
||||||
|
`--routeviews` is a separately selected P0 provider action. `-b all` never enables it, and `-l` does not change its fixed limit of 300 sequential requests or 300 seconds.
|
||||||
|
|
||||||
|
- Guest access runs at the documented rate of one request per second. A configured `routeviews.key` selects PeeringDB-verified authenticated access and the documented 10-request-per-second allowance.
|
||||||
|
- A domain run queries only harvested IPs backed by sourced IP-to-ASN attribution. Harvested IPs without that attribution are not sent, and bare ASN findings are not expanded into complete prefix inventories.
|
||||||
|
- An explicit run target may be an ASN, IP, or CIDR. IP lookups retain only the most-specific returned prefix, including every origin for a multi-origin prefix.
|
||||||
|
- Returned prefixes are not queried recursively or promoted into DNS or direct-action scope.
|
||||||
|
|
||||||
|
Treat cloud and CDN routes as relationship evidence. Route origins and RPKI states do not establish ownership, authorization, or reachability.
|
||||||
|
|
||||||
Use an owned or explicitly authorized domain for active examples. Do not substitute universities, public companies, bounty targets, or reserved example domains for recurring active scans.
|
Use an owned or explicitly authorized domain for active examples. Do not substitute universities, public companies, bounty targets, or reserved example domains for recurring active scans.
|
||||||
|
|
||||||
|
|||||||
+36
-7
@@ -2,10 +2,14 @@
|
|||||||
|
|
||||||
`harvestview` serves the web application at `/` and one versioned API for local automation.
|
`harvestview` serves the web application at `/` and one versioned API for local automation.
|
||||||
|
|
||||||
|

|
||||||
|
|
||||||
## Start the service
|
## Start the service
|
||||||
|
|
||||||
Set a long random API key before startup:
|
Set a long random API key before startup:
|
||||||
|
|
||||||
|
Network activity: local-only with the default loopback binding.
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
export THEHARVESTER_API_KEY='replace-with-a-long-random-value'
|
export THEHARVESTER_API_KEY='replace-with-a-long-random-value'
|
||||||
uv run harvestview
|
uv run harvestview
|
||||||
@@ -58,6 +62,8 @@ Provider credentials remain in theHarvester's server-side configuration. Request
|
|||||||
|
|
||||||
Source names and capability selectors share the `sources` array. Multiple capabilities select the union of matching sources and do not filter fields returned by those sources.
|
Source names and capability selectors share the `sources` array. Multiple capabilities select the union of matching sources and do not filter fields returned by those sources.
|
||||||
|
|
||||||
|
Network activity: the API request is local, but the queued run contacts the selected providers and any enabled action targets.
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
run_id="$(curl -s http://127.0.0.1:5000/api/v1/runs \
|
run_id="$(curl -s http://127.0.0.1:5000/api/v1/runs \
|
||||||
-X POST \
|
-X POST \
|
||||||
@@ -77,13 +83,26 @@ curl -s "http://127.0.0.1:5000/api/v1/runs/$run_id" \
|
|||||||
| jq '{status, evidence_status, results, source_executions, action_executions, artifacts}'
|
| jq '{status, evidence_status, results, source_executions, action_executions, artifacts}'
|
||||||
```
|
```
|
||||||
|
|
||||||
Run submission is asynchronous. Lifecycle status is `queued`, `running`, `cancelling`, `cancelled`, `completed`, or `failed`. Terminal evidence status is reported separately as `complete`, `partial`, or `failed` when evidence exists.
|
Run submission is asynchronous. Read the two status fields separately:
|
||||||
|
|
||||||
|
| Field | Values | Meaning |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| `status` | `queued`, `running`, `cancelling`, `cancelled`, `completed`, `failed` | Worker lifecycle state. |
|
||||||
|
| `evidence_status` | `complete`, `partial`, `failed` | Quality of terminal evidence when it exists. |
|
||||||
|
|
||||||
`source_workers` is the same positive concurrency used by CLI `-j` or `--source-workers` and HarvestView. It defaults to three, is reduced when fewer sources are selected, and never skips sources or limits their results.
|
`source_workers` is the same positive concurrency used by CLI `-j` or `--source-workers` and HarvestView. It defaults to three, is reduced when fewer sources are selected, and never skips sources or limits their results.
|
||||||
|
|
||||||
P1 DNS and P2 direct options are fields on the same run request. The OpenAPI schema shows their current defaults, limits, and descriptions. The server uses the operator-selected target and does not impose a public-only egress policy.
|
P1 DNS and P2 direct options are fields on the same run request. The OpenAPI schema shows their current defaults, limits, and descriptions. The server uses the operator-selected target and does not impose a public-only egress policy.
|
||||||
|
|
||||||
RouteViews is the explicit P0 `routeviews` action. When selected for a domain run, it automatically enriches harvested IPs that have sourced IP-to-ASN attribution. It also accepts an AS-prefixed ASN or IP address supplied as the run target. Harvested IPs without that attribution are not sent, and bare ASN findings are not expanded into complete prefix inventories. For IP pivots, only the most-specific matching prefix is retained, including every origin when that prefix is multi-origin. An explicit ASN target still requests its complete prefix inventory. The fixed internal budget is 300 sequential requests and 300 seconds; `limit` does not change it. A server-side `routeviews.key` is used automatically for PeeringDB-verified authenticated access at the documented 10-request-per-second allowance; otherwise the action uses guest access at one request per second. Provider credentials cannot be supplied in a run request. Returned prefixes remain external relationships and are never scheduled as DNS or P2 targets. The CLI also accepts a literal CIDR target.
|
### Query RouteViews
|
||||||
|
|
||||||
|
RouteViews is the explicit P0 `routeviews` action.
|
||||||
|
|
||||||
|
- A domain run enriches only harvested IPs that have sourced IP-to-ASN attribution. Harvested IPs without that attribution are not sent, and bare ASN findings are not expanded into complete prefix inventories.
|
||||||
|
- A run may target an AS-prefixed ASN or IP address. The CLI also accepts a literal CIDR. An IP pivot keeps the most-specific matching prefix and every origin for a multi-origin prefix. An explicit ASN target requests its complete prefix inventory.
|
||||||
|
- The fixed budget is 300 sequential requests and 300 seconds. `limit` does not change it.
|
||||||
|
- A server-side `routeviews.key` selects PeeringDB-verified authenticated access at the documented 10-request-per-second allowance. Without a key, the action uses guest access at one request per second. Requests cannot supply provider credentials.
|
||||||
|
- Returned prefixes remain external relationships. They are never scheduled as DNS or P2 targets.
|
||||||
|
|
||||||
```json
|
```json
|
||||||
{
|
{
|
||||||
@@ -97,13 +116,15 @@ RouteViews is the explicit P0 `routeviews` action. When selected for a domain ru
|
|||||||
|
|
||||||
Screenshots and DNS brute force can run directly against an authorized hostname without repeating discovery. Submit an empty `sources` array and select one action:
|
Screenshots and DNS brute force can run directly against an authorized hostname without repeating discovery. Submit an empty `sources` array and select one action:
|
||||||
|
|
||||||
|
Network activity: target-facing for screenshots and resolver-facing for DNS brute force. The API request itself is local.
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
curl -s http://127.0.0.1:5000/api/v1/runs \
|
curl -s http://127.0.0.1:5000/api/v1/runs \
|
||||||
-X POST \
|
-X POST \
|
||||||
-H "X-API-Key: $THEHARVESTER_API_KEY" \
|
-H "X-API-Key: $THEHARVESTER_API_KEY" \
|
||||||
-H 'Content-Type: application/json' \
|
-H 'Content-Type: application/json' \
|
||||||
-d '{
|
-d '{
|
||||||
"target": "subdomain.example.com",
|
"target": "replace-with-an-authorized-hostname",
|
||||||
"sources": [],
|
"sources": [],
|
||||||
"screenshot": true
|
"screenshot": true
|
||||||
}' \
|
}' \
|
||||||
@@ -116,7 +137,7 @@ The action catalog and run request use the same names. For example, set `takeove
|
|||||||
|
|
||||||
```json
|
```json
|
||||||
{
|
{
|
||||||
"target": "api.example.com",
|
"target": "replace-with-an-authorized-hostname",
|
||||||
"sources": [],
|
"sources": [],
|
||||||
"api_scan": true,
|
"api_scan": true,
|
||||||
"api_scan_paths": ["/api/v2", "/health"]
|
"api_scan_paths": ["/api/v2", "/health"]
|
||||||
@@ -127,6 +148,8 @@ Every custom API scan entry must be a URL path beginning with `/`. The API does
|
|||||||
|
|
||||||
Virtual host discovery is the `vhost` action. An action-only run must provide both a literal-IP endpoint and at least one in-scope hostname candidate:
|
Virtual host discovery is the `vhost` action. An action-only run must provide both a literal-IP endpoint and at least one in-scope hostname candidate:
|
||||||
|
|
||||||
|
Network activity: target-facing literal-IP requests with the candidate in SNI and HTTP `Host`.
|
||||||
|
|
||||||
```json
|
```json
|
||||||
{
|
{
|
||||||
"target": "authorized.example",
|
"target": "authorized.example",
|
||||||
@@ -142,7 +165,7 @@ HarvestView's subdomain action buttons call this route and create a separate run
|
|||||||
|
|
||||||
## Import and export
|
## Import and export
|
||||||
|
|
||||||
Import records existing evidence and never contacts the target. For one run, send the same JSONL written by `theHarvester -f NAME`:
|
These operations are local-only. Import records existing evidence without contacting providers or targets. For one run, send the same JSONL written by `theHarvester -f NAME`:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
curl -s "http://127.0.0.1:5000/api/v1/runs/import?filename=report.jsonl" \
|
curl -s "http://127.0.0.1:5000/api/v1/runs/import?filename=report.jsonl" \
|
||||||
@@ -168,7 +191,13 @@ curl -s "http://127.0.0.1:5000/api/v1/runs/import-database?filename=stash.sqlite
|
|||||||
| jq
|
| jq
|
||||||
```
|
```
|
||||||
|
|
||||||
The server checks the SQLite header, integrity, schema, and each completed run before copying it. Original run IDs are preserved. Exact duplicates are skipped, while a reused ID with different evidence is rejected. Close the source process or checkpoint its WAL before uploading the database. Screenshot metadata is imported, but screenshot files must be copied separately. The default upload ceiling is 1 GiB and can be changed with `THEHARVESTER_MAX_DATABASE_IMPORT_BYTES`.
|
Before importing SQLite:
|
||||||
|
|
||||||
|
- Close the source process or checkpoint its WAL.
|
||||||
|
- Expect the server to check the SQLite header, integrity, schema, and each completed run before copying it.
|
||||||
|
- Original run IDs are preserved. Exact duplicates are skipped; a reused ID with different evidence is rejected.
|
||||||
|
- Screenshot metadata is imported, but screenshot files must be copied separately.
|
||||||
|
- The default upload ceiling is 1 GiB. Change it with `THEHARVESTER_MAX_DATABASE_IMPORT_BYTES`.
|
||||||
|
|
||||||
Export every completed run as a consistent database that can be imported elsewhere:
|
Export every completed run as a consistent database that can be imported elsewhere:
|
||||||
|
|
||||||
@@ -178,7 +207,7 @@ curl -s "http://127.0.0.1:5000/api/v1/runs/export-database" \
|
|||||||
-o theharvester-completed-runs.sqlite
|
-o theharvester-completed-runs.sqlite
|
||||||
```
|
```
|
||||||
|
|
||||||
The export is rebuilt from canonical completed evidence, so it excludes queue state, cancellation state, worker leases, and legacy observations. It includes screenshot metadata but not screenshot files. The server checkpoints and closes the temporary database before download; no manual WAL handling is required.
|
The export contains canonical completed evidence and screenshot metadata. It excludes queue state, cancellation state, worker leases, and legacy observations. Screenshot files are also excluded. The server checkpoints and closes the temporary database before download, so no manual WAL handling is required.
|
||||||
|
|
||||||
Export one normalized result set in the same streamable format:
|
Export one normalized result set in the same streamable format:
|
||||||
|
|
||||||
|
|||||||
@@ -1,6 +1,14 @@
|
|||||||
# Results and local data
|
# Results and local data
|
||||||
|
|
||||||
theHarvester can print findings, write reports, retain selected records in SQLite, save screenshots, and expose durable run records through the API. These outputs have different schemas and sensitivity.
|
Choose the output that matches the next task:
|
||||||
|
|
||||||
|
| Output | Use it for | Important limit |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| Terminal | Interactive review | Not a stable automation interface. |
|
||||||
|
| JSONL | Automation, one-run interchange, and provenance | One summary record followed by normalized findings. |
|
||||||
|
| SQLite | Local history and bulk transfer of completed runs | Contains sensitive evidence across runs. |
|
||||||
|
| JSON or XML | Compatibility with older consumers | Does not preserve per-item source attribution. |
|
||||||
|
| REST API | Lifecycle state, normalized results, and local integrations | Requires API authentication. |
|
||||||
|
|
||||||
## Terminal output
|
## Terminal output
|
||||||
|
|
||||||
@@ -10,6 +18,8 @@ The CLI groups findings by result type. It can also print separate enrichment, s
|
|||||||
|
|
||||||
Use `-f NAME` to write a durable run report:
|
Use `-f NAME` to write a durable run report:
|
||||||
|
|
||||||
|
Network activity: provider-facing passive discovery plus local report writes.
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
uv run theHarvester -d example.com -b crtsh,certspotter -f report
|
uv run theHarvester -d example.com -b crtsh,certspotter -f report
|
||||||
```
|
```
|
||||||
@@ -46,28 +56,28 @@ The database persists across runs. Account for it in engagement cleanup and rete
|
|||||||
|
|
||||||
Completed CLI executions store one normalized terminal record keyed by run UUID. API executions use the same database by default and may override its path with `THEHARVESTER_RUN_DB`. Lifecycle rows keep queue, cancellation, and worker state separate from terminal evidence. Imported JSONL is stored without executing discovery, and source attribution is rebuilt from each finding's `sources` array. A SQLite import copies every completed run after validating the database and keeps the original run IDs.
|
Completed CLI executions store one normalized terminal record keyed by run UUID. API executions use the same database by default and may override its path with `THEHARVESTER_RUN_DB`. Lifecycle rows keep queue, cancellation, and worker state separate from terminal evidence. Imported JSONL is stored without executing discovery, and source attribution is rebuilt from each finding's `sources` array. A SQLite import copies every completed run after validating the database and keeps the original run IDs.
|
||||||
|
|
||||||
The normalized persistence model can represent active-action provenance and artifact metadata through six core tables:
|
Six tables hold completed evidence:
|
||||||
|
|
||||||
- `runs`: one finite enumeration run;
|
- `runs`: one finite enumeration run.
|
||||||
- `executions`: each passive source or active action represented by the model;
|
- `executions`: each passive source or active action represented by the model.
|
||||||
- `results`: deduplicated hostnames, IPs, emails, URLs, and structured outputs;
|
- `results`: deduplicated hostnames, IPs, emails, URLs, and structured outputs.
|
||||||
- `result_origins`: which execution produced each result;
|
- `result_origins`: the execution that produced each result.
|
||||||
- `asn_attributions`: sourced organization labels linking an ASN result to the exact hostname or IP result supplied by the same execution; and
|
- `asn_attributions`: sourced organization labels linking an ASN result to the exact hostname or IP supplied by the same execution.
|
||||||
- `artifacts`: files such as screenshots, linked to their creating action and subject result.
|
- `artifacts`: files such as screenshots, linked to their creating action and subject result.
|
||||||
|
|
||||||
Virtual-host evidence stays inside this model. `results` holds one `hostname` row, `result_origins` links it to the `vhost` action execution, and the result's `details_json` contains the canonical endpoint observation array. If one hostname is distinct on several IP endpoints, it remains one result with several observations.
|
Virtual-host evidence uses the same model. `results` holds one `hostname` row, `result_origins` links it to the `vhost` action execution, and `details_json` contains the endpoint observations. A hostname that is distinct on several IP endpoints remains one result with several observations.
|
||||||
|
|
||||||
Runtime collection records passive source executions plus DNS, takeover, Shodan, and API endpoint scan executions and origins. Screenshot actions attach file metadata to their captured hostname or URL without creating screenshot findings.
|
Runtime collection records passive source executions plus DNS, takeover, Shodan, and API endpoint scan executions and origins. Screenshot actions attach file metadata to their captured hostname or URL without creating screenshot findings.
|
||||||
|
|
||||||
RouteViews creates `prefix` results with `scope: external-relationship` and `routeviews` action provenance. Native observations distinguish one ASN-prefix origin claim, one collector/peer BGP route, and one RPKI validation state. They are routing evidence, not registration, ownership, authorization, reachability, or expanded target scope.
|
RouteViews creates `prefix` results with `scope: external-relationship` and `routeviews` action provenance. Its observations distinguish ASN-prefix origin claims, collector/peer BGP routes, and RPKI validation states. Treat them as routing evidence, not registration, ownership, authorization, reachability, or expanded target scope.
|
||||||
|
|
||||||
URLScan, ONYPHE, and Shodan can attach a provider organization label to an ASN. SQLite stores each relationship in `asn_attributions`; JSONL, the API, CLI output, and HarvestView project the same typed observation. Labels remain time-bound provider evidence, so missing or conflicting values are retained rather than replaced by one ASN owner property. Shodan's documented `org` field is used for the organization label; its separate `isp` field remains part of the existing Shodan payload and is not treated as equivalent.
|
URLScan, ONYPHE, and Shodan can attach a provider organization label to an ASN. SQLite stores each relationship in `asn_attributions`; JSONL, the API, CLI output, and HarvestView expose the same typed observation. These labels are time-bound provider evidence. Missing or conflicting values remain separate instead of being replaced by one ASN owner property. Shodan's documented `org` field supplies the organization label; its `isp` field remains part of the Shodan payload and is not treated as equivalent.
|
||||||
|
|
||||||
Every discovered URL is stored as the `url` result kind. Its source or action origins identify whether it came from BuiltWith, GitLab, RocketReach, API scanning, or another producer; provider-specific URL kinds are not stored.
|
Every discovered URL is stored as the `url` result kind. Its source or action origins identify whether it came from BuiltWith, GitLab, RocketReach, API scanning, or another producer; provider-specific URL kinds are not stored.
|
||||||
|
|
||||||
Hostname and IP evidence use the `hostname` and `ip` result kinds in SQLite, JSONL, the API, and HarvestView. A hostname may be the authorized target itself or a subordinate name, so the result kind does not claim that every value is a subdomain.
|
Hostname and IP evidence use the `hostname` and `ip` result kinds in SQLite, JSONL, the API, and HarvestView. A hostname may be the authorized target itself or a subordinate name, so the result kind does not claim that every value is a subdomain.
|
||||||
|
|
||||||
Two operational tables support the API without changing those six evidence concepts: `run_records` stores queue and lifecycle state, and `run_worker_leases` prevents two local workers from claiming the same queue. Runless rows are stored in `legacy_observations`. SQLite upgrades supported schemas automatically during normal initialization.
|
Two operational tables support the API: `run_records` stores queue and lifecycle state, and `run_worker_leases` prevents two local workers from claiming the same queue. Runless rows are stored in `legacy_observations`. SQLite upgrades supported schemas during normal initialization.
|
||||||
|
|
||||||
## Screenshots
|
## Screenshots
|
||||||
|
|
||||||
|
|||||||
@@ -34,20 +34,25 @@ The first existing file wins.
|
|||||||
|
|
||||||
Check the [README source matrix](https://github.com/laramies/theHarvester/blob/dev/README.md#discovery-sources). Configure credentials for that provider or choose a keyless source. `-q` suppresses missing-key notices; it does not make keyed providers work without credentials.
|
Check the [README source matrix](https://github.com/laramies/theHarvester/blob/dev/README.md#discovery-sources). Configure credentials for that provider or choose a keyless source. `-q` suppresses missing-key notices; it does not make keyed providers work without credentials.
|
||||||
|
|
||||||
## Provider errors, timeouts, or empty results
|
## A provider fails, times out, or returns no results
|
||||||
|
|
||||||
Rerun one source with a small limit:
|
Rerun one source with a small limit and save its summary:
|
||||||
|
|
||||||
|
Network activity: provider-facing lookup plus local report writes.
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
uv run theHarvester -d example.com -b source-name -l 10
|
uv run theHarvester -d example.com -b source-name -l 10 -f diagnostic
|
||||||
|
jq -c 'select(.type == "summary") | {evidence_status, source_executions}' diagnostic.jsonl
|
||||||
```
|
```
|
||||||
|
|
||||||
Then check:
|
Read the source outcome before treating an empty result as a failure. A completed `no-results` outcome means the provider conversation finished normally. A `partial`, `failed`, or `rate-limited` outcome names a separate coverage problem.
|
||||||
|
|
||||||
|
If the source did not complete normally, check:
|
||||||
|
|
||||||
- provider status and current API documentation;
|
- provider status and current API documentation;
|
||||||
- credential validity and subscription access;
|
- credential validity and subscription access;
|
||||||
- provider rate limits or temporary blocking of shared CI/cloud addresses;
|
- provider rate limits or temporary blocking of shared CI/cloud addresses;
|
||||||
- whether the source legitimately has no results for the target.
|
- whether the provider documents the target or query type as supported.
|
||||||
|
|
||||||
Do not post credentials, private targets, account details, or raw provider responses in a public issue.
|
Do not post credentials, private targets, account details, or raw provider responses in a public issue.
|
||||||
|
|
||||||
@@ -62,6 +67,8 @@ uv run theHarvester -d "$AUTHORIZED_DOMAIN" -b crtsh -r resolvers.txt
|
|||||||
|
|
||||||
If theHarvester reports invalid resolvers, remove hostnames, comments, blank values, or `host:port` entries; the resolver list accepts IP addresses only. Check local firewall and DNS policy before substituting public resolvers.
|
If theHarvester reports invalid resolvers, remove hostnames, comments, blank values, or `host:port` entries; the resolver list accepts IP addresses only. Check local firewall and DNS policy before substituting public resolvers.
|
||||||
|
|
||||||
|
The input is accepted when the invalid-resolver message is gone. A valid resolver list does not guarantee that a name has DNS records.
|
||||||
|
|
||||||
## Screenshots and Chromium
|
## Screenshots and Chromium
|
||||||
|
|
||||||
Install the browser used by Playwright:
|
Install the browser used by Playwright:
|
||||||
@@ -87,6 +94,8 @@ Then open [http://127.0.0.1:5000/docs](http://127.0.0.1:5000/docs).
|
|||||||
- `429`: a reverse proxy or remote provider applied its own rate limit. `harvestview` has no built-in request limiter.
|
- `429`: a reverse proxy or remote provider applied its own rate limit. `harvestview` has no built-in request limiter.
|
||||||
- `503` when creating a run: the execution worker is disabled or unavailable.
|
- `503` when creating a run: the execution worker is disabled or unavailable.
|
||||||
|
|
||||||
|
After correcting the cause, retry `GET /api/v1/sources`. A successful authenticated response returns the source catalog.
|
||||||
|
|
||||||
## Docker
|
## Docker
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
|
|||||||
@@ -4,6 +4,37 @@ A web server can host several sites on one IP address. The site selected by the
|
|||||||
|
|
||||||
This is a P2 direct action. It sends requests to harvested IP addresses or to one literal-IP endpoint supplied by the operator. Use it only when the target, addresses, ports, and technique are covered by the assessment authorization.
|
This is a P2 direct action. It sends requests to harvested IP addresses or to one literal-IP endpoint supplied by the operator. Use it only when the target, addresses, ports, and technique are covered by the assessment authorization.
|
||||||
|
|
||||||
|
## Run a bounded sweep
|
||||||
|
|
||||||
|
Network activity: provider-facing discovery followed by target-facing requests to harvested IP endpoints.
|
||||||
|
|
||||||
|
Harvest hostnames and IP addresses, then run the sweep with its bounded defaults:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
AUTHORIZED_DOMAIN='replace-with-a-domain-you-control'
|
||||||
|
|
||||||
|
uv run theHarvester \
|
||||||
|
-d "$AUTHORIZED_DOMAIN" \
|
||||||
|
-b rapiddns \
|
||||||
|
--vhost \
|
||||||
|
-f report
|
||||||
|
```
|
||||||
|
|
||||||
|
`rapiddns` is used here because it can return both hostnames and literal IP addresses. If the selected sources return only hostnames, enable a DNS action that contributes IP evidence or supply `--vhost-endpoint`.
|
||||||
|
|
||||||
|
Repeat `--vhost-candidate` to add names that are already authorized but were not returned by the selected sources:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
uv run theHarvester \
|
||||||
|
-d "$AUTHORIZED_DOMAIN" \
|
||||||
|
-b rapiddns \
|
||||||
|
--vhost \
|
||||||
|
--vhost-candidate "admin.${AUTHORIZED_DOMAIN}" \
|
||||||
|
--vhost-candidate "preview.${AUTHORIZED_DOMAIN}"
|
||||||
|
```
|
||||||
|
|
||||||
|
Virtual host discovery uses direct transport and rejects `--proxies`. HTTPS certificate verification is enabled by default. `--vhost-insecure` disables verification and records `tls_verified: false` in the evidence. Use it only when the engagement requires unverified TLS and the endpoint is authorized.
|
||||||
|
|
||||||
## What gets probed
|
## What gets probed
|
||||||
|
|
||||||
`--vhost` uses the run's collected evidence:
|
`--vhost` uses the run's collected evidence:
|
||||||
@@ -21,6 +52,8 @@ The shared request cap may stop the sweep before it reaches every endpoint. When
|
|||||||
|
|
||||||
An explicit endpoint replaces the harvested endpoint set:
|
An explicit endpoint replaces the harvested endpoint set:
|
||||||
|
|
||||||
|
Network activity: target-facing requests to the literal IP endpoint.
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
AUTHORIZED_DOMAIN='replace-with-a-domain-you-control'
|
AUTHORIZED_DOMAIN='replace-with-a-domain-you-control'
|
||||||
AUTHORIZED_IP='replace-with-an-authorized-ip'
|
AUTHORIZED_IP='replace-with-an-authorized-ip'
|
||||||
@@ -42,28 +75,13 @@ For each endpoint, theHarvester makes a literal-IP context request and at least
|
|||||||
|
|
||||||
HTTPS candidate and control requests send the same hostname in TLS SNI and the HTTP `Host` header. HTTP requests send the hostname only in `Host`. Redirects are recorded as evidence but are not followed.
|
HTTPS candidate and control requests send the same hostname in TLS SNI and the HTTP `Host` header. HTTP requests send the hostname only in `Host`. Redirects are recorded as evidence but are not followed.
|
||||||
|
|
||||||
```mermaid
|
The sweep first turns harvested evidence and any operator override into one bounded set of comparable requests:
|
||||||
flowchart TD
|
|
||||||
A[Collect hostnames and literal IPs] --> B[Keep candidates inside the exact target scope]
|

|
||||||
A --> C[Build HTTPS endpoints, then HTTP endpoints]
|
|
||||||
B --> D[Select the next endpoint and candidates that fit the shared cap]
|
The response set then enters a separate classifier. A status, redirect, or request-phase difference can be accepted immediately. A body-only difference must repeat before it becomes a finding:
|
||||||
C --> D
|
|
||||||
D --> E[Request the literal-IP context]
|

|
||||||
D --> F[Request three unknown controls for each candidate shape]
|
|
||||||
D --> G[Request the candidate with aligned SNI and Host]
|
|
||||||
E --> H{Are both baselines usable and controls stable?}
|
|
||||||
F --> H
|
|
||||||
G --> H
|
|
||||||
H -->|No| I[Indeterminate]
|
|
||||||
H -->|Yes| J{Does the candidate match either baseline?}
|
|
||||||
J -->|Yes| K[Default]
|
|
||||||
J -->|No, status or redirect differs| L[Distinct]
|
|
||||||
J -->|Only the body differs| M[Repeat the candidate request]
|
|
||||||
M --> N{Does the body difference repeat?}
|
|
||||||
N -->|Yes| L
|
|
||||||
N -->|No| I
|
|
||||||
L --> O[Store structured virtual host evidence]
|
|
||||||
```
|
|
||||||
|
|
||||||
Before comparison, the classifier replaces an exact reflection of the current authority in the response body or `Location` header. A generic error page that merely repeats the requested hostname should not become a discovery.
|
Before comparison, the classifier replaces an exact reflection of the current authority in the response body or `Location` header. A generic error page that merely repeats the requested hostname should not become a discovery.
|
||||||
|
|
||||||
@@ -123,36 +141,7 @@ Its `stop_reason` explains the outcome in more detail:
|
|||||||
|
|
||||||
Cancellation requested by the operator is different from a runtime limit. It propagates through the worker lifecycle and closes active connections.
|
Cancellation requested by the operator is different from a runtime limit. It propagates through the worker lifecycle and closes active connections.
|
||||||
|
|
||||||
## Basic CLI use
|
## Read terminal output
|
||||||
|
|
||||||
Harvest hostnames and IP addresses, then run the bounded sweep:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
AUTHORIZED_DOMAIN='replace-with-a-domain-you-control'
|
|
||||||
|
|
||||||
uv run theHarvester \
|
|
||||||
-d "$AUTHORIZED_DOMAIN" \
|
|
||||||
-b rapiddns \
|
|
||||||
--vhost \
|
|
||||||
-f report
|
|
||||||
```
|
|
||||||
|
|
||||||
`rapiddns` is used here because it can return both hostnames and literal IP addresses. If the selected sources return only hostnames, enable a DNS action that contributes IP evidence or supply `--vhost-endpoint`.
|
|
||||||
|
|
||||||
Repeat `--vhost-candidate` to add names that are already authorized but were not returned by the selected sources:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
uv run theHarvester \
|
|
||||||
-d "$AUTHORIZED_DOMAIN" \
|
|
||||||
-b rapiddns \
|
|
||||||
--vhost \
|
|
||||||
--vhost-candidate "admin.${AUTHORIZED_DOMAIN}" \
|
|
||||||
--vhost-candidate "preview.${AUTHORIZED_DOMAIN}"
|
|
||||||
```
|
|
||||||
|
|
||||||
Virtual host discovery uses direct transport. It rejects `--proxies`. HTTPS certificate verification is enabled by default. `--vhost-insecure` disables verification and records `tls_verified: false` in the evidence. Use that option only when the engagement requires it and the endpoint is authorized.
|
|
||||||
|
|
||||||
### Reading terminal output
|
|
||||||
|
|
||||||
The summary reports confirmed endpoint observations and coverage:
|
The summary reports confirmed endpoint observations and coverage:
|
||||||
|
|
||||||
@@ -170,6 +159,8 @@ HarvestView exposes a virtual host discovery checkbox plus optional endpoint and
|
|||||||
|
|
||||||
The same run can be submitted to `POST /api/v1/runs`:
|
The same run can be submitted to `POST /api/v1/runs`:
|
||||||
|
|
||||||
|
Network activity: the API request is local, but the queued run performs provider-facing discovery and target-facing virtual host requests.
|
||||||
|
|
||||||
```json
|
```json
|
||||||
{
|
{
|
||||||
"target": "authorized.example",
|
"target": "authorized.example",
|
||||||
@@ -188,6 +179,9 @@ JSONL is unversioned. After the summary line, it stores one canonical finding fo
|
|||||||
|
|
||||||
If a hostname is distinct on two endpoints, both endpoint records stay under that one hostname finding:
|
If a hostname is distinct on two endpoints, both endpoint records stay under that one hostname finding:
|
||||||
|
|
||||||
|
<details>
|
||||||
|
<summary>Full JSONL finding example</summary>
|
||||||
|
|
||||||
```json
|
```json
|
||||||
{
|
{
|
||||||
"type": "hostname",
|
"type": "hostname",
|
||||||
@@ -255,6 +249,8 @@ If a hostname is distinct on two endpoints, both endpoint records stay under tha
|
|||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
|
</details>
|
||||||
|
|
||||||
The `context_*` fields contain the literal-IP response, while `control_*` contains the stable unknown-host response. A distinct candidate must differ from both. `confirmation_body_sha256` is present only when a repeated response confirmed a body-only difference. These fields let a reviewer verify the recorded `distinct_signals` without retaining every synthetic control request. The source list is empty because virtual host discovery is an action, not a passive source.
|
The `context_*` fields contain the literal-IP response, while `control_*` contains the stable unknown-host response. A distinct candidate must differ from both. `confirmation_body_sha256` is present only when a repeated response confirmed a body-only difference. These fields let a reviewer verify the recorded `distinct_signals` without retaining every synthetic control request. The source list is empty because virtual host discovery is an action, not a passive source.
|
||||||
|
|
||||||
SQLite uses the same shape without inventing another evidence concept: `results` holds one `hostname` row, `result_origins` links it to the `vhost` execution, and that result's details hold the endpoint observation array. JSONL export and API run details expose the array as native JSON. `vhost` is an action name, not a result kind.
|
SQLite uses the same shape without inventing another evidence concept: `results` holds one `hostname` row, `result_origins` links it to the `vhost` execution, and that result's details hold the endpoint observation array. JSONL export and API run details expose the array as native JSON. `vhost` is an action name, not a result kind.
|
||||||
|
|||||||
@@ -1 +1,3 @@
|
|||||||
[Repository](https://github.com/laramies/theHarvester) · [Releases](https://github.com/laramies/theHarvester/releases) · [Issues](https://github.com/laramies/theHarvester/issues) · [Contributing](https://github.com/laramies/theHarvester/blob/dev/CONTRIBUTING.md) · [Security](https://github.com/laramies/theHarvester/blob/dev/SECURITY.md) · [License](https://github.com/laramies/theHarvester/blob/dev/README/COPYING)
|
[Repository](https://github.com/laramies/theHarvester) · [Releases](https://github.com/laramies/theHarvester/releases) · [Issues](https://github.com/laramies/theHarvester/issues) · [Contributing](https://github.com/laramies/theHarvester/blob/dev/CONTRIBUTING.md) · [Security](https://github.com/laramies/theHarvester/blob/dev/SECURITY.md) · [License](https://github.com/laramies/theHarvester/blob/dev/README/COPYING)
|
||||||
|
|
||||||
|
Reviewed wiki changes belong in [`docs/wiki/`](https://github.com/laramies/theHarvester/tree/dev/docs/wiki). The live GitHub wiki is the published copy.
|
||||||
|
|||||||
@@ -3,12 +3,12 @@
|
|||||||
**Get started**
|
**Get started**
|
||||||
|
|
||||||
- [Installation](Installation)
|
- [Installation](Installation)
|
||||||
- [Quick Start](Quick-Start)
|
- [Configuration and API Keys](Configuration-and-API-Keys)
|
||||||
- [Responsible Use and Scope](Responsible-Use-and-Scope)
|
- [Responsible Use and Scope](Responsible-Use-and-Scope)
|
||||||
|
- [Quick Start](Quick-Start)
|
||||||
|
|
||||||
**Operate**
|
**Operate**
|
||||||
|
|
||||||
- [Configuration and API Keys](Configuration-and-API-Keys)
|
|
||||||
- [Operator Workflows](Operator-Workflows)
|
- [Operator Workflows](Operator-Workflows)
|
||||||
- [Virtual Host Discovery](Virtual-Host-Discovery)
|
- [Virtual Host Discovery](Virtual-Host-Discovery)
|
||||||
- [Results and Local Data](Results-and-Local-Data)
|
- [Results and Local Data](Results-and-Local-Data)
|
||||||
|
|||||||
+103
-8
@@ -6,9 +6,11 @@ from pathlib import Path
|
|||||||
|
|
||||||
import yaml
|
import yaml
|
||||||
|
|
||||||
from theHarvester.lib.source_catalog import SOURCE_SPECS
|
from theHarvester.lib.completed_result import parse_result_jsonl
|
||||||
|
from theHarvester.lib.source_catalog import ACTION_ACTIVITIES, RESULT_CAPABILITIES, SOURCE_SPECS
|
||||||
|
|
||||||
OPTIONAL_API_KEY_SOURCES = {'hackertarget', 'mojeek', 'windvane'}
|
OPTIONAL_API_KEY_SOURCES = {'hackertarget', 'mojeek', 'windvane'}
|
||||||
|
SOURCE_ROUTE_ORDER = ('subdomains', 'emails', 'ips', 'asns', 'urls', 'people', 'breaches')
|
||||||
API_KEY_SOURCE_ALIASES = {
|
API_KEY_SOURCE_ALIASES = {
|
||||||
'github': {'github-code'},
|
'github': {'github-code'},
|
||||||
'pentestTools': {'pentesttools'},
|
'pentestTools': {'pentesttools'},
|
||||||
@@ -98,7 +100,7 @@ def _declared_source_contracts() -> dict[str, set[str]]:
|
|||||||
|
|
||||||
|
|
||||||
def _source_matrix(readme: str) -> str:
|
def _source_matrix(readme: str) -> str:
|
||||||
return readme.split('<summary><strong>View the source and result matrix</strong></summary>', 1)[1].split('</details>', 1)[0]
|
return readme.split('<summary><strong>View all 58 discovery sources</strong></summary>', 1)[1].split('</details>', 1)[0]
|
||||||
|
|
||||||
|
|
||||||
def _documented_source_rows(readme: str) -> dict[str, list[str]]:
|
def _documented_source_rows(readme: str) -> dict[str, list[str]]:
|
||||||
@@ -121,7 +123,10 @@ def _documented_source_links(readme: str) -> list[tuple[str, str]]:
|
|||||||
|
|
||||||
|
|
||||||
def _documented_source_contracts(readme: str) -> dict[str, set[str]]:
|
def _documented_source_contracts(readme: str) -> dict[str, set[str]]:
|
||||||
return {source: {route.strip() for route in cells[0].split(',')} for source, cells in _documented_source_rows(readme).items()}
|
return {
|
||||||
|
source: {route.strip().removesuffix(' only') for route in cells[0].split(',')}
|
||||||
|
for source, cells in _documented_source_rows(readme).items()
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
def _documented_source_activities(readme: str) -> dict[str, str]:
|
def _documented_source_activities(readme: str) -> dict[str, str]:
|
||||||
@@ -142,7 +147,8 @@ def test_readme_matches_declared_source_contracts() -> None:
|
|||||||
documented = _documented_source_contracts(readme)
|
documented = _documented_source_contracts(readme)
|
||||||
declared = _declared_source_contracts()
|
declared = _declared_source_contracts()
|
||||||
|
|
||||||
assert '| Source | Result routes | Activity | Credentials |' in readme
|
assert _source_matrix(readme).count('| Source | Returns | Activity | API key |') == 1
|
||||||
|
assert 'Credentials |' not in _source_matrix(readme)
|
||||||
assert len(declared) == 58
|
assert len(declared) == 58
|
||||||
assert len(documented) == 58
|
assert len(documented) == 58
|
||||||
assert documented == declared
|
assert documented == declared
|
||||||
@@ -153,6 +159,23 @@ def test_readme_matches_declared_source_contracts() -> None:
|
|||||||
assert {'securitytrails', 'shodaninternetdb'}.isdisjoint(documented)
|
assert {'securitytrails', 'shodaninternetdb'}.isdisjoint(documented)
|
||||||
|
|
||||||
|
|
||||||
|
def test_readme_source_matrix_is_one_table_and_ordered() -> None:
|
||||||
|
readme = Path('README.md').read_text()
|
||||||
|
matrix = _source_matrix(readme)
|
||||||
|
rows = _documented_source_rows(readme)
|
||||||
|
names = list(rows)
|
||||||
|
|
||||||
|
assert '#### Subdomain-only sources' not in matrix
|
||||||
|
assert '#### Sources that return other results' not in matrix
|
||||||
|
assert names == sorted(names, key=str.casefold)
|
||||||
|
|
||||||
|
route_rank = {route: index for index, route in enumerate(SOURCE_ROUTE_ORDER)}
|
||||||
|
for source, (routes, *_) in rows.items():
|
||||||
|
documented_order = [route.strip().removesuffix(' only') for route in routes.split(',')]
|
||||||
|
assert documented_order == sorted(documented_order, key=route_rank.__getitem__)
|
||||||
|
assert routes.endswith(' only') == (len(SOURCE_SPECS[source].capabilities) == 1)
|
||||||
|
|
||||||
|
|
||||||
def test_readme_api_key_markers_match_configuration() -> None:
|
def test_readme_api_key_markers_match_configuration() -> None:
|
||||||
readme = Path('README.md').read_text()
|
readme = Path('README.md').read_text()
|
||||||
requirements = _documented_api_key_requirements(readme)
|
requirements = _documented_api_key_requirements(readme)
|
||||||
@@ -182,6 +205,16 @@ def test_wiki_navigation_and_readme_links_resolve() -> None:
|
|||||||
assert all(Path(target).is_file() for target in readme_wiki_links)
|
assert all(Path(target).is_file() for target in readme_wiki_links)
|
||||||
|
|
||||||
|
|
||||||
|
def test_wiki_examples_use_copy_safe_operator_language() -> None:
|
||||||
|
quick_start = Path('docs/wiki/Quick-Start.md').read_text()
|
||||||
|
configuration = Path('docs/wiki/Configuration-and-API-Keys.md').read_text()
|
||||||
|
|
||||||
|
assert 'inert test data' not in quick_start
|
||||||
|
assert 'Passive providers still receive the target string.' in quick_start
|
||||||
|
assert '${EDITOR:-vi}' not in configuration
|
||||||
|
assert 'vi ~/.theHarvester/api-keys.yaml' in configuration
|
||||||
|
|
||||||
|
|
||||||
def test_readme_architecture_diagrams_are_local_and_accessible() -> None:
|
def test_readme_architecture_diagrams_are_local_and_accessible() -> None:
|
||||||
readme = Path('README.md').read_text()
|
readme = Path('README.md').read_text()
|
||||||
diagrams = (
|
diagrams = (
|
||||||
@@ -189,24 +222,78 @@ def test_readme_architecture_diagrams_are_local_and_accessible() -> None:
|
|||||||
'theHarvester discovery routes and enrichment',
|
'theHarvester discovery routes and enrichment',
|
||||||
Path('docs/images/run-evidence-architecture.svg'),
|
Path('docs/images/run-evidence-architecture.svg'),
|
||||||
'run-evidence-architecture',
|
'run-evidence-architecture',
|
||||||
('subdomains · emails · IPs', 'Shodan host detail', 'RouteViews routes', 'vhost · screenshots · takeover'),
|
('58 discovery adapters', 'CompletedResult evidence contract', 'Terminal · JSONL · SQLite · REST'),
|
||||||
),
|
),
|
||||||
(
|
(
|
||||||
'HarvestView run desk architecture',
|
'HarvestView run desk architecture',
|
||||||
Path('docs/images/harvestview-architecture.svg'),
|
Path('docs/images/harvestview-architecture.svg'),
|
||||||
'harvestview-architecture',
|
'harvestview-architecture',
|
||||||
('Authenticated REST API', 'queued → running → terminal', 'Isolated run worker', 'JSONL / SQLite export'),
|
('Authenticated REST API', 'queued · running', 'Isolated run worker', 'JSONL · one run'),
|
||||||
),
|
),
|
||||||
)
|
)
|
||||||
|
|
||||||
for alt, svg, slug, expected_text in diagrams:
|
for alt, svg, slug, expected_text in diagrams:
|
||||||
svg_text = svg.read_text()
|
svg_text = svg.read_text()
|
||||||
assert f'' in readme
|
assert f'[]({svg})' in readme
|
||||||
|
assert 'role="img"' in svg_text
|
||||||
|
assert f'<title id="{slug}-title">' in svg_text
|
||||||
|
assert f'<desc id="{slug}-desc">' in svg_text
|
||||||
|
assert 'viewBox="0 0 960 640"' in svg_text
|
||||||
|
assert '@media (prefers-color-scheme: light)' in svg_text
|
||||||
|
assert "font: 600 16px 'Geist'" in svg_text
|
||||||
|
assert all(text in svg_text for text in expected_text)
|
||||||
|
|
||||||
|
run_diagram = diagrams[0][1].read_text()
|
||||||
|
harvestview_diagram = diagrams[1][1].read_text()
|
||||||
|
assert all(capability in run_diagram for capability in RESULT_CAPABILITIES)
|
||||||
|
assert all(action in run_diagram for action in ACTION_ACTIVITIES)
|
||||||
|
assert f'{len(SOURCE_SPECS)} discovery adapters' in run_diagram
|
||||||
|
assert f'{len(ACTION_ACTIVITIES)} explicit actions' in harvestview_diagram
|
||||||
|
|
||||||
|
|
||||||
|
def test_wiki_diagrams_are_local_accessible_and_used_deliberately() -> None:
|
||||||
|
raw_root = 'https://raw.githubusercontent.com/laramies/theHarvester/dev/'
|
||||||
|
diagrams = (
|
||||||
|
(
|
||||||
|
Path('docs/wiki/Virtual-Host-Discovery.md'),
|
||||||
|
'Bounded virtual-host sweep',
|
||||||
|
Path('docs/images/vhost-sweep-overview.svg'),
|
||||||
|
'vhost-sweep-overview',
|
||||||
|
('exact target scope', 'Literal-IP endpoint pool', 'shared request cap', '3 shape-matched controls'),
|
||||||
|
),
|
||||||
|
(
|
||||||
|
Path('docs/wiki/Virtual-Host-Discovery.md'),
|
||||||
|
'Virtual-host response classifier',
|
||||||
|
Path('docs/images/vhost-classifier.svg'),
|
||||||
|
'vhost-classifier',
|
||||||
|
('All responses usable', 'Candidate matches', 'Repeat candidate request', 'Retain hostname'),
|
||||||
|
),
|
||||||
|
(
|
||||||
|
Path('docs/wiki/Operator-Workflows.md'),
|
||||||
|
'theHarvester discovery routes and enrichment',
|
||||||
|
Path('docs/images/run-evidence-architecture.svg'),
|
||||||
|
'run-evidence-architecture',
|
||||||
|
(),
|
||||||
|
),
|
||||||
|
(
|
||||||
|
Path('docs/wiki/Rest-API.md'),
|
||||||
|
'HarvestView run desk architecture',
|
||||||
|
Path('docs/images/harvestview-architecture.svg'),
|
||||||
|
'harvestview-architecture',
|
||||||
|
(),
|
||||||
|
),
|
||||||
|
)
|
||||||
|
|
||||||
|
for page, alt, svg, slug, expected_text in diagrams:
|
||||||
|
svg_text = svg.read_text()
|
||||||
|
assert f'' in page.read_text()
|
||||||
assert 'role="img"' in svg_text
|
assert 'role="img"' in svg_text
|
||||||
assert f'<title id="{slug}-title">' in svg_text
|
assert f'<title id="{slug}-title">' in svg_text
|
||||||
assert f'<desc id="{slug}-desc">' in svg_text
|
assert f'<desc id="{slug}-desc">' in svg_text
|
||||||
assert all(text in svg_text for text in expected_text)
|
assert all(text in svg_text for text in expected_text)
|
||||||
|
|
||||||
|
assert '```mermaid' not in Path('docs/wiki/Virtual-Host-Discovery.md').read_text()
|
||||||
|
|
||||||
|
|
||||||
def test_virtual_host_wiki_examples_match_the_structured_result_contract() -> None:
|
def test_virtual_host_wiki_examples_match_the_structured_result_contract() -> None:
|
||||||
page = Path('docs/wiki/Virtual-Host-Discovery.md').read_text()
|
page = Path('docs/wiki/Virtual-Host-Discovery.md').read_text()
|
||||||
@@ -271,8 +358,16 @@ def test_operator_docs_cover_portable_database_export() -> None:
|
|||||||
|
|
||||||
def test_readme_explains_jsonl_record_and_structured_evidence_parsing() -> None:
|
def test_readme_explains_jsonl_record_and_structured_evidence_parsing() -> None:
|
||||||
readme = Path('README.md').read_text()
|
readme = Path('README.md').read_text()
|
||||||
|
example = re.search(r'```jsonl\n(.*?)\n```', readme, flags=re.DOTALL)
|
||||||
|
assert example is not None
|
||||||
|
summary, findings = parse_result_jsonl(example.group(1))
|
||||||
|
|
||||||
assert '{"sources":[],"type":"hostname","value":"api.example.com"}' in readme
|
finding_types = ('asn', 'breach', 'email', 'hostname', 'ip', 'url')
|
||||||
|
assert summary['counts'] == {result_type: 1 for result_type in finding_types}
|
||||||
|
assert summary['result_count'] == len(findings) == 6
|
||||||
|
assert tuple(finding['type'] for finding in findings) == finding_types
|
||||||
|
assert all(finding['sources'] for finding in findings)
|
||||||
|
assert {execution['source'] for execution in summary['source_executions']} == {'haveibeenpwned', 'zoomeye'}
|
||||||
for result_kind in ('hostname', 'ip', 'asn', 'email', 'url', 'person', 'breach'):
|
for result_kind in ('hostname', 'ip', 'asn', 'email', 'url', 'person', 'breach'):
|
||||||
assert f'select(.type == "{result_kind}")' in readme
|
assert f'select(.type == "{result_kind}")' in readme
|
||||||
assert 'select(.type == "person") | .value | fromjson' in readme
|
assert 'select(.type == "person") | .value | fromjson' in readme
|
||||||
|
|||||||
Reference in New Issue
Block a user