mirror of
https://github.com/laramies/theHarvester.git
synced 2026-08-17 19:35:40 +02:00
docs: keep wiki focused on current behavior
This commit is contained in:
@@ -41,13 +41,13 @@ apikeys:
|
||||
|
||||
Do not commit populated configuration files. Prefer provider credentials scoped to the minimum access the provider supports.
|
||||
|
||||
The [README source matrix](https://github.com/laramies/theHarvester/blob/dev/README.md#discovery-sources) is the canonical source list. It shows whether each source requires a key, accepts an optional key, or has no key setting.
|
||||
The [README source matrix](https://github.com/laramies/theHarvester/blob/dev/README.md#discovery-sources) lists each source's result routes, activity class, and credential requirement. The executable source catalog is the authoritative inventory.
|
||||
|
||||
Provider pricing, quotas, and terms change frequently. Check the provider's current documentation for these details.
|
||||
|
||||
`censys.token` is a Censys Platform Personal Access Token. Set `organization_id` when searches should use an entitled organization. This source uses the Global Search API, which is unavailable to Free accounts because they are limited to asset lookups. The retired Search API ID and secret fields are not accepted.
|
||||
`censys.token` is a Censys Platform Personal Access Token. Set `organization_id` when searches should use an entitled organization. This source uses the Global Search API, which is unavailable to Free accounts because they are limited to asset lookups. Search API ID and secret fields are not accepted.
|
||||
|
||||
`hibpverified` queries [HIBP's authenticated verified-domain endpoint](https://haveibeenpwned.com/API/v3#BreachedDomain). It is selected by its name, the `breaches` capability, and `all`. Without a configured HIBP API key it is skipped like other unavailable keyed sources. Live use requires a user-owned paid HIBP API key and a user-owned domain verified in that account. The keyless `haveibeenpwned` source continues to query only the public breach catalogue.
|
||||
`hibpverified` queries [HIBP's authenticated verified-domain endpoint](https://haveibeenpwned.com/API/v3#BreachedDomain). It is selected by its name, the `breaches` capability, and `all`. Without a configured HIBP API key it is skipped like other unavailable keyed sources. Live use requires a user-owned paid HIBP API key and a user-owned domain verified in that account. The keyless `haveibeenpwned` source queries only the public breach catalogue.
|
||||
|
||||
`routeviews.key` is optional. RouteViews provides authenticated API keys to verified PeeringDB users. `--routeviews` uses the authenticated endpoint and documented 10-request-per-second allowance when the key is configured; otherwise it uses guest access at one request per second. If RouteViews rejects a configured key, the action fails without retrying as a guest; remove the key to select guest access. RouteViews does not document this as a paid subscription.
|
||||
|
||||
|
||||
@@ -25,7 +25,7 @@ An adapter normally provides:
|
||||
|
||||
Do not return fields the provider did not supply. Normalize and deduplicate before returning results.
|
||||
|
||||
Return `None` when the provider conversation completed normally, including a valid zero-result response. Return an immutable `SourceExecutionReport` with a stable provider-specific reason for another terminal condition: `completed` for a successful early stop such as reaching the requested result limit, `failed` for provider or transport failure, `rate-limited` for a terminal rate limit, or `partial` when the provider itself confirms incomplete coverage. Do not add mutable `execution_status` or `stop_reason` fields to an adapter. The source runner rejects adapters that expose either removed field before execution and rechecks before evidence collection. The source runner owns finalization: it promotes any incomplete report with retained normalized evidence to `partial`, and records a normal zero-result completion as `completed` with `no-results`.
|
||||
Return `None` when the provider conversation completed normally, including a valid zero-result response. Return an immutable `SourceExecutionReport` with a stable provider-specific reason for another terminal condition: `completed` for a successful early stop such as reaching the requested result limit, `failed` for provider or transport failure, `rate-limited` for a terminal rate limit, or `partial` when the provider confirms incomplete coverage. Adapters must not define mutable `execution_status` or `stop_reason` fields. The source runner checks for either field before execution and again before evidence collection. It owns finalization, promotes incomplete reports with retained normalized evidence to `partial`, and records a normal zero-result completion as `completed` with `no-results`.
|
||||
|
||||
### Own the provider conversation
|
||||
|
||||
@@ -42,7 +42,7 @@ The completion check is an offline test in which a later page depends on state e
|
||||
|
||||
## 3. Register the source
|
||||
|
||||
Add one catalog entry in [`theHarvester/lib/source_catalog.py`](https://github.com/laramies/theHarvester/blob/dev/theHarvester/lib/source_catalog.py) and one factory entry in [`theHarvester/lib/source_runner.py`](https://github.com/laramies/theHarvester/blob/dev/theHarvester/lib/source_runner.py). The catalog supplies CLI help, source selection, and activity classification; the factory constructs the adapter; the runner collects declared result routes and persists them through the existing completed-result flow.
|
||||
Add one catalog entry in [`theHarvester/lib/source_catalog.py`](https://github.com/laramies/theHarvester/blob/dev/theHarvester/lib/source_catalog.py) and one factory entry in [`theHarvester/lib/source_runner.py`](https://github.com/laramies/theHarvester/blob/dev/theHarvester/lib/source_runner.py). The catalog supplies CLI help, source selection, and activity classification. The factory constructs the adapter. The runner collects declared result routes and persists them with the completed run.
|
||||
|
||||
Keep the public source identifier stable and use the same spelling everywhere.
|
||||
|
||||
@@ -74,6 +74,6 @@ Tests must not require external network access or real provider credentials.
|
||||
|
||||
## 6. Update operator documentation
|
||||
|
||||
Add the source to the README source/result matrix with its actual output columns and key requirement. The matrix contract test checks that documented result types match the catalog entry.
|
||||
Add the source to the README matrix with its result routes, activity class, and credential requirement. The matrix contract test checks those values against the catalog entry.
|
||||
|
||||
In the pull request, link the provider API documentation and explain any intentional exception to shared transport behavior.
|
||||
|
||||
@@ -16,7 +16,7 @@ Use the [README source matrix](https://github.com/laramies/theHarvester/blob/dev
|
||||
uv run theHarvester -d example.com -b crtsh,certspotter -f report
|
||||
```
|
||||
|
||||
Use `report.jsonl` for automation, provenance, and run interchange. The same command also writes legacy `report.json` and `report.xml` compatibility files. See [Results and Local Data](Results-and-Local-Data).
|
||||
Use `report.jsonl` for automation, provenance, and run interchange. The same command also writes `report.json` and `report.xml` compatibility files. See [Results and Local Data](Results-and-Local-Data).
|
||||
|
||||
## DNS resolution
|
||||
|
||||
@@ -45,7 +45,7 @@ AUTHORIZED_DOMAIN='replace-with-a-domain-you-control'
|
||||
uv run theHarvester -d "$AUTHORIZED_DOMAIN" -b crtsh -r -s
|
||||
```
|
||||
|
||||
Shodan output is enrichment after discovery and is separate from what most discovery sources add to the consolidated result columns.
|
||||
Shodan host enrichment runs after discovery and is separate from the `shodan` source's subdomain results.
|
||||
|
||||
## DNS brute force
|
||||
|
||||
|
||||
@@ -24,7 +24,7 @@ This queries two passive certificate sources and prints consolidated findings. P
|
||||
uv run theHarvester -d example.com -b crtsh,certspotter -f report
|
||||
```
|
||||
|
||||
This writes `report.jsonl` for automation and interchange. It also writes legacy `report.json` and `report.xml` compatibility reports. See [Results and Local Data](Results-and-Local-Data).
|
||||
This writes `report.jsonl` for automation and interchange. It also writes `report.json` and `report.xml` compatibility reports. See [Results and Local Data](Results-and-Local-Data).
|
||||
|
||||
## Resolve discovered hosts
|
||||
|
||||
|
||||
@@ -24,7 +24,7 @@ The following options require additional care:
|
||||
| `--screenshot DIR` | Opens discovered web services in a browser. |
|
||||
| `-a`, `--api-scan` | Requests common API paths from the target. |
|
||||
|
||||
Use `--dns-resolvers IPS_OR_FILE` to select resolver addresses for DNS brute force, reverse lookup, or recursive DNS without also enabling hostname resolution. The compatible `--dns-resolve` value still selects resolvers and enables hostname resolution.
|
||||
Use `--dns-resolvers IPS_OR_FILE` to select resolver addresses for DNS brute force, reverse lookup, or recursive DNS without enabling hostname resolution. `--dns-resolve [IPS_OR_FILE]` selects resolvers and enables hostname resolution.
|
||||
|
||||
`--routeviews` is a separately selected P0 provider action. It is never enabled by `-b all` and ignores `-l`; one run is internally bounded to 300 sequential requests and 300 seconds. With no configured key it uses the documented guest rate of one request per second. A configured `routeviews.key` selects PeeringDB-verified authenticated access and the documented 10-request-per-second allowance. When selected for a domain run, it automatically queries harvested IPs backed by sourced IP-to-ASN attribution. It also accepts an ASN, IP, or CIDR supplied as the run target. Harvested IPs without that attribution are not sent, and bare ASN findings are not expanded into complete prefix inventories. IP lookups retain only the most-specific returned prefix, including all origins for a multi-origin prefix. The action does not recursively query returned prefixes and never promotes returned CIDRs into DNS or direct-action scope. Cloud and CDN routes can be useful relationship evidence, but route origins and RPKI states do not establish ownership or authorization.
|
||||
|
||||
|
||||
@@ -188,7 +188,7 @@ curl -s "http://127.0.0.1:5000/api/v1/runs/$run_id/export" \
|
||||
-o results.jsonl
|
||||
```
|
||||
|
||||
The first line is the `summary` record, including evidence status, source and action outcomes, and artifacts. Each remaining line is one normalized finding with `type`, `value`, `sources`, and optional `actions`. A hostname confirmed by the `vhost` action adds native endpoint observations; a RouteViews `prefix` adds native origin, route, and RPKI observations with fixed external-relationship scope. This keeps the file easy to stream with `jq -c` and makes API exports importable again without a format conversion. Lifecycle details and the submitted request remain available from `GET /api/v1/runs/{run_id}`.
|
||||
The first line is the `summary` record, including evidence status, source and action outcomes, and artifacts. Each remaining line is one normalized finding with `type`, `value`, `sources`, and optional `actions`. A hostname confirmed by the `vhost` action adds native endpoint observations; a RouteViews `prefix` adds native origin, route, and RPKI observations with fixed external-relationship scope. The file can be streamed with `jq -c` or imported through the API without conversion. Lifecycle details and the submitted request are available from `GET /api/v1/runs/{run_id}`.
|
||||
|
||||
## Security boundary
|
||||
|
||||
|
||||
@@ -20,19 +20,19 @@ The recommended automation output is `report.jsonl`. Its first record summarizes
|
||||
jq -c 'select(.type != "summary") | {type, value, sources, actions}' report.jsonl
|
||||
```
|
||||
|
||||
The same `-f report` command also creates `report.json` and `report.xml` for compatibility.
|
||||
The same `-f report` command creates `report.json` and `report.xml` compatibility reports.
|
||||
|
||||
## Legacy JSON and XML
|
||||
## JSON and XML compatibility reports
|
||||
|
||||
- **JSON** is one object and contains the broader result set. `cmd`, `hosts`, and `shodan` are always present; other fields appear when non-empty.
|
||||
- **XML** contains the command, emails, hosts, and virtual hosts. Use JSON for other result types.
|
||||
- Current JSON and XML reports do not record which source found each item.
|
||||
- JSON and XML reports do not record which source found each item.
|
||||
|
||||
When virtual host discovery runs, JSON's `vhosts` array and XML's `<vhost>` entries contain confirmed hostnames only. They do not include endpoint or baseline evidence; use JSONL or API run details for that structured data.
|
||||
|
||||
Host values may be plain hostnames. When DNS resolution is enabled, they can also use the `hostname:IP` form.
|
||||
|
||||
The repository [README output section](https://github.com/laramies/theHarvester/blob/dev/README.md#report-formats) documents the current formats and provides copyable `jq` examples for JSONL.
|
||||
The repository [README output section](https://github.com/laramies/theHarvester/blob/dev/README.md#output-and-local-data) summarizes the formats and provides copyable `jq` examples for JSONL.
|
||||
|
||||
## SQLite database
|
||||
|
||||
@@ -57,7 +57,7 @@ The normalized persistence model can represent active-action provenance and arti
|
||||
|
||||
Virtual-host evidence stays inside this model. `results` holds one `hostname` row, `result_origins` links it to the `vhost` action execution, and the result's `details_json` contains the canonical endpoint observation array. If one hostname is distinct on several IP endpoints, it remains one result with several observations.
|
||||
|
||||
Current runtime collection populates passive source executions plus DNS, takeover, Shodan, and API endpoint scan executions and origins. Screenshot actions attach file metadata to their captured hostname or URL without creating fake screenshot findings.
|
||||
Runtime collection records passive source executions plus DNS, takeover, Shodan, and API endpoint scan executions and origins. Screenshot actions attach file metadata to their captured hostname or URL without creating screenshot findings.
|
||||
|
||||
RouteViews creates `prefix` results with `scope: external-relationship` and `routeviews` action provenance. Native observations distinguish one ASN-prefix origin claim, one collector/peer BGP route, and one RPKI validation state. They are routing evidence, not registration, ownership, authorization, reachability, or expanded target scope.
|
||||
|
||||
@@ -67,7 +67,7 @@ Every discovered URL is stored as the `url` result kind. Its source or action or
|
||||
|
||||
Hostname and IP evidence use the `hostname` and `ip` result kinds in SQLite, JSONL, the API, and HarvestView. A hostname may be the authorized target itself or a subordinate name, so the result kind does not claim that every value is a subdomain.
|
||||
|
||||
Two operational tables support the API without changing those six evidence concepts: `run_records` stores queue and lifecycle state, and `run_worker_leases` prevents two local workers from claiming the same queue. Older runless rows remain in `legacy_observations`. SQLite upgrades supported schemas automatically during normal initialization.
|
||||
Two operational tables support the API without changing those six evidence concepts: `run_records` stores queue and lifecycle state, and `run_worker_leases` prevents two local workers from claiming the same queue. Runless rows are stored in `legacy_observations`. SQLite upgrades supported schemas automatically during normal initialization.
|
||||
|
||||
## Screenshots
|
||||
|
||||
|
||||
@@ -11,4 +11,4 @@ Released changes are recorded in:
|
||||
- [CHANGELOG.md](https://github.com/laramies/theHarvester/blob/dev/CHANGELOG.md)
|
||||
- [GitHub Releases](https://github.com/laramies/theHarvester/releases)
|
||||
|
||||
This page keeps the historical `Roadmap` URL valid without duplicating work that belongs in issues and milestones.
|
||||
This page points to the live project artifacts so plans are not duplicated in the wiki.
|
||||
|
||||
@@ -259,7 +259,7 @@ The `context_*` fields contain the literal-IP response, while `control_*` contai
|
||||
|
||||
SQLite uses the same shape without inventing another evidence concept: `results` holds one `hostname` row, `result_origins` links it to the `vhost` execution, and that result's details hold the endpoint observation array. JSONL export and API run details expose the array as native JSON. `vhost` is an action name, not a result kind.
|
||||
|
||||
Legacy JSON and XML keep their existing virtual host name lists. They do not carry the endpoint and response evidence. Use JSONL or `GET /api/v1/runs/{run_id}` when automation needs the structured observations.
|
||||
JSON and XML compatibility reports contain virtual host name lists but not endpoint or response evidence. Use JSONL or `GET /api/v1/runs/{run_id}` when automation needs the structured observations.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
|
||||
Reference in New Issue
Block a user