diff --git a/README.md b/README.md index 3f1a5b00..e22d06c7 100644 --- a/README.md +++ b/README.md @@ -90,7 +90,9 @@ Open [http://127.0.0.1:5000/docs](http://127.0.0.1:5000/docs) for interactive Sw The service rate limit defaults to five requests per minute and can be changed with `--rate-limit`. The `/additional/*` routes require `THEHARVESTER_API_KEY` on the server and the same value in the `X-API-Key` request header. -The core `/query`, `/sources`, and `/dnsbrute` routes are not authenticated. Keep the service bound to localhost unless you place it behind appropriate authentication, access controls, and TLS. Docker Compose publishes port `5000` on every host interface unless you narrow the port mapping: +The core `/query`, `/sources`, and `/dnsbrute` routes do not require authentication. Keep the service bound to localhost. If you require remote access, add authentication, access controls, and TLS. + +Docker Compose publishes port `5000` on every host interface unless you narrow the port mapping: ```bash docker compose up --build @@ -98,76 +100,84 @@ docker compose up --build ## Discovery sources -Saved JSON reports expose separate fields for hosts, emails, IP addresses, ASNs, URLs or links, and people when those results are available. The result-type columns below describe only that consolidated CLI report. They do not include every field parsed from a provider response. Empty fields may be omitted, and reports do not retain per-source attribution. +The table shows which result types each source can add to the JSON report. It covers the consolidated CLI report only. Some adapters parse fields that the report does not store. -A checkmark means the current CLI can add that result type to its consolidated report. The **Separate output** column identifies REST endpoints or optional actions whose results are not part of those source columns. In the **API key** column, **✓** means credentials are required, **Optional** means a key can unlock additional access, and **—** means the source has no key setting. +The report groups findings by result type. It does not record which source found each item. Empty optional fields may be omitted. + +A checkmark means the source can add that result type. The **Separate output** column lists REST endpoints and optional actions that return other data. + +Read the **API key** column as follows: + +- **✓**: credentials are required. +- **Optional**: a key can provide additional access. +- **No**: the source has no key setting.
View the source and result matrix | Source | Hosts | Emails | IPs | ASNs | URLs / links | People | Separate REST/action output (not consolidated report) | API key | | --- | :---: | :---: | :---: | :---: | :---: | :---: | --- | :---: | -| `baidu` | ✓ | ✓ | — | — | — | — | — | — | -| `bevigil` | ✓ | — | — | — | ✓ | — | — | ✓ | -| `bitbucket` | ✓ | ✓ | — | — | — | — | — | ✓ | -| `bufferoverun` | ✓ | — | ✓ | — | — | — | — | ✓ | -| `builtwith` | ✓ | — | — | — | ✓ | — | `POST /additional/tech-stack` response | ✓ | -| `brave` | ✓ | ✓ | — | — | — | — | — | ✓ | -| `censys` | ✓ | ✓ | — | — | — | — | — | ✓ | -| `certspotter` | ✓ | — | — | — | — | — | — | — | -| `chaos` | ✓ | — | — | — | — | — | — | ✓ | -| `commoncrawl` | ✓ | — | — | — | — | — | — | — | -| `criminalip` | ✓ | — | ✓ | ✓ | — | — | — | ✓ | -| `crtsh` | ✓ | — | — | — | — | — | — | — | -| `dehashed` | — | — | ✓ | — | — | — | — | ✓ | -| `dnsdumpster` | ✓ | — | ✓ | — | — | — | — | ✓ | -| `duckduckgo` | ✓ | ✓ | — | — | — | — | — | — | -| `dymo` | ✓ | — | — | — | — | — | — | ✓ | -| `fofa` | ✓ | — | ✓ | — | — | — | — | ✓ | -| `fullhunt` | ✓ | — | — | — | — | — | — | ✓ | -| `github-code` | ✓ | ✓ | — | — | — | — | — | ✓ | -| `gitlab` | ✓ | ✓ | — | — | — | — | — | — | -| `hackertarget` | ✓ | — | — | — | — | — | — | Optional | -| `haveibeenpwned` | — | — | — | — | — | — | `POST /additional/breaches` response | ✓ | -| `hudsonrock` | ✓ | ✓ | ✓ | — | — | — | — | — | -| `hunter` | ✓ | ✓ | — | — | — | — | — | ✓ | -| `hunterhow` | ✓ | — | — | — | — | — | — | ✓ | -| `intelx` | — | ✓ | — | — | ✓ | — | — | ✓ | -| `leakix` | ✓ | ✓ | — | — | — | — | — | Optional | -| `leaklookup` | — | ✓ | — | — | — | — | `POST /additional/leaks` response | ✓ | -| `mojeek` | ✓ | ✓ | — | — | — | — | — | Optional | -| `netlas` | ✓ | — | — | — | — | — | — | ✓ | -| `onyphe` | ✓ | — | ✓ | ✓ | — | — | — | ✓ | -| `otx` | ✓ | — | ✓ | — | — | — | — | — | -| `pentesttools` | ✓ | — | — | — | — | — | — | ✓ | -| `projectdiscovery` | ✓ | — | — | — | — | — | — | ✓ | -| `rapiddns` | ✓ | — | — | — | — | — | — | — | -| `robtex` | ✓ | — | ✓ | — | — | — | — | — | -| `rocketreach` | — | ✓ | — | — | ✓ | — | — | ✓ | -| `securityscorecard` | ✓ | — | ✓ | — | — | — | `POST /additional/security-score` response | ✓ | -| `securityTrails` | ✓ | — | ✓ | — | — | — | — | ✓ | -| `sherlockeye` | ✓ | ✓ | ✓ | — | — | — | — | ✓ | -| `shodan` | ✓ | — | — | — | — | — | `-s` / `--shodan` host-enrichment output | ✓ | -| `shodanInternetDB` | ✓ | — | ✓ | — | — | — | — | — | -| `subdomaincenter` | ✓ | — | — | — | — | — | — | — | -| `subdomainfinderc99` | ✓ | — | — | — | — | — | — | — | -| `thc` | ✓ | — | — | — | — | — | — | — | -| `threatcrowd` | ✓ | — | ✓ | — | — | — | — | — | -| `tomba` | ✓ | ✓ | — | — | — | — | — | ✓ | -| `urlscan` | ✓ | — | ✓ | ✓ | ✓ | — | — | — | -| `venacus` | — | ✓ | ✓ | — | ✓ | ✓ | — | ✓ | -| `virustotal` | ✓ | — | — | — | — | — | — | ✓ | -| `waybackarchive` | ✓ | — | — | — | — | — | — | — | -| `whoisxml` | ✓ | — | — | — | — | — | — | ✓ | -| `windvane` | ✓ | ✓ | ✓ | — | — | — | — | Optional | -| `yahoo` | ✓ | ✓ | — | — | — | — | — | — | -| `zoomeye` | ✓ | ✓ | ✓ | ✓ | ✓ | — | — | ✓ | +| `baidu` | ✓ | ✓ | No | No | No | No | No | No | +| `bevigil` | ✓ | No | No | No | ✓ | No | No | ✓ | +| `bitbucket` | ✓ | ✓ | No | No | No | No | No | ✓ | +| `bufferoverun` | ✓ | No | ✓ | No | No | No | No | ✓ | +| `builtwith` | ✓ | No | No | No | ✓ | No | `POST /additional/tech-stack` response | ✓ | +| `brave` | ✓ | ✓ | No | No | No | No | No | ✓ | +| `censys` | ✓ | ✓ | No | No | No | No | No | ✓ | +| `certspotter` | ✓ | No | No | No | No | No | No | No | +| `chaos` | ✓ | No | No | No | No | No | No | ✓ | +| `commoncrawl` | ✓ | No | No | No | No | No | No | No | +| `criminalip` | ✓ | No | ✓ | ✓ | No | No | No | ✓ | +| `crtsh` | ✓ | No | No | No | No | No | No | No | +| `dehashed` | No | No | ✓ | No | No | No | No | ✓ | +| `dnsdumpster` | ✓ | No | ✓ | No | No | No | No | ✓ | +| `duckduckgo` | ✓ | ✓ | No | No | No | No | No | No | +| `dymo` | ✓ | No | No | No | No | No | No | ✓ | +| `fofa` | ✓ | No | ✓ | No | No | No | No | ✓ | +| `fullhunt` | ✓ | No | No | No | No | No | No | ✓ | +| `github-code` | ✓ | ✓ | No | No | No | No | No | ✓ | +| `gitlab` | ✓ | ✓ | No | No | No | No | No | No | +| `hackertarget` | ✓ | No | No | No | No | No | No | Optional | +| `haveibeenpwned` | No | No | No | No | No | No | `POST /additional/breaches` response | ✓ | +| `hudsonrock` | ✓ | ✓ | ✓ | No | No | No | No | No | +| `hunter` | ✓ | ✓ | No | No | No | No | No | ✓ | +| `hunterhow` | ✓ | No | No | No | No | No | No | ✓ | +| `intelx` | No | ✓ | No | No | ✓ | No | No | ✓ | +| `leakix` | ✓ | ✓ | No | No | No | No | No | Optional | +| `leaklookup` | No | ✓ | No | No | No | No | `POST /additional/leaks` response | ✓ | +| `mojeek` | ✓ | ✓ | No | No | No | No | No | Optional | +| `netlas` | ✓ | No | No | No | No | No | No | ✓ | +| `onyphe` | ✓ | No | ✓ | ✓ | No | No | No | ✓ | +| `otx` | ✓ | No | ✓ | No | No | No | No | No | +| `pentesttools` | ✓ | No | No | No | No | No | No | ✓ | +| `projectdiscovery` | ✓ | No | No | No | No | No | No | ✓ | +| `rapiddns` | ✓ | No | No | No | No | No | No | No | +| `robtex` | ✓ | No | ✓ | No | No | No | No | No | +| `rocketreach` | No | ✓ | No | No | ✓ | No | No | ✓ | +| `securityscorecard` | ✓ | No | ✓ | No | No | No | `POST /additional/security-score` response | ✓ | +| `securityTrails` | ✓ | No | ✓ | No | No | No | No | ✓ | +| `sherlockeye` | ✓ | ✓ | ✓ | No | No | No | No | ✓ | +| `shodan` | ✓ | No | No | No | No | No | `-s` / `--shodan` host-enrichment output | ✓ | +| `shodanInternetDB` | ✓ | No | ✓ | No | No | No | No | No | +| `subdomaincenter` | ✓ | No | No | No | No | No | No | No | +| `subdomainfinderc99` | ✓ | No | No | No | No | No | No | No | +| `thc` | ✓ | No | No | No | No | No | No | No | +| `threatcrowd` | ✓ | No | ✓ | No | No | No | No | No | +| `tomba` | ✓ | ✓ | No | No | No | No | No | ✓ | +| `urlscan` | ✓ | No | ✓ | ✓ | ✓ | No | No | No | +| `venacus` | No | ✓ | ✓ | No | ✓ | ✓ | No | ✓ | +| `virustotal` | ✓ | No | No | No | No | No | No | ✓ | +| `waybackarchive` | ✓ | No | No | No | No | No | No | No | +| `whoisxml` | ✓ | No | No | No | No | No | No | ✓ | +| `windvane` | ✓ | ✓ | ✓ | No | No | No | No | Optional | +| `yahoo` | ✓ | ✓ | No | No | No | No | No | No | +| `zoomeye` | ✓ | ✓ | ✓ | ✓ | ✓ | No | No | ✓ |
Provider pricing is intentionally omitted because plans and quotas change frequently. See [Configuration and API Keys](docs/wiki/Configuration-and-API-Keys.md) and each provider's current documentation. -The runtime registry also reports the legacy identifiers `linkedin`, `linkedin_links`, `netcraft`, `omnisint`, `sublist3r`, and `zoomeyeapi`. They have no active CLI handlers in the current code and are therefore not presented as usable sources in this table. +The runtime registry also reports the legacy identifiers `linkedin`, `linkedin_links`, `netcraft`, `omnisint`, `sublist3r`, and `zoomeyeapi`. These identifiers have no active CLI handlers. The table does not present them as usable sources. ## Configuration @@ -241,7 +251,7 @@ Read [CONTRIBUTING.md](CONTRIBUTING.md) for the development setup, required chec - Use [GitHub Issues](https://github.com/laramies/theHarvester/issues) for reproducible bugs and focused feature requests. - Report suspected vulnerabilities according to [SECURITY.md](SECURITY.md), not in public issues. -- [Christian Martorella (@laramies)](https://twitter.com/laramies) created theHarvester — [cmartorella@edge-security.com](mailto:cmartorella@edge-security.com). +- [Christian Martorella (@laramies)](https://twitter.com/laramies) created theHarvester No [cmartorella@edge-security.com](mailto:cmartorella@edge-security.com). - [Matt Brown (@NotoriousRebel1)](https://twitter.com/NotoriousRebel1) and [Jay "L1ghtn1ng" Townsend (@jay_townsend1)](https://twitter.com/jay_townsend1) maintain and develop the project. - [Lee Baird (@discoverscripts)](https://twitter.com/discoverscripts) is a main contributor. - Thanks to John Matherly for Shodan and Ahmed Aboul Ela for the bundled subdomain dictionaries. diff --git a/docs/wiki/Configuration-and-API-Keys.md b/docs/wiki/Configuration-and-API-Keys.md index 3f3a6447..40de3ae2 100644 --- a/docs/wiki/Configuration-and-API-Keys.md +++ b/docs/wiki/Configuration-and-API-Keys.md @@ -35,7 +35,9 @@ apikeys: Do not commit populated configuration files. Prefer provider credentials scoped to the minimum access the provider supports. -The [README source matrix](https://github.com/laramies/theHarvester/blob/dev/README.md#discovery-sources) is the canonical list of current sources and whether a key is required, optional, or unused. Provider pricing, quotas, and terms change frequently; use each provider's documentation rather than copying those values into this wiki. +The [README source matrix](https://github.com/laramies/theHarvester/blob/dev/README.md#discovery-sources) is the canonical source list. It shows whether each source requires a key, accepts an optional key, or has no key setting. + +Provider pricing, quotas, and terms change frequently. Check the provider's current documentation for these details. ## Proxies diff --git a/docs/wiki/Contributing-and-Security.md b/docs/wiki/Contributing-and-Security.md index 116815ae..c9a62eea 100644 --- a/docs/wiki/Contributing-and-Security.md +++ b/docs/wiki/Contributing-and-Security.md @@ -2,9 +2,9 @@ Canonical repository guidance lives with the code: -- [CONTRIBUTING.md](https://github.com/laramies/theHarvester/blob/dev/CONTRIBUTING.md) — development setup, focused changes, safe tests, and pull requests. -- [SECURITY.md](https://github.com/laramies/theHarvester/blob/dev/SECURITY.md) — private vulnerability-reporting guidance and scope. -- [GitHub Issues](https://github.com/laramies/theHarvester/issues) — reproducible bugs and focused feature requests. +- [CONTRIBUTING.md](https://github.com/laramies/theHarvester/blob/dev/CONTRIBUTING.md): development setup, focused changes, safe tests, and pull requests. +- [SECURITY.md](https://github.com/laramies/theHarvester/blob/dev/SECURITY.md): private vulnerability-reporting guidance and scope. +- [GitHub Issues](https://github.com/laramies/theHarvester/issues): reproducible bugs and focused feature requests. Do not disclose suspected vulnerabilities, credentials, private target data, or exploit details in a public issue or wiki page. Follow `SECURITY.md` and include no vulnerability details if you must ask maintainers for a private reporting contact. diff --git a/docs/wiki/Home.md b/docs/wiki/Home.md index 17b69195..102cd591 100644 --- a/docs/wiki/Home.md +++ b/docs/wiki/Home.md @@ -1,8 +1,10 @@ # theHarvester wiki -theHarvester gathers open-source intelligence about a domain or organization from search engines, certificate transparency logs, DNS datasets, code repositories, threat-intelligence platforms, and other public sources. +theHarvester gathers open-source intelligence about a domain or organization. It queries search engines, certificate transparency logs, DNS datasets, code repositories, threat-intelligence platforms, and other public sources. -Use it for the early reconnaissance stage of an authorized security assessment. Passive providers still receive the search target, while DNS brute force, DNS resolution, takeover checks, screenshots, and API-path scanning create additional network activity. Use those features only on systems you own or are explicitly authorized to test. +Use it during the early reconnaissance stage of an authorized security assessment. Passive providers still receive the search target. + +DNS brute force, DNS resolution, takeover checks, screenshots, and API-path scanning create additional network activity. Use these features only on systems you own or are explicitly authorized to test. ## Start here diff --git a/docs/wiki/Quick-Start.md b/docs/wiki/Quick-Start.md index 87b0d920..9344a8a0 100644 --- a/docs/wiki/Quick-Start.md +++ b/docs/wiki/Quick-Start.md @@ -46,6 +46,8 @@ uv run theHarvester -d "$AUTHORIZED_DOMAIN" -b crtsh -r resolvers.txt The [README source matrix](https://github.com/laramies/theHarvester/blob/dev/README.md#discovery-sources) shows the result types and credential requirements for every current source. -Avoid starting with `-b all`. It contacts many independent services, increases runtime, consumes quotas, and makes provider-specific failures harder to diagnose. Add sources in small groups that match the result types you need. +Do not start with `-b all`. It contacts many independent services and can consume quotas. It also increases runtime and makes provider failures harder to isolate. + +Choose a small group of sources that provides the result types you need. Use `theHarvester -h` for the current option and source list. diff --git a/docs/wiki/Responsible-Use-and-Scope.md b/docs/wiki/Responsible-Use-and-Scope.md index a4f14f57..c361b296 100644 --- a/docs/wiki/Responsible-Use-and-Scope.md +++ b/docs/wiki/Responsible-Use-and-Scope.md @@ -1,6 +1,6 @@ # Responsible use and scope -Use theHarvester only on targets you own or are explicitly authorized to assess. Authorization should identify the target, permitted techniques, time window, data handling rules, and any third-party restrictions. +Use theHarvester only on targets you own or are explicitly authorized to assess. The authorization should name the target, permitted techniques, time window, data-handling rules, and third-party restrictions. ## Passive does not mean invisible @@ -35,4 +35,6 @@ Results may contain private infrastructure, employee addresses, account identifi ## Service exposure -`restfulHarvest` core query routes are not authenticated. The optional `/additional/*` routes use `THEHARVESTER_API_KEY`, but that does not protect `/query`, `/sources`, or `/dnsbrute`. Keep the service on localhost or place it behind appropriate authentication, network controls, and TLS. +The `restfulHarvest` core query routes do not require authentication. `THEHARVESTER_API_KEY` protects the optional `/additional/*` routes only. It does not protect `/query`, `/sources`, or `/dnsbrute`. + +Keep the service on localhost. If you require remote access, add authentication, network controls, and TLS. diff --git a/docs/wiki/Rest-API.md b/docs/wiki/Rest-API.md index febc64f5..eecb1bbd 100644 --- a/docs/wiki/Rest-API.md +++ b/docs/wiki/Rest-API.md @@ -86,4 +86,6 @@ These routes may also require provider credentials in the request body or local `THEHARVESTER_API_KEY` protects only `/additional/*`. It does not authenticate `/query`, `/sources`, or `/dnsbrute`. -Keep the default localhost binding. If remote access is required, place the service behind authentication, network allowlists, TLS, request logging, and an appropriate rate limit. The supplied Docker Compose configuration binds host port `5000` on every interface unless you narrow the mapping. +Keep the default localhost binding. If you require remote access, add authentication, network allowlists, TLS, request logging, and an appropriate rate limit. + +The supplied Docker Compose configuration binds host port `5000` on every interface unless you narrow the mapping. diff --git a/docs/wiki/Results-and-Local-Data.md b/docs/wiki/Results-and-Local-Data.md index 2ee87410..c7ecea0c 100644 --- a/docs/wiki/Results-and-Local-Data.md +++ b/docs/wiki/Results-and-Local-Data.md @@ -4,7 +4,7 @@ theHarvester can print findings, write reports, retain selected records in SQLit ## Terminal output -The CLI prints consolidated result sections plus separately selected enrichment such as Shodan. Terminal output is intended for operators, not as a stable machine-readable contract. +The CLI groups findings by result type. It can also print separate enrichment, such as Shodan output. Use terminal output for operators, not as a stable automation interface. ## JSON and XML reports @@ -18,9 +18,11 @@ This creates `report.json` and `report.xml`. - **JSON** is one object and contains the broader result set. `cmd`, `hosts`, and `shodan` are always present; other fields appear when non-empty. - **XML** contains the command, emails, hosts, and virtual hosts. Use JSON for other result types. -- Consolidated reports do not retain per-source attribution. +- Current JSON and XML reports do not record which source found each item. -Host values may be plain hostnames or `hostname:IP` pairs when DNS resolution is enabled. The repository [README output section](https://github.com/laramies/theHarvester/blob/dev/README.md#report-formats) documents the current fields and provides copyable `jq` examples. +Host values may be plain hostnames. When DNS resolution is enabled, they can also use the `hostname:IP` form. + +The repository [README output section](https://github.com/laramies/theHarvester/blob/dev/README.md#report-formats) documents the current fields and provides copyable `jq` examples. ## SQLite database diff --git a/tests/test_readme.py b/tests/test_readme.py index 48ba8cc2..a4b31546 100644 --- a/tests/test_readme.py +++ b/tests/test_readme.py @@ -101,7 +101,7 @@ def _documented_source_contracts(readme: str) -> dict[str, set[str]]: contracts: dict[str, set[str]] = {} for source, cells in _documented_source_rows(readme).items(): markers = cells[:6] - assert set(markers) <= {'✓', '—'} + assert set(markers) <= {'✓', 'No'} contracts[source] = {column for column, marker in zip(RESULT_COLUMNS, markers, strict=True) if marker == '✓'} return contracts @@ -129,8 +129,8 @@ def test_readme_matches_executable_source_contracts() -> None: def test_readme_api_key_markers_match_configuration() -> None: requirements = _documented_api_key_requirements(Path('README.md').read_text()) - assert set(requirements.values()) <= {'✓', 'Optional', '—'} - assert {source for source, marker in requirements.items() if marker != '—'} == _configured_api_key_sources() + assert set(requirements.values()) <= {'✓', 'Optional', 'No'} + assert {source for source, marker in requirements.items() if marker != 'No'} == _configured_api_key_sources() assert {source for source, marker in requirements.items() if marker == 'Optional'} == OPTIONAL_API_KEY_SOURCES