diff --git a/.gitignore b/.gitignore index f0f290a7..69f4781d 100644 --- a/.gitignore +++ b/.gitignore @@ -3,6 +3,7 @@ *.sqlite *.html !theHarvester/lib/api/static/harvestview/index.html +!docs/diagrams/*.html *.htm *.vscode *.xml diff --git a/README.md b/README.md index 6c72030a..9053978b 100644 --- a/README.md +++ b/README.md @@ -15,17 +15,31 @@ It is built for the early reconnaissance stage of authorized security assessment - **Useful result types:** collect hostnames, email addresses, IP addresses, URLs, ASNs, and people. - **Enrichment after discovery:** optionally enrich routing evidence through RouteViews, resolve DNS, query Shodan, check for subdomain takeovers, brute-force DNS names, scan common API paths, and capture screenshots. - **CLI and browser-accessible API:** use the command line interactively or run the FastAPI service for automation and interactive Swagger/ReDoc documentation. -- **Repeatable output:** print results, write JSON, XML, and JSONL reports, and retain host, email, and IP findings in a local SQLite database. +- **Repeatable output:** print results, write provenance-aware JSONL, retain run evidence in SQLite, and generate legacy JSON and XML reports when needed. - **Operational controls:** select individual sources, set result limits, use HTTP or SOCKS proxies, choose DNS resolvers, and suppress missing-key noise. Source availability, quotas, and response formats are controlled by third parties and can change independently of theHarvester. +![theHarvester run evidence architecture](docs/images/run-evidence-architecture.svg) + +The editable diagram source is [`docs/diagrams/run-evidence-architecture.html`](docs/diagrams/run-evidence-architecture.html). + +## Releases and packages + +| Channel | Current version | Get it | +| --- | --- | --- | +| GitHub release | `4.11.1` | [Release notes and source archive](https://github.com/laramies/theHarvester/releases/tag/4.11.1) | +| Kali Rolling | `4.11.1-0kali1` | [Kali package tracker](https://pkg.kali.org/pkg/theharvester) | +| Other distributions | Varies | [Repology package list](https://repology.org/project/theharvester/versions) | +| Development | `dev` | [Current development branch](https://github.com/laramies/theHarvester/tree/dev) | + +The [PyPI project](https://pypi.org/project/theHarvester/) currently reports `0.0.1` and does not track this repository's current releases. + ## Quick start -theHarvester requires Python 3.12 or newer and uses [uv](https://docs.astral.sh/uv/) for dependency management. +theHarvester requires Python 3.12 or newer. From a source checkout: ```bash -curl -LsSf https://astral.sh/uv/install.sh | sh git clone https://github.com/laramies/theHarvester.git cd theHarvester uv sync @@ -68,12 +82,14 @@ uv run theHarvester -d example.com -b emails,ips,urls --no-hosts -f non-host-res `--no-hosts` skips sources whose only declared route is `subdomains`. Mixed sources still run, but their hostname getter is not called; emails, IPs, URLs, ASNs, people, and breach names remain available. Hostname and virtual-host records are omitted from terminal, JSON, XML, JSONL, SQLite, API, and HarvestView output. The option cannot be combined with Shodan enrichment, DNS resolution/lookup/brute force/recursion, takeover checks, screenshots, or virtual-host discovery. Target-only API endpoint interaction remains available because it does not depend on harvested hostnames. HarvestView and `POST /api/v1/runs` expose the same option as `no_hosts`. -Save JSON, XML, and JSONL reports: +Save a durable JSONL report: ```bash uv run theHarvester -d example.com -b crtsh,certspotter -f report ``` +This writes `report.jsonl` for automation and interchange. The same command also writes legacy `report.json` and `report.xml` compatibility reports. + Resolve discovered hosts for an authorized domain with the default resolver list: ```bash @@ -278,21 +294,7 @@ Treat collected OSINT as potentially sensitive. Keep report files, screenshots, ### Report formats -The JSON report is a single object. Host entries remain plain hostnames or `hostname:address[,address...]` values when DNS resolution is enabled. DNS resolution and DNS brute force retain candidates only when A, AAAA, or CNAME evidence is available; CNAME-only candidates remain plain hostnames in existing CLI, REST, JSON, and XML output. - -`Checker.check()` and `DnsForce.run()` retain their existing `(resolved, hosts, addresses)` return shape. Normalized A, AAAA, and CNAME values are available through each object's `records` mapping. - -| Field | Availability | Contents | -| --- | --- | --- | -| `cmd` | Always | Command-line arguments used for the run. | -| `hosts` | Always | Discovered hosts; an empty array when none are found. | -| `shodan` | Always | Shodan host objects with canonical IP `value` and structured `details`; an empty array when Shodan is not used. | -| `ips`, `emails`, `vhosts`, `asns`, `prefixes` | When non-empty | Network and contact findings. RouteViews prefixes are external routing relationships, not claimed target scope. | -| `urls` | When non-empty | Discovered URLs from every URL-producing source or action. | -| `people`, `twitter_people`, `linkedin_people` | When non-empty | People and profile findings. | -| `takeover_results` | When non-empty | Optional takeover-check results. | - -The XML report contains the command, emails, hosts, and virtual hosts. Use JSON when you need the additional result types above. +Use JSONL for automation, run interchange, and structured evidence. Legacy JSON and XML remain available for consumers that depend on their existing grouped schemas. The JSONL report is finalized after the selected one-shot actions finish. The first line identifies the run with its UUID, target, UTC timestamps, and result counts. Each later line is one sorted, deduplicated finding. When you concatenate report files, treat each summary line as the start of a new run. @@ -309,7 +311,7 @@ Virtual-host observations do not use that string encoding. Each confirmed name r RouteViews evidence also uses native observations. Each `prefix` finding has `scope: "external-relationship"`, `actions: ["routeviews"]`, and observed-origin, BGP route, or RPKI validation records. These records describe provider-observed routing, never registration, ownership, authorization, reachability, or target scope. -ASN organization labels from URLScan, ONYPHE, and the Shodan host action are also native observations. Each label remains tied to its provider and the exact hostname or IP that supplied the relationship. ONYPHE's physical hosting and logical WHOIS labels remain separate observations. Conflicting labels are retained for review; organization text never becomes an ASN owner field or a pivot filter. Before RouteViews runs, the exact source-attributed IP relationship—not the organization label—selects automatic network pivots. +ASN organization labels from URLScan, ONYPHE, and the Shodan host action are also native observations. Each label remains tied to its provider and the exact hostname or IP that supplied the relationship. ONYPHE's physical hosting and logical WHOIS labels remain separate observations. Conflicting labels are retained for review; organization text never becomes an ASN owner field or a pivot filter. Before RouteViews runs, the exact source-attributed IP relationship, not the organization label, selects automatic network pivots. Parse recursive DNS findings as JSON objects: @@ -343,34 +345,23 @@ List every JSONL finding as tab-separated type and value columns: jq -r 'select(.type != "summary") | [.type, .value] | @tsv' report.jsonl ``` -List discovered hosts with [`jq`](https://jqlang.org/): +#### Legacy JSON and XML -```bash -jq -r '.hosts[]?' report.json -``` +The legacy JSON report is one object. Host entries remain plain hostnames or use `hostname:address[,address...]` when DNS resolution is enabled. DNS resolution and brute force retain candidates only when A, AAAA, or CNAME evidence is available. CNAME-only candidates remain plain hostnames in CLI, REST, JSON, and XML output. -Count common result types while safely handling omitted fields: +`Checker.check()` and `DnsForce.run()` retain their existing `(resolved, hosts, addresses)` return shape. Their `records` mappings contain normalized A, AAAA, and CNAME values. -```bash -jq '{ - hosts: (.hosts // [] | length), - emails: (.emails // [] | length), - ips: (.ips // [] | length), - asns: (.asns // [] | length) -}' report.json -``` +| Field | Availability | Contents | +| --- | --- | --- | +| `cmd` | Always | Command-line arguments used for the run. | +| `hosts` | Always | Discovered hosts; an empty array when none are found. | +| `shodan` | Always | Shodan host objects with canonical IP `value` and structured `details`; an empty array when Shodan is not used. | +| `ips`, `emails`, `vhosts`, `asns`, `prefixes` | When non-empty | Network and contact findings. RouteViews prefixes are external routing relationships, not claimed target scope. | +| `urls` | When non-empty | Discovered URLs from every URL-producing source or action. | +| `people`, `twitter_people`, `linkedin_people` | When non-empty | People and profile findings. | +| `takeover_results` | When non-empty | Optional takeover-check results. | -Export common findings as tab-separated values: - -```bash -jq -r '( - ["type", "value"], - (.hosts[]? | ["host", .]), - (.emails[]? | ["email", .]), - (.ips[]? | ["ip", .]), - (.asns[]? | ["asn", .]) -) | @tsv' report.json > findings.tsv -``` +The XML report contains only the command, emails, hosts, and virtual hosts. JSON and XML group findings by type without source attribution, lifecycle outcomes, or structured action evidence. ## Development and contributing @@ -380,7 +371,7 @@ Read [CONTRIBUTING.md](CONTRIBUTING.md) for the development setup, required chec - Use [GitHub Issues](https://github.com/laramies/theHarvester/issues) for reproducible bugs and focused feature requests. - Report suspected vulnerabilities according to [SECURITY.md](SECURITY.md), not in public issues. -- [Christian Martorella (@laramies)](https://twitter.com/laramies) created theHarvester No [cmartorella@edge-security.com](mailto:cmartorella@edge-security.com). +- [Christian Martorella (@laramies)](https://twitter.com/laramies) created theHarvester. Contact: [cmartorella@edge-security.com](mailto:cmartorella@edge-security.com). - [Matt Brown (@NotoriousRebel1)](https://twitter.com/NotoriousRebel1) and [Jay "L1ghtn1ng" Townsend (@jay_townsend1)](https://twitter.com/jay_townsend1) maintain and develop the project. - [Lee Baird (@discoverscripts)](https://twitter.com/discoverscripts) is a main contributor. - Thanks to John Matherly for Shodan and Ahmed Aboul Ela for the bundled subdomain dictionaries. diff --git a/docs/diagrams/run-evidence-architecture.html b/docs/diagrams/run-evidence-architecture.html new file mode 100644 index 00000000..e599ba1b --- /dev/null +++ b/docs/diagrams/run-evidence-architecture.html @@ -0,0 +1,111 @@ + + + + + + One run, one evidence model + + + + + One run, one evidence model + theHarvester architecture showing CLI, HarvestView, and REST requests moving through a validated run contract, the source catalog and enumeration pipeline, then into one normalized CompletedResult written to JSONL, SQLite, terminal output, and the API. + + + + + + + + + THEHARVESTER ARCHITECTURE + One run, one evidence model + Every interface and activity class converges on the same normalized result. + + + + CONTROL + + + + EXECUTION + + + + EVIDENCE + + + + + + + + + + ENTRY POINTS + Operator interfaces + CLI · HarvestView · REST + one finite run + + + SHARED INPUT + Validated run contract + target · sources · limits · authorization + + + REGISTRY + Source catalog + routes · keys · class + + + FINITE WORK + Enumeration pipeline + + Source runner · P0 + Explicit actions + P0 RouteViews · P1 DNS + P2 direct interaction + + + NORMALIZED RECORD + CompletedResult + findings · origins · outcomes + artifacts · evidence status + partial evidence stays attributable + + + DURABLE OUTPUTS + JSONL · SQLite + terminal · API · HarvestView + + + + internal handoff + + evidence boundary + + operator output + P0 provider · P1 DNS · P2 direct interaction + + + diff --git a/docs/images/run-evidence-architecture.svg b/docs/images/run-evidence-architecture.svg new file mode 100644 index 00000000..04f5273f --- /dev/null +++ b/docs/images/run-evidence-architecture.svg @@ -0,0 +1,97 @@ + + + One run, one evidence model + theHarvester architecture showing CLI, HarvestView, and REST requests moving through a validated run contract, the source catalog and enumeration pipeline, then into one normalized CompletedResult written to JSONL, SQLite, terminal output, and the API. + + + + + + + + + THEHARVESTER ARCHITECTURE + One run, one evidence model + Every interface and activity class converges on the same normalized result. + + + + CONTROL + + + + EXECUTION + + + + EVIDENCE + + + + + + + + + + ENTRY POINTS + Operator interfaces + CLI · HarvestView · REST + one finite run + + + SHARED INPUT + Validated run contract + target · sources · limits · authorization + + + REGISTRY + Source catalog + routes · keys · class + + + FINITE WORK + Enumeration pipeline + + Source runner · P0 + Explicit actions + P0 RouteViews · P1 DNS + P2 direct interaction + + + NORMALIZED RECORD + CompletedResult + findings · origins · outcomes + artifacts · evidence status + partial evidence stays attributable + + + DURABLE OUTPUTS + JSONL · SQLite + terminal · API · HarvestView + + + + internal handoff + + evidence boundary + + operator output + P0 provider · P1 DNS · P2 direct interaction + diff --git a/docs/wiki/How-to-add-a-new-module.md b/docs/wiki/How-to-add-a-new-module.md index 4abc615f..fa50fe77 100644 --- a/docs/wiki/How-to-add-a-new-module.md +++ b/docs/wiki/How-to-add-a-new-module.md @@ -57,7 +57,7 @@ If the source accepts an API key: Never log credentials or include real keys in tests, examples, commits, issues, or pull requests. -## 5. Add focused coverage when possible +## 5. Add focused coverage [The Baidu discovery tests](https://github.com/laramies/theHarvester/blob/dev/tests/discovery/test_baidusearch.py) are a small example that can be copied and adapted. They replace network fetching with `pytest` `monkeypatch` and assert normalized results. diff --git a/docs/wiki/Installation.md b/docs/wiki/Installation.md index c5cac5ac..6810bd4a 100644 --- a/docs/wiki/Installation.md +++ b/docs/wiki/Installation.md @@ -1,6 +1,6 @@ # Installation -theHarvester requires Python 3.12 or newer. Choose one installation lane and finish with its verification step. +theHarvester requires Python 3.12 or newer. Choose one installation lane and finish with its verification step. The README lists the [current GitHub and distribution versions](https://github.com/laramies/theHarvester#releases-and-packages). ## Kali Linux package @@ -12,14 +12,13 @@ sudo apt install theharvester theHarvester -h ``` -If the installed command or available sources differ from the repository, update Kali first and check the packaged version. Package releases can lag behind the current `master` or `dev` branch. +If the installed command or available sources differ from the repository, update Kali first and check the packaged version. Distribution packages can lag behind the current stable release or `dev` branch. -## Source installation with uv +## Source checkout -Install [`uv`](https://docs.astral.sh/uv/getting-started/installation/), clone the repository, and install the locked runtime dependencies: +Clone the repository and install the locked runtime dependencies: ```bash -curl -LsSf https://astral.sh/uv/install.sh | sh git clone https://github.com/laramies/theHarvester.git cd theHarvester uv sync diff --git a/docs/wiki/Operator-Workflows.md b/docs/wiki/Operator-Workflows.md index 85837ed7..59ec4e89 100644 --- a/docs/wiki/Operator-Workflows.md +++ b/docs/wiki/Operator-Workflows.md @@ -16,7 +16,7 @@ Use the [README source matrix](https://github.com/laramies/theHarvester/blob/dev uv run theHarvester -d example.com -b crtsh,certspotter -f report ``` -Use `report.json` for automation and `report.xml` for the smaller legacy host/email representation. See [Results and Local Data](Results-and-Local-Data). +Use `report.jsonl` for automation, provenance, and run interchange. The same command also writes legacy `report.json` and `report.xml` compatibility files. See [Results and Local Data](Results-and-Local-Data). ## DNS resolution diff --git a/docs/wiki/Quick-Start.md b/docs/wiki/Quick-Start.md index 880d7cee..6610d307 100644 --- a/docs/wiki/Quick-Start.md +++ b/docs/wiki/Quick-Start.md @@ -24,7 +24,7 @@ This queries two passive certificate sources and prints consolidated findings. P uv run theHarvester -d example.com -b crtsh,certspotter -f report ``` -This writes `report.json` and `report.xml` in the current directory. JSON contains more result types and is the better automation format. See [Results and Local Data](Results-and-Local-Data). +This writes `report.jsonl` for automation and interchange. It also writes legacy `report.json` and `report.xml` compatibility reports. See [Results and Local Data](Results-and-Local-Data). ## Resolve discovered hosts diff --git a/docs/wiki/Rest-API.md b/docs/wiki/Rest-API.md index 0b56373a..71f0a68a 100644 --- a/docs/wiki/Rest-API.md +++ b/docs/wiki/Rest-API.md @@ -78,8 +78,7 @@ curl -s "http://127.0.0.1:5000/api/v1/runs/$run_id" \ Run submission is asynchronous. Lifecycle status is `queued`, `running`, `cancelling`, `cancelled`, `completed`, or `failed`. Terminal evidence status is reported separately as `complete`, `partial`, or `failed` when evidence exists. -`source_workers` is the same positive concurrency used by CLI `-j` or `--source-workers` and HarvestView. It defaults -to three, is reduced when fewer sources are selected, and never skips sources or limits their results. +`source_workers` is the same positive concurrency used by CLI `-j` or `--source-workers` and HarvestView. It defaults to three, is reduced when fewer sources are selected, and never skips sources or limits their results. P1 DNS and P2 direct options are fields on the same run request. The OpenAPI schema shows their current defaults, limits, and descriptions. The server uses the operator-selected target and does not impose a public-only egress policy. diff --git a/docs/wiki/Results-and-Local-Data.md b/docs/wiki/Results-and-Local-Data.md index 638751f8..08beaefc 100644 --- a/docs/wiki/Results-and-Local-Data.md +++ b/docs/wiki/Results-and-Local-Data.md @@ -6,15 +6,23 @@ theHarvester can print findings, write reports, retain selected records in SQLit The CLI groups findings by result type. It can also print separate enrichment, such as Shodan output. Use terminal output for operators, not as a stable automation interface. -## JSON and XML reports +## JSONL reports -Use `-f NAME` to write both formats: +Use `-f NAME` to write a durable run report: ```bash uv run theHarvester -d example.com -b crtsh,certspotter -f report ``` -This creates `report.json` and `report.xml`. +The recommended automation output is `report.jsonl`. Its first record summarizes the run, evidence status, source and action outcomes, and artifacts. Each remaining record is one normalized finding with producer attribution. The API can import this file without executing discovery. + +```bash +jq -c 'select(.type != "summary") | {type, value, sources, actions}' report.jsonl +``` + +The same `-f report` command also creates `report.json` and `report.xml` for compatibility. + +## Legacy JSON and XML - **JSON** is one object and contains the broader result set. `cmd`, `hosts`, and `shodan` are always present; other fields appear when non-empty. - **XML** contains the command, emails, hosts, and virtual hosts. Use JSON for other result types. @@ -24,7 +32,7 @@ When virtual host discovery runs, JSON's `vhosts` array and XML's `` entr Host values may be plain hostnames. When DNS resolution is enabled, they can also use the `hostname:IP` form. -The repository [README output section](https://github.com/laramies/theHarvester/blob/dev/README.md#report-formats) documents the current fields and provides copyable `jq` examples. +The repository [README output section](https://github.com/laramies/theHarvester/blob/dev/README.md#report-formats) documents the current formats and provides copyable `jq` examples for JSONL. ## SQLite database diff --git a/docs/wiki/_Footer.md b/docs/wiki/_Footer.md index 80192887..f1e17db4 100644 --- a/docs/wiki/_Footer.md +++ b/docs/wiki/_Footer.md @@ -1 +1 @@ -[Repository](https://github.com/laramies/theHarvester) · [Releases](https://github.com/laramies/theHarvester/releases) · [Issues](https://github.com/laramies/theHarvester/issues) · [Contributing](https://github.com/laramies/theHarvester/blob/dev/CONTRIBUTING.md) · [Security](https://github.com/laramies/theHarvester/blob/dev/SECURITY.md) · [GPL-2.0 license metadata](https://github.com/laramies/theHarvester/blob/dev/pyproject.toml) +[Repository](https://github.com/laramies/theHarvester) · [Releases](https://github.com/laramies/theHarvester/releases) · [Issues](https://github.com/laramies/theHarvester/issues) · [Contributing](https://github.com/laramies/theHarvester/blob/dev/CONTRIBUTING.md) · [Security](https://github.com/laramies/theHarvester/blob/dev/SECURITY.md) · [License](https://github.com/laramies/theHarvester/blob/dev/pyproject.toml) diff --git a/tests/test_readme.py b/tests/test_readme.py index 91a88833..883889b8 100644 --- a/tests/test_readme.py +++ b/tests/test_readme.py @@ -116,6 +116,18 @@ def test_wiki_navigation_and_readme_links_resolve() -> None: assert all(Path(target).is_file() for target in readme_wiki_links) +def test_readme_architecture_diagram_is_local_and_accessible() -> None: + readme = Path('README.md').read_text() + html = Path('docs/diagrams/run-evidence-architecture.html') + svg = Path('docs/images/run-evidence-architecture.svg') + + assert f'![theHarvester run evidence architecture]({svg})' in readme + assert f'[`{html}`]({html})' in readme + assert 'role="img"' in svg.read_text() + assert '' in svg.read_text() + assert '<desc id="run-evidence-architecture-desc">' in svg.read_text() + + def test_virtual_host_wiki_examples_match_the_structured_result_contract() -> None: page = Path('docs/wiki/Virtual-Host-Discovery.md').read_text() @@ -131,9 +143,7 @@ def test_virtual_host_wiki_examples_match_the_structured_result_contract() -> No json_examples = re.findall(r'```json\n(.*?)\n```', page, flags=re.DOTALL) finding = next( - json.loads(example) - for example in json_examples - if '"type": "hostname"' in example and '"observations"' in example + json.loads(example) for example in json_examples if '"type": "hostname"' in example and '"observations"' in example ) assert finding['value'] == 'admin.authorized.example' assert finding['actions'] == ['vhost'] @@ -155,6 +165,20 @@ def test_readme_preserves_project_social_attribution() -> None: assert f'@{handle}' in readme +def test_operator_docs_recommend_jsonl_and_assume_uv_is_available() -> None: + readme = Path('README.md').read_text() + installation = Path('docs/wiki/Installation.md').read_text() + quick_start = Path('docs/wiki/Quick-Start.md').read_text() + workflows = Path('docs/wiki/Operator-Workflows.md').read_text() + + assert '## Releases and packages' in readme + assert 'https://github.com/laramies/theHarvester/releases/tag/' in readme + assert 'https://pkg.kali.org/pkg/theharvester' in readme + assert 'curl -LsSf https://astral.sh/uv/install.sh' not in readme + assert 'curl -LsSf https://astral.sh/uv/install.sh' not in installation + assert all('`report.jsonl`' in page and 'automation' in page for page in (quick_start, workflows)) + + def test_readme_explains_jsonl_record_and_structured_evidence_parsing() -> None: readme = Path('README.md').read_text()