WEBDIENSTE/theHarvester
1
0
Fork 0
mirror of https://github.com/laramies/theHarvester.git synced 2026-09-26 11:34:53 +02:00
Code Issues Packages Projects Releases Wiki Activity

Merge pull request #2584 from laramies/dev [skip ci]

Browse Source 
Sync dev into master
This commit is contained in:
Matt
2026-08-24 16:14:28 -04:00
committed by GitHub
parent b8055faed0 54834f991e
commit cca653fd96
95 changed files with 3367 additions and 848 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CONTEXT.md
+7
View File
@@ -31,6 +31,7 @@ When a change alters one of these boundaries, update this document and the neare
- The source catalog is the authority for canonical source names, aliases, credentials, result capabilities, and activity class. Explicit source names and capability selectors form a union; `all` selects every cataloged P0 source once.
- A source adapter returns `None` for ordinary completion or an immutable `SourceExecutionReport` when it must preserve an explicit outcome or stop reason. The central source runner owns observation collection, result counting, no-result classification, exception handling, and final source status.
- A result limit of zero removes the shared local cap on results and pages. Adapters continue until the provider is exhausted, but source-owned quotas, protocol maxima, response limits, and runtime safety bounds still apply. A source that stops at one of those boundaries must report an explicit partial outcome and stop reason.
- Source execution statuses are `completed`, `partial`, `failed`, `rate-limited`, and `skipped`. Mutable adapter fields such as `execution_status` and `stop_reason` are outside this release contract and are rejected, including when an adapter raises or is cancelled.
- Run lifecycle statuses are `queued`, `running`, `cancelling`, `cancelled`, `completed`, and `failed`. Terminal evidence status is independently `complete`, `partial`, or `failed`; retained evidence survives a later cancellation or process failure.
- Run schedules support one-time, hourly, daily, weekly, and monthly recurrence. Daily, weekly, and monthly occurrences preserve the selected local wall-clock time; a monthly day that does not exist falls on that month’s final day.
@@ -39,6 +40,7 @@ When a change alters one of these boundaries, update this document and the neare
### Evidence and portability
- A merged result is canonical by result kind and value. Source and action provenance, typed observations, execution outcomes, timestamps, and artifact metadata remain attached through deduplication and persistence.
- API run details derive per-source hostname yields from persisted normalized provenance. The counts cover observed, unique, shared, DNS-resolved, and unique DNS-resolved hostnames. A hostname is unique when no other source in the run reported it. It is DNS-resolved when the run retained an A, AAAA, or CNAME answer. These counts measure marginal coverage and current DNS evidence, not independent corroboration or service reachability.
- JSONL is the primary single-run interchange format: one summary record followed by sorted finding records. It retains terminal evidence status, producer outcomes, provenance, and supported structured observations.
- SQLite is the canonical local multi-run store. Portable SQLite export contains every finalized evidence record, preserves original run IDs and canonical structured evidence, and excludes queue, cancellation, worker-lease, and legacy-observation state. Screenshot metadata travels with evidence; screenshot files remain separately managed artifacts.
- Legacy JSON and XML remain supported grouped reports for existing consumers. They are presentation formats and do not replace JSONL or SQLite when lossless provenance and structured evidence are required.
@@ -53,6 +55,7 @@ When a change alters one of these boundaries, update this document and the neare
### Deferred boundaries
- Cross-run change detection, alerts, and automatic reactions remain separate future product decisions. A scheduled occurrence only submits finite enumeration runs.
- Cross-run source ranking remains a reporting decision. One run's hostname-yield summary does not automatically select, disable, or rank sources.
- Distributed workers, multi-host operation, PostgreSQL, and hosted multi-user authorization require measured demand and new decisions. The release remains SQLite-first and local-operator focused.
- Automatic scope expansion remains deferred. Evidence can suggest a later target, while the operator controls every scope change.
@@ -210,6 +213,10 @@ _Avoid_: Cancelled run, process killed
One attempt to run one canonical discovery source within an enumeration run, with an explicit completion status and summary counts.
_Avoid_: Source result, provider response
**Source hostname yield**:
One source's normalized hostname counts within one enumeration run: observed, unique, shared, DNS-resolved, and unique DNS-resolved. A hostname is unique when exactly one source reported it in that run. It is DNS-resolved when the run's resolution action retained an A, AAAA, or CNAME answer. The counts measure marginal coverage and current DNS evidence, not independent corroboration, ownership, or service reachability.
_Avoid_: Source quality score, authoritative result count, independent confirmation
**Source capability**:
A declared class of normalized result that a source can contribute to consolidated enumeration output, independent of whether one source execution yields any data.
_Avoid_: Guaranteed result, module return type, source category
README.md
+16 -2
View File
@@ -68,6 +68,8 @@ uv run theHarvester -d example.com -b emails,urls,certspotter
Capability selectors form a union and choose which sources run. They do not discard other result types returned by those sources. Available selectors are `subdomains`, `emails`, `ips`, `asns`, `urls`, `people`, and `breaches`. `-b all` runs every cataloged P0 passive source. P1 DNS and P2 direct sources require explicit selection.
Pass `--limit 0` to remove the shared per-source result cap and local page ceilings. Each adapter then runs until its provider is exhausted. Provider quotas and runtime safeguards still apply. If a provider or safety limit stops a source after it has retained results, the run keeps them and records the source as partial with the stop reason.
Exclude hostname results while retaining other result types:
```bash
@@ -140,7 +142,7 @@ docker compose down
| `GET /api/v1/sources` | List registered discovery sources and capabilities. |
| `POST /api/v1/runs` | Submit a finite enumeration run. |
| `GET /api/v1/runs` | List durable run records. |
| `GET /api/v1/runs/{run_id}` | Retrieve lifecycle state, normalized results, and source outcomes. |
| `GET /api/v1/runs/{run_id}` | Retrieve lifecycle state, normalized results, source outcomes, and hostname yields. |
| `POST /api/v1/runs/{run_id}/cancel` | Cancel queued or running work. |
| `POST /api/v1/runs/import` | Import JSONL evidence without executing discovery. |
| `POST /api/v1/runs/import-database` | Import completed runs from a theHarvester SQLite database. |
@@ -167,7 +169,7 @@ Result types in this table always appear in this order: `subdomains`, `emails`,
The `shodan` source contributes subdomains. Shodan host enrichment through `-s` or `--shodan` is a separate action and is not a source result route.
<details>
<summary><strong>View all 58 discovery sources</strong></summary>
<summary><strong>View all 59 discovery sources</strong></summary>
| Source | Returns | Activity | API key |
| --- | --- | :---: | :---: |
@@ -218,6 +220,7 @@ The `shodan` source contributes subdomains. Shodan host enrichment through `-s`
| [`shodanct`](https://ctl.shodan.io/) | subdomains only | P0 | No |
| [`shodanInternetDB`](https://internetdb.shodan.io/) | subdomains, ips | P1 | No |
| [`sourcegraph`](https://sourcegraph.com/search) | subdomains only | P0 | No |
| [`subdomainapi`](https://api.subdomain.app/) | subdomains only | P0 | No |
| [`subdomaincenter`](https://www.subdomain.center/) | subdomains only | P0 | No |
| [`subdomainfinderc99`](https://subdomainfinder.c99.nl/) | subdomains only | P1 | No |
| [`thc`](https://ip.thc.org/) | subdomains only | P0 | No |
@@ -296,6 +299,17 @@ jq -r 'select(.type != "summary") | [.type, .value] | @tsv' report.jsonl
The `subdomains` capability produces `hostname` records because a result can be the target hostname itself. Read [Results and local data](docs/wiki/Results-and-Local-Data.md) for the complete JSONL and evidence contract.
### Compare source yield
`harvest-yields` reads completed runs from an existing SQLite results database. For each source, it reports observed values, values unique within a run, and hostnames with retained DNS answers. With no arguments, it reads the standard local database:
```bash
uv run harvest-yields
uv run harvest-yields --database results.sqlite --kind ip --format json
```
The command does not run discovery or DNS resolution. Compare runs collected with the same targets, source set, limit, and collection window. The [results guide](docs/wiki/Results-and-Local-Data.md) explains the fields and benchmark method.
### SQLite, JSON, and XML
CLI and API runs use the same SQLite evidence model. JSONL moves one run at a time. SQLite import and export handle completed runs in bulk but exclude queue and worker state. Screenshot files remain separate from their metadata.
docs/adr/0003-run-worker-lifecycle.md
+4 -2
View File
@@ -6,7 +6,7 @@ Status: accepted
The HTTP application owns a durable run record separate from theHarvester's optional terminal `RunResult` evidence. A submission receives its stable ID while queued. One local worker claims one queued run at a time and executes the finite theHarvester core in an isolated child process.
Lifecycle transitions are `queued -> running -> completed|failed`, `queued -> cancelled`, and `running -> cancelling -> cancelled`. A finite whole-run deadline applies when selected; resolution, reverse, and recursive DNS runs default to unlimited so their complete candidate sets can finish, while other runs retain the 1800-second default. Running cancellation first requests cooperative termination, waits a short grace period, and then forces termination if needed. Queued cancellation is an atomic transition that prevents the worker claim.
Lifecycle transitions are `queued -> running -> completed|failed`, `queued -> cancelled`, and `running -> cancelling -> cancelled`. A finite whole-run deadline applies when selected. Resolution, reverse, and recursive DNS runs default to unlimited so their complete candidate sets can finish; other runs retain the 1800-second default. A result limit of zero removes the shared per-source cap on results and pages. Provider quotas, protocol maxima, response-size guards, and runtime safety bounds still apply. Running cancellation first requests cooperative termination, waits a short grace period, and then forces termination if needed. Queued cancellation is an atomic transition that prevents the worker claim.
Evidence already persisted remains attached after failure or cancellation. Terminal evidence status (`complete`, `partial`, or `failed`) is reported independently from orchestration lifecycle status. On service restart, queued runs may resume; orphaned running or cancelling records become failed because their process ownership cannot be proven.
@@ -16,4 +16,6 @@ The existing core is a finite one-shot enumerator and its result object is creat
## Consequences
Run records need one small SQLite table and a lifecycle API. Child-process boundaries make deadline and forced cancellation reliable across blocking provider code. Work is serialized by design. Imported evidence enters as an already completed run record and never enters the queue. Portable database export copies finalized evidence with its original run IDs while omitting lifecycle, cancellation, and worker-lease state.
Run records need one small SQLite table and a lifecycle API. Child-process boundaries make deadline and forced cancellation reliable across blocking provider code. Work is serialized by design. Imported evidence enters as an already completed run record and never enters the queue. Portable database export copies finalized evidence with its original run IDs while omitting lifecycle, cancellation, and worker-lease state. If a provider or safety boundary stops a source after it retained evidence, the source reports a partial outcome and exact stop reason.
Run details derive per-source observed, unique, shared, DNS-resolved, and unique DNS-resolved hostname counts from persisted result origins. DNS resolution therefore retains hostname origins alongside returned IP evidence. The feature uses the existing provenance model rather than a tracker table or second evidence path. Imports and ordinary runs share that model, sources with no matching hostnames remain visible with zero counts, and cross-run ranking and automatic source selection remain deferred.
docs/images/run-evidence-architecture.svg
+1 -1
View File
@@ -99,7 +99,7 @@
<rect x="332" y="144" width="200" height="144" rx="8" class="node"/>
<rect x="344" y="156" width="72" height="20" rx="4" class="tag"/>
<text x="380" y="172" class="tag-text">SOURCES</text>
<text x="348" y="208" class="node-title">58 discovery adapters</text>
<text x="348" y="208" class="node-title">59 discovery adapters</text>
<text x="348" y="232" class="node-copy-strong">search · CT · DNS data</text>
<text x="348" y="248" class="node-copy-strong">code · archives</text>
<text x="348" y="264" class="node-copy-strong">threat intelligence</text>
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 11 KiB

After

Width:  |  Height:  |  Size: 11 KiB

docs/wiki/Operator-Workflows.md
+2
View File
@@ -16,6 +16,8 @@ uv run theHarvester -d example.com -b crtsh,certspotter,commoncrawl
Use the [README source matrix](https://github.com/laramies/theHarvester/blob/dev/README.md#discovery-sources) to choose complementary sources. Adding every source usually increases noise, rate-limit failures, and runtime more than it improves a focused run.
Pass `--limit 0` to remove the shared per-source result cap and local page ceilings. Each adapter then runs until its provider is exhausted. Provider quotas, protocol maxima, response-size guards, and runtime limits still apply. If a provider or safety limit stops a source after it retained evidence, the run keeps that evidence and records the source as partial with the stop reason.
## Save results for automation
Network activity: provider-facing discovery plus local report writes.
docs/wiki/Rest-API.md
+6 -2
View File
@@ -33,7 +33,7 @@ Treat the runtime OpenAPI document as the exact request and response reference.
| `GET /api/v1/sources` | List discovery sources, capabilities, activity classes, and credential names. |
| `POST /api/v1/runs` | Submit one finite enumeration run. |
| `GET /api/v1/runs` | List run records with `limit` and `offset` pagination. |
| `GET /api/v1/runs/{run_id}` | Retrieve lifecycle state, options, results, source outcomes, and artifacts. |
| `GET /api/v1/runs/{run_id}` | Retrieve lifecycle state, options, results, source outcomes, source yields, and artifacts. |
| `POST /api/v1/runs/{run_id}/cancel` | Cancel queued work or request cancellation of running work. |
| `POST /api/v1/runs/import` | Import a JSONL result file without executing discovery. |
| `POST /api/v1/runs/import-database` | Import completed runs from a theHarvester SQLite database. |
@@ -122,7 +122,7 @@ run_id="$(curl -s http://127.0.0.1:5000/api/v1/runs \
curl -s "http://127.0.0.1:5000/api/v1/runs/$run_id" \
-H "X-API-Key: $THEHARVESTER_API_KEY" \
| jq '{status, evidence_status, results, source_executions, action_executions, artifacts}'
| jq '{status, evidence_status, results, source_executions, source_yields, action_executions, artifacts}'
```
Run submission is asynchronous. Read the two status fields separately:
@@ -134,6 +134,10 @@ Run submission is asynchronous. Read the two status fields separately:
`source_workers` is the same positive concurrency used by CLI `-j` or `--source-workers` and HarvestView. It defaults to three, is reduced when fewer sources are selected, and never skips sources or limits their results.
`limit` defaults to 500 per source. A value of `0` removes the shared cap on results and pages, so adapters continue until their provider is exhausted. There is no numeric maximum. Provider quotas, protocol maxima, response-size guards, and runtime limits still apply. If a provider or safety limit stops a source after it retained results, the run keeps those results and records the source as partial with the stop reason.
`source_yields` reports normalized hostname contributions within the run. `unique_result_count` counts hostnames reported by exactly one selected source. When `dns_resolve` ran, `resolved_hostname_count` and `unique_resolved_hostname_count` show which of those hostnames had retained A, AAAA, or CNAME answers. Read these counts with the source's execution status and stop reason. The [results guide](Results-and-Local-Data#compare-source-hostname-yield) explains how to compare fixed runs over time.
P1 DNS and P2 direct options are fields on the same run request. The OpenAPI schema shows their current defaults, limits, and descriptions. The server uses the operator-selected target and does not impose a public-only egress policy.
### Query RouteViews
docs/wiki/Results-and-Local-Data.md
+34
View File
@@ -154,6 +154,40 @@ Two operational tables support the API: `run_records` stores queue and lifecycle
`GET /api/v1/runs/{run_id}` returns lifecycle state plus a normalized `results` array. Each result has `type`, `value`, `sources`, and `actions`. A `hostname` found through the `vhost` action has native endpoint observations; a `prefix` found through RouteViews has native origin, route, and RPKI observations with fixed external-relationship scope. Run-level source and action outcomes remain available in `source_executions` and `action_executions`, while file metadata is returned through `artifacts`. JSONL imports or exports one run. SQLite import and `GET /api/v1/runs/export-database` move completed runs in bulk without queue, cancellation, or worker-lease state. Treat runtime `/docs`, `/redoc`, and OpenAPI as the exact request and response reference.
### Compare source hostname yield
Run details derive `source_yields` from persisted normalized hostname provenance. Each selected source has these fields:
- `observed_result_count`: distinct hostnames attributed to the source.
- `unique_result_count`: hostnames no other source in that run reported.
- `shared_result_count`: hostnames also reported by at least one other source.
- `resolved_hostname_count`: attributed hostnames for which the run's `dns-resolve` action retained an A, AAAA, or CNAME answer.
- `unique_resolved_hostname_count`: resolved hostnames attributed to only this source.
`unique_result_count` measures a source's marginal coverage without depending on source order. `unique_resolved_hostname_count` limits that count to hostnames with current DNS evidence. Neither count proves that a provider is authoritative or independent. A DNS answer also does not prove service reachability.
Read the counts with the matching source execution status and stop reason. A source that failed, was rate-limited or skipped, or stopped at a provider boundary cannot be compared with a source that completed with zero results. Sources in the same certificate-transparency or passive-DNS family may overlap because they depend on the same upstream evidence.
To compare runs, keep the authorized target, source set, requested limit, release version, resolver set, and collection window fixed. Set the limit to `0` only when the comparison should have no shared local result cap. Save completed runs in SQLite and repeat the test across several authorized targets and dates. Compare median unique count, median unique-resolved count, resolution rate, and successful-run rate. Record provider and adapter ceilings as execution evidence instead of treating truncated runs as zero yield. Do not commit target results. Add cross-run aggregation only after you have enough comparable runs to justify it.
#### Analyze yields from SQLite
The installed `harvest-yields` command reads an existing results database. By default, it reads `~/.local/share/theHarvester/stash.sqlite` and reports hostname yields. If the selected database does not exist, the command exits without creating it. Use the flags below to select another database, result kind, or completed run:
```console
harvest-yields
harvest-yields --database results.sqlite
harvest-yields --database results.sqlite --kind hostname
harvest-yields --database results.sqlite --kind ip
harvest-yields --database results.sqlite --kind asn
harvest-yields --database results.sqlite --run-id 11111111-1111-4111-8111-111111111111
harvest-yields --database results.sqlite --format json
```
Without `--run-id`, the command adds each run's source yields. The top-level `run_count` shows how many runs were selected. Each source row has its own `run_count`, including executions that produced no results. `UNIQUE/RUN` divides the summed unique count by that source's run count and is the default ranking key. The JSON field is `unique_result_count_per_run`.
"Unique" always means unique within one run, so aggregate totals add the per-run counts instead of recalculating uniqueness across targets or dates. Hostname output also includes resolved and unique-resolved counts plus `UNIQUE-RESOLVED/RUN`, named `unique_resolved_hostname_count_per_run` in JSON. Other result kinds omit the DNS-specific fields.
## Handling and sharing
- Store results only where the engagement permits.
pyproject.toml
+1
View File
@@ -49,6 +49,7 @@ dev = [
[project.scripts]
theHarvester = "theHarvester.theHarvester:main"
harvestview = "theHarvester.harvestview:main"
harvest-yields = "theHarvester.source_yields:main"
[tool.pytest.ini_options]
minversion = "8.3.3"
tests/discovery/test_apisguru.py
+15 -12
View File
@@ -563,7 +563,7 @@ async def test_malformed_matching_directory_entry_is_failed(
@pytest.mark.asyncio
async def test_directory_entry_scan_is_bounded(monkeypatch: pytest.MonkeyPatch) -> None:
async def test_directory_entry_scan_has_no_local_entry_cap(monkeypatch: pytest.MonkeyPatch) -> None:
spec_urls = [f'https://api.apis.guru/v2/specs/example.com/service-{index}/1.0/openapi.json' for index in range(2)]
directory = {
f'example.com:service-{index}': {
@@ -584,6 +584,11 @@ async def test_directory_entry_scan_is_bounded(monkeypatch: pytest.MonkeyPatch)
status=200,
headers={},
),
FetcherResponse(
body={'openapi': '3.0.3', 'servers': [{'url': 'https://second.example.com'}]},
status=200,
headers={},
),
]
requested_urls: list[str] = []
@@ -591,16 +596,14 @@ async def test_directory_entry_scan_is_bounded(monkeypatch: pytest.MonkeyPatch)
requested_urls.append(kwargs['url'])
return responses.pop(0)
monkeypatch.setattr(apisguru.SearchApisGuru, 'MAX_DIRECTORY_ENTRIES', 1)
monkeypatch.setattr(apisguru.AsyncFetcher, 'fetch_json', fake_fetch)
search = apisguru.SearchApisGuru('example.com', limit=5)
report = await search.process()
assert requested_urls == ['https://api.apis.guru/v2/example.com.json', spec_urls[0]]
assert await search.get_hostnames() == {'api.example.com'}
assert report.status == 'failed'
assert report.stop_reason == 'directory-entry-limit'
assert requested_urls == ['https://api.apis.guru/v2/example.com.json', *spec_urls]
assert await search.get_hostnames() == {'api.example.com', 'second.example.com'}
assert report is None
@pytest.mark.asyncio
@@ -840,7 +843,9 @@ async def test_apex_only_hostname_is_not_counted_as_a_retained_result(monkeypatc
@pytest.mark.asyncio
async def test_scalar_results_are_hard_bounded(monkeypatch: pytest.MonkeyPatch) -> None:
async def test_scalar_results_use_the_requested_limit_without_a_protective_clamp(
monkeypatch: pytest.MonkeyPatch,
) -> None:
spec_url = 'https://api.apis.guru/v2/specs/example.com/1.0/openapi.json'
responses = [
FetcherResponse(
@@ -872,16 +877,14 @@ async def test_scalar_results_are_hard_bounded(monkeypatch: pytest.MonkeyPatch)
async def fake_fetch(**_kwargs: Any) -> FetcherResponse:
return responses.pop(0)
monkeypatch.setattr(apisguru.SearchApisGuru, 'MAX_RESULTS_PER_ROUTE', 1)
monkeypatch.setattr(apisguru.AsyncFetcher, 'fetch_json', fake_fetch)
search = apisguru.SearchApisGuru('example.com', limit=5)
report = await search.process()
assert await search.get_hostnames() == {'one.example.com'}
assert await search.get_urls() == {'https://one.example.com'}
assert report.status == 'failed'
assert report.stop_reason == 'result-cap'
assert await search.get_hostnames() == {'one.example.com', 'two.example.com'}
assert await search.get_urls() == {'https://one.example.com', 'https://two.example.com'}
assert report is None
@pytest.mark.asyncio
tests/discovery/test_arquivo.py
+78 -6
View File
@@ -1,3 +1,4 @@
import asyncio
import logging
from typing import Any
from urllib.parse import parse_qs, urlparse
@@ -6,6 +7,7 @@ import pytest
from theHarvester.discovery import arquivo
from theHarvester.lib.core import FetcherResponse
from theHarvester.lib.source_execution import SourceExecutionReport
@pytest.mark.asyncio
@@ -51,19 +53,58 @@ async def test_process_collects_scoped_hosts_from_one_cdx_request(monkeypatch: p
@pytest.mark.asyncio
async def test_process_bounds_the_provider_limit(monkeypatch: pytest.MonkeyPatch) -> None:
async def test_process_paginates_to_the_operator_limit(monkeypatch: pytest.MonkeyPatch) -> None:
requested_urls: list[str] = []
responses = ['{"url": "https://one.example.com"}\n{"url": "https://two.example.com"}', '{"url": "https://three.example.com"}']
async def fake_fetch_all(urls: list[str], **_kwargs: Any) -> list[FetcherResponse]:
requested_urls.extend(urls)
return [FetcherResponse(body='', status=200, headers={})]
return [FetcherResponse(body=responses.pop(0), status=200, headers={})]
monkeypatch.setattr(arquivo.AsyncFetcher, 'fetch_all', fake_fetch_all)
monkeypatch.setattr(arquivo.SearchArquivo, 'PAGE_SIZE', 2)
search = arquivo.SearchArquivo('example.com', 100_001)
search = arquivo.SearchArquivo('example.com', 3)
await search.process()
assert parse_qs(urlparse(requested_urls[0]).query)['limit'] == ['10000']
queries = [parse_qs(urlparse(url).query) for url in requested_urls]
assert [query['limit'] for query in queries] == [['2'], ['1']]
assert 'offset' not in queries[0]
assert queries[1]['offset'] == ['2']
assert await search.get_hostnames() == {'one.example.com', 'two.example.com', 'three.example.com'}
@pytest.mark.asyncio
async def test_unlimited_request_pages_until_the_provider_is_exhausted(monkeypatch: pytest.MonkeyPatch) -> None:
requested_urls: list[str] = []
responses = ['{"url": "https://one.example.com"}\n{"url": "https://two.example.com"}', '']
async def fake_fetch_all(urls: list[str], **_kwargs: Any) -> list[FetcherResponse]:
requested_urls.extend(urls)
return [FetcherResponse(body=responses.pop(0), status=200, headers={})]
monkeypatch.setattr(arquivo.AsyncFetcher, 'fetch_all', fake_fetch_all)
monkeypatch.setattr(arquivo.SearchArquivo, 'PAGE_SIZE', 2)
report = await arquivo.SearchArquivo('example.com', None).process()
queries = [parse_qs(urlparse(url).query) for url in requested_urls]
assert [query['limit'] for query in queries] == [['2'], ['2']]
assert queries[1]['offset'] == ['2']
assert report is None
@pytest.mark.asyncio
async def test_unlimited_request_reports_repeated_page(monkeypatch: pytest.MonkeyPatch) -> None:
body = '{"url": "https://one.example.com"}'
async def fake_fetch_all(_urls: list[str], **_kwargs: Any) -> list[FetcherResponse]:
return [FetcherResponse(body=body, status=200, headers={})]
monkeypatch.setattr(arquivo.AsyncFetcher, 'fetch_all', fake_fetch_all)
monkeypatch.setattr(arquivo.SearchArquivo, 'PAGE_SIZE', 1)
assert await arquivo.SearchArquivo('example.com', None).process() == SourceExecutionReport('partial', 'repeated-page')
@pytest.mark.asyncio
@@ -83,14 +124,45 @@ async def test_process_reports_http_and_malformed_responses(
with caplog.at_level(logging.INFO, logger=arquivo.__name__):
first = arquivo.SearchArquivo('example.com', 500)
await first.process()
first_report = await first.process()
second = arquivo.SearchArquivo('example.com', 500)
await second.process()
second_report = await second.process()
assert first_report == SourceExecutionReport('failed', 'http-429')
assert second_report == SourceExecutionReport('failed', 'invalid-response')
assert await first.get_hostnames() == set()
assert await second.get_hostnames() == set()
assert 'Arquivo.pt request failed with HTTP 429' in caplog.text
assert 'Arquivo.pt returned malformed CDX data' in caplog.text
@pytest.mark.asyncio
async def test_later_arquivo_failure_preserves_partial_evidence(monkeypatch: pytest.MonkeyPatch) -> None:
responses = [
FetcherResponse(body='{"url": "https://one.example.com"}', status=200, headers={}),
FetcherResponse(body='unavailable', status=503, headers={}),
]
async def fake_fetch_all(_urls: list[str], **_kwargs: Any) -> list[FetcherResponse]:
return [responses.pop(0)]
monkeypatch.setattr(arquivo.AsyncFetcher, 'fetch_all', fake_fetch_all)
monkeypatch.setattr(arquivo.SearchArquivo, 'PAGE_SIZE', 1)
search = arquivo.SearchArquivo('example.com', None)
assert await search.process() == SourceExecutionReport('partial', 'http-503')
assert await search.get_hostnames() == {'one.example.com'}
@pytest.mark.asyncio
async def test_arquivo_cancellation_propagates(monkeypatch: pytest.MonkeyPatch) -> None:
async def cancel(*_args: Any, **_kwargs: Any) -> list[FetcherResponse]:
raise asyncio.CancelledError('operator-stop')
monkeypatch.setattr(arquivo.AsyncFetcher, 'fetch_all', cancel)
with pytest.raises(asyncio.CancelledError, match='operator-stop'):
await arquivo.SearchArquivo('example.com', None).process()
pytestmark = pytest.mark.provider_contract('arquivo')
tests/discovery/test_baidusearch.py
+38 -1
View File
@@ -196,6 +196,25 @@ def patch_http(monkeypatch: pytest.MonkeyPatch, responses: list[baidusearch.Fetc
class TestBaiduSearch:
@pytest.mark.asyncio
async def test_unlimited_stops_when_provider_repeats_a_page(self, monkeypatch: pytest.MonkeyPatch) -> None:
state = patch_browser(
monkeypatch,
[
PageResponse('one.example.com'),
PageResponse('two.example.com'),
PageResponse('two.example.com'),
],
)
search = baidusearch.SearchBaidu(word='example.com', limit=None)
report = await search.process()
assert report == baidusearch.SourceExecutionReport('partial', 'repeated-page')
assert len(state.calls) == 3
assert state.delays == [1.0, 1.0]
assert await search.get_hostnames() == ['one.example.com', 'two.example.com']
@pytest.mark.asyncio
async def test_process_queries_site_first_and_reuses_one_browser(self, monkeypatch: pytest.MonkeyPatch) -> None:
state = patch_browser(
@@ -267,7 +286,7 @@ class TestBaiduSearch:
assert len(state.calls) == 2
assert await search.get_hostnames() == ['api.example.com']
assert report.status == 'failed'
assert report.status == 'partial'
assert report.stop_reason == 'security-verification'
@pytest.mark.asyncio
@@ -329,6 +348,24 @@ class TestBaiduSearch:
assert http.delays == [1.0]
assert http.closed
@pytest.mark.asyncio
async def test_http_fallback_reports_repeated_page(self, monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.setattr(baidusearch, 'playwright_api', None)
patch_http(monkeypatch, [http_response('api.example.com'), http_response('api.example.com')])
report = await baidusearch.SearchBaidu(word='example.com', limit=None).process()
assert report == baidusearch.SourceExecutionReport('partial', 'repeated-page')
@pytest.mark.asyncio
async def test_http_fallback_reports_malformed_response(self, monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.setattr(baidusearch, 'playwright_api', None)
patch_http(monkeypatch, [baidusearch.FetcherResponse(body={}, status=200, headers={})])
report = await baidusearch.SearchBaidu(word='example.com', limit=10).process()
assert report == baidusearch.SourceExecutionReport('failed', 'invalid-response')
@pytest.mark.asyncio
async def test_cancellation_survives_cleanup_failures(self, monkeypatch: pytest.MonkeyPatch) -> None:
cancellation = asyncio.CancelledError('operator-stop')
tests/discovery/test_brave.py
+3 -3
View File
@@ -440,7 +440,7 @@ async def test_brave_rate_limit_does_not_skip_to_the_next_page(
@pytest.mark.asyncio
async def test_brave_reports_maximum_page_offset_as_truncation(
async def test_brave_reports_provider_offset_boundary_as_truncation_when_unlimited(
monkeypatch: pytest.MonkeyPatch,
brave_credentials: InMemoryCredentialAdapter,
) -> None:
@@ -452,11 +452,11 @@ async def test_brave_reports_maximum_page_offset_as_truncation(
return _response([_result(len(requests))], more=True)
monkeypatch.setattr(bravesearch.AsyncFetcher, 'fetch_json', fake_fetch)
search = bravesearch.SearchBrave('example.com', 1_000, credential_adapter=brave_credentials)
search = bravesearch.SearchBrave('example.com', None, credential_adapter=brave_credentials)
report = await search.process()
assert [request['offset'] for request in requests] == [[str(offset)] for offset in range(10)]
assert report == SourceExecutionReport('partial', 'pagination-limit')
assert report == SourceExecutionReport('partial', 'provider-limit')
pytestmark = pytest.mark.provider_contract('brave')
tests/discovery/test_censys.py
+67 -2
View File
@@ -1,10 +1,11 @@
from __future__ import annotations
import asyncio
import contextlib
import sys
import types
from collections.abc import AsyncIterator
from pathlib import Path
from typing import Any
from typing import TYPE_CHECKING, Any
import pytest
@@ -23,6 +24,9 @@ from theHarvester.discovery import censysearch
from theHarvester.discovery.constants import MissingKey
from theHarvester.lib.core import FetcherResponse
if TYPE_CHECKING:
from collections.abc import AsyncIterator
@pytest.mark.asyncio
async def test_missing_platform_token_raises(monkeypatch) -> None:
@@ -140,6 +144,67 @@ async def test_search_calls_platform_api_directly_and_follows_page_tokens(monkey
assert report is None
@pytest.mark.asyncio
async def test_unlimited_search_follows_tokens_until_the_provider_terminates(monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.setattr(censysearch.Core, 'censys_key', lambda: ('platform-token', None))
calls: list[dict[str, object]] = []
responses = [
FetcherResponse({'result': {'hits': [], 'next_page_token': 'next-page'}}, 200, {}),
FetcherResponse({'result': {'hits': [], 'next_page_token': ''}}, 200, {}),
]
async def fake_post_fetch(_url: str, **kwargs: object) -> FetcherResponse:
calls.append(kwargs)
return responses.pop(0)
monkeypatch.setattr(censysearch.AsyncFetcher, 'post_fetch', fake_post_fetch)
search = censysearch.SearchCensys('example.com', limit=None)
assert await search.process() is None
assert [call['json_body'] for call in calls] == [
{
'query': 'cert.names: "example.com"',
'fields': ['cert.names', 'cert.parsed.subject.email_address'],
'page_size': 100,
},
{
'query': 'cert.names: "example.com"',
'fields': ['cert.names', 'cert.parsed.subject.email_address'],
'page_size': 100,
'page_token': 'next-page',
},
]
@pytest.mark.parametrize(
('hit', 'expected_status'),
[
({'certificate_v1': {'resource': {'names': ['api.example.com']}}}, 'partial'),
({'host_v1': {'resource': {'ip': '192.0.2.1'}}}, 'failed'),
],
)
@pytest.mark.asyncio
async def test_repeated_cursor_status_reflects_retained_evidence(
monkeypatch: pytest.MonkeyPatch,
hit: dict[str, object],
expected_status: str,
) -> None:
monkeypatch.setattr(censysearch.Core, 'censys_key', lambda: ('platform-token', None))
responses = [
FetcherResponse({'result': {'hits': [hit], 'next_page_token': 'same'}}, 200, {}),
FetcherResponse({'result': {'hits': [], 'next_page_token': 'same'}}, 200, {}),
]
async def fake_post_fetch(*_args: object, **_kwargs: object) -> FetcherResponse:
return responses.pop(0)
monkeypatch.setattr(censysearch.AsyncFetcher, 'post_fetch', fake_post_fetch)
report = await censysearch.SearchCensys('example.com', limit=None).process()
assert report.status == expected_status
assert report.stop_reason == 'repeated-cursor'
@pytest.mark.asyncio
async def test_session_setup_failure_reports_transport_error(monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.setattr(censysearch.Core, 'censys_key', lambda: ('platform-token', 'org-id'))
tests/discovery/test_certspotter.py
+10 -9
View File
@@ -76,7 +76,7 @@ class TestCertspotterSearch:
]
@pytest.mark.asyncio
async def test_search_preserves_results_at_page_limit(
async def test_search_continues_until_provider_exhaustion_without_a_page_cap(
self,
monkeypatch: pytest.MonkeyPatch,
caplog: pytest.LogCaptureFixture,
@@ -86,22 +86,23 @@ class TestCertspotterSearch:
async def fake_fetch_all(*_args: Any, **_kwargs: Any) -> list[list[dict[str, Any]]]:
nonlocal requests
requests += 1
if requests > 2:
if requests > 3:
return [[]]
return [[{'id': f'cursor-{requests}', 'dns_names': [f'host-{requests}.example.com']}]]
monkeypatch.setattr(certspottersearch.AsyncFetcher, 'fetch_all', fake_fetch_all)
monkeypatch.setattr(certspottersearch.SearchCertspoter, 'MAX_PAGES', 2, raising=False)
search = certspottersearch.SearchCertspoter(TestCertspotter.domain())
with caplog.at_level(logging.WARNING, logger=certspottersearch.__name__):
report = await search.process()
assert requests == 2
assert await search.get_hostnames() == {'host-1.example.com', 'host-2.example.com'}
assert report.status == 'partial'
assert report.stop_reason == 'page-limit'
assert 'page limit reached; results may be incomplete' in caplog.text
assert requests == 4
assert await search.get_hostnames() == {
'host-1.example.com',
'host-2.example.com',
'host-3.example.com',
}
assert report is None
assert 'page limit reached' not in caplog.text
@pytest.mark.asyncio
async def test_search_returns_only_normalized_scoped_names(self, monkeypatch: pytest.MonkeyPatch) -> None:
tests/discovery/test_commoncrawl.py
+30 -17
View File
@@ -99,6 +99,25 @@ async def test_process_uses_unique_indexes_from_latest_catalog_year_window_and_s
assert all(old_endpoint not in url for url in requested_urls)
def test_unlimited_index_selection_has_no_local_history_window() -> None:
catalog = [
{
'id': 'CC-MAIN-2026-30',
'cdx-api': 'https://index.commoncrawl.org/CC-MAIN-2026-30-index',
'to': '2026-07-12T00:00:00',
},
{
'id': 'CC-MAIN-2012',
'cdx-api': 'https://index.commoncrawl.org/CC-MAIN-2012-index',
'to': '2012-12-31T00:00:00',
},
]
indexes = commoncrawl.SearchCommoncrawl._select_indexes(catalog, include_all=True)
assert [index['id'] for index in indexes] == ['CC-MAIN-2026-30', 'CC-MAIN-2012']
@pytest.mark.asyncio
async def test_process_rejects_untrusted_catalog_endpoints(
monkeypatch: pytest.MonkeyPatch,
@@ -161,10 +180,7 @@ async def test_process_requests_provider_pages_sequentially(monkeypatch: pytest.
@pytest.mark.asyncio
async def test_process_caps_provider_page_counts_and_reports_truncation(
monkeypatch: pytest.MonkeyPatch,
caplog: pytest.LogCaptureFixture,
) -> None:
async def test_unlimited_process_exhausts_provider_reported_page_count(monkeypatch: pytest.MonkeyPatch) -> None:
catalog = [
{
'id': 'CC-MAIN-2026-30',
@@ -179,21 +195,18 @@ async def test_process_caps_provider_page_counts_and_reports_truncation(
return [catalog]
query = parse_qs(urlsplit(urls[0]).query)
if query.get('showNumPages') == ['true']:
return ['{"pages": 1000000, "pageSize": 5, "blocks": 5000000}']
requested_pages.extend(int(parse_qs(urlsplit(url).query)['page'][0]) for url in urls)
return ['{"url":"https://api.example.com/"}' for _url in urls]
return ['{"pages": 3, "pageSize": 5, "blocks": 3}']
page = int(query['page'][0])
requested_pages.append(page)
return [f'{{"url":"https://host-{page}.example.com/"}}']
monkeypatch.setattr(commoncrawl.AsyncFetcher, 'fetch_all', fake_fetch_all)
monkeypatch.setattr(commoncrawl.SearchCommoncrawl, 'MAX_PAGES_PER_QUERY', 2)
search = commoncrawl.SearchCommoncrawl('example.com', limit=None)
report = await search.process()
search = commoncrawl.SearchCommoncrawl('example.com', limit=50)
with caplog.at_level(logging.WARNING, logger=commoncrawl.__name__):
report = await search.process()
assert requested_pages == [0, 1, 0, 1]
assert 'Common Crawl page limit reached for index CC-MAIN-2026-30; results may be incomplete' in caplog.text
assert report.status == 'partial'
assert report.stop_reason == 'page-limit'
assert requested_pages == [0, 1, 2, 0, 1, 2]
assert await search.get_hostnames() == {'host-0.example.com', 'host-1.example.com', 'host-2.example.com'}
assert report is None
@pytest.mark.asyncio
@@ -424,7 +437,7 @@ async def test_process_retains_partial_results_at_the_runtime_limit(monkeypatch:
assert await search.get_hostnames() == {'api.example.com'}
assert report.status == 'failed'
assert report.status == 'partial'
assert report.stop_reason == 'runtime-limit'
tests/discovery/test_criminalip.py
+69 -3
View File
@@ -1,10 +1,11 @@
#!/usr/bin/env python3
# coding=utf-8
import asyncio
import logging
import pytest
from theHarvester.discovery import criminalip
from theHarvester.lib.source_execution import SourceExecutionReport
@pytest.mark.asyncio
@@ -18,8 +19,9 @@ async def test_failed_response_body_is_not_logged(monkeypatch, caplog) -> None:
monkeypatch.setattr(criminalip.AsyncFetcher, 'post_fetch', fake_post_fetch)
caplog.set_level(logging.INFO, logger=criminalip.__name__)
await criminalip.SearchCriminalIP('example.com').process()
report = await criminalip.SearchCriminalIP('example.com').process()
assert report == SourceExecutionReport('failed', 'provider-error')
assert 'provider-secret-payload' not in caplog.text
assert '500' in caplog.text
@@ -32,7 +34,9 @@ async def test_parser_handles_missing_legacy_fields(monkeypatch) -> None:
payload = {
'data': {
'certificates': [{'subject': 'www.example.com'}],
'connected_domain_subdomain': [{'main_domain': {'domain': 'example.com'}, 'subdomains': [{'domain': 'api.example.com'}]}],
'connected_domain_subdomain': [
{'main_domain': {'domain': 'example.com'}, 'subdomains': [{'domain': 'api.example.com'}]}
],
'connected_ip': [{'ip': '192.0.2.34'}],
'connected_ip_info': [
{
@@ -122,4 +126,66 @@ async def test_do_search_uses_v2_report_endpoint(monkeypatch) -> None:
assert all('/v1/domain/report/' not in url for url in called_urls)
@pytest.mark.asyncio
async def test_waiting_scan_reports_runtime_limit(monkeypatch) -> None:
monkeypatch.setattr(criminalip.Core, 'criminalip_key', lambda: 'test-key')
monkeypatch.setattr(criminalip.Core, 'get_user_agent', lambda: 'test-agent')
status_calls = 0
async def fake_post_fetch(*_args, **_kwargs):
return {'status': 200, 'data': {'scan_id': 12345}}
async def fake_fetch_all(*_args, **_kwargs):
nonlocal status_calls
status_calls += 1
return [{'status': 200, 'data': {'scan_percentage': 50}}]
async def no_sleep(*_args, **_kwargs):
return None
monkeypatch.setattr(criminalip.AsyncFetcher, 'post_fetch', fake_post_fetch)
monkeypatch.setattr(criminalip.AsyncFetcher, 'fetch_all', fake_fetch_all)
monkeypatch.setattr(criminalip.asyncio, 'sleep', no_sleep)
report = await criminalip.SearchCriminalIP('example.com').process()
assert status_calls == 10
assert report == SourceExecutionReport('partial', 'runtime-limit')
@pytest.mark.asyncio
async def test_polling_cancellation_propagates(monkeypatch) -> None:
monkeypatch.setattr(criminalip.Core, 'criminalip_key', lambda: 'test-key')
monkeypatch.setattr(criminalip.Core, 'get_user_agent', lambda: 'test-agent')
async def fake_post_fetch(*_args, **_kwargs):
return {'status': 200, 'data': {'scan_id': 12345}}
async def fake_fetch_all(*_args, **_kwargs):
return [{'status': 200, 'data': {'scan_percentage': 50}}]
async def cancel(*_args, **_kwargs):
raise asyncio.CancelledError
monkeypatch.setattr(criminalip.AsyncFetcher, 'post_fetch', fake_post_fetch)
monkeypatch.setattr(criminalip.AsyncFetcher, 'fetch_all', fake_fetch_all)
monkeypatch.setattr(criminalip.asyncio, 'sleep', cancel)
with pytest.raises(asyncio.CancelledError):
await criminalip.SearchCriminalIP('example.com').process()
@pytest.mark.asyncio
async def test_provider_timeout_returns_explicit_transport_error(monkeypatch) -> None:
monkeypatch.setattr(criminalip.Core, 'criminalip_key', lambda: 'test-key')
monkeypatch.setattr(criminalip.Core, 'get_user_agent', lambda: 'test-agent')
async def timeout(*_args, **_kwargs):
raise TimeoutError
monkeypatch.setattr(criminalip.AsyncFetcher, 'post_fetch', timeout)
assert await criminalip.SearchCriminalIP('example.com').process() == SourceExecutionReport('failed', 'transport-error')
pytestmark = pytest.mark.provider_contract('criminalip')
tests/discovery/test_fofa.py
+21 -6
View File
@@ -1,8 +1,9 @@
from __future__ import annotations
import asyncio
import base64
import contextlib
from collections.abc import AsyncIterator
from typing import Any
from typing import TYPE_CHECKING, Any
import pytest
@@ -11,6 +12,9 @@ from theHarvester.discovery.constants import MissingKey
from theHarvester.lib.core import FetcherResponse
from theHarvester.lib.source_execution import SourceExecutionReport
if TYPE_CHECKING:
from collections.abc import AsyncIterator
@pytest.mark.provider_contract('fofa')
@pytest.mark.asyncio
@@ -122,12 +126,23 @@ async def test_failures_are_structured(
assert report == SourceExecutionReport(status, reason)
@pytest.mark.parametrize(
('results', 'expected_status'),
[
([['api.example.com', '192.0.2.1']], 'partial'),
([['outside.test', 'not-an-ip']], 'failed'),
],
)
@pytest.mark.asyncio
async def test_repeated_cursor_stops_without_a_third_request(monkeypatch: pytest.MonkeyPatch) -> None:
async def test_repeated_cursor_status_reflects_retained_evidence(
monkeypatch: pytest.MonkeyPatch,
results: list[list[str]],
expected_status: str,
) -> None:
monkeypatch.setattr(fofa.Core, 'fofa_key', lambda: ('test-key', 'operator@example.com'))
responses = [
FetcherResponse({'error': False, 'results': [['api.example.com', '192.0.2.1']], 'next': 'same'}, 200, {}),
FetcherResponse({'error': False, 'results': [['mail.example.com', '192.0.2.2']], 'next': 'same'}, 200, {}),
FetcherResponse({'error': False, 'results': results, 'next': 'same'}, 200, {}),
FetcherResponse({'error': False, 'results': results, 'next': 'same'}, 200, {}),
]
@contextlib.asynccontextmanager
@@ -142,7 +157,7 @@ async def test_repeated_cursor_stops_without_a_third_request(monkeypatch: pytest
search = fofa.SearchFofa('example.com', 10)
report = await search.process()
assert report == SourceExecutionReport('failed', 'repeated-cursor')
assert report == SourceExecutionReport(expected_status, 'repeated-cursor')
assert responses == []
tests/discovery/test_githubcode_contract.py
+109 -4
View File
@@ -1,11 +1,16 @@
from __future__ import annotations
import asyncio
import contextlib
from collections.abc import AsyncIterator
from typing import Any
from typing import TYPE_CHECKING, Any
import pytest
from theHarvester.discovery import githubcode
if TYPE_CHECKING:
from collections.abc import AsyncIterator
class FakeResponse:
status = 200
@@ -14,7 +19,7 @@ class FakeResponse:
self.payload = payload
self.links = links
async def __aenter__(self) -> 'FakeResponse':
async def __aenter__(self) -> FakeResponse:
return self
async def __aexit__(self, *_args: Any) -> None:
@@ -38,7 +43,7 @@ def install_github_responses(monkeypatch: pytest.MonkeyPatch):
def __init__(self, *, headers: dict[str, str]) -> None:
pass
async def __aenter__(self) -> 'FakeSession':
async def __aenter__(self) -> FakeSession:
return self
async def __aexit__(self, *_args: Any) -> None:
@@ -208,4 +213,104 @@ async def test_github_code_malformed_page_terminates_without_following_paginatio
assert await search.get_hostnames() == []
@pytest.mark.asyncio
async def test_github_code_unlimited_pagination_cycle_preserves_partial_evidence(install_github_responses) -> None:
requested_urls = install_github_responses(
FakeResponse(
{'items': [{'text_matches': [{'fragment': 'api.example.com'}]}]},
{'next': {'url': 'https://api.github.com/search/code?q=example.com&page=2'}},
),
FakeResponse(
{'items': [{'text_matches': [{'fragment': 'docs.example.com'}]}]},
{'next': {'url': 'https://api.github.com/search/code?q=example.com&page=1'}},
),
)
search = githubcode.SearchGithubCode('example.com', limit=None)
report = await search.process()
assert requested_urls == [
'https://api.github.com/search/code?q="example.com"&page=1',
'https://api.github.com/search/code?q="example.com"&page=2',
]
assert await search.get_hostnames() == ['api.example.com', 'docs.example.com']
assert report == githubcode.SourceExecutionReport('partial', 'repeated-page')
@pytest.mark.asyncio
async def test_github_code_unlimited_repeated_content_stops_before_counting_duplicates(
install_github_responses,
) -> None:
requested_urls = install_github_responses(
FakeResponse(
{
'items': [
{'text_matches': [{'fragment': 'api.example.com'}, {'fragment': 'docs.example.com'}]},
]
},
{'next': {'url': 'https://api.github.com/search/code?q=example.com&page=2'}},
),
FakeResponse(
{
'items': [
{'text_matches': [{'fragment': 'docs.example.com'}, {'fragment': 'api.example.com'}]},
]
},
{'next': {'url': 'https://api.github.com/search/code?q=example.com&page=3'}},
),
)
search = githubcode.SearchGithubCode('example.com', limit=None)
report = await search.process()
assert requested_urls == [
'https://api.github.com/search/code?q="example.com"&page=1',
'https://api.github.com/search/code?q="example.com"&page=2',
]
assert search.counter == 2
assert await search.get_hostnames() == ['api.example.com', 'docs.example.com']
assert report == githubcode.SourceExecutionReport('partial', 'repeated-page')
@pytest.mark.parametrize('fragments', [[], ['api.example.com']], ids=['no-evidence', 'partial-evidence'])
@pytest.mark.asyncio
async def test_github_code_persistent_exceptions_stop_with_truthful_report(
monkeypatch: pytest.MonkeyPatch,
fragments: list[str],
) -> None:
monkeypatch.setattr(githubcode.Core, 'github_key', staticmethod(lambda: 'test-token'))
monkeypatch.setattr(githubcode, 'get_delay', lambda: 0)
search = githubcode.SearchGithubCode('example.com', limit=None)
search.total_results = ' '.join(fragments)
search.counter = len(fragments)
search.max_retries = 1
calls = 0
async def fail_search(*_args: Any, **_kwargs: Any) -> tuple[str, dict, int, Any]:
nonlocal calls
calls += 1
raise RuntimeError('persistent provider failure')
monkeypatch.setattr(search, 'do_search', fail_search)
report = await search.process()
assert calls == 2
assert report == githubcode.SourceExecutionReport('partial' if fragments else 'failed', 'transport-error')
@pytest.mark.asyncio
async def test_github_code_cancellation_propagates(monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.setattr(githubcode.Core, 'github_key', staticmethod(lambda: 'test-token'))
search = githubcode.SearchGithubCode('example.com', limit=None)
async def cancel_search(*_args: Any, **_kwargs: Any) -> tuple[str, dict, int, Any]:
raise asyncio.CancelledError('operator-stop')
monkeypatch.setattr(search, 'do_search', cancel_search)
with pytest.raises(asyncio.CancelledError, match='operator-stop'):
await search.process()
pytestmark = pytest.mark.provider_contract('github-code')
tests/discovery/test_gitlabsearch.py
+116 -11
View File
@@ -1,13 +1,21 @@
from __future__ import annotations
import json
import sys
from pathlib import Path
from typing import Any
from typing import TYPE_CHECKING, Any
from urllib.parse import parse_qs, urlparse
import pytest
from theHarvester import __main__ as theharvester_main
from theHarvester.discovery import gitlabsearch
from theHarvester.lib.completed_result import CompletedResult
from theHarvester.lib.core import FetcherResponse
from theHarvester.lib.source_execution import SourceExecutionReport
if TYPE_CHECKING:
from pathlib import Path
from theHarvester.lib.completed_result import CompletedResult
@pytest.mark.asyncio
@@ -53,12 +61,12 @@ async def test_public_discovery_normalizes_evidence_and_uses_bounded_requests(
url = next(iter(urls))
requests.append({'url': url, 'headers': headers, 'proxy': proxy})
responses = {
'https://gitlab.com/api/v4/projects?search=example.test&per_page=20': json.dumps(projects),
'https://gitlab.com/api/v4/projects?search=example.test&per_page=100&page=1': json.dumps(projects),
'https://gitlab.com/api/v4/projects/group%2Fproject/repository/files/README.md/raw?ref=feature%2Freadme': (
'Contact Admin@Example.TEST. at docs.example.test; ignore admin@notexample.test'
),
'https://gitlab.com/api/v4/projects?search=*.example.test&per_page=20': '[]',
'https://gitlab.com/api/v4/users?search=example.test&per_page=10': json.dumps(users),
'https://gitlab.com/api/v4/projects?search=*.example.test&per_page=100&page=1': '[]',
'https://gitlab.com/api/v4/users?search=example.test&per_page=100&page=1': json.dumps(users),
}
if url not in responses:
raise AssertionError(f'unexpected GitLab request: {url}')
@@ -73,10 +81,10 @@ async def test_public_discovery_normalizes_evidence_and_uses_bounded_requests(
assert requests == [
{'url': url, 'headers': {'User-agent': 'UA'}, 'proxy': True}
for url in (
'https://gitlab.com/api/v4/projects?search=example.test&per_page=20',
'https://gitlab.com/api/v4/projects?search=example.test&per_page=100&page=1',
'https://gitlab.com/api/v4/projects/group%2Fproject/repository/files/README.md/raw?ref=feature%2Freadme',
'https://gitlab.com/api/v4/projects?search=*.example.test&per_page=20',
'https://gitlab.com/api/v4/users?search=example.test&per_page=10',
'https://gitlab.com/api/v4/projects?search=*.example.test&per_page=100&page=1',
'https://gitlab.com/api/v4/users?search=example.test&per_page=100&page=1',
)
]
assert await search.get_hostnames() == {
@@ -129,6 +137,103 @@ async def test_decoded_pages_are_accepted_without_silent_slicing(monkeypatch: py
assert 'user-11@example.test' in emails
@pytest.mark.asyncio
async def test_unlimited_search_follows_project_and_user_pagination(monkeypatch: pytest.MonkeyPatch) -> None:
requests: list[tuple[str, str, int, int]] = []
async def fake_fetch_all(urls: list[str] | set[str], **kwargs: Any) -> list[FetcherResponse]:
assert kwargs['include_metadata'] is True
assert kwargs['json'] is True
parsed = urlparse(next(iter(urls)))
query = parse_qs(parsed.query)
term = query['search'][0]
page = int(query['page'][0])
per_page = int(query['per_page'][0])
endpoint = parsed.path.rsplit('/', 1)[-1]
requests.append((endpoint, term, page, per_page))
records: dict[tuple[str, str, int], tuple[list[dict[str, object]], str]] = {
('projects', 'example.test', 1): ([{'description': 'first.example.test'}], '2'),
('projects', 'example.test', 2): ([{'description': 'second.example.test'}], ''),
('projects', '*.example.test', 1): ([{'description': 'wildcard.example.test'}], ''),
('users', 'example.test', 1): ([{'public_email': 'first@example.test'}], '2'),
('users', 'example.test', 2): ([{'public_email': 'second@example.test'}], ''),
}
body, next_page = records[(endpoint, term, page)]
return [FetcherResponse(body, 200, {'x-next-page': next_page})]
monkeypatch.setattr(gitlabsearch.AsyncFetcher, 'fetch_all', fake_fetch_all)
search = gitlabsearch.SearchGitlab('example.test', None)
assert await search.process() is None
assert await search.get_hostnames() == {
'first.example.test',
'second.example.test',
'wildcard.example.test',
}
assert await search.get_emails() == {'first@example.test', 'second@example.test'}
assert requests == [
('projects', 'example.test', 1, 100),
('projects', 'example.test', 2, 100),
('projects', '*.example.test', 1, 100),
('users', 'example.test', 1, 100),
('users', 'example.test', 2, 100),
]
@pytest.mark.asyncio
async def test_unlimited_search_reports_repeated_gitlab_cursor(monkeypatch: pytest.MonkeyPatch) -> None:
async def fake_fetch_all(*_args: Any, **_kwargs: Any) -> list[FetcherResponse]:
return [FetcherResponse([{'description': 'api.example.test'}], 200, {'x-next-page': '1'})]
monkeypatch.setattr(gitlabsearch.AsyncFetcher, 'fetch_all', fake_fetch_all)
search = gitlabsearch.SearchGitlab('example.test', None)
assert await search.process() == SourceExecutionReport('partial', 'repeated-cursor')
@pytest.mark.asyncio
async def test_unlimited_search_reports_gitlab_provider_bound(monkeypatch: pytest.MonkeyPatch) -> None:
calls = 0
async def fake_fetch_all(*_args: Any, **_kwargs: Any) -> list[FetcherResponse]:
nonlocal calls
calls += 1
if calls == 1:
return [FetcherResponse([{'description': 'api.example.test'}], 200, {'x-next-page': '2'})]
return [FetcherResponse({'message': 'Pagination limit reached'}, 400, {})]
monkeypatch.setattr(gitlabsearch.AsyncFetcher, 'fetch_all', fake_fetch_all)
search = gitlabsearch.SearchGitlab('example.test', None)
assert await search.process() == SourceExecutionReport('partial', 'provider-limit')
assert await search.get_hostnames() == {'api.example.test'}
@pytest.mark.asyncio
async def test_finite_limit_stops_each_gitlab_query(monkeypatch: pytest.MonkeyPatch) -> None:
requests: list[tuple[str, str, int, int]] = []
async def fake_fetch_all(urls: list[str] | set[str], **_kwargs: Any) -> list[FetcherResponse]:
parsed = urlparse(next(iter(urls)))
query = parse_qs(parsed.query)
endpoint = parsed.path.rsplit('/', 1)[-1]
term = query['search'][0]
page = int(query['page'][0])
per_page = int(query['per_page'][0])
requests.append((endpoint, term, page, per_page))
return [FetcherResponse([{}], 200, {'x-next-page': '2'})]
monkeypatch.setattr(gitlabsearch.AsyncFetcher, 'fetch_all', fake_fetch_all)
search = gitlabsearch.SearchGitlab('example.test', 1)
assert await search.process() == SourceExecutionReport('completed', 'result-limit')
assert requests == [
('projects', 'example.test', 1, 1),
('projects', '*.example.test', 1, 1),
('users', 'example.test', 1, 1),
]
@pytest.mark.asyncio
async def test_gitlab_urls_reach_completed_jsonl(
monkeypatch: pytest.MonkeyPatch,
@@ -147,8 +252,8 @@ async def test_gitlab_urls_reach_completed_jsonl(
completed_results.append(result)
class FakeGitlab:
def __init__(self, domain: str) -> None:
assert domain == 'example.test'
def __init__(self, domain: str, limit: int | None) -> None:
assert (domain, limit) == ('example.test', 500)
async def process(self, _proxy: bool) -> None:
return None
tests/discovery/test_hunterhow.py
+22
View File
@@ -76,6 +76,28 @@ async def test_process_paginates_to_limit_and_keeps_scoped_hostnames(monkeypatch
assert session_exited is True
@pytest.mark.asyncio
async def test_unlimited_search_has_no_local_history_window(monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.setattr(searchhunterhow.Core, 'hunterhow_key', staticmethod(lambda: 'test-key'))
calls: list[dict[str, Any]] = []
@contextlib.asynccontextmanager
async def fake_open_session(**_kwargs: Any) -> AsyncIterator[object]:
yield object()
async def fake_fetch(**kwargs: Any) -> FetcherResponse:
calls.append(kwargs)
return FetcherResponse({'code': 200, 'data': {'total': 0, 'list': []}}, 200, {})
monkeypatch.setattr(searchhunterhow.AsyncFetcher, 'open_session', fake_open_session)
monkeypatch.setattr(searchhunterhow.AsyncFetcher, 'fetch', fake_fetch)
report = await searchhunterhow.SearchHunterHow('example.com', limit=None).process()
assert report is None
assert calls[0]['params']['start_time'] == '1970-01-01'
@pytest.mark.parametrize('key', [None, '', ' '])
def test_missing_or_empty_key_fails_before_transport(monkeypatch: pytest.MonkeyPatch, key: str | None) -> None:
monkeypatch.setattr(searchhunterhow.Core, 'hunterhow_key', staticmethod(lambda: key))
tests/discovery/test_huntersearch.py
+42 -4
View File
@@ -6,6 +6,7 @@ import pytest
from theHarvester.discovery import huntersearch
from theHarvester.discovery.constants import MissingKey
from theHarvester.lib.core import FetcherResponse
from theHarvester.lib.source_execution import SourceExecutionReport
@pytest.mark.parametrize('key', [None, ' '])
@@ -34,7 +35,6 @@ async def test_hunter_http_failures_return_no_results(monkeypatch, caplog, statu
assert await search.get_emails() == []
assert await search.get_hostnames() == []
assert f'Hunter request failed with HTTP {status}' in caplog.text
assert 'provider detail' not in caplog.text
@@ -196,6 +196,35 @@ async def test_free_hunter_search_honors_limit_and_offset(monkeypatch) -> None:
]
@pytest.mark.asyncio
async def test_free_hunter_unlimited_reports_saturated_provider_boundary(monkeypatch) -> None:
responses = iter(
[
{'data': {'plan_name': 'Free', 'requests': {'searches': {'available': 10, 'used': 0}}}},
{
'data': {
'emails': [
{'value': f'user{index}@example.test', 'sources': [{'domain': f'user{index}.example.test'}]}
for index in range(10)
]
}
},
]
)
async def fake_fetch_all(*_args, **_kwargs):
return [FetcherResponse(body=next(responses), status=200, headers={})]
monkeypatch.setattr(huntersearch.Core, 'hunter_key', lambda: 'test-key')
monkeypatch.setattr(huntersearch.Core, 'get_user_agent', lambda: 'test-agent')
monkeypatch.setattr(huntersearch.AsyncFetcher, 'fetch_all', fake_fetch_all)
search = huntersearch.SearchHunter('example.test', None, 0)
assert await search.process() == SourceExecutionReport('partial', 'provider-limit')
assert len(await search.get_hostnames()) == 10
@pytest.mark.asyncio
async def test_paid_hunter_search_stops_before_exceeding_quota(monkeypatch) -> None:
requests: list[str] = []
@@ -203,6 +232,13 @@ async def test_paid_hunter_search_stops_before_exceeding_quota(monkeypatch) -> N
[
{'data': {'plan_name': 'Growth', 'requests': {'searches': {'available': 1, 'used': 0}}}},
{'data': {'total': 250}},
{
'data': {
'emails': [
{'value': 'alice@example.test', 'sources': [{'domain': 'api.example.test'}]},
]
}
},
]
)
@@ -215,14 +251,16 @@ async def test_paid_hunter_search_stops_before_exceeding_quota(monkeypatch) -> N
monkeypatch.setattr(huntersearch.AsyncFetcher, 'fetch_all', fake_fetch_all)
search = huntersearch.SearchHunter('example.test', 250, 0)
await search.process()
report = await search.process()
assert requests == [
'https://api.hunter.io/v2/account?api_key=test-key',
'https://api.hunter.io/v2/email-count?domain=example.test',
'https://api.hunter.io/v2/domain-search?domain=example.test&api_key=test-key&limit=100&offset=0',
]
assert await search.get_emails() == []
assert await search.get_hostnames() == []
assert report == SourceExecutionReport('partial', 'quota-exhausted')
assert await search.get_emails() == ['alice@example.test']
assert await search.get_hostnames() == ['api.example.test']
pytestmark = pytest.mark.provider_contract('hunter')
tests/discovery/test_intelxsearch.py
+45 -6
View File
@@ -9,8 +9,9 @@ from theHarvester.lib.completed_result import CompletedResult, ResultObservation
class _Response:
def __init__(self, payload: object) -> None:
def __init__(self, payload: object, status: int = 200) -> None:
self.payload = payload
self.status = status
async def __aenter__(self):
return self
@@ -24,8 +25,9 @@ class _Response:
class _Session:
def __init__(self, result: object, search_result: object | None = None) -> None:
self.result = result
self.result = list(result) if isinstance(result, list) else result
self.search_result = {'success': True, 'id': 'search-id'} if search_result is None else search_result
self.search_request: dict[str, object] | None = None
async def __aenter__(self):
return self
@@ -33,11 +35,12 @@ class _Session:
async def __aexit__(self, *_args) -> None:
return None
def post(self, *_args, **_kwargs) -> _Response:
def post(self, *_args, **kwargs) -> _Response:
self.search_request = kwargs.get('json')
return _Response(self.search_result)
def get(self, *_args, **_kwargs) -> _Response:
return _Response(self.result)
return _Response(self.result.pop(0) if isinstance(self.result, list) else self.result)
def test_blank_key_is_missing(monkeypatch: pytest.MonkeyPatch) -> None:
@@ -50,6 +53,7 @@ def test_blank_key_is_missing(monkeypatch: pytest.MonkeyPatch) -> None:
@pytest.mark.asyncio
async def test_process_exposes_flat_normalized_in_scope_results(monkeypatch: pytest.MonkeyPatch) -> None:
result = {
'status': 1,
'selectors': [
{'selectorvalue': 'ADMIN@Example.COM'},
{'selectorvalue': 'bad local@example.com'},
@@ -64,7 +68,7 @@ async def test_process_exposes_flat_normalized_in_scope_results(monkeypatch: pyt
{'selectorvalue': 'http://['},
{'selectorvalue': None},
None,
]
],
}
session = _Session(result)
@@ -83,6 +87,41 @@ async def test_process_exposes_flat_normalized_in_scope_results(monkeypatch: pyt
assert await search.get_urls() == ['https://portal.example.com/path']
@pytest.mark.asyncio
async def test_unlimited_process_collects_provider_pages_until_terminal_status(monkeypatch: pytest.MonkeyPatch) -> None:
pages = [
{'status': 0, 'selectors': [{'selectorvalue': 'one.example.com'}]},
{'status': 1, 'selectors': [{'selectorvalue': 'two.example.com'}]},
]
session = _Session(pages)
monkeypatch.setattr(intelxsearch.Core, 'intelx_key', staticmethod(lambda: 'test-key'))
monkeypatch.setattr(intelxsearch.aiohttp, 'ClientSession', lambda: session)
search = intelxsearch.SearchIntelx('example.com', limit=None)
assert await search.process() is None
assert await search.get_hostnames() == ['one.example.com', 'two.example.com']
assert session.search_request['maxresults'] == intelxsearch.SearchIntelx.UNLIMITED_QUERY_RESULTS
@pytest.mark.asyncio
async def test_finite_process_stops_after_requested_selector_count(monkeypatch: pytest.MonkeyPatch) -> None:
pages = [
{'status': 0, 'selectors': [{'selectorvalue': 'one.example.com'}, {'selectorvalue': 'two.example.com'}]},
{'status': 1, 'selectors': [{'selectorvalue': 'three.example.com'}]},
]
session = _Session(pages)
monkeypatch.setattr(intelxsearch.Core, 'intelx_key', staticmethod(lambda: 'test-key'))
monkeypatch.setattr(intelxsearch.aiohttp, 'ClientSession', lambda: session)
search = intelxsearch.SearchIntelx('example.com', limit=2)
assert await search.process() is None
assert await search.get_hostnames() == ['one.example.com', 'two.example.com']
@pytest.mark.asyncio
@pytest.mark.parametrize(
('search_result', 'result'),
@@ -125,7 +164,7 @@ async def test_orchestrator_stores_intelx_subdomains_without_dns(monkeypatch: py
completed_results.append(result)
class _Intelx:
def __init__(self, _domain: str) -> None:
def __init__(self, _domain: str, _limit: int | None) -> None:
pass
async def process(self, _proxy: bool) -> None:
tests/discovery/test_netlas.py
+51
View File
@@ -67,6 +67,57 @@ async def test_process_uses_current_api_contract_and_keeps_scoped_results(monkey
assert session_exited is True
@pytest.mark.asyncio
async def test_unlimited_request_covers_the_provider_count_error_margin(monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.setattr(netlas.Core, 'netlas_key', staticmethod(lambda: 'test-key'))
@contextlib.asynccontextmanager
async def fake_open_session(**_kwargs: Any) -> AsyncIterator[object]:
yield object()
count_calls: list[dict[str, Any]] = []
download_calls: list[dict[str, Any]] = []
async def fake_fetch(*args: Any, **kwargs: Any) -> FetcherResponse:
count_calls.append({'url': args[0] if args else kwargs['url'], **kwargs})
return FetcherResponse({'count': 12_345}, 200, {})
async def fake_post_fetch(*_args: Any, **kwargs: Any) -> FetcherResponse:
download_calls.append(kwargs)
return FetcherResponse([], 200, {})
monkeypatch.setattr(netlas.AsyncFetcher, 'open_session', fake_open_session)
monkeypatch.setattr(netlas.AsyncFetcher, 'fetch', fake_fetch)
monkeypatch.setattr(netlas.AsyncFetcher, 'post_fetch', fake_post_fetch)
report = await netlas.SearchNetlas('example.com', None).process()
assert count_calls[0]['url'] == 'https://app.netlas.io/api/domains_count/'
assert count_calls[0]['params'] == {'q': '*.example.com'}
assert download_calls[0]['json_body']['size'] == 12_727
assert report is None
@pytest.mark.asyncio
async def test_unlimited_request_rejects_an_invalid_provider_count(monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.setattr(netlas.Core, 'netlas_key', staticmethod(lambda: 'test-key'))
@contextlib.asynccontextmanager
async def fake_open_session(**_kwargs: Any) -> AsyncIterator[object]:
yield object()
async def fake_fetch(*_args: Any, **_kwargs: Any) -> FetcherResponse:
return FetcherResponse({'count': 'many'}, 200, {})
monkeypatch.setattr(netlas.AsyncFetcher, 'open_session', fake_open_session)
monkeypatch.setattr(netlas.AsyncFetcher, 'fetch', fake_fetch)
report = await netlas.SearchNetlas('example.com', None).process()
assert report.status == 'failed'
assert report.stop_reason == 'invalid-response'
@pytest.mark.parametrize('key', [None, '', ' '])
def test_missing_or_blank_key_fails_closed(monkeypatch: pytest.MonkeyPatch, key: str | None) -> None:
monkeypatch.setattr(netlas.Core, 'netlas_key', staticmethod(lambda: key))
tests/discovery/test_onyphe.py
+10 -5
View File
@@ -1,7 +1,8 @@
from __future__ import annotations
import asyncio
import contextlib
from collections.abc import AsyncIterator
from typing import Any
from typing import TYPE_CHECKING, Any
import pytest
@@ -9,6 +10,9 @@ from theHarvester.discovery import onyphe
from theHarvester.discovery.constants import MissingKey
from theHarvester.lib.core import FetcherResponse
if TYPE_CHECKING:
from collections.abc import AsyncIterator
@pytest.mark.provider_contract('onyphe')
@pytest.mark.asyncio
@@ -196,7 +200,8 @@ async def test_operator_limit_is_not_reported_as_a_provider_limit(monkeypatch: p
@pytest.mark.asyncio
async def test_search_api_reports_its_documented_total_boundary(monkeypatch: pytest.MonkeyPatch) -> None:
@pytest.mark.parametrize('limit', [None, 10_001])
async def test_search_api_reports_its_documented_total_boundary(monkeypatch: pytest.MonkeyPatch, limit: int | None) -> None:
monkeypatch.setattr(onyphe.Core, 'onyphe_key', lambda: 'test-key')
calls: list[dict[str, Any]] = []
response = FetcherResponse(
@@ -220,13 +225,13 @@ async def test_search_api_reports_its_documented_total_boundary(monkeypatch: pyt
monkeypatch.setattr(onyphe.AsyncFetcher, 'open_session', fake_open_session)
monkeypatch.setattr(onyphe.AsyncFetcher, 'fetch', fake_fetch)
search = onyphe.SearchOnyphe('example.com', 10_001)
search = onyphe.SearchOnyphe('example.com', limit)
report = await search.process()
assert [call['params']['page'] for call in calls] == [1]
assert [call['params']['size'] for call in calls] == [10_000]
assert await search.get_hostnames() == {'api.example.com'}
assert report.status == 'failed'
assert report.status == 'partial'
assert report.stop_reason == 'provider-limit'
tests/discovery/test_pentesttools.py
+35 -3
View File
@@ -1,8 +1,10 @@
import asyncio
import logging
import pytest
from theHarvester.discovery import pentesttools
from theHarvester.lib.source_execution import SourceExecutionReport
@pytest.mark.asyncio
@@ -107,8 +109,9 @@ async def test_status_error_payload_is_not_logged(monkeypatch, caplog) -> None:
monkeypatch.setattr(pentesttools.asyncio, 'sleep', no_sleep)
caplog.set_level(logging.INFO, logger=pentesttools.__name__)
await pentesttools.SearchPentestTools('example.com').process()
report = await pentesttools.SearchPentestTools('example.com').process()
assert report == SourceExecutionReport('failed', 'provider-error')
assert 'provider-secret-payload' not in caplog.text
assert 'private target data' not in caplog.text
assert 'did not finish successfully' in caplog.text
@@ -134,7 +137,6 @@ async def test_malformed_start_response_completes_without_evidence(monkeypatch,
assert not await search.get_hostnames()
assert not await search.get_ips()
assert 'malformed' in caplog.text
@@ -159,12 +161,42 @@ async def test_waiting_scan_stops_after_ten_status_checks(monkeypatch, caplog) -
monkeypatch.setattr(pentesttools.asyncio, 'sleep', no_sleep)
caplog.set_level(logging.INFO, logger=pentesttools.__name__)
await pentesttools.SearchPentestTools('example.test').process()
report = await pentesttools.SearchPentestTools('example.test').process()
assert status_calls == 10
assert report == SourceExecutionReport('partial', 'runtime-limit')
assert 'still waiting after 10 status checks' in caplog.text
@pytest.mark.asyncio
async def test_polling_cancellation_propagates(monkeypatch) -> None:
monkeypatch.setattr(pentesttools.Core, 'pentest_tools_key', lambda: 'test-key')
async def fake_post_fetch(**_kwargs):
return {'data': {'created_id': 420323}}
async def cancel(*_args, **_kwargs):
raise asyncio.CancelledError
monkeypatch.setattr(pentesttools.AsyncFetcher, 'post_fetch', fake_post_fetch)
monkeypatch.setattr(pentesttools.asyncio, 'sleep', cancel)
with pytest.raises(asyncio.CancelledError):
await pentesttools.SearchPentestTools('example.test').process()
@pytest.mark.asyncio
async def test_provider_timeout_returns_explicit_transport_error(monkeypatch) -> None:
monkeypatch.setattr(pentesttools.Core, 'pentest_tools_key', lambda: 'test-key')
async def timeout(**_kwargs):
raise TimeoutError
monkeypatch.setattr(pentesttools.AsyncFetcher, 'post_fetch', timeout)
assert await pentesttools.SearchPentestTools('example.test').process() == SourceExecutionReport('failed', 'transport-error')
@pytest.mark.asyncio
async def test_malformed_status_response_completes_without_evidence(monkeypatch, caplog) -> None:
monkeypatch.setattr(pentesttools.Core, 'pentest_tools_key', lambda: 'test-key')
tests/discovery/test_securityscorecard.py
+43
View File
@@ -157,6 +157,49 @@ async def test_malformed_asset_rows_preserve_valid_partial_results(monkeypatch:
assert report.stop_reason == 'invalid-response'
@pytest.mark.parametrize(
('entry', 'expected_status', 'expected_hosts'),
[
({'domain': 'api.example.com'}, 'partial', {'api.example.com'}),
({'domain': 'outside.test'}, 'failed', set()),
],
)
@pytest.mark.asyncio
async def test_unlimited_repeated_asset_page_stops_with_truthful_report(
monkeypatch: pytest.MonkeyPatch,
entry: dict[str, str],
expected_status: str,
expected_hosts: set[str],
) -> None:
monkeypatch.setattr(securityscorecard.Core, 'securityscorecard_key', staticmethod(lambda: 'test-key'))
monkeypatch.setattr(securityscorecard.SearchSecurityScorecard, 'PAGE_SIZE', 1)
calls = 0
@contextlib.asynccontextmanager
async def fake_open_session(**_kwargs: Any) -> AsyncIterator[object]:
yield object()
async def fake_fetch(**_kwargs: Any) -> FetcherResponse:
return FetcherResponse({}, 200, {})
async def fake_post_fetch(*_args: Any, **_kwargs: Any) -> FetcherResponse:
nonlocal calls
calls += 1
return FetcherResponse({'entries': [entry], 'size': 2}, 200, {})
monkeypatch.setattr(securityscorecard.AsyncFetcher, 'open_session', fake_open_session)
monkeypatch.setattr(securityscorecard.AsyncFetcher, 'fetch', fake_fetch)
monkeypatch.setattr(securityscorecard.AsyncFetcher, 'post_fetch', fake_post_fetch)
search = securityscorecard.SearchSecurityScorecard('example.com', None)
report = await search.process()
assert calls == 2
assert await search.get_hostnames() == expected_hosts
assert report.status == expected_status
assert report.stop_reason == 'repeated-page'
@pytest.mark.asyncio
async def test_cancellation_closes_provider_session(monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.setattr(securityscorecard.Core, 'securityscorecard_key', staticmethod(lambda: 'test-key'))
tests/discovery/test_shodan_engine.py
+46
View File
@@ -393,6 +393,52 @@ class TestShodanEngine:
assert hosts['198.51.100.21']['services'][0]['tls'] == {'subject_cn': 'cert.example.test'}
assert report is None
@pytest.mark.asyncio
@pytest.mark.parametrize(
('hostnames', 'expected_status'),
[(('api.example.test',), 'partial'), (('outside.invalid',), 'failed')],
)
async def test_shodan_repeated_search_page_is_not_counted_toward_provider_total(
self,
monkeypatch,
hostnames,
expected_status,
):
from theHarvester.discovery import shodansearch
from theHarvester.lib.core import FetcherResponse
search_calls = []
repeated_match = {
'ip_str': '198.51.100.20',
'hostnames': list(hostnames),
'port': 443,
'transport': 'tcp',
}
async def fetch_json(url, *, params, **_kwargs):
if not url.endswith('/search'):
return FetcherResponse(body=None, status=404, headers={})
search_calls.append((params['query'], params['page']))
if params['query'] == 'hostname:example.test':
return FetcherResponse(body={'matches': [repeated_match], 'total': 3}, status=200, headers={})
return FetcherResponse(body={'matches': [], 'total': 0}, status=200, headers={})
monkeypatch.setattr(shodansearch.Core, 'shodan_key', lambda: 'test-key')
monkeypatch.setattr(shodansearch.AsyncFetcher, 'fetch_json', fetch_json)
patch_resolution(monkeypatch, shodansearch)
search = shodansearch.SearchShodan('example.test')
report = await search.process()
assert search_calls == [
('hostname:example.test', 1),
('hostname:example.test', 2),
('ssl:example.test', 1),
]
assert await search.get_hostnames() == ({'api.example.test'} if expected_status == 'partial' else set())
assert report.status == expected_status
assert report.stop_reason == 'repeated-page'
@pytest.mark.asyncio
async def test_shodan_discovery_counts_service_only_evidence_as_a_result(self, monkeypatch):
from theHarvester.discovery import shodansearch
tests/discovery/test_sourcegraph.py
+14 -14
View File
@@ -95,7 +95,7 @@ async def test_sourcegraph_uses_fixed_chunk_query_and_collects_descendants(
event('done', {}),
)
calls = install_stream(monkeypatch, records)
search = sourcegraph.SearchSourcegraph(' Scope.TEST. ', limit=1)
search = sourcegraph.SearchSourcegraph(' Scope.TEST. ', limit=500)
report = await search.process(proxy=True)
@@ -364,7 +364,7 @@ async def test_sourcegraph_rejects_events_after_done(monkeypatch: pytest.MonkeyP
@pytest.mark.asyncio
async def test_sourcegraph_preserves_prefix_at_hostname_limit(monkeypatch: pytest.MonkeyPatch) -> None:
async def test_sourcegraph_honors_finite_result_limit_after_terminal_stream(monkeypatch: pytest.MonkeyPatch) -> None:
install_stream(
monkeypatch,
(
@@ -381,36 +381,36 @@ async def test_sourcegraph_preserves_prefix_at_hostname_limit(monkeypatch: pytes
event('done', {}),
),
)
monkeypatch.setattr(sourcegraph.SearchSourcegraph, 'MAX_HOSTNAMES', 1)
search = sourcegraph.SearchSourcegraph('scope.test', limit=500)
search = sourcegraph.SearchSourcegraph('scope.test', limit=1)
report = await search.process()
assert await search.get_hostnames() == ['one.scope.test']
assert report.status == 'failed'
assert report.stop_reason == 'response-limit'
assert report.status == 'completed'
assert report.stop_reason == 'result-limit'
@pytest.mark.asyncio
async def test_sourcegraph_preserves_prefix_at_event_limit(monkeypatch: pytest.MonkeyPatch) -> None:
async def test_sourcegraph_unlimited_mode_has_no_event_or_hostname_cap(monkeypatch: pytest.MonkeyPatch) -> None:
hostnames = ' '.join(f'host-{number}.scope.test' for number in range(10_001))
install_stream(
monkeypatch,
(
event(
'matches',
[{'type': 'content', 'chunkMatches': [{'content': 'api.scope.test'}]}],
[{'type': 'content', 'chunkMatches': [{'content': hostnames}]}],
),
event('progress', {'done': False, 'skipped': []}),
*(event('filters', []) for _ in range(10_000)),
event('progress', {'done': True, 'skipped': []}),
event('done', {}),
),
)
monkeypatch.setattr(sourcegraph.SearchSourcegraph, 'MAX_EVENTS', 1)
search = sourcegraph.SearchSourcegraph('scope.test', limit=500)
search = sourcegraph.SearchSourcegraph('scope.test', limit=None)
report = await search.process()
assert await search.get_hostnames() == ['api.scope.test']
assert report.status == 'failed'
assert report.stop_reason == 'response-limit'
assert len(await search.get_hostnames()) == 10_001
assert report is None
@pytest.mark.asyncio
tests/discovery/test_subdomainapi.py
+116
View File
@@ -0,0 +1,116 @@
import asyncio
from typing import Any
import pytest
from theHarvester.discovery import subdomainapi
from theHarvester.lib.core import FetcherResponse
from theHarvester.lib.source_execution import SourceExecutionReport
@pytest.mark.asyncio
async def test_process_keeps_every_returned_scoped_name_and_reports_the_provider_limit(
monkeypatch: pytest.MonkeyPatch,
) -> None:
captured: dict[str, Any] = {}
async def fake_fetch_all(urls: list[str], **kwargs: Any) -> list[FetcherResponse]:
captured.update({'urls': urls, **kwargs})
return [
FetcherResponse(
body={
'domain': 'example.com',
'count': 6,
'total': 10_006,
'subdomains': [
'API.Example.COM.',
'api.example.com',
'deep.api.example.com',
'*.wild.example.com',
'example.com',
'outside.test',
],
},
status=200,
headers={},
)
]
monkeypatch.setattr(subdomainapi.AsyncFetcher, 'fetch_all', fake_fetch_all)
search = subdomainapi.SearchSubdomainApi('example.com')
report = await search.process(proxy=True)
assert await search.get_hostnames() == {'api.example.com', 'deep.api.example.com'}
assert report == SourceExecutionReport('partial', 'provider-limit')
assert captured == {
'urls': ['https://api.subdomain.app/v1/query?domain=example.com'],
'headers': {'User-Agent': subdomainapi.Core.get_user_agent()},
'proxy': True,
'json': True,
'include_metadata': True,
}
@pytest.mark.asyncio
async def test_process_accepts_a_valid_empty_response(monkeypatch: pytest.MonkeyPatch) -> None:
async def fake_fetch_all(*_args: Any, **_kwargs: Any) -> list[FetcherResponse]:
return [
FetcherResponse(
body={'domain': 'example.com', 'count': 0, 'total': 0, 'subdomains': []},
status=200,
headers={},
)
]
monkeypatch.setattr(subdomainapi.AsyncFetcher, 'fetch_all', fake_fetch_all)
search = subdomainapi.SearchSubdomainApi('example.com')
assert await search.process() is None
assert await search.get_hostnames() == set()
@pytest.mark.parametrize(
('response', 'expected'),
[
(None, SourceExecutionReport('failed', 'transport-error')),
(FetcherResponse({}, 403, {}), SourceExecutionReport('failed', 'access-denied')),
(FetcherResponse({}, 429, {}), SourceExecutionReport('rate-limited', 'http-429')),
(FetcherResponse({}, 503, {}), SourceExecutionReport('failed', 'http-503')),
(FetcherResponse([], 200, {}), SourceExecutionReport('failed', 'invalid-response')),
(
FetcherResponse(
{'domain': 'example.com', 'count': 2, 'total': 2, 'subdomains': ['api.example.com']},
200,
{},
),
SourceExecutionReport('failed', 'invalid-response'),
),
],
)
@pytest.mark.asyncio
async def test_process_reports_provider_and_response_failures(
monkeypatch: pytest.MonkeyPatch,
response: FetcherResponse | None,
expected: SourceExecutionReport,
) -> None:
async def fake_fetch_all(*_args: Any, **_kwargs: Any) -> list[FetcherResponse | None]:
return [response]
monkeypatch.setattr(subdomainapi.AsyncFetcher, 'fetch_all', fake_fetch_all)
assert await subdomainapi.SearchSubdomainApi('example.com').process() == expected
@pytest.mark.asyncio
async def test_process_propagates_cancellation(monkeypatch: pytest.MonkeyPatch) -> None:
async def cancelled_fetch(*_args: Any, **_kwargs: Any) -> list[FetcherResponse]:
raise asyncio.CancelledError
monkeypatch.setattr(subdomainapi.AsyncFetcher, 'fetch_all', cancelled_fetch)
with pytest.raises(asyncio.CancelledError):
await subdomainapi.SearchSubdomainApi('example.com').process()
pytestmark = pytest.mark.provider_contract('subdomainapi')
tests/discovery/test_thc.py
+30 -3
View File
@@ -1,5 +1,4 @@
#!/usr/bin/env python3
# coding=utf-8
"""Tests for the THC (ip.thc.org) discovery source.
THC provides multiple endpoints:
@@ -9,8 +8,10 @@ THC provides multiple endpoints:
API documentation: https://ip.thc.org/docs/
"""
from types import TracebackType
from typing import Any, Self
from __future__ import annotations
from typing import TYPE_CHECKING, Any, Self
from urllib.parse import parse_qs, urlparse
import httpx
@@ -19,6 +20,9 @@ import pytest
from theHarvester.discovery import thc
from theHarvester.lib.core import Core
if TYPE_CHECKING:
from types import TracebackType
class FakeResponse:
def __init__(self, text: str, status: int = 200, headers: dict[str, str] | None = None) -> None:
@@ -176,6 +180,27 @@ class TestThcSubdomainSearch:
result_list = list(result)
assert len(result_list) == len(set(result_list))
@pytest.mark.asyncio
async def test_unlimited_uses_provider_max_and_reports_saturation(
self,
monkeypatch: pytest.MonkeyPatch,
) -> None:
requested_urls: list[str] = []
class RecordingSession(FakeSession):
def get(self, url: str) -> FakeResponse:
requested_urls.append(url)
return FakeResponse('one.example.com\ntwo.example.com\n')
monkeypatch.setattr(thc.SearchThc, 'PROVIDER_MAX_RESULTS', 2)
monkeypatch.setattr(thc.aiohttp, 'ClientSession', RecordingSession)
report = await thc.SearchThc(self.domain(), None).process()
assert parse_qs(urlparse(requested_urls[0]).query)['limit'] == ['2']
assert report.status == 'partial'
assert report.stop_reason == 'provider-limit'
@pytest.mark.asyncio
@pytest.mark.parametrize(
('outcomes', 'message'),
@@ -425,12 +450,14 @@ class TestThcIntegration:
async def test_module_can_be_imported(self) -> None:
"""Import the THC discovery module."""
from theHarvester.discovery import thc as thc_module
assert thc_module is not None
@pytest.mark.asyncio
async def test_search_class_exists(self) -> None:
"""Expose the ``SearchThc`` adapter."""
from theHarvester.discovery import thc as thc_module
assert hasattr(thc_module, 'SearchThc')
@pytest.mark.asyncio
tests/discovery/test_tombasearch.py
+36 -4
View File
@@ -6,6 +6,7 @@ import pytest
from theHarvester.discovery import tombasearch
from theHarvester.discovery.constants import MissingKey
from theHarvester.lib.core import FetcherResponse
from theHarvester.lib.source_execution import SourceExecutionReport
def tomba_page(first: int, count: int = 50) -> dict:
@@ -56,7 +57,6 @@ async def test_tomba_http_failures_return_no_results(monkeypatch, caplog, status
assert await search.get_emails() == []
assert await search.get_hostnames() == []
assert f'Tomba request failed with HTTP {status}' in caplog.text
assert 'provider detail' not in caplog.text
@@ -264,6 +264,33 @@ async def test_free_tomba_search_honors_limit_and_start(monkeypatch) -> None:
}
@pytest.mark.asyncio
async def test_free_tomba_unlimited_reports_saturated_provider_boundary(monkeypatch) -> None:
responses = iter(
[
{
'data': {
'pricing': {'name': 'Free'},
'requests': {'domains': {'available': 10, 'used': 0}},
}
},
tomba_page(0, 10),
]
)
async def fake_fetch_all(*_args, **_kwargs):
return [FetcherResponse(body=next(responses), status=200, headers={})]
monkeypatch.setattr(tombasearch.Core, 'tomba_key', lambda: ('test-key', 'test-secret'))
monkeypatch.setattr(tombasearch.Core, 'get_user_agent', lambda: 'test-agent')
monkeypatch.setattr(tombasearch.AsyncFetcher, 'fetch_all', fake_fetch_all)
search = tombasearch.SearchTomba('example.test', None, 0)
assert await search.process() == SourceExecutionReport('partial', 'provider-limit')
assert len(await search.get_hostnames()) == 10
@pytest.mark.asyncio
async def test_paid_tomba_search_stops_before_exceeding_quota(monkeypatch) -> None:
requests: list[str] = []
@@ -276,6 +303,8 @@ async def test_paid_tomba_search_stops_before_exceeding_quota(monkeypatch) -> No
}
},
{'data': {'total': 120}},
tomba_page(0),
tomba_page(50),
]
)
@@ -288,14 +317,17 @@ async def test_paid_tomba_search_stops_before_exceeding_quota(monkeypatch) -> No
monkeypatch.setattr(tombasearch.AsyncFetcher, 'fetch_all', fake_fetch_all)
search = tombasearch.SearchTomba('example.test', 120, 0)
await search.process()
report = await search.process()
assert requests == [
'https://api.tomba.io/v1/me',
'https://api.tomba.io/v1/email-count?domain=example.test',
'https://api.tomba.io/v1/domain-search?domain=example.test&limit=50&page=1',
'https://api.tomba.io/v1/domain-search?domain=example.test&limit=50&page=2',
]
assert await search.get_emails() == []
assert await search.get_hostnames() == []
assert report == SourceExecutionReport('partial', 'quota-exhausted')
assert len(await search.get_emails()) == 100
assert len(await search.get_hostnames()) == 100
pytestmark = pytest.mark.provider_contract('tomba')
tests/discovery/test_urlscan.py
+23 -7
View File
@@ -1,8 +1,9 @@
from __future__ import annotations
import asyncio
import contextlib
from collections.abc import AsyncIterator
from datetime import UTC, datetime, timedelta
from typing import Any
from typing import TYPE_CHECKING, Any
import pytest
@@ -10,6 +11,9 @@ from theHarvester.discovery import urlscan
from theHarvester.lib.core import FetcherResponse
from theHarvester.lib.source_execution import SourceExecutionReport
if TYPE_CHECKING:
from collections.abc import AsyncIterator
class ProviderSession:
def __init__(self) -> None:
@@ -375,16 +379,28 @@ async def test_missing_cursor_stops_after_first_page(monkeypatch: pytest.MonkeyP
assert report == SourceExecutionReport('failed', 'invalid-cursor')
@pytest.mark.parametrize(
('domains', 'expected_status', 'expected_hostnames'),
[
(('first.example.com', 'second.example.com'), 'partial', {'first.example.com', 'second.example.com'}),
(('outside.test', 'other.test'), 'failed', set()),
],
)
@pytest.mark.asyncio
async def test_repeated_cursor_stops_without_a_third_request(monkeypatch: pytest.MonkeyPatch) -> None:
async def test_repeated_cursor_status_reflects_retained_evidence(
monkeypatch: pytest.MonkeyPatch,
domains: tuple[str, str],
expected_status: str,
expected_hostnames: set[str],
) -> None:
responses = [
FetcherResponse(
body={'results': [{'page': {'domain': 'first.example.com'}, 'sort': [1, 'same']}]},
body={'results': [{'page': {'domain': domains[0]}, 'sort': [1, 'same']}]},
status=200,
headers={},
),
FetcherResponse(
body={'results': [{'page': {'domain': 'second.example.com'}, 'sort': [1, 'same']}]},
body={'results': [{'page': {'domain': domains[1]}, 'sort': [1, 'same']}]},
status=200,
headers={},
),
@@ -402,8 +418,8 @@ async def test_repeated_cursor_stops_without_a_third_request(monkeypatch: pytest
report = await search.process()
assert calls == 2
assert await search.get_hostnames() == {'first.example.com', 'second.example.com'}
assert report == SourceExecutionReport('failed', 'repeated-cursor')
assert await search.get_hostnames() == expected_hostnames
assert report == SourceExecutionReport(expected_status, 'repeated-cursor')
@pytest.mark.asyncio
tests/discovery/test_virustotal.py
+17 -5
View File
@@ -180,12 +180,24 @@ async def test_later_rate_limit_preserves_partial_results(monkeypatch: pytest.Mo
assert report.stop_reason == 'http-429'
@pytest.mark.parametrize(
('identifiers', 'expected_status', 'expected_hostnames'),
[
(('outside.test', 'api.example.com'), 'partial', {'api.example.com'}),
(('outside.test', 'other.test'), 'failed', set()),
],
)
@pytest.mark.asyncio
async def test_repeated_cursor_stops_without_spending_more_quota(monkeypatch: pytest.MonkeyPatch) -> None:
async def test_repeated_cursor_status_reflects_retained_evidence(
monkeypatch: pytest.MonkeyPatch,
identifiers: tuple[str, str],
expected_status: str,
expected_hostnames: set[str],
) -> None:
monkeypatch.setattr(virustotal.Core, 'virustotal_key', staticmethod(lambda: 'test-key'))
responses = [
FetcherResponse({'data': [{'id': 'outside.test', 'attributes': {}}], 'meta': {'cursor': 'same'}}, 200, {}),
FetcherResponse({'data': [{'id': 'api.example.com', 'attributes': {}}], 'meta': {'cursor': 'same'}}, 200, {}),
FetcherResponse({'data': [{'id': identifiers[0], 'attributes': {}}], 'meta': {'cursor': 'same'}}, 200, {}),
FetcherResponse({'data': [{'id': identifiers[1], 'attributes': {}}], 'meta': {'cursor': 'same'}}, 200, {}),
]
@contextlib.asynccontextmanager
@@ -201,8 +213,8 @@ async def test_repeated_cursor_stops_without_spending_more_quota(monkeypatch: py
report = await search.process()
assert await search.get_hostnames() == {'api.example.com'}
assert report.status == 'failed'
assert await search.get_hostnames() == expected_hostnames
assert report.status == expected_status
assert report.stop_reason == 'repeated-cursor'
assert responses == []
tests/discovery/test_waybackarchive.py
+29 -12
View File
@@ -54,7 +54,8 @@ async def test_process_stops_when_a_resume_key_repeats(monkeypatch: pytest.Monke
search = waybackarchive.SearchWaybackarchive('example.com')
report = await search.process()
assert report is None
assert report.status == 'failed'
assert report.stop_reason == 'repeated-cursor'
wildcard_requests = [query for query in requests if query['url'] == ['*.example.com']]
assert len(wildcard_requests) == 2
assert await search.get_hostnames() == {'api.example.com'}
@@ -167,7 +168,7 @@ async def test_process_ignores_empty_html_and_non_text_responses(
@pytest.mark.asyncio
async def test_process_respects_the_per_query_page_bound(
async def test_process_continues_until_provider_exhaustion_without_a_page_bound(
monkeypatch: pytest.MonkeyPatch,
caplog: pytest.LogCaptureFixture,
) -> None:
@@ -176,24 +177,24 @@ async def test_process_respects_the_per_query_page_bound(
async def fake_fetch_all(urls: list[str], **_kwargs: object) -> list[str]:
nonlocal wildcard_requests
query = parse_qs(urlparse(urls[0]).query)
if query['url'] == ['*.example.com']:
if query['url'] == ['*.example.com'] and wildcard_requests < 3:
wildcard_requests += 1
return [f'https://host-{wildcard_requests}.example.com/path\n\npage-{wildcard_requests}']
return ['']
monkeypatch.setattr(waybackarchive.AsyncFetcher, 'fetch_all', fake_fetch_all)
monkeypatch.setattr(waybackarchive.SearchWaybackarchive, 'MAX_PAGES_PER_QUERY', 2)
search = waybackarchive.SearchWaybackarchive('example.com')
with caplog.at_level(logging.INFO, logger=waybackarchive.__name__):
report = await search.process()
assert wildcard_requests == 2
assert await search.get_hostnames() == {'host-1.example.com', 'host-2.example.com'}
assert 'Wayback Archive page limit reached for pattern *.example.com; results may be incomplete' in caplog.text
assert report is not None
assert report.status == 'partial'
assert report.stop_reason == 'page-limit'
assert wildcard_requests == 3
assert await search.get_hostnames() == {
'host-1.example.com',
'host-2.example.com',
'host-3.example.com',
}
assert 'Wayback Archive page limit reached' not in caplog.text
assert report is None
@pytest.mark.asyncio
@@ -219,12 +220,28 @@ async def test_process_retains_partial_results_at_the_runtime_limit(
report = await search.process()
assert await search.get_hostnames() == {'api.example.com'}
assert report.status == 'failed'
assert report.status == 'partial'
assert report.stop_reason == 'runtime-limit'
assert 'Wayback Archive page 1: hosts=1' in caplog.text
assert 'example.com' not in caplog.text
@pytest.mark.asyncio
async def test_process_reports_a_runtime_limit_before_collecting_results(monkeypatch: pytest.MonkeyPatch) -> None:
async def fake_fetch_all(_urls: list[str], **_kwargs: object) -> list[str]:
await asyncio.Event().wait()
monkeypatch.setattr(waybackarchive.AsyncFetcher, 'fetch_all', fake_fetch_all)
monkeypatch.setattr(waybackarchive.SearchWaybackarchive, 'RUNTIME_SECONDS', 0.01)
search = waybackarchive.SearchWaybackarchive('example.com')
report = await asyncio.wait_for(search.process(), timeout=0.1)
assert await search.get_hostnames() == set()
assert report.status == 'failed'
assert report.stop_reason == 'runtime-limit'
@pytest.mark.asyncio
async def test_process_stops_at_the_requested_result_limit(monkeypatch: pytest.MonkeyPatch) -> None:
requested_urls: list[str] = []
tests/discovery/test_whoisxml.py
+37
View File
@@ -192,6 +192,43 @@ async def test_later_page_failure_preserves_partial_results(monkeypatch: pytest.
assert report.stop_reason == 'http-429'
@pytest.mark.parametrize(
('domains', 'expected_status'),
[
(('api.example.com', 'mail.example.com'), 'partial'),
(('outside.test', 'other.test'), 'failed'),
],
)
@pytest.mark.asyncio
async def test_repeated_cursor_status_reflects_retained_evidence(
monkeypatch: pytest.MonkeyPatch,
domains: tuple[str, str],
expected_status: str,
) -> None:
monkeypatch.setattr(whoisxml.Core, 'whoisxml_key', staticmethod(lambda: 'test-key'))
responses = [
FetcherResponse(
{'result': {'nextPageSearchAfter': 'same', 'records': [{'domain': domains[0]}]}},
200,
{},
),
FetcherResponse(
{'result': {'nextPageSearchAfter': 'same', 'records': [{'domain': domains[1]}]}},
200,
{},
),
]
async def fake_fetch(**_kwargs: Any) -> FetcherResponse:
return responses.pop(0)
monkeypatch.setattr(whoisxml.AsyncFetcher, 'fetch', fake_fetch)
report = await whoisxml.SearchWhoisXML('example.com', 10).process()
assert report.status == expected_status
assert report.stop_reason == 'repeated-cursor'
@pytest.mark.asyncio
async def test_cancellation_closes_provider_session(
monkeypatch: pytest.MonkeyPatch,
tests/discovery/test_windvane.py
+104
View File
@@ -4,6 +4,7 @@ import socket
import pytest
from theHarvester.discovery import windvane
from theHarvester.lib.source_execution import SourceExecutionReport
@pytest.mark.asyncio
@@ -81,4 +82,107 @@ async def test_keyless_provider_failure_does_not_guess_dns_names(monkeypatch) ->
assert await search.get_ips() == set()
@pytest.mark.asyncio
async def test_authenticated_unlimited_search_follows_all_endpoint_pagination(monkeypatch) -> None:
monkeypatch.setattr(windvane.Core, 'windvane_key', lambda: 'test-key')
requests: list[tuple[str, int, int]] = []
async def fake_post_fetch(url, headers=None, data=None, proxy=False):
endpoint = url.rsplit('/', 1)[-1]
page_request = json.loads(data)['page_request']
page = page_request['page']
requests.append((endpoint, page, page_request['count']))
last_pages = {'ListSubDomain': 4, 'ListDNS': 3, 'ListEmail': 2}
last_page = last_pages[endpoint]
if page > last_page:
return json.dumps({'code': 0, 'data': {'list': [], 'has_more': False}})
if endpoint == 'ListSubDomain':
item = {'domain': f'sub-{page}.example.test'}
elif endpoint == 'ListDNS':
item = {'domain': f'dns-{page}.example.test', 'answer': f'203.0.113.{page}', 'answer_type': 'A'}
else:
item = {'email': f'user-{page}@example.test'}
return json.dumps({'code': 0, 'data': {'list': [item], 'has_more': page < last_page}})
monkeypatch.setattr(windvane.AsyncFetcher, 'post_fetch', fake_post_fetch)
search = windvane.SearchWindvane('example.test', None)
assert await search.process() is None
assert 'sub-4.example.test' in await search.get_hostnames()
assert 'dns-3.example.test' in await search.get_hostnames()
assert 'user-2@example.test' in await search.get_emails()
assert requests == [
('ListSubDomain', 1, 30),
('ListSubDomain', 2, 30),
('ListSubDomain', 3, 30),
('ListSubDomain', 4, 30),
('ListDNS', 1, 30),
('ListDNS', 2, 30),
('ListDNS', 3, 30),
('ListEmail', 1, 50),
('ListEmail', 2, 50),
]
@pytest.mark.asyncio
async def test_keyless_unlimited_search_reports_repeated_page(monkeypatch) -> None:
monkeypatch.setattr(windvane.Core, 'windvane_key', lambda: None)
async def fake_post_fetch(*_args, **_kwargs):
return json.dumps(
{
'code': 0,
'data': {'list': [{'domain': 'api.example.test'}], 'has_more': True},
}
)
monkeypatch.setattr(windvane.AsyncFetcher, 'post_fetch', fake_post_fetch)
search = windvane.SearchWindvane('example.test', None)
assert await search.process() == SourceExecutionReport('partial', 'repeated-page')
assert await search.get_hostnames() == {'api.example.test'}
@pytest.mark.asyncio
async def test_keyless_unlimited_search_reports_provider_bound(monkeypatch) -> None:
monkeypatch.setattr(windvane.Core, 'windvane_key', lambda: None)
calls = 0
async def fake_post_fetch(*_args, **_kwargs):
nonlocal calls
calls += 1
if calls == 1:
return json.dumps(
{
'code': 0,
'data': {'list': [{'domain': 'api.example.test'}], 'has_more': True},
}
)
return json.dumps({'code': 1, 'message': 'Unauthenticated request limit reached'})
monkeypatch.setattr(windvane.AsyncFetcher, 'post_fetch', fake_post_fetch)
search = windvane.SearchWindvane('example.test', None)
assert await search.process() == SourceExecutionReport('partial', 'provider-limit')
assert await search.get_hostnames() == {'api.example.test'}
@pytest.mark.asyncio
async def test_finite_limit_stops_each_windvane_endpoint(monkeypatch) -> None:
monkeypatch.setattr(windvane.Core, 'windvane_key', lambda: 'test-key')
requests: list[tuple[str, int, int]] = []
async def fake_post_fetch(url, headers=None, data=None, proxy=False):
endpoint = url.rsplit('/', 1)[-1]
page_request = json.loads(data)['page_request']
requests.append((endpoint, page_request['page'], page_request['count']))
return json.dumps({'code': 0, 'data': {'list': [{}], 'has_more': True}})
monkeypatch.setattr(windvane.AsyncFetcher, 'post_fetch', fake_post_fetch)
search = windvane.SearchWindvane('example.test', 1)
assert await search.process() == SourceExecutionReport('completed', 'result-limit')
assert requests == [('ListSubDomain', 1, 1), ('ListDNS', 1, 1), ('ListEmail', 1, 1)]
pytestmark = pytest.mark.provider_contract('windvane')
tests/discovery/test_yahoosearch.py
+65 -6
View File
@@ -1,8 +1,11 @@
import asyncio
from typing import Any
import pytest
from theHarvester.discovery import yahoosearch
from theHarvester.lib.core import FetcherResponse
from theHarvester.lib.source_execution import SourceExecutionReport
@pytest.mark.asyncio
@@ -41,26 +44,82 @@ async def test_yahoo_uses_exact_pages_and_normalizes_evidence(monkeypatch: pytes
assert await search.get_hostnames() == ['blog.example.com', 'example.com']
@pytest.mark.asyncio
async def test_yahoo_unlimited_stops_when_provider_repeats_a_page(monkeypatch: pytest.MonkeyPatch) -> None:
requests: list[dict[str, Any]] = []
responses = iter(['one.example.com', 'two.example.com', 'two.example.com'])
async def fake_fetch(**kwargs: Any) -> FetcherResponse:
requests.append(kwargs)
return FetcherResponse(next(responses), 200, {})
monkeypatch.setattr(yahoosearch.AsyncFetcher, 'fetch', fake_fetch)
search = yahoosearch.SearchYahoo('example.com', None)
report = await search.process()
assert report == SourceExecutionReport('partial', 'repeated-page')
assert [request['url'] for request in requests] == [
'https://search.yahoo.com/search?p=%40example.com&b=0&pz=10',
'https://search.yahoo.com/search?p=%40example.com&b=10&pz=10',
'https://search.yahoo.com/search?p=%40example.com&b=20&pz=10',
]
assert await search.get_hostnames() == ['one.example.com', 'two.example.com']
@pytest.mark.parametrize(
'response',
['', None, '<html>Access denied at api.example.net</html>'],
ids=['empty', 'malformed', 'blocked'],
('response', 'expected_report'),
[
('', None),
(None, SourceExecutionReport('failed', 'transport-error')),
(FetcherResponse({}, 200, {}), SourceExecutionReport('failed', 'invalid-response')),
('<html>Access denied at api.example.net</html>', SourceExecutionReport('failed', 'access-denied')),
],
ids=['empty', 'missing', 'malformed', 'blocked'],
)
@pytest.mark.asyncio
async def test_yahoo_unusable_responses_return_no_evidence(
monkeypatch: pytest.MonkeyPatch,
response: str | None,
response: object,
expected_report: SourceExecutionReport | None,
) -> None:
async def fake_fetch_all(urls: list[str] | set[str], **_kwargs: Any) -> list[str | None]:
async def fake_fetch_all(urls: list[str] | set[str], **_kwargs: Any) -> list[object]:
return [response] * len(urls)
monkeypatch.setattr(yahoosearch.AsyncFetcher, 'fetch_all', fake_fetch_all)
search = yahoosearch.SearchYahoo('example.com', 20)
await search.process()
report = await search.process()
assert report == expected_report
assert await search.get_emails() == []
assert await search.get_hostnames() == []
@pytest.mark.asyncio
async def test_yahoo_later_http_failure_is_partial(monkeypatch: pytest.MonkeyPatch) -> None:
async def fake_fetch_all(*_args: Any, **_kwargs: Any) -> list[FetcherResponse]:
return [
FetcherResponse('one.example.com', 200, {}),
FetcherResponse('unavailable', 503, {}),
]
monkeypatch.setattr(yahoosearch.AsyncFetcher, 'fetch_all', fake_fetch_all)
report = await yahoosearch.SearchYahoo('example.com', 20).process()
assert report == SourceExecutionReport('partial', 'http-503')
@pytest.mark.asyncio
async def test_yahoo_cancellation_propagates(monkeypatch: pytest.MonkeyPatch) -> None:
async def cancel(**_kwargs: Any) -> FetcherResponse:
raise asyncio.CancelledError('operator-stop')
monkeypatch.setattr(yahoosearch.AsyncFetcher, 'fetch', cancel)
with pytest.raises(asyncio.CancelledError, match='operator-stop'):
await yahoosearch.SearchYahoo('example.com', None).process()
pytestmark = pytest.mark.provider_contract('yahoo')
tests/discovery/test_zoomeyesearch.py
+27
View File
@@ -157,6 +157,33 @@ async def test_empty_pages_do_not_hide_later_provider_results(monkeypatch: pytes
assert report is None
@pytest.mark.asyncio
async def test_unlimited_search_uses_provider_total(monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.setattr(zoomeyesearch.Core, 'zoomeye_key', staticmethod(lambda: 'test-key'))
calls: list[dict[str, Any]] = []
responses = [
FetcherResponse({'code': 60000, 'total': 2, 'data': [{'hostname': 'one.example.com'}]}, 200, {}),
FetcherResponse({'code': 60000, 'total': 2, 'data': [{'hostname': 'two.example.com'}]}, 200, {}),
]
@contextlib.asynccontextmanager
async def fake_open_session(**_kwargs: Any) -> AsyncIterator[object]:
yield object()
async def fake_post_fetch(*_args: Any, **kwargs: Any) -> FetcherResponse:
calls.append(kwargs['json_body'])
return responses.pop(0)
monkeypatch.setattr(zoomeyesearch.AsyncFetcher, 'open_session', fake_open_session)
monkeypatch.setattr(zoomeyesearch.AsyncFetcher, 'post_fetch', fake_post_fetch)
monkeypatch.setattr(zoomeyesearch.SearchZoomEye, 'PAGE_SIZE', 1)
search = zoomeyesearch.SearchZoomEye('example.com', None)
assert await search.process() is None
assert [call['page'] for call in calls] == [1, 2]
assert await search.get_hostnames() == {'one.example.com', 'two.example.com'}
@pytest.mark.asyncio
async def test_numbered_pages_keep_a_stable_size_and_slice_the_final_page(monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.setattr(zoomeyesearch.Core, 'zoomeye_key', staticmethod(lambda: 'test-key'))
tests/lib/test_api_v1.py
+105
View File
@@ -509,6 +509,16 @@ def test_dns_limits_default_to_unlimited_and_keep_explicit_values() -> None:
assert (explicit.dns_recursive_query_limit, explicit.dns_recursive_runtime_seconds) == (12, 1.5)
def test_result_limit_accepts_unlimited_and_has_no_numeric_ceiling() -> None:
from theHarvester.lib.api.run_models import RunRequest
unlimited = RunRequest(target='example.test', sources=['crtsh'], limit=0)
large = RunRequest(target='example.test', sources=['crtsh'], limit=1_000_000)
assert unlimited.limit == 0
assert large.limit == 1_000_000
def test_run_detail_exposes_one_normalized_evidence_surface(tmp_path, monkeypatch) -> None:
from theHarvester.lib.api import api
@@ -966,6 +976,101 @@ def test_api_jsonl_round_trip_preserves_source_attribution(tmp_path, monkeypatch
assert reimported.json()['source_executions'] == [source_execution]
def test_run_detail_reports_unique_hostname_contribution_per_source(tmp_path, monkeypatch) -> None:
from theHarvester.lib.api import api
summary = {
'type': 'summary',
'run_id': '31ced47a-6354-4f18-b633-2583400c4aad',
'target': 'example.test',
'started_at': '2026-08-23T12:00:00Z',
'completed_at': '2026-08-23T12:01:00Z',
'evidence_status': 'partial',
'result_count': 2,
'counts': {'hostname': 2},
'source_executions': [
{
'source': 'crtsh',
'status': 'completed',
'duration_ms': 1,
'result_count': 1,
'error_type': None,
'stop_reason': None,
},
{
'source': 'subdomainapi',
'status': 'partial',
'duration_ms': 2,
'result_count': 2,
'error_type': None,
'stop_reason': 'provider-limit',
},
],
'action_executions': [
{
'action': 'dns-resolve',
'status': 'completed',
'duration_ms': 3,
'result_count': 1,
'error_type': None,
'stop_reason': None,
}
],
}
payload = '\n'.join(
(
json.dumps(summary),
json.dumps(
{
'type': 'hostname',
'value': 'shared.example.test',
'sources': ['crtsh', 'subdomainapi'],
}
),
json.dumps(
{
'type': 'hostname',
'value': 'unique.example.test',
'sources': ['subdomainapi'],
'actions': ['dns-resolve'],
}
),
'',
)
)
monkeypatch.setenv('THEHARVESTER_API_KEY', 'test-key')
monkeypatch.setenv('THEHARVESTER_RUN_DB', str(tmp_path / 'runs.sqlite'))
monkeypatch.setenv('THEHARVESTER_RUN_WORKER', 'disabled')
with TestClient(api.app) as client:
response = client.post(
'/api/v1/runs/import',
params={'filename': 'source-yields.jsonl'},
headers={'X-API-Key': 'test-key'},
content=payload,
)
assert response.status_code == 201
assert response.json()['source_yields'] == [
{
'source': 'crtsh',
'observed_result_count': 1,
'unique_result_count': 0,
'shared_result_count': 1,
'resolved_hostname_count': 0,
'unique_resolved_hostname_count': 0,
},
{
'source': 'subdomainapi',
'observed_result_count': 2,
'unique_result_count': 1,
'shared_result_count': 1,
'resolved_hostname_count': 1,
'unique_resolved_hostname_count': 1,
},
]
def test_api_jsonl_export_uses_evidence_timestamps_not_lifecycle_timestamps(tmp_path, monkeypatch) -> None:
from theHarvester.lib.api import api
from theHarvester.lib.api.run_models import RunRequest
tests/lib/test_completed_persistence.py
+60
View File
@@ -1165,6 +1165,16 @@ async def test_source_yields_distinguish_unique_and_shared_results(tmp_path) ->
SourceExecution('certspotter', 'completed', 8.0, 2),
SourceExecution('empty-source', 'completed', 5.0, 0),
),
active_evidence=ActiveEvidence(
(
ActionExecution.finish(
action='dns-resolve',
status='completed',
duration_ms=4,
groups={'hostname': ['api.example.com', 'mail.example.com']},
),
)
),
)
await store.save_run(result)
@@ -1176,18 +1186,68 @@ async def test_source_yields_distinguish_unique_and_shared_results(tmp_path) ->
'observed_result_count': 2,
'unique_result_count': 1,
'shared_result_count': 1,
'resolved_hostname_count': 1,
'unique_resolved_hostname_count': 0,
},
{
'source': 'crtsh',
'observed_result_count': 2,
'unique_result_count': 1,
'shared_result_count': 1,
'resolved_hostname_count': 2,
'unique_resolved_hostname_count': 1,
},
{
'source': 'empty-source',
'observed_result_count': 0,
'unique_result_count': 0,
'shared_result_count': 0,
'resolved_hostname_count': 0,
'unique_resolved_hostname_count': 0,
},
]
@pytest.mark.asyncio
async def test_source_yields_can_measure_hostname_contributions_only(tmp_path) -> None:
store = ResultStore(tmp_path / 'stash.sqlite')
await store.initialize()
result = CompletedResult.finish(
run_id=UUID('86eafad7-3308-4786-8e85-1e15b24afafc'),
target='example.com',
started_at=datetime(2026, 8, 23, 12, 0, tzinfo=UTC),
completed_at=datetime(2026, 8, 23, 12, 1, tzinfo=UTC),
groups={'hostname': ['shared.example.com'], 'url': ['https://unique.example.com/']},
observations=(
ResultObservation('first', 'hostname', 'shared.example.com'),
ResultObservation('second', 'hostname', 'shared.example.com'),
ResultObservation('first', 'url', 'https://unique.example.com/'),
),
source_executions=(
SourceExecution('first', 'completed', 10, 2),
SourceExecution('second', 'completed', 10, 1),
),
)
await store.save_run(result)
yields = await store.source_yields(result.run_id, kind='hostname')
assert [item.to_dict() for item in yields] == [
{
'source': 'first',
'observed_result_count': 1,
'unique_result_count': 0,
'shared_result_count': 1,
'resolved_hostname_count': 0,
'unique_resolved_hostname_count': 0,
},
{
'source': 'second',
'observed_result_count': 1,
'unique_result_count': 0,
'shared_result_count': 1,
'resolved_hostname_count': 0,
'unique_resolved_hostname_count': 0,
},
]
tests/lib/test_enumeration.py
+11
View File
@@ -45,6 +45,17 @@ def test_enumeration_options_preserve_explicit_transport_values() -> None:
assert options.source_workers == 7
def test_zero_result_limit_means_unlimited() -> None:
options = EnumerationOptions(domain='example.com', source='crtsh', limit=0)
assert options.limit is None
def test_negative_result_limit_is_rejected() -> None:
with pytest.raises(ValueError, match='result limit cannot be negative'):
EnumerationOptions(domain='example.com', source='crtsh', limit=-1)
def test_routeviews_is_an_explicit_passive_action_independent_of_source_limits() -> None:
options = EnumerationOptions.from_namespace(Namespace(domain='example.com', source=None, limit=25, routeviews=True))
tests/lib/test_harvestview_ui.py
+6
View File
@@ -24,6 +24,9 @@ def test_harvestview_owns_root_and_issues_an_http_only_session(tmp_path, monkeyp
assert f'value="{",".join(DEFAULT_DNS_RESOLVERS)}"' in root.text
assert 'Resolve with the configured resolver addresses.' in root.text
assert 'id="source-workers" name="source_workers" type="number" min="1"' in root.text
assert 'id="run-limit" name="limit" type="number" min="0" value="500"' in root.text
assert '0 means unlimited; positive values apply per selected source.' in root.text
assert 'id="run-limit" name="limit" type="number" min="0" max="10000"' not in root.text
assert legacy.status_code == 404
cookie = root.headers['set-cookie']
assert 'theharvester-api-key=' in cookie
@@ -55,6 +58,7 @@ def test_harvestview_assets_load_outside_the_repository_directory(tmp_path, monk
assert "request.deadline_seconds === null ? 'Unlimited'" in response.text
assert "deadline_seconds: form.get('deadline_seconds') ? Number(form.get('deadline_seconds')) : null" in response.text
assert "source_workers: Number(form.get('source_workers'))" in response.text
assert "request.limit === 0 ? 'Unlimited'" in response.text
def test_harvestview_exposes_local_schedule_page_and_assets(tmp_path, monkeypatch) -> None:
@@ -77,6 +81,8 @@ def test_harvestview_exposes_local_schedule_page_and_assets(tmp_path, monkeypatc
assert 'href="/schedules">Schedules</a>' in root.text
assert schedules.status_code == 200
assert '<h1 id="builder-title">Create a schedule</h1>' in schedules.text
assert 'id="run-limit" type="number" min="0" value="500"' in schedules.text
assert 'id="run-limit" type="number" min="0" max="10000"' not in schedules.text
assert '/static/harvestview/schedules.css?v=' in schedules.text
assert '/static/harvestview/schedules.js?v=' in schedules.text
assert stylesheet.status_code == 200
tests/lib/test_run_backend.py
+28
View File
@@ -990,6 +990,34 @@ def test_routeviews_child_receives_explicit_action_without_source_limit_controls
assert received_options[0].limit == 9_999
def test_persisted_unlimited_result_limit_becomes_no_worker_cap(tmp_path, monkeypatch) -> None:
from theHarvester import __main__ as main_module
from theHarvester.lib.api import run_worker
from theHarvester.lib.api.run_models import RunRequest
from theHarvester.lib.api.run_store import RunStore
from theHarvester.lib.completed_result import CompletedResult
received_options = []
async def fake_start(options, **_kwargs):
received_options.append(options)
now = datetime.now(UTC)
return (CompletedResult.finish(target=options.domain, started_at=now, completed_at=now, groups={}),)
monkeypatch.setattr(main_module, 'start', fake_start)
async def scenario() -> None:
store = RunStore(tmp_path / 'runs.sqlite')
created = await store.create(RunRequest(target='example.test', sources=['crtsh'], limit=0))
assert created['request']['limit'] == 0
assert await store.claim_next() is not None
await run_worker._child_execute(created['run_id'], store.database)
asyncio.run(scenario())
assert received_options[0].limit is None
@pytest.mark.parametrize(
('field', 'value', 'option'),
[
tests/lib/test_source_runner.py
+30 -4
View File
@@ -99,7 +99,7 @@ def test_source_factories_match_the_catalog() -> None:
('fofa', 'theHarvester.lib.source_runner.fofa.SearchFofa', ('example.test', 25), {}),
('fullhunt', 'theHarvester.lib.source_runner.fullhuntsearch.SearchFullHunt', ('example.test',), {}),
('github-code', 'theHarvester.lib.source_runner.githubcode.SearchGithubCode', ('example.test', 25), {}),
('gitlab', 'theHarvester.lib.source_runner.gitlabsearch.SearchGitlab', ('example.test',), {}),
('gitlab', 'theHarvester.lib.source_runner.gitlabsearch.SearchGitlab', ('example.test', 25), {}),
(
'hackertarget',
'theHarvester.lib.source_runner.hackertarget.SearchHackerTarget',
@@ -121,7 +121,7 @@ def test_source_factories_match_the_catalog() -> None:
('hudsonrock', 'theHarvester.lib.source_runner.hudsonrocksearch.SearchHudsonRock', ('example.test',), {}),
('hunter', 'theHarvester.lib.source_runner.huntersearch.SearchHunter', ('example.test', 25, 5), {}),
('hunterhow', 'theHarvester.lib.source_runner.searchhunterhow.SearchHunterHow', ('example.test', 25), {}),
('intelx', 'theHarvester.lib.source_runner.intelxsearch.SearchIntelx', ('example.test',), {}),
('intelx', 'theHarvester.lib.source_runner.intelxsearch.SearchIntelx', ('example.test', 25), {}),
('leakix', 'theHarvester.lib.source_runner.leakix.SearchLeakix', ('example.test',), {}),
('leaklookup', 'theHarvester.lib.source_runner.leaklookup.SearchLeakLookup', ('example.test',), {}),
('mojeek', 'theHarvester.lib.source_runner.mojeek.SearchMojeek', ('example.test', 25), {}),
@@ -182,7 +182,7 @@ def test_source_factories_match_the_catalog() -> None:
('example.test',),
{},
),
('thc', 'theHarvester.lib.source_runner.thc.SearchThc', ('example.test',), {}),
('thc', 'theHarvester.lib.source_runner.thc.SearchThc', ('example.test', 25), {}),
('tomba', 'theHarvester.lib.source_runner.tombasearch.SearchTomba', ('example.test', 25, 5), {}),
('urlscan', 'theHarvester.lib.source_runner.urlscan.SearchUrlscan', ('example.test', 25), {}),
('virustotal', 'theHarvester.lib.source_runner.virustotal.SearchVirustotal', ('example.test', 25), {}),
@@ -193,7 +193,7 @@ def test_source_factories_match_the_catalog() -> None:
{},
),
('whoisxml', 'theHarvester.lib.source_runner.whoisxml.SearchWhoisXML', ('example.test', 25), {}),
('windvane', 'theHarvester.lib.source_runner.windvane.SearchWindvane', ('example.test',), {}),
('windvane', 'theHarvester.lib.source_runner.windvane.SearchWindvane', ('example.test', 25), {}),
('yahoo', 'theHarvester.lib.source_runner.yahoosearch.SearchYahoo', ('example.test', 25), {}),
('zoomeye', 'theHarvester.lib.source_runner.zoomeyesearch.SearchZoomEye', ('example.test', 25), {}),
],
@@ -218,6 +218,31 @@ def test_factory_constructor_shapes(
assert calls == [(expected_args, expected_kwargs)]
@pytest.mark.parametrize(
('source', 'patch_target'),
[
('gitlab', 'theHarvester.lib.source_runner.gitlabsearch.SearchGitlab'),
('windvane', 'theHarvester.lib.source_runner.windvane.SearchWindvane'),
],
)
def test_unlimited_limit_reaches_gitlab_and_windvane_factories(
monkeypatch: pytest.MonkeyPatch,
source: str,
patch_target: str,
) -> None:
calls: list[tuple[object, ...]] = []
def constructor(*args: object) -> object:
calls.append(args)
return object()
monkeypatch.setattr(patch_target, constructor)
create_source(SourceRequest(source, 'example.test', None, 0, False, True))
assert calls == [('example.test', None)]
@pytest.mark.asyncio
async def test_runner_normalizes_only_declared_apis_guru_routes(monkeypatch: pytest.MonkeyPatch) -> None:
class FakeApisGuru:
@@ -433,6 +458,7 @@ async def test_runner_reports_normal_zero_yield_as_completed_no_results(monkeypa
(SourceExecutionReport('completed', 'result-limit'), False, 'completed', 'result-limit'),
(SourceExecutionReport('failed', 'provider-failure'), True, 'partial', 'provider-failure'),
(SourceExecutionReport('failed', 'provider-failure'), False, 'failed', 'provider-failure'),
(SourceExecutionReport('rate-limited', 'http-429'), True, 'partial', 'http-429'),
(SourceExecutionReport('rate-limited', 'http-429'), False, 'rate-limited', 'http-429'),
],
)
tests/test_harvestview_launcher.py
+2 -1
View File
@@ -7,12 +7,13 @@ import pytest
from theHarvester import harvestview
def test_project_scripts_expose_harvestview() -> None:
def test_project_scripts_expose_commands() -> None:
scripts = tomllib.loads(Path('pyproject.toml').read_text(encoding='utf-8'))['project']['scripts']
assert scripts == {
'theHarvester': 'theHarvester.theHarvester:main',
'harvestview': 'theHarvester.harvestview:main',
'harvest-yields': 'theHarvester.source_yields:main',
}
tests/test_main.py
+31
View File
@@ -23,6 +23,7 @@ from theHarvester.lib.hostchecker import HostDnsRecords
from theHarvester.lib.network_evidence import PrefixOriginObservation, RpkiValidationObservation
from theHarvester.lib.recursive_dns import RecursiveDNSClassification, RecursiveDNSFinding, RecursiveDNSResult
from theHarvester.lib.routeviews import RouteViewsCancelled, RouteViewsResult
from theHarvester.lib.source_catalog import resolve_sources
from theHarvester.lib.source_execution import SourceExecutionReport
from theHarvester.lib.takeover_evidence import TakeoverCandidateOutcome
from theHarvester.lib.virtual_host import (
@@ -67,6 +68,7 @@ async def test_cli_help_explains_proxy_and_direct_action_scope(
assert 'Candidate names are never resolved through DNS.' in help_text
assert '-j SOURCE_WORKERS' in help_text
assert '--source-workers SOURCE_WORKERS' in help_text
assert '0 continues to provider exhaustion with no local result or page-count cap' in help_text
assert 'Indicators are not confirmed takeovers.' in help_text
@@ -676,6 +678,9 @@ async def test_rapiddns_hostnames_honor_explicit_dns_resolution(monkeypatch: pyt
assert dns_execution.error_type == 'TimeoutError'
assert dns_execution.stop_reason == 'query-errors'
assert {(observation.kind, observation.value) for observation in dns_execution.observations} == {
('hostname', 'api.example.com'),
('hostname', 'crt.example.com'),
('hostname', 'reported.example.com'),
('ip', '192.0.2.10'),
('ip', '192.0.2.21'),
('ip', '192.0.2.30'),
@@ -1492,6 +1497,32 @@ async def test_source_progress_waits_for_runner_admission(
assert '[*] Searching Dymo.' not in interim_output
@pytest.mark.asyncio
async def test_unlimited_subdomain_selection_passes_no_result_cap_to_every_source(
monkeypatch: pytest.MonkeyPatch,
) -> None:
captured: tuple[source_runner.SourceJob, ...] = ()
async def capture_jobs(
jobs: tuple[source_runner.SourceJob, ...],
**_kwargs: object,
) -> tuple[source_runner.SourceOutcome, ...]:
nonlocal captured
captured = jobs
return ()
monkeypatch.setattr(theharvester_main, 'run_source_jobs', capture_jobs)
monkeypatch.setattr(theharvester_main, 'ResultStore', _NoopResultStore)
await theharvester_main.start(
EnumerationOptions(domain='example.test', source='subdomains', limit=0, quiet=True),
return_completed_result=True,
)
assert [job.request.source for job in captured] == resolve_sources('subdomains')
assert all(job.request.limit is None for job in captured)
@pytest.mark.asyncio
async def test_source_completion_reports_verbose_terminal_summary(
monkeypatch: pytest.MonkeyPatch,
tests/test_mojeek.py
+115
View File
@@ -1,3 +1,4 @@
import asyncio
from typing import Any
import pytest
@@ -7,6 +8,120 @@ from theHarvester.lib.core import FetcherResponse
class TestMojeekSearch:
@pytest.mark.asyncio
async def test_unlimited_keyless_stops_when_provider_repeats_a_page(self, monkeypatch: pytest.MonkeyPatch) -> None:
calls: list[dict[str, Any]] = []
responses = iter(
[
FetcherResponse(body='<ul class="results-standard">one.example.com</ul>', status=200, headers={}),
FetcherResponse(body='<ul class="results-standard">two.example.com</ul>', status=200, headers={}),
FetcherResponse(body='<ul class="results-standard">two.example.com</ul>', status=200, headers={}),
]
)
async def fake_fetch(**kwargs: Any) -> FetcherResponse:
calls.append(kwargs)
return next(responses)
async def fake_sleep(_delay: float) -> None:
return None
monkeypatch.setattr(mojeek.Core, 'mojeek_key', staticmethod(lambda: ''))
monkeypatch.setattr(mojeek.AsyncFetcher, 'fetch', fake_fetch)
monkeypatch.setattr(mojeek.asyncio, 'sleep', fake_sleep)
search = mojeek.SearchMojeek(word='example.com', limit=None)
report = await search.process()
assert report == mojeek.SourceExecutionReport('partial', 'repeated-page')
assert [call['url'] for call in calls] == [
'https://www.mojeek.com/search?q=example.com&s=0',
'https://www.mojeek.com/search?q=example.com&s=10',
'https://www.mojeek.com/search?q=example.com&s=20',
]
assert await search.get_hostnames() == ['one.example.com', 'two.example.com']
@pytest.mark.asyncio
async def test_unlimited_keyed_api_stops_when_provider_exhausts_results(self, monkeypatch: pytest.MonkeyPatch) -> None:
requests: list[list[str]] = []
responses = iter(
[
FetcherResponse(body={'response': {'results': [{'url': 'https://one.example.com'}]}}, status=200, headers={}),
FetcherResponse(body={'response': {'results': [{'url': 'https://two.example.com'}]}}, status=200, headers={}),
FetcherResponse(body={'response': {'results': []}}, status=200, headers={}),
]
)
async def fake_fetch_all(urls: list[str], **_kwargs: Any) -> list[FetcherResponse]:
requests.append(urls)
return [next(responses)]
monkeypatch.setattr(mojeek.Core, 'mojeek_key', staticmethod(lambda: 'test-key'))
monkeypatch.setattr(mojeek.AsyncFetcher, 'fetch_all', fake_fetch_all)
search = mojeek.SearchMojeek(word='example.com', limit=None)
report = await search.process()
assert report is None
assert requests == [
['https://api.mojeek.com/search?api_key=test-key&q=example.com&fmt=json&s=1'],
['https://api.mojeek.com/search?api_key=test-key&q=example.com&fmt=json&s=11'],
['https://api.mojeek.com/search?api_key=test-key&q=example.com&fmt=json&s=21'],
]
assert await search.get_hostnames() == ['one.example.com', 'two.example.com']
@pytest.mark.asyncio
async def test_unlimited_keyed_api_reports_repeated_page(self, monkeypatch: pytest.MonkeyPatch) -> None:
page = FetcherResponse(
body={'response': {'results': [{'url': 'https://one.example.com'}]}},
status=200,
headers={},
)
async def fake_fetch_all(_urls: list[str], **_kwargs: Any) -> list[FetcherResponse]:
return [page]
monkeypatch.setattr(mojeek.Core, 'mojeek_key', staticmethod(lambda: 'test-key'))
monkeypatch.setattr(mojeek.AsyncFetcher, 'fetch_all', fake_fetch_all)
report = await mojeek.SearchMojeek(word='example.com', limit=None).process()
assert report == mojeek.SourceExecutionReport('partial', 'repeated-page')
@pytest.mark.asyncio
async def test_keyless_later_http_failure_is_partial(self, monkeypatch: pytest.MonkeyPatch) -> None:
responses = iter(
[
FetcherResponse(body='<ul class="results-standard">one.example.com</ul>', status=200, headers={}),
FetcherResponse(body='unavailable', status=503, headers={}),
]
)
async def fake_fetch(**_kwargs: Any) -> FetcherResponse:
return next(responses)
async def fake_sleep(_delay: float) -> None:
return None
monkeypatch.setattr(mojeek.Core, 'mojeek_key', staticmethod(lambda: ''))
monkeypatch.setattr(mojeek.AsyncFetcher, 'fetch', fake_fetch)
monkeypatch.setattr(mojeek.asyncio, 'sleep', fake_sleep)
report = await mojeek.SearchMojeek(word='example.com', limit=None).process()
assert report == mojeek.SourceExecutionReport('partial', 'http-503')
@pytest.mark.asyncio
async def test_mojeek_cancellation_propagates(self, monkeypatch: pytest.MonkeyPatch) -> None:
async def cancel(**_kwargs: Any) -> FetcherResponse:
raise asyncio.CancelledError('operator-stop')
monkeypatch.setattr(mojeek.Core, 'mojeek_key', staticmethod(lambda: ''))
monkeypatch.setattr(mojeek.AsyncFetcher, 'fetch', cancel)
with pytest.raises(asyncio.CancelledError, match='operator-stop'):
await mojeek.SearchMojeek(word='example.com', limit=None).process()
@pytest.mark.asyncio
async def test_keyless_pages_are_sequential_and_stop_after_first_empty_page(
self,
tests/test_readme.py
+5 -4
View File
@@ -66,6 +66,7 @@ SOURCE_PROVIDER_LINKS = {
'shodanInternetDB': 'https://internetdb.shodan.io/',
'shodanct': 'https://ctl.shodan.io/',
'sourcegraph': 'https://sourcegraph.com/search',
'subdomainapi': 'https://api.subdomain.app/',
'subdomaincenter': 'https://www.subdomain.center/',
'subdomainfinderc99': 'https://subdomainfinder.c99.nl/',
'thc': 'https://ip.thc.org/',
@@ -102,7 +103,7 @@ def _declared_source_contracts() -> dict[str, set[str]]:
def _source_matrix(readme: str) -> str:
return readme.split('<summary><strong>View all 58 discovery sources</strong></summary>', 1)[1].split('</details>', 1)[0]
return readme.split('<summary><strong>View all 59 discovery sources</strong></summary>', 1)[1].split('</details>', 1)[0]
def _documented_source_rows(readme: str) -> dict[str, list[str]]:
@@ -151,8 +152,8 @@ def test_readme_matches_declared_source_contracts() -> None:
assert _source_matrix(readme).count('| Source | Returns | Activity | API key |') == 1
assert 'Credentials |' not in _source_matrix(readme)
assert len(declared) == 58
assert len(documented) == 58
assert len(declared) == 59
assert len(documented) == 59
assert documented == declared
source_links = _documented_source_links(readme)
assert len(source_links) == len(declared)
@@ -224,7 +225,7 @@ def test_readme_architecture_diagrams_are_local_and_accessible() -> None:
'theHarvester discovery routes and enrichment',
Path('docs/images/run-evidence-architecture.svg'),
'run-evidence-architecture',
('58 discovery adapters', 'CompletedResult evidence contract', 'Terminal · JSONL · SQLite · REST'),
('59 discovery adapters', 'CompletedResult evidence contract', 'Terminal · JSONL · SQLite · REST'),
),
(
'HarvestView run desk architecture',
tests/test_source_yields_cli.py
+230
View File
@@ -0,0 +1,230 @@
import asyncio
import json
import tomllib
from datetime import UTC, datetime, timedelta
from pathlib import Path
from uuid import UUID
import pytest
from theHarvester import source_yields
from theHarvester.lib import database as database_module
from theHarvester.lib.active_evidence import ActionExecution, ActiveEvidence
from theHarvester.lib.completed_result import CompletedResult, ResultObservation, SourceExecution
from theHarvester.lib.database import ResultStore
from theHarvester.lib.evidence_types import RESULT_KINDS
RUN_ONE = UUID('11111111-1111-4111-8111-111111111111')
RUN_TWO = UUID('22222222-2222-4222-8222-222222222222')
def test_project_installs_harvest_yields_command() -> None:
project = tomllib.loads(Path('pyproject.toml').read_text(encoding='utf-8'))
assert project['project']['scripts']['harvest-yields'] == 'theHarvester.source_yields:main'
def _completed_run(
run_id: UUID,
*,
observations: tuple[ResultObservation, ...],
resolved_hostnames: tuple[str, ...] = (),
) -> CompletedResult:
started_at = datetime(2026, 8, 23, 12, tzinfo=UTC) + timedelta(minutes=int(str(run_id)[0]))
sources = sorted({observation.source for observation in observations})
active_evidence = (
ActiveEvidence(
executions=(
ActionExecution.finish(
action='dns-resolve',
status='completed',
duration_ms=1,
groups={'hostname': resolved_hostnames},
),
)
)
if resolved_hostnames
else ActiveEvidence()
)
return CompletedResult.finish(
run_id=run_id,
target='example.test',
started_at=started_at,
completed_at=started_at + timedelta(seconds=1),
groups={
observation.kind: [item.value for item in observations if item.kind == observation.kind]
for observation in observations
},
source_executions=tuple(
SourceExecution(
source=source,
status='completed',
duration_ms=1,
result_count=sum(observation.source == source for observation in observations),
)
for source in sources
),
observations=observations,
active_evidence=active_evidence,
)
async def _create_database(database: Path) -> None:
store = ResultStore(database)
await store.initialize()
await store.save_run(
_completed_run(
RUN_ONE,
observations=(
ResultObservation('alpha', 'hostname', 'shared.example.test'),
ResultObservation('alpha', 'hostname', 'unique-alpha.example.test'),
ResultObservation('beta', 'hostname', 'shared.example.test'),
ResultObservation('beta', 'hostname', 'unique-beta.example.test'),
),
resolved_hostnames=('shared.example.test', 'unique-alpha.example.test'),
)
)
await store.save_run(
_completed_run(
RUN_TWO,
observations=(
ResultObservation('alpha', 'hostname', 'second-shared.example.test'),
ResultObservation('beta', 'hostname', 'second-shared.example.test'),
ResultObservation('gamma', 'hostname', 'second-gamma.example.test'),
ResultObservation('alpha', 'ip', '192.0.2.1'),
ResultObservation('beta', 'ip', '192.0.2.1'),
ResultObservation('gamma', 'ip', '198.51.100.2'),
ResultObservation('alpha', 'asn', 'AS64496'),
),
resolved_hostnames=('second-shared.example.test',),
)
)
await store.dispose()
def test_missing_database_fails_without_creating_file(
tmp_path: Path,
capsys: pytest.CaptureFixture[str],
) -> None:
database = tmp_path / 'missing.sqlite'
with pytest.raises(SystemExit) as error:
source_yields.main(['--database', str(database)])
assert error.value.code == 2
assert 'database does not exist' in capsys.readouterr().err
assert not database.exists()
def test_default_database_uses_the_standard_result_store(
tmp_path: Path,
monkeypatch: pytest.MonkeyPatch,
capsys: pytest.CaptureFixture[str],
) -> None:
database = tmp_path / 'stash.sqlite'
asyncio.run(_create_database(database))
monkeypatch.setattr(database_module, '_DEFAULT_DATABASE', database)
assert source_yields.main([]) == 0
assert capsys.readouterr().out.startswith('Kind: hostname\nRun count: 2\n')
def test_default_table_ranks_by_unique_per_run_and_aligns_columns(
tmp_path: Path,
capsys: pytest.CaptureFixture[str],
) -> None:
database = tmp_path / 'runs.sqlite'
asyncio.run(_create_database(database))
assert source_yields.main(['--database', str(database)]) == 0
assert capsys.readouterr().out.splitlines() == [
'Kind: hostname',
'Run count: 2',
'SOURCE RUNS OBSERVED UNIQUE UNIQUE/RUN SHARED RESOLVED UNIQUE-RESOLVED UNIQUE-RESOLVED/RUN',
'gamma 1 1 1 1.00 0 0 0 0.00',
'alpha 2 3 1 0.50 2 3 1 0.50',
'beta 2 3 1 0.50 2 2 0 0.00',
]
@pytest.mark.parametrize('kind', sorted(RESULT_KINDS))
def test_kind_accepts_every_result_kind_and_only_hostname_shows_resolution_columns(
tmp_path: Path,
capsys: pytest.CaptureFixture[str],
kind: str,
) -> None:
database = tmp_path / 'runs.sqlite'
asyncio.run(_create_database(database))
assert source_yields.main(['--database', str(database), '--kind', kind]) == 0
lines = capsys.readouterr().out.splitlines()
assert lines[0] == f'Kind: {kind}'
assert ('RESOLVED' in lines[2]) is (kind == 'hostname')
if kind == 'ip':
assert [line.split() for line in lines[3:]] == [
['gamma', '1', '1', '1', '1.00', '0'],
['alpha', '2', '1', '0', '0.00', '1'],
['beta', '2', '1', '0', '0.00', '1'],
]
def test_run_id_selects_one_run(
tmp_path: Path,
capsys: pytest.CaptureFixture[str],
) -> None:
database = tmp_path / 'runs.sqlite'
asyncio.run(_create_database(database))
assert source_yields.main(['--database', str(database), '--run-id', str(RUN_ONE)]) == 0
lines = capsys.readouterr().out.splitlines()
assert lines[1] == 'Run count: 1'
assert [line.split() for line in lines[3:]] == [
['alpha', '1', '2', '1', '1.00', '1', '2', '1', '1.00'],
['beta', '1', '2', '1', '1.00', '1', '1', '0', '0.00'],
]
def test_unknown_run_id_fails_instead_of_reporting_an_empty_run(
tmp_path: Path,
capsys: pytest.CaptureFixture[str],
) -> None:
database = tmp_path / 'runs.sqlite'
asyncio.run(_create_database(database))
missing_run = UUID('33333333-3333-4333-8333-333333333333')
with pytest.raises(SystemExit) as error:
source_yields.main(['--database', str(database), '--run-id', str(missing_run)])
assert error.value.code == 2
assert 'completed result not found' in capsys.readouterr().err
@pytest.mark.parametrize('kind', ['hostname', 'ip'])
def test_json_format_is_machine_readable_and_uses_kind_specific_fields(
tmp_path: Path,
capsys: pytest.CaptureFixture[str],
kind: str,
) -> None:
database = tmp_path / 'runs.sqlite'
asyncio.run(_create_database(database))
assert source_yields.main(['--database', str(database), '--kind', kind, '--format', 'json']) == 0
payload = json.loads(capsys.readouterr().out)
assert payload['kind'] == kind
assert payload['run_count'] == 2
assert [row['source'] for row in payload['source_yields']] == ['gamma', 'alpha', 'beta']
resolution_fields = {
'resolved_hostname_count',
'unique_resolved_hostname_count',
'unique_resolved_hostname_count_per_run',
}
assert all(resolution_fields <= row.keys() for row in payload['source_yields']) is (kind == 'hostname')
assert {row['source']: row['run_count'] for row in payload['source_yields']} == {'alpha': 2, 'beta': 2, 'gamma': 1}
assert {row['source']: row['unique_result_count_per_run'] for row in payload['source_yields']} == (
{'alpha': 0.5, 'beta': 0.5, 'gamma': 1.0} if kind == 'hostname' else {'alpha': 0.0, 'beta': 0.0, 'gamma': 1.0}
)
theHarvester/__main__.py
+12 -9
View File
@@ -159,7 +159,10 @@ async def start(
parser.add_argument(
'-l',
'--limit',
help='Maximum results requested from each source that supports result limits (default: 500).',
help=(
'Maximum results requested from each source that supports result limits; 0 continues to provider '
'exhaustion with no local result or page-count cap (default: 500).'
),
default=DEFAULT_RESULT_LIMIT,
type=int,
)
@@ -504,11 +507,11 @@ async def start(
engines: list = []
# If the user specifies
full: list = []
resolved_screenshot_hosts: set[str] = set()
resolved_hostnames: set[str] = set()
reported_host_ip_pairs: set[tuple[str, str]] = set()
ips: list = []
host_ip: list = []
limit: int = args.limit
limit: int | None = args.limit
routeviews_enabled = args.routeviews
shodan = args.shodan
start: int = args.start
@@ -747,7 +750,7 @@ async def start(
action='dns-resolve',
status=status,
duration_ms=dns_resolution_duration_ms,
groups={'ip': dns_resolution_ips},
groups={'hostname': resolved_hostnames, 'ip': dns_resolution_ips},
error_type=error_type,
stop_reason=stop_reason,
)
@@ -837,7 +840,7 @@ async def start(
dns_resolution_ips.update(_normalize_ip_addresses(temp_ips))
all_ip.extend(temp_ips)
full.extend(resolved_pair)
resolved_screenshot_hosts.update(resolved_hosts)
resolved_hostnames.update(resolved_hosts)
try:
retain_results(await full_hosts_checker.check())
@@ -1026,7 +1029,7 @@ async def start(
recursive_ips = [address for finding in recursive_result.findings for address in finding.records.addresses]
all_hosts.extend(recursive_hosts)
all_ip.extend(recursive_ips)
resolved_screenshot_hosts.update(recursive_hosts)
resolved_hostnames.update(recursive_hosts)
for finding in recursive_result.findings:
if finding.records.addresses:
full.extend(f'{finding.hostname}:{address}' for address in finding.records.addresses)
@@ -1253,7 +1256,7 @@ async def start(
await checkpoint_completed_result()
await persist_result(finish_completed_result())
raise
resolved_screenshot_hosts.update(hosts)
resolved_hostnames.update(hosts)
normalized_brute_hosts = _normalize_hosts_for_storage(hosts, word)
normalized_brute_ips = _normalize_ip_addresses(ips)
temp = set()
@@ -1676,9 +1679,9 @@ async def start(
output_logger.info(f'\nScreenshots can be found in: {screen_shotter.output}{screen_shotter.slash}')
output_logger.info('Filtering domains for ones we can reach')
if not engines:
unique_resolved_domains = resolved_screenshot_hosts | {word}
unique_resolved_domains = resolved_hostnames | {word}
elif dnsresolve != '':
unique_resolved_domains = resolved_screenshot_hosts
unique_resolved_domains = resolved_hostnames
else:
# Technically not resolved in this case, which is not ideal
# You should always use dns resolve when doing screenshotting
theHarvester/discovery/apisguru.py
+9 -29
View File
@@ -3,7 +3,6 @@ import re
from email.errors import HeaderParseError
from email.headerregistry import Address
from ipaddress import ip_address
from itertools import islice
from urllib.parse import unquote, urlsplit, urlunsplit
from theHarvester.lib.core import AsyncFetcher, FetcherResponse, ResponseStreamError
@@ -16,32 +15,26 @@ class SearchApisGuru:
Results include descendant hostnames, contact emails, concrete API base URLs,
and related in-scope URLs. The adapter does not resolve IP addresses or expand
OpenAPI paths into operation endpoints. It checks up to 1,000 matching specs
for up to ten minutes; ``--limit`` caps stored results, not spec traversal.
OpenAPI paths into operation endpoints. It checks matching specs for up to
ten minutes; ``--limit`` caps stored results, not spec traversal.
The shared fetcher caps each JSON response at 16 MiB. Oversized specs are
skipped and leave the source marked partial.
"""
DIRECTORY_ROOT = 'https://api.apis.guru/v2'
MAX_DIRECTORY_ENTRIES = 1000
MAX_SPEC_ITEMS = 1000
MAX_RESULTS_PER_ROUTE = 1000
MAX_URL_LENGTH = 4096
REQUEST_TIMEOUT = 60
MAX_RUNTIME_SECONDS = 600
def __init__(self, word: str, limit: int) -> None:
def __init__(self, word: str, limit: int | None) -> None:
self.word = self._domain(word)
requested_limit = max(0, limit)
self.result_limit = min(requested_limit, self.MAX_RESULTS_PER_ROUTE)
self.result_limit_is_protective = requested_limit > self.MAX_RESULTS_PER_ROUTE
self.result_limit = max(0, limit) if limit is not None else None
self.totalhosts: set[str] = set()
self.totalemails: set[str] = set()
self.urls: set[str] = set()
self.proxy = False
self._report: SourceExecutionReport | None = None
self.result_limit_reached = False
self.protective_limit_reached = False
@staticmethod
def _domain(value: str) -> str:
@@ -95,11 +88,8 @@ class SearchApisGuru:
return bool(domain and normalize_scoped_hostname(domain, self.word))
def _retain(self, values: set[str], value: str) -> None:
if value not in values and len(values) >= self.result_limit:
if self.result_limit_is_protective:
self.protective_limit_reached = True
else:
self.result_limit_reached = True
if self.result_limit is not None and value not in values and len(values) >= self.result_limit:
self.result_limit_reached = True
else:
values.add(value)
@@ -175,9 +165,7 @@ class SearchApisGuru:
self._add_host(host)
if isinstance(host, str) and isinstance(schemes, list):
path = base_path if isinstance(base_path, str) and base_path.startswith('/') else ''
if len(schemes) > self.MAX_SPEC_ITEMS:
self.protective_limit_reached = True
for scheme in islice(schemes, self.MAX_SPEC_ITEMS):
for scheme in schemes:
if isinstance(scheme, str) and scheme.lower() in {'http', 'https'}:
self._add_url(f'{scheme.lower()}://{host}{path}')
elif not isinstance(scheme, str):
@@ -185,9 +173,7 @@ class SearchApisGuru:
servers = spec.get('servers')
if isinstance(servers, list):
if len(servers) > self.MAX_SPEC_ITEMS:
self.protective_limit_reached = True
for server in islice(servers, self.MAX_SPEC_ITEMS):
for server in servers:
if not isinstance(server, dict):
malformed = True
elif not isinstance(server.get('url'), str):
@@ -283,11 +269,10 @@ class SearchApisGuru:
self._stop('failed', 'invalid-response')
return
directory_limit_reached = len(directory) > self.MAX_DIRECTORY_ENTRIES
spec_urls: list[str] = []
malformed = False
seen_spec_urls: set[str] = set()
for api_id, api in islice(directory.items(), self.MAX_DIRECTORY_ENTRIES):
for api_id, api in directory.items():
if not isinstance(api_id, str):
continue
provider_domain = api_id.partition(':')[0]
@@ -354,12 +339,8 @@ class SearchApisGuru:
self._stop('failed', 'invalid-response')
elif spec_failure is not None:
self._stop('failed', spec_failure)
elif self.protective_limit_reached:
self._stop('failed', 'result-cap')
elif self.result_limit_reached:
self._stop('completed', 'result-limit')
elif directory_limit_reached:
self._stop('failed', 'directory-entry-limit')
async def get_hostnames(self) -> set[str]:
return self.totalhosts
@@ -374,7 +355,6 @@ class SearchApisGuru:
self.proxy = proxy
self._report = None
self.result_limit_reached = False
self.protective_limit_reached = False
try:
async with asyncio.timeout(self.MAX_RUNTIME_SECONDS):
await self.do_search()
theHarvester/discovery/arquivo.py
+68 -42
View File
@@ -1,68 +1,94 @@
import asyncio
import json
import logging
from urllib.parse import urlencode, urlsplit
from theHarvester.lib.core import AsyncFetcher, Core, FetcherResponse
from theHarvester.lib.hostnames import normalize_scoped_hostname
from theHarvester.lib.source_execution import SourceExecutionReport
logger = logging.getLogger(__name__)
class SearchArquivo:
MAX_RESULTS = 10_000
PAGE_SIZE = 10_000
def __init__(self, word: str, limit: int) -> None:
def __init__(self, word: str, limit: int | None) -> None:
self.word = word.strip().lower().rstrip('.')
self.limit = min(max(limit, 1), self.MAX_RESULTS)
self.limit = limit
self.totalhosts: set[str] = set()
self.proxy = False
async def process(self, proxy: bool = False) -> None:
async def process(self, proxy: bool = False) -> SourceExecutionReport | None:
self.proxy = proxy
query = urlencode(
{
offset = 0
previous_page = None
report = None
while self.limit is None or offset < self.limit:
page_size = self.PAGE_SIZE if self.limit is None else min(self.PAGE_SIZE, self.limit - offset)
parameters = {
'url': self.word,
'matchType': 'domain',
'output': 'json',
'fields': 'url',
'limit': self.limit,
'limit': page_size,
}
)
try:
responses: list[FetcherResponse | None] = await AsyncFetcher.fetch_all(
[f'https://arquivo.pt/wayback/cdx?{query}'],
headers={'User-agent': Core.get_user_agent()},
proxy=self.proxy,
include_metadata=True,
)
except Exception as error:
logger.info(f'Arquivo.pt request failed: {error}')
return
response = responses[0] if responses else None
if response is None:
logger.info('Arquivo.pt request failed')
return
if not 200 <= response.status < 300:
logger.info(f'Arquivo.pt request failed with HTTP {response.status}')
return
if not isinstance(response.body, str):
logger.info('Arquivo.pt returned malformed CDX data')
return
for line in response.body.splitlines():
if offset:
parameters['offset'] = offset
query = urlencode(parameters)
try:
item = json.loads(line)
except json.JSONDecodeError, TypeError:
continue
if not isinstance(item, dict) or not isinstance(url := item.get('url'), str):
continue
try:
hostname = urlsplit(url).hostname
except ValueError:
continue
if (normalized := normalize_scoped_hostname(hostname, self.word)) and normalized != self.word:
self.totalhosts.add(normalized)
responses: list[FetcherResponse | None] = await AsyncFetcher.fetch_all(
[f'https://arquivo.pt/wayback/cdx?{query}'],
headers={'User-agent': Core.get_user_agent()},
proxy=self.proxy,
include_metadata=True,
)
except asyncio.CancelledError:
raise
except Exception as error:
logger.info(f'Arquivo.pt request failed: {error}')
return SourceExecutionReport('partial' if self.totalhosts else 'failed', 'transport-error')
response = responses[0] if responses else None
if response is None:
logger.info('Arquivo.pt request failed')
return SourceExecutionReport('partial' if self.totalhosts else 'failed', 'transport-error')
if not 200 <= response.status < 300:
logger.info(f'Arquivo.pt request failed with HTTP {response.status}')
return SourceExecutionReport('partial' if self.totalhosts else 'failed', f'http-{response.status}')
if not isinstance(response.body, str):
logger.info('Arquivo.pt returned malformed CDX data')
return SourceExecutionReport('partial' if self.totalhosts else 'failed', 'invalid-response')
if response.body == previous_page:
return SourceExecutionReport('partial', 'repeated-page')
previous_page = response.body
lines = response.body.splitlines()
malformed = False
for line in lines:
try:
item = json.loads(line)
except json.JSONDecodeError, TypeError:
malformed = True
continue
if not isinstance(item, dict) or not isinstance(url := item.get('url'), str):
malformed = True
continue
try:
hostname = urlsplit(url).hostname
except ValueError:
malformed = True
continue
if (normalized := normalize_scoped_hostname(hostname, self.word)) and normalized != self.word:
self.totalhosts.add(normalized)
if malformed:
report = SourceExecutionReport('partial' if self.totalhosts else 'failed', 'invalid-response')
offset += len(lines)
if len(lines) < page_size:
break
if report is not None and self.totalhosts and report.status == 'failed':
return SourceExecutionReport('partial', report.stop_reason)
return report
async def get_hostnames(self) -> set[str]:
return self.totalhosts
theHarvester/discovery/baidusearch.py
+47 -27
View File
@@ -50,8 +50,9 @@ class SearchBaidu:
return SourceExecutionReport('failed', 'no-response')
return None
async def _http_search(self, urls: list[str], proxy: str | bool) -> SourceExecutionReport | None:
async def _http_search(self, urls, proxy: str | bool) -> SourceExecutionReport | None:
headers = {'Host': self.server, 'User-Agent': Core.get_browser_user_agent()}
seen_bodies: set[str] = set()
try:
async with AsyncFetcher.open_session(headers=headers, proxy=proxy, request_timeout=60) as session:
for page_number, url in enumerate(urls):
@@ -64,16 +65,23 @@ class SearchBaidu:
include_metadata=True,
)
if not isinstance(response, FetcherResponse):
return SourceExecutionReport('failed', 'transport-error')
body = response.body if isinstance(response.body, str) else ''
return SourceExecutionReport('partial' if self.total_results else 'failed', 'transport-error')
if not isinstance(response.body, str):
return SourceExecutionReport('partial' if self.total_results else 'failed', 'invalid-response')
body = response.body
if report := self._response_report(response.status, body, response.headers.get('location', '')):
return report
return SourceExecutionReport('partial', report.stop_reason) if self.total_results else report
if body in seen_bodies:
return SourceExecutionReport('partial', 'repeated-page')
seen_bodies.add(body)
self.total_results += f' {body}'
except asyncio.CancelledError:
raise
except Exception:
return SourceExecutionReport('failed', 'transport-error')
return SourceExecutionReport('partial' if self.total_results else 'failed', 'transport-error')
return None
def __init__(self, word, limit) -> None:
def __init__(self, word, limit: int | None) -> None:
self.word = word
self.total_results = ''
self.server = 'www.baidu.com'
@@ -82,23 +90,27 @@ class SearchBaidu:
async def do_search(self) -> SourceExecutionReport | None:
base_url = f'https://{self.server}/s'
urls = []
for offset in range(0, self.limit, 10):
params: dict[str, str | int] = {
'ie': 'utf-8',
'f': 8,
'tn': 'baidu',
'wd': f'site:{self.word}',
'rqlang': 'en',
'rsv_enter': 1,
'rsv_dl': 'tb_enter',
}
if offset:
params['pn'] = offset
urls.append(f'{base_url}?{urlencode(params)}')
def urls():
offset = 0
while self.limit is None or offset < self.limit:
params: dict[str, str | int] = {
'ie': 'utf-8',
'f': 8,
'tn': 'baidu',
'wd': f'site:{self.word}',
'rqlang': 'en',
'rsv_enter': 1,
'rsv_dl': 'tb_enter',
}
if offset:
params['pn'] = offset
yield f'{base_url}?{urlencode(params)}'
offset += 10
page_urls = urls()
if playwright_api is None:
return await self._http_search(urls, self.proxy)
return await self._http_search(page_urls, self.proxy)
proxy_url, _proxy_type = AsyncFetcher._resolve_proxy(self.proxy)
manager = playwright_api.async_playwright()
@@ -107,6 +119,7 @@ class SearchBaidu:
report = None
primary_error: BaseException | None = None
cleanup_errors: list[BaseException] = []
seen_bodies: set[str] = set()
try:
playwright = await manager.__aenter__()
manager_entered = True
@@ -116,16 +129,23 @@ class SearchBaidu:
)
context = await browser.new_context(user_agent=Core.get_browser_user_agent())
page = await context.new_page()
for page_number, url in enumerate(urls):
for page_number, url in enumerate(page_urls):
if page_number:
await asyncio.sleep(self.REQUEST_DELAY_SECONDS)
response = await page.goto(url, wait_until='domcontentloaded', timeout=60_000)
if response is None:
report = SourceExecutionReport('failed', 'transport-error')
report = SourceExecutionReport('partial' if self.total_results else 'failed', 'transport-error')
break
body = await page.content()
if report := self._response_report(response.status, body, page.url):
if response_report := self._response_report(response.status, body, page.url):
report = (
SourceExecutionReport('partial', response_report.stop_reason) if self.total_results else response_report
)
break
if body in seen_bodies:
report = SourceExecutionReport('partial', 'repeated-page')
break
seen_bodies.add(body)
self.total_results += f' {body}'
except BaseException as error:
primary_error = error
@@ -153,12 +173,12 @@ class SearchBaidu:
isinstance(final_error, ValueError) and str(final_error) == 'startupinfo is not supported'
):
if not self.total_results:
return await self._http_search(urls, proxy_url or False)
return SourceExecutionReport('failed', 'transport-error')
return await self._http_search(urls(), proxy_url or False)
return SourceExecutionReport('partial', 'transport-error')
if final_error is not None:
raise final_error
if report is not None and report.stop_reason == 'transport-error' and not self.total_results:
return await self._http_search(urls, proxy_url or False)
return await self._http_search(urls(), proxy_url or False)
return report
async def process(self, proxy: bool = False) -> SourceExecutionReport | None:
theHarvester/discovery/bravesearch.py
+11 -7
View File
@@ -23,7 +23,11 @@ class SearchBrave:
embedded use independent of operator configuration files.
"""
def __init__(self, word: str, limit: int, credential_adapter: CredentialAdapter | None = None) -> None:
# Brave documents offsets 0 through 9; this is a provider boundary, not a
# theHarvester result cap.
MAX_OFFSET = 9
def __init__(self, word: str, limit: int | None, credential_adapter: CredentialAdapter | None = None) -> None:
self.word = word
self.results: list[dict[str, Any]] = []
self.totalresults = ''
@@ -55,12 +59,12 @@ class SearchBrave:
queries = [f'"{self.word}"', f'site:{self.word}']
for query in queries:
if len(self.results) >= self.limit:
if self.limit is not None and len(self.results) >= self.limit:
break
try:
for offset in range(10):
remaining = self.limit - len(self.results)
if remaining <= 0:
for offset in range(self.MAX_OFFSET + 1):
remaining = self.limit - len(self.results) if self.limit is not None else 20
if self.limit is not None and remaining <= 0:
break
params = {
'q': query,
@@ -139,14 +143,14 @@ class SearchBrave:
self.totalresults += result_text + '\n'
self.results.extend(results)
if len(self.results) >= self.limit:
if self.limit is not None and len(self.results) >= self.limit:
return SourceExecutionReport('completed', 'result-limit')
if not more_results_available:
break
await asyncio.sleep(get_delay())
else:
return SourceExecutionReport('partial', 'pagination-limit')
return SourceExecutionReport('partial', 'provider-limit')
except ResponseStreamError as error:
return SourceExecutionReport('failed', error.reason)
theHarvester/discovery/censysearch.py
+12 -7
View File
@@ -13,7 +13,7 @@ class SearchCensys:
MAX_RESULTS_PER_PAGE = 100
SERVER = 'https://api.platform.censys.io/v3/global/search/query'
def __init__(self, domain: str, limit: int = 500) -> None:
def __init__(self, domain: str, limit: int | None = 500) -> None:
self.word = domain
token, self.organization_id = Core.censys_key()
if not isinstance(token, str) or not token.strip():
@@ -55,7 +55,7 @@ class SearchCensys:
return False
async def do_search(self) -> SourceExecutionReport | None:
if self.limit <= 0:
if self.limit is not None and self.limit <= 0:
return None
headers = {'Accept': 'application/json', 'Authorization': f'Bearer {self.token}'}
@@ -70,11 +70,13 @@ class SearchCensys:
malformed = False
async with AsyncFetcher.open_session(headers=headers, proxy=self.proxy, request_timeout=720) as session:
while records_seen < self.limit:
while self.limit is None or records_seen < self.limit:
body = {
'query': f'cert.names: "{self.word}"',
'fields': ['cert.names', 'cert.parsed.subject.email_address'],
'page_size': min(self.MAX_RESULTS_PER_PAGE, self.limit - records_seen),
'page_size': min(self.MAX_RESULTS_PER_PAGE, self.limit - records_seen)
if self.limit is not None
else self.MAX_RESULTS_PER_PAGE,
}
if page_token is not None:
body['page_token'] = page_token
@@ -108,11 +110,11 @@ class SearchCensys:
return SourceExecutionReport('failed', 'invalid-response')
for hit in hits:
if records_seen >= self.limit:
if self.limit is not None and records_seen >= self.limit:
break
malformed = self._parse_hit(hit) or malformed
records_seen += 1
if records_seen >= self.limit:
if self.limit is not None and records_seen >= self.limit:
if malformed:
return SourceExecutionReport('failed', 'invalid-response')
return None
@@ -121,7 +123,10 @@ class SearchCensys:
return SourceExecutionReport('failed', 'invalid-response')
return None
if next_page_token in seen_tokens:
return SourceExecutionReport('failed', 'repeated-cursor')
return SourceExecutionReport(
'partial' if self.totalhosts or self.emails else 'failed',
'repeated-cursor',
)
seen_tokens.add(next_page_token)
page_token = next_page_token
return None
theHarvester/discovery/certspottersearch.py
+1 -7
View File
@@ -13,9 +13,6 @@ class SearchCertspoter:
API reference: https://sslmate.com/help/reference/ct_search_api_v1
"""
# ponytail: hard cap protects against endless unique cursors; raise only if real targets exceed 1,000 pages.
MAX_PAGES = 1000
def __init__(self, word) -> None:
self.word = word.strip().lower().rstrip('.')
self.totalhosts: set = set()
@@ -31,7 +28,7 @@ class SearchCertspoter:
cursor = None
seen_cursors: set[str] = set()
try:
for _ in range(self.MAX_PAGES):
while True:
params = {
'domain': self.word,
'include_subdomains': 'true',
@@ -118,9 +115,6 @@ class SearchCertspoter:
break
seen_cursors.add(next_cursor)
cursor = next_cursor
else:
self._mark_incomplete('page-limit')
logger.warning('Cert Spotter page limit reached; results may be incomplete.')
except ConnectionError:
self._mark_incomplete('connection-error')
logger.warning('Cert Spotter network connection failed; results may be incomplete.')
theHarvester/discovery/commoncrawl.py
+13 -24
View File
@@ -22,12 +22,10 @@ class SearchCommoncrawl:
MAX_RECORDS_PER_REQUEST = 50
MAX_CONSECUTIVE_PAGE_ERRORS = 3
RUNTIME_SECONDS = 120.0
# Protect the shared index service even when its page count is unexpectedly large.
MAX_PAGES_PER_QUERY = 100
def __init__(self, word, limit: int = 500) -> None:
def __init__(self, word, limit: int | None = 500) -> None:
self.word = word.lower().rstrip('.')
self.limit = max(limit, 0)
self.limit = max(limit, 0) if limit is not None else None
self.totalhosts: set[str] = set()
self.proxy = False
self.hostname = 'https://index.commoncrawl.org'
@@ -66,7 +64,7 @@ class SearchCommoncrawl:
return ''
@classmethod
def _select_indexes(cls, catalog: list[object]) -> list[dict]:
def _select_indexes(cls, catalog: list[object], *, include_all: bool = False) -> list[dict]:
dated_indexes: list[tuple[datetime, dict]] = []
for entry in catalog:
if not isinstance(entry, dict):
@@ -98,12 +96,12 @@ class SearchCommoncrawl:
if not dated_indexes:
return []
cutoff = max(timestamp for timestamp, _ in dated_indexes) - cls.INDEX_LOOKBACK
cutoff = None if include_all else max(timestamp for timestamp, _ in dated_indexes) - cls.INDEX_LOOKBACK
selected: list[dict] = []
endpoints: set[str] = set()
for timestamp, entry in sorted(dated_indexes, key=lambda item: item[0], reverse=True):
endpoint = entry['cdx-api']
if timestamp >= cutoff and endpoint not in endpoints:
if (cutoff is None or timestamp >= cutoff) and endpoint not in endpoints:
endpoints.add(endpoint)
selected.append(entry)
return selected
@@ -121,7 +119,7 @@ class SearchCommoncrawl:
logger.error('Common Crawl API error: invalid index catalog')
return SourceExecutionReport('failed', 'invalid-catalog')
indexes = self._select_indexes(catalog_response[0])
indexes = self._select_indexes(catalog_response[0], include_all=self.limit is None)
if not indexes:
logger.error('Common Crawl API error: index catalog contains no usable entries')
return SourceExecutionReport('failed', 'no-usable-indexes')
@@ -136,7 +134,6 @@ class SearchCommoncrawl:
successful_queries = 0
failed_queries = 0
page_limit_reached = False
query_number = 0
for index in indexes:
endpoint = index['cdx-api']
@@ -158,12 +155,13 @@ class SearchCommoncrawl:
if isinstance(page_count, bool) or not isinstance(page_count, int) or page_count < 0:
raise ValueError('invalid page count')
query_succeeded = page_count == 0
page_limit = min(page_count, self.MAX_PAGES_PER_QUERY)
first_page = 0
consecutive_page_errors = 0
while first_page < page_limit:
remaining = self.limit - len(self.totalhosts)
if remaining == 0:
while first_page < page_count:
remaining = (
self.limit - len(self.totalhosts) if self.limit is not None else self.MAX_RECORDS_PER_REQUEST
)
if self.limit is not None and remaining == 0:
return None
page_url = f'{endpoint}?{urlencode({"url": query, "output": "json", "pageSize": self.PAGE_SIZE, "page": first_page, "limit": min(remaining, self.MAX_RECORDS_PER_REQUEST)})}'
first_page += 1
@@ -179,7 +177,7 @@ class SearchCommoncrawl:
domain = self._extract_domain_from_url(record.get('url', ''))
if domain.endswith(f'.{self.word}') or domain == self.word:
self.totalhosts.add(domain)
if len(self.totalhosts) >= self.limit:
if self.limit is not None and len(self.totalhosts) >= self.limit:
return None
except ValueError as error:
message = str(error)
@@ -194,12 +192,6 @@ class SearchCommoncrawl:
consecutive_page_errors += 1
if consecutive_page_errors >= self.MAX_CONSECUTIVE_PAGE_ERRORS:
break
if page_count > page_limit:
page_limit_reached = True
logger.warning(
f'Common Crawl page limit reached for index {index.get("id", "unknown")}; '
'results may be incomplete'
)
if query_had_errors:
failed_queries += 1
except Exception as error:
@@ -213,9 +205,6 @@ class SearchCommoncrawl:
return SourceExecutionReport('partial', 'query-errors')
logger.warning(f'Common Crawl failed all {query_total} queries')
return SourceExecutionReport('failed', 'all-queries-failed')
if page_limit_reached:
return SourceExecutionReport('partial', 'page-limit')
except Exception as error:
logger.error(f'Common Crawl API error: {error}')
return SourceExecutionReport('failed', 'unexpected-error')
@@ -233,4 +222,4 @@ class SearchCommoncrawl:
logger.info(
f'Common Crawl runtime limit reached after {self.RUNTIME_SECONDS:g}s; preserved {len(self.totalhosts)} hosts'
)
return SourceExecutionReport('failed', 'runtime-limit')
return SourceExecutionReport('partial' if self.totalhosts else 'failed', 'runtime-limit')
theHarvester/discovery/criminalip.py
+34 -14
View File
@@ -5,6 +5,7 @@ from urllib.parse import urlparse
from theHarvester.discovery.constants import MissingKey, get_delay
from theHarvester.lib.core import AsyncFetcher, Core
from theHarvester.lib.source_execution import SourceExecutionReport
logger = logging.getLogger(__name__)
@@ -85,7 +86,7 @@ class SearchCriminalIP:
for nested_value in value.values():
self._collect_hosts_from_value(nested_value)
async def do_search(self) -> None:
async def do_search(self) -> SourceExecutionReport | None:
# https://www.criminalip.io/developer/api/post-domain-scan
# https://www.criminalip.io/developer/api/get-domain-status-id
# https://www.criminalip.io/developer/api/get-v2-domain-report-id
@@ -103,15 +104,19 @@ class SearchCriminalIP:
# {'data': {'scan_id': scan_id}, 'message': 'api success', 'status': 200}
if not isinstance(response, dict):
logger.info(f'CriminalIP scan response has unexpected type: {type(response).__name__}')
return
return SourceExecutionReport('failed', 'invalid-response')
if response.get('status') != 200:
logger.info(f'CriminalIP scan request failed with status {response.get("status")}')
return
if response.get('status') == 429:
return SourceExecutionReport('rate-limited', 'http-429')
if response.get('status') in {401, 403}:
return SourceExecutionReport('failed', 'access-denied')
return SourceExecutionReport('failed', 'provider-error')
scan_id = response.get('data', {}).get('scan_id')
if scan_id is None:
logger.info('CriminalIP scan response did not include a scan_id')
return
return SourceExecutionReport('failed', 'invalid-response')
scan_percentage = 0
counter = 0
@@ -127,25 +132,29 @@ class SearchCriminalIP:
status = status_response[0] if isinstance(status_response, list) and len(status_response) > 0 else {}
if not isinstance(status, dict):
logger.info(f'CriminalIP status response has unexpected type: {type(status).__name__}')
return
return SourceExecutionReport('failed', 'invalid-response')
if status.get('status') != 200:
logger.info(f'CriminalIP status request failed with status {status.get("status")}')
return
if status.get('status') == 429:
return SourceExecutionReport('rate-limited', 'http-429')
if status.get('status') in {401, 403}:
return SourceExecutionReport('failed', 'access-denied')
return SourceExecutionReport('failed', 'provider-error')
# Expected format:
# {"data": {"scan_percentage": 100}, "message": "api success", "status": 200}
scan_percentage = status.get('data', {}).get('scan_percentage')
if scan_percentage is None:
logger.info('CriminalIP status response did not include scan_percentage')
return
return SourceExecutionReport('failed', 'invalid-response')
if scan_percentage == 100:
break
if scan_percentage == -2:
logger.info(f'CriminalIP failed to scan: {self.word} does not exist, verify manually')
return
return None
if scan_percentage == -1:
logger.info('CriminalIP scan failed with scan_percentage -1')
return
return SourceExecutionReport('failed', 'provider-error')
# Wait for scan to finish
if counter >= 5:
await asyncio.sleep(20 * get_delay())
@@ -156,7 +165,7 @@ class SearchCriminalIP:
logger.info(
'Ten iterations have occurred in CriminalIP waiting for scan to finish, returning to prevent infinite loop.'
)
return
return SourceExecutionReport('partial', 'runtime-limit')
report_url = f'https://api.criminalip.io/v2/domain/report/{scan_id}'
scan_response = await AsyncFetcher.fetch_all(
@@ -168,15 +177,21 @@ class SearchCriminalIP:
scan = scan_response[0] if isinstance(scan_response, list) and len(scan_response) > 0 else {}
if not isinstance(scan, dict):
logger.info(f'CriminalIP report response has unexpected type: {type(scan).__name__}')
return
return SourceExecutionReport('failed', 'invalid-response')
if scan.get('status') != 200:
logger.info(f'CriminalIP report request failed with status {scan.get("status")}')
return
if scan.get('status') == 429:
return SourceExecutionReport('rate-limited', 'http-429')
if scan.get('status') in {401, 403}:
return SourceExecutionReport('failed', 'access-denied')
return SourceExecutionReport('failed', 'provider-error')
try:
await self.parser(scan)
except Exception as e:
logger.info(f'CriminalIP report parsing failed with {type(e).__name__}')
return SourceExecutionReport('failed', 'invalid-response')
return None
async def parser(self, jlines):
# TODO when new scope field is added to parse lines for potential new scope!
@@ -314,6 +329,11 @@ class SearchCriminalIP:
async def get_ips(self) -> set:
return self.totalips
async def process(self, proxy: bool = False) -> None:
async def process(self, proxy: bool = False) -> SourceExecutionReport | None:
self.proxy = proxy
await self.do_search()
try:
return await self.do_search()
except asyncio.CancelledError:
raise
except Exception:
return SourceExecutionReport('failed', 'transport-error')
theHarvester/discovery/fofa.py
+9 -6
View File
@@ -15,8 +15,8 @@ class SearchFofa:
MAX_PAGE_SIZE = 10_000
def __init__(self, word: str, limit: int) -> None:
if isinstance(limit, bool) or not isinstance(limit, int) or limit <= 0:
def __init__(self, word: str, limit: int | None) -> None:
if limit is not None and (isinstance(limit, bool) or not isinstance(limit, int) or limit <= 0):
raise ValueError('FOFA limit must be a positive integer')
self.word = word
self.limit = limit
@@ -81,8 +81,8 @@ class SearchFofa:
headers={'User-Agent': Core.get_user_agent()},
proxy=self.proxy,
) as session:
while records_seen < self.limit:
remaining = self.limit - records_seen
while self.limit is None or records_seen < self.limit:
remaining = self.limit - records_seen if self.limit is not None else self.MAX_PAGE_SIZE
params: dict[str, str | int] = {
'email': self.email,
'key': self.api_key,
@@ -110,7 +110,7 @@ class SearchFofa:
if not isinstance(results, list):
return SourceExecutionReport('failed', 'invalid-response')
page_results = results[:remaining]
page_results = results[:remaining] if self.limit is not None else results
records_seen += len(page_results)
if self._store_results(page_results):
report = SourceExecutionReport('failed', 'invalid-response')
@@ -118,7 +118,10 @@ class SearchFofa:
if not results or not isinstance(next_cursor, str) or not next_cursor:
break
if next_cursor in seen_cursors:
return SourceExecutionReport('failed', 'repeated-cursor')
return SourceExecutionReport(
'partial' if self.totalhosts or self.totalips else 'failed',
'repeated-cursor',
)
seen_cursors.add(next_cursor)
cursor = next_cursor
except Exception:
theHarvester/discovery/githubcode.py
+32 -11
View File
@@ -5,6 +5,7 @@ from typing import TYPE_CHECKING, Any, NamedTuple
from theHarvester.discovery.constants import MissingKey, get_delay
from theHarvester.lib.core import AsyncFetcher, Core
from theHarvester.lib.source_execution import SourceExecutionReport, SourceReportStatus
from theHarvester.parsers import myparser
if TYPE_CHECKING:
@@ -29,7 +30,7 @@ class ErrorResult(NamedTuple):
class SearchGithubCode:
def __init__(self, word, limit) -> None:
def __init__(self, word, limit: int | None) -> None:
try:
self.word = word
self.total_results = ''
@@ -127,35 +128,48 @@ class SearchGithubCode:
logger.info(f'Error performing search: {e}')
return '', {}, 500, {}
async def process(self, proxy: bool = False) -> None:
def _failure_report(self, reason: str, *, empty_status: SourceReportStatus = 'failed') -> SourceExecutionReport:
status = 'partial' if self.counter else empty_status
return SourceExecutionReport(status, reason)
async def process(self, proxy: bool = False) -> SourceExecutionReport | None:
try:
self.proxy = proxy
visited_pages: set[int] = set()
seen_page_content: set[tuple[str, ...]] = set()
async with AsyncFetcher.open_session(headers=self.headers, proxy=self.proxy) as session:
while self.counter < self.limit and self.page != 0:
while (self.limit is None or self.counter < self.limit) and self.page != 0:
try:
visited_pages.add(self.page)
api_response = await self.do_search(self.page, session)
result = await self.handle_response(api_response)
if isinstance(result, SuccessResult):
if self.limit is None and result.fragments:
page_content = tuple(sorted(result.fragments))
if page_content in seen_page_content:
logger.info('\tRepeated result page detected; exiting to avoid infinite loop.')
self.page = 0
return self._failure_report('repeated-page')
seen_page_content.add(page_content)
# Reset retry counter on any successful response
self.retry_count = 0
logger.info(f'\tSearching {self.counter} results.')
remaining = self.limit - self.counter
remaining = self.limit - self.counter if self.limit is not None else len(result.fragments)
fragments = result.fragments[:remaining]
if not fragments:
self.page = 0
break
self.total_results += f'{" ".join(fragments)} '
self.counter += len(fragments)
if self.counter >= self.limit:
if self.limit is not None and self.counter >= self.limit:
self.page = 0
break
next_or_last = result.next_page or result.last_page
# Break if pagination does not advance to avoid infinite loop
if next_or_last == self.page:
logger.info('\tNo page advancement detected; exiting to avoid infinite loop.')
if next_or_last in visited_pages:
logger.info('\tPagination cycle detected; exiting to avoid infinite loop.')
self.page = 0
break
return self._failure_report('repeated-page')
self.page = next_or_last
await asyncio.sleep(get_delay())
elif isinstance(result, RetryResult):
@@ -163,7 +177,7 @@ class SearchGithubCode:
if self.retry_count > self.max_retries:
logger.info('\tMaximum retries reached; exiting to avoid infinite loop.')
self.page = 0
break
return self._failure_report('rate-limited', empty_status='rate-limited')
sleepy_time = get_delay() + result.time
logger.info(f'\tRetrying page in {sleepy_time} seconds...')
await asyncio.sleep(sleepy_time)
@@ -171,12 +185,19 @@ class SearchGithubCode:
# On error, stop to avoid endless retries on a bad state
logger.info(f'\tGitHub code API request failed with status {result.status_code}')
self.page = 0
break
reason = 'access-denied' if result.status_code in {401, 403} else 'provider-error'
return self._failure_report(reason)
except Exception as e:
logger.info(f'Error processing page: {e}')
self.retry_count += 1
if self.retry_count > self.max_retries:
self.page = 0
return self._failure_report('transport-error')
await asyncio.sleep(get_delay())
except Exception as e:
logger.info(f'An exception has occurred in githubcode process: {e}')
return self._failure_report('transport-error')
return None
async def get_emails(self):
try:
theHarvester/discovery/gitlabsearch.py
+123 -59
View File
@@ -3,8 +3,10 @@ import logging
import re
from urllib.parse import quote
from theHarvester.lib.core import AsyncFetcher, Core
from theHarvester.discovery.provider_response import provider_http_error
from theHarvester.lib.core import AsyncFetcher, Core, FetcherResponse
from theHarvester.lib.hostnames import normalize_scoped_hostname
from theHarvester.lib.source_execution import SourceExecutionReport
logger = logging.getLogger(__name__)
@@ -12,8 +14,11 @@ logger = logging.getLogger(__name__)
class SearchGitlab:
"""Search public GitLab project metadata, README files, and user profiles."""
def __init__(self, word) -> None:
def __init__(self, word: str, limit: int | None = None) -> None:
if limit is not None and (isinstance(limit, bool) or not isinstance(limit, int) or limit <= 0):
raise ValueError('GitLab limit must be a positive integer')
self.word = word
self.limit = limit
self.totalhosts: set = set()
self.totalemails: set = set()
self.totalurls: set = set()
@@ -32,6 +37,15 @@ class SearchGitlab:
return {}
return {}
@staticmethod
def _combine_reports(
current: SourceExecutionReport | None,
candidate: SourceExecutionReport | None,
) -> SourceExecutionReport | None:
if current is None or (current.status == 'completed' and candidate is not None):
return candidate
return current
def _extract_domains_from_text(self, text: str) -> set:
"""Extract domain names that match the target domain."""
domains: set[str] = set()
@@ -64,42 +78,74 @@ class SearchGitlab:
self.totalemails.update(emails)
return bool(hosts or emails)
async def search_projects(self) -> None:
async def _fetch_page(
self,
endpoint: str,
term: str,
page: int,
per_page: int,
) -> tuple[list[object], str | None, SourceExecutionReport | None]:
url = f'{self.hostname}/api/v4/{endpoint}?search={term}&per_page={per_page}&page={page}'
response = await AsyncFetcher.fetch_all(
[url],
headers={'User-agent': Core.get_user_agent()},
proxy=self.proxy,
json=True,
include_metadata=True,
)
if not response:
return [], None, SourceExecutionReport('failed', 'transport-error')
payload = response[0]
headers: dict[str, str] | None = None
if isinstance(payload, FetcherResponse):
if error := provider_http_error(payload):
if page > 1 and payload.status == 400:
return [], None, SourceExecutionReport('partial', 'provider-limit')
return [], None, SourceExecutionReport(*error)
headers = payload.headers
payload = payload.body
records = self._safe_parse_json(payload)
if not isinstance(records, list):
return [], None, SourceExecutionReport('failed', 'invalid-response')
if headers is not None and 'x-next-page' in headers:
next_page = headers['x-next-page'].strip() or None
else:
next_page = str(page + 1) if len(records) >= per_page else None
return records, next_page, None
async def search_projects(self) -> SourceExecutionReport | None:
"""Search GitLab projects for references to the target domain."""
try:
headers = {'User-agent': Core.get_user_agent()}
# Search for projects mentioning our domain
search_terms = [self.word, f'*.{self.word}']
report = None
for term in search_terms:
# Search projects
projects_url = f'{self.hostname}/api/v4/projects?search={term}&per_page=20'
response = await AsyncFetcher.fetch_all([projects_url], headers=headers, proxy=self.proxy)
if not response or not isinstance(response, list) or not response[0]:
continue
try:
projects = self._safe_parse_json(response[0])
if not isinstance(projects, list):
continue
for project in projects:
page = 1
records_seen = 0
seen_pages: set[str] = set()
seen_cursors: set[str] = set()
while self.limit is None or records_seen < self.limit:
per_page = min(100, self.limit - records_seen) if self.limit is not None else 100
projects, next_page, page_report = await self._fetch_page('projects', term, page, per_page)
if page_report is not None:
report = self._combine_reports(report, page_report)
break
signature = json.dumps(projects, sort_keys=True, default=str)
if signature in seen_pages:
report = self._combine_reports(report, SourceExecutionReport('partial', 'repeated-page'))
break
seen_pages.add(signature)
accepted = projects[: self.limit - records_seen] if self.limit is not None else projects
records_seen += len(accepted)
for project in accepted:
if not isinstance(project, dict):
continue
# Extract information from project metadata
description = project.get('description', '') or ''
name = project.get('name', '') or ''
path = project.get('path_with_namespace', '') or ''
web_url = project.get('web_url', '') or ''
# Look for domains in description and name
all_text = f'{description} {name} {path}'
project_is_relevant = self._add_text_evidence(all_text)
# Try to get README content for more detailed search
project_id = project.get('id')
default_branch = project.get('default_branch')
if project_id and isinstance(default_branch, str) and default_branch:
@@ -119,35 +165,46 @@ class SearchGitlab:
if project_is_relevant and isinstance(web_url, str) and web_url.strip():
self.totalurls.add(web_url.strip())
except Exception as e:
logger.info(f'Failed to parse GitLab projects response: {e}')
if self.limit is not None and records_seen >= self.limit:
report = self._combine_reports(report, SourceExecutionReport('completed', 'result-limit'))
break
if next_page is None:
break
if next_page in seen_cursors or next_page == str(page):
report = self._combine_reports(report, SourceExecutionReport('partial', 'repeated-cursor'))
break
seen_cursors.add(next_page)
try:
page = int(next_page)
except ValueError:
report = self._combine_reports(report, SourceExecutionReport('failed', 'invalid-response'))
break
return report
except Exception as e:
logger.info(f'GitLab API projects search error: {e}')
return SourceExecutionReport('failed', 'transport-error')
async def search_users(self) -> None:
async def search_users(self) -> SourceExecutionReport | None:
"""Search GitLab users for references to the target domain."""
try:
headers = {'User-agent': Core.get_user_agent()}
# Search for users mentioning our domain
users_url = f'{self.hostname}/api/v4/users?search={self.word}&per_page=10'
response = await AsyncFetcher.fetch_all([users_url], headers=headers, proxy=self.proxy)
if not response or not isinstance(response, list) or not response[0]:
return
try:
users = self._safe_parse_json(response[0])
if not isinstance(users, list):
return
for user in users:
page = 1
records_seen = 0
seen_pages: set[str] = set()
seen_cursors: set[str] = set()
while self.limit is None or records_seen < self.limit:
per_page = min(100, self.limit - records_seen) if self.limit is not None else 100
users, next_page, report = await self._fetch_page('users', self.word, page, per_page)
if report is not None:
return report
signature = json.dumps(users, sort_keys=True, default=str)
if signature in seen_pages:
return SourceExecutionReport('partial', 'repeated-page')
seen_pages.add(signature)
accepted = users[: self.limit - records_seen] if self.limit is not None else users
records_seen += len(accepted)
for user in accepted:
if not isinstance(user, dict):
continue
# Extract information from user metadata
name = user.get('name', '') or ''
username = user.get('username', '') or ''
bio = user.get('bio', '') or ''
@@ -155,13 +212,10 @@ class SearchGitlab:
website_url = user.get('website_url', '') or ''
public_email = user.get('public_email', '') or ''
# Look for domains in user info
user_hosts = self._extract_domains_from_text(f'{name} {username} {bio}')
website_hosts = self._extract_domains_from_text(website_url) if isinstance(website_url, str) else set()
user_hosts.update(website_hosts)
self.totalhosts.update(user_hosts)
# Check email
user_emails: set[str] = set()
if public_email:
user_emails = self._extract_emails_from_text(public_email)
@@ -173,16 +227,26 @@ class SearchGitlab:
if user_is_relevant and isinstance(web_url, str) and web_url.strip():
self.totalurls.add(web_url.strip())
except Exception as e:
logger.info(f'Failed to parse GitLab users response: {e}')
if self.limit is not None and records_seen >= self.limit:
return SourceExecutionReport('completed', 'result-limit')
if next_page is None:
return None
if next_page in seen_cursors or next_page == str(page):
return SourceExecutionReport('partial', 'repeated-cursor')
seen_cursors.add(next_page)
try:
page = int(next_page)
except ValueError:
return SourceExecutionReport('failed', 'invalid-response')
return None
except Exception as e:
logger.info(f'GitLab API users search error: {e}')
return SourceExecutionReport('failed', 'transport-error')
async def do_search(self) -> None:
await self.search_projects()
await self.search_users()
async def do_search(self) -> SourceExecutionReport | None:
project_report = await self.search_projects()
user_report = await self.search_users()
return self._combine_reports(project_report, user_report)
async def get_hostnames(self) -> set:
return self.totalhosts
@@ -193,6 +257,6 @@ class SearchGitlab:
async def get_urls(self) -> set:
return self.totalurls
async def process(self, proxy: bool = False) -> None:
async def process(self, proxy: bool = False) -> SourceExecutionReport | None:
self.proxy = proxy
await self.do_search()
return await self.do_search()
theHarvester/discovery/huntersearch.py
+27 -13
View File
@@ -3,15 +3,16 @@ import logging
from theHarvester.discovery.constants import MissingKey
from theHarvester.lib.core import AsyncFetcher, Core, FetcherResponse
from theHarvester.lib.source_execution import SourceExecutionReport
logger = logging.getLogger(__name__)
class SearchHunter:
def __init__(self, word, limit, start) -> None:
def __init__(self, word, limit: int | None, start) -> None:
self.word = word
self.requested_limit = limit
self.limit = min(limit, 10)
self.limit = min(limit, 10) if limit is not None else 10
self.start = start
key = Core.hunter_key()
self.key = key.strip() if key else ''
@@ -46,14 +47,14 @@ class SearchHunter:
return None
return metadata.body
async def do_search(self) -> None:
async def do_search(self) -> SourceExecutionReport | None:
# First determine if a user account is not a free account, this call is free
is_free = True
headers = {'User-Agent': Core.get_user_agent()}
acc_info_url = f'https://api.hunter.io/v2/account?api_key={self.key}'
response = await self._fetch_json(acc_info_url, headers)
if response is None:
return
return None
is_free = is_free if 'plan_name' in response['data'].keys() and response['data']['plan_name'].lower() == 'free' else False
# Extract the total number of requests that are available for an account
@@ -64,6 +65,13 @@ class SearchHunter:
response = await self._fetch_json(self.database, headers)
if response is not None:
self.emails, self.hostnames = await self.parse_resp(json_resp=response)
entries = response.get('data', {}).get('emails', [])
if (
isinstance(entries, list)
and len(entries) >= self.limit
and (self.requested_limit is None or self.requested_limit > self.limit)
):
return SourceExecutionReport('partial', 'provider-limit')
else:
# Determine the total number of emails that are available
# As the most emails you can get within one query are 100
@@ -71,31 +79,36 @@ class SearchHunter:
hunter_dinfo_url = f'https://api.hunter.io/v2/email-count?domain={self.word}'
response = await self._fetch_json(hunter_dinfo_url, headers)
if response is None:
return
total_results = min(max(0, response['data']['total'] - self.start), self.requested_limit)
return None
available_results = max(0, response['data']['total'] - self.start)
total_results = (
min(available_results, self.requested_limit) if self.requested_limit is not None else available_results
)
total_number_reqs = (total_results + 99) // 100
# Parse out meta field within initial JSON response to determine the total number of results
if total_requests_avail < total_number_reqs:
quota_exhausted = total_requests_avail < total_number_reqs
if quota_exhausted:
logger.info('WARNING: account does not have enough requests to gather all emails')
logger.info(
f'Total requests available: {total_requests_avail}, total requests needed to be made: {total_number_reqs}'
)
logger.info('RETURNING current results, if you would still like to run this module comment out the if request')
return
# max number of emails you can get per request is 100
# increments of 100 with offset determining where to start
# See docs for more details: https://hunter.io/api-documentation/v2#domain-search
result_end = self.start + total_results
result_end = self.start + min(total_results, max(total_requests_avail, 0) * 100)
for offset in range(self.start, result_end, 100):
page_limit = min(100, result_end - offset)
req_url = f'https://api.hunter.io/v2/domain-search?domain={self.word}&api_key={self.key}&limit={page_limit}&offset={offset}'
response = await self._fetch_json(req_url, headers)
if response is None:
return
return None
temp_emails, temp_hostnames = await self.parse_resp(response)
self.emails.extend(temp_emails)
self.hostnames.extend(temp_hostnames)
await asyncio.sleep(1)
if quota_exhausted:
return SourceExecutionReport('partial', 'quota-exhausted')
return None
async def parse_resp(self, json_resp):
emails = list(sorted({email['value'] for email in json_resp['data']['emails']}))
@@ -111,12 +124,13 @@ class SearchHunter:
)
return emails, domains
async def process(self, proxy: bool = False) -> None:
async def process(self, proxy: bool = False) -> SourceExecutionReport | None:
self.proxy = proxy
try:
await self.do_search() # Only need to do it once.
return await self.do_search() # Only need to do it once.
except AttributeError, KeyError, TypeError:
logger.info('Hunter returned malformed data')
return SourceExecutionReport('failed', 'invalid-response')
async def get_emails(self):
return self.emails
theHarvester/discovery/intelxsearch.py
+100 -52
View File
@@ -1,8 +1,6 @@
import asyncio
import logging
from email.errors import HeaderParseError
from email.headerregistry import Address
from typing import Any
from urllib.parse import urlparse
import aiohttp
@@ -10,79 +8,130 @@ import aiohttp
from theHarvester.discovery.constants import MissingKey
from theHarvester.lib.core import Core
from theHarvester.lib.hostnames import normalize_scoped_hostname
from theHarvester.lib.source_execution import SourceExecutionReport
from theHarvester.parsers import intelxparser
logger = logging.getLogger(__name__)
class SearchIntelx:
"""Search the Intelligence X Phonebook API.
"""Search the Intelligence X Phonebook API."""
API documentation: https://github.com/IntelligenceX/SDK
"""
PAGE_SIZE = 1000
UNLIMITED_QUERY_RESULTS = 2**31 - 1
MAX_PENDING_POLLS = 30
MAX_RUNTIME_SECONDS = 60.0
def __init__(self, word) -> None:
def __init__(self, word: str, limit: int | None = None) -> None:
if limit is not None and (isinstance(limit, bool) or not isinstance(limit, int) or limit <= 0):
raise ValueError('IntelX limit must be a positive integer or None')
self.word = word
self.key = Core.intelx_key()
if not isinstance(self.key, str) or not self.key.strip():
raise MissingKey('Intelx')
self.database = 'https://2.intelx.io'
self.results: dict[str, Any] = {}
self.results: dict[str, list[object]] = {'selectors': []}
self.emails: list[str] = []
self.hostnames: list[str] = []
self.urls: list[str] = []
self.limit: int = 10000
self.limit = limit
self.proxy = False
self.offset = 0
async def do_search(self) -> None:
async def do_search(self) -> SourceExecutionReport | None:
headers = {'x-key': self.key, 'User-Agent': Core.get_user_agent(), 'Content-Type': 'application/json'}
search_limit = self.limit if self.limit is not None else self.UNLIMITED_QUERY_RESULTS
data = {
'term': self.word,
'buckets': [],
'lookuplevel': 0,
'maxresults': search_limit,
'timeout': 5,
'datefrom': '',
'dateto': '',
'sort': 4,
'media': 0,
'terminate': [],
'target': 0,
}
collected = 0
pending_polls = 0
try:
headers = {
'x-key': self.key,
'User-Agent': Core.get_user_agent(),
'Content-Type': 'application/json',
}
data = {
'term': self.word,
'buckets': [],
'lookuplevel': 0,
'maxresults': self.limit,
'timeout': 5,
'datefrom': '',
'dateto': '',
'sort': 4, # Sort by date descending for faster relevant results
'media': 0,
'terminate': [],
'target': 0,
}
async with aiohttp.ClientSession() as session:
async with session.post(f'{self.database}/phonebook/search', headers=headers, json=data) as total_resp:
search_data = await total_resp.json()
if not search_data['success']:
logger.info('IntelX search request failed')
return
async with asyncio.timeout(self.MAX_RUNTIME_SECONDS):
async with aiohttp.ClientSession() as session:
async with session.post(f'{self.database}/phonebook/search', headers=headers, json=data) as response:
if response.status in {401, 403}:
return SourceExecutionReport('failed', 'access-denied')
if response.status == 429:
return SourceExecutionReport('rate-limited', 'http-429')
if not 200 <= response.status < 300:
return SourceExecutionReport('failed', f'http-{response.status}')
search_data = await response.json()
if (
not isinstance(search_data, dict)
or search_data.get('success') is False
or not isinstance(search_data.get('id'), str)
or not search_data['id']
):
return SourceExecutionReport('failed', 'invalid-response')
phonebook_id = search_data['id']
while self.limit is None or collected < self.limit:
page_size = min(self.PAGE_SIZE, self.limit - collected) if self.limit is not None else self.PAGE_SIZE
async with session.get(
f'{self.database}/phonebook/search/result',
headers=headers,
params={'id': phonebook_id, 'limit': page_size},
) as response:
if response.status in {401, 403}:
return SourceExecutionReport('failed', 'access-denied')
if response.status == 429:
return SourceExecutionReport('rate-limited', 'http-429')
if not 200 <= response.status < 300:
return SourceExecutionReport('failed', f'http-{response.status}')
page = await response.json()
if (
not isinstance(page, dict)
or isinstance(page.get('status'), bool)
or not isinstance(page.get('status'), int)
):
return SourceExecutionReport('failed', 'invalid-response')
status = page['status']
if status == 2:
return SourceExecutionReport('failed', 'search-not-found')
if status == 4:
return SourceExecutionReport('failed', 'provider-error')
if status not in {0, 1, 3}:
return SourceExecutionReport('failed', 'invalid-response')
selectors = page.get('selectors', [])
if not isinstance(selectors, list):
return SourceExecutionReport('failed', 'invalid-response')
if status == 3:
pending_polls += 1
if pending_polls >= self.MAX_PENDING_POLLS:
return SourceExecutionReport('partial', 'runtime-limit')
await asyncio.sleep(1)
continue
if status == 0 and not selectors:
return SourceExecutionReport('failed', 'invalid-response')
pending_polls = 0
retained = selectors[:page_size]
self.results['selectors'].extend(retained)
collected += len(retained)
if status == 1 or (self.limit is not None and collected >= self.limit):
return None
except TimeoutError:
return SourceExecutionReport('partial', 'runtime-limit')
except asyncio.CancelledError:
raise
except aiohttp.ClientError, OSError:
return SourceExecutionReport('failed', 'transport-error')
return None
await asyncio.sleep(2) # Reduced sleep time as 5s is excessive
async with session.get(
f'{self.database}/phonebook/search/result?id={phonebook_id}&limit={self.limit}&offset={self.offset}',
headers=headers,
) as resp:
self.results = await resp.json()
except Exception as e:
logger.info(f'An exception has occurred in Intelx: {e}')
async def process(self, proxy: bool = False):
async def process(self, proxy: bool = False) -> SourceExecutionReport | None:
self.proxy = proxy
await self.do_search()
report = await self.do_search()
intelx_parser = intelxparser.Parser()
raw_emails, raw_selectors = await intelx_parser.parse_dictionaries(self.results)
emails: set[str] = set()
urls: set[str] = set()
hostnames: set[str] = set()
for email in raw_emails:
if email.count('@') != 1:
continue
@@ -92,7 +141,6 @@ class SearchIntelx:
continue
if address.username and (normalized_domain := normalize_scoped_hostname(address.domain, self.word)):
emails.add(f'{address.username}@{normalized_domain}')
for selector in raw_selectors:
selector = selector.strip()
try:
@@ -104,10 +152,10 @@ class SearchIntelx:
hostnames.add(normalized_hostname)
if parsed.scheme in {'http', 'https'} and parsed.netloc:
urls.add(selector)
self.emails = sorted(emails)
self.urls = sorted(urls)
self.hostnames = sorted(hostnames)
return report
async def get_emails(self) -> list[str]:
return self.emails
theHarvester/discovery/mojeek.py
+92 -43
View File
@@ -11,7 +11,7 @@ logger = logging.getLogger(__name__)
class SearchMojeek:
REQUEST_DELAY_SECONDS = 1.0
def __init__(self, word, limit) -> None:
def __init__(self, word, limit: int | None) -> None:
self.word = word
self.limit = limit
self.total_results = ''
@@ -31,12 +31,80 @@ class SearchMojeek:
logger.info('[*] Mojeek: No API key found, using default scraping mode.')
def _stop(self, status: SourceReportStatus, reason: str) -> None:
self._report = SourceExecutionReport(status, reason)
self._report = SourceExecutionReport('partial' if self.total_results else status, reason)
def _api_page_results(self, response: FetcherResponse) -> list[str] | None:
if response.status == 403:
self._stop('failed', 'access-denied')
return None
if response.status == 429:
self._stop('rate-limited', 'http-429')
return None
if not 200 <= response.status < 300:
self._stop('failed', f'http-{response.status}')
return None
data = response.body.get('response', response.body) if isinstance(response.body, dict) else None
if not isinstance(data, dict):
self._stop('failed', 'invalid-response')
return None
status = data.get('status')
if isinstance(status, str) and 'denied' in status.casefold():
self._stop('failed', 'access-denied')
return None
if 'status' in data and not isinstance(status, str):
self._stop('failed', 'invalid-response')
return None
results = data.get('results')
if not isinstance(results, list):
self._stop('failed', 'invalid-response')
return None
parsed_results: list[str] = []
for result in results:
if not isinstance(result, dict):
self._stop('failed', 'invalid-response')
return None
url_value = result.get('url')
title_value = result.get('title')
description_value = result.get('desc')
url = url_value.replace('\\/', '/') if isinstance(url_value, str) else ''
title = title_value if isinstance(title_value, str) else ''
description = description_value if isinstance(description_value, str) else ''
if not any((url, title, description)):
self._stop('failed', 'invalid-response')
return None
parsed_results.append(f'{url} {title} {description}')
return parsed_results
async def _search_api(self, headers: dict[str, str]) -> None:
if self.limit is None:
seen_pages: set[tuple[str, ...]] = set()
offset = 1
while True:
url = f'https://{self.api_server}/search?api_key={self.api_key}&q={self.word}&fmt=json&s={offset}'
responses = await AsyncFetcher.fetch_all(
[url], headers=headers, proxy=self.proxy, json=True, include_metadata=True
)
if len(responses) != 1 or not isinstance(responses[0], FetcherResponse):
self._stop('failed', 'transport-error')
return
parsed_results = self._api_page_results(responses[0])
if parsed_results is None:
return
if not parsed_results:
return
signature = tuple(parsed_results)
if signature in seen_pages:
self._stop('partial', 'repeated-page')
return
seen_pages.add(signature)
self.total_results += f' {" ".join(parsed_results)} '
offset += 10
return
result_limit = self.limit
urls = [
f'https://{self.api_server}/search?api_key={self.api_key}&q={self.word}&fmt=json&s={num}'
for num in range(1, self.limit, 10)
for num in range(1, result_limit, 10)
]
responses = await AsyncFetcher.fetch_all(
urls,
@@ -45,55 +113,30 @@ class SearchMojeek:
json=True,
include_metadata=True,
)
seen_finite_pages: set[tuple[str, ...]] = set()
for response in responses:
if not isinstance(response, FetcherResponse):
self._stop('failed', 'transport-error')
return
if response.status == 403:
self._stop('failed', 'access-denied')
parsed_results = self._api_page_results(response)
if parsed_results is None:
return
if response.status == 429:
self._stop('rate-limited', 'http-429')
if not parsed_results:
break
signature = tuple(parsed_results)
if signature in seen_finite_pages:
self._stop('partial', 'repeated-page')
return
if not 200 <= response.status < 300:
self._stop('failed', f'http-{response.status}')
return
data = response.body.get('response', response.body) if isinstance(response.body, dict) else None
if not isinstance(data, dict):
self._stop('failed', 'invalid-response')
return
status = data.get('status')
if isinstance(status, str) and 'denied' in status.casefold():
self._stop('failed', 'access-denied')
return
if 'status' in data and not isinstance(status, str):
self._stop('failed', 'invalid-response')
return
results = data.get('results')
if not isinstance(results, list):
self._stop('failed', 'invalid-response')
return
for result in results:
if not isinstance(result, dict):
self._stop('failed', 'invalid-response')
return
url_value = result.get('url')
title_value = result.get('title')
description_value = result.get('desc')
url = url_value.replace('\\/', '/') if isinstance(url_value, str) else ''
title = title_value if isinstance(title_value, str) else ''
description = description_value if isinstance(description_value, str) else ''
if not any((url, title, description)):
self._stop('failed', 'invalid-response')
return
self.total_results += f' {url} {title} {description} '
seen_finite_pages.add(signature)
self.total_results += f' {" ".join(parsed_results)} '
logger.info('[*] Mojeek: API search completed successfully.')
async def _search_keyless(self, headers: dict[str, str]) -> None:
urls = [f'https://{self.server}/search?q={self.word}&s={num}' for num in range(0, self.limit, 10)]
for page, url in enumerate(urls):
seen_bodies: set[str] = set()
offset = 0
page = 0
while self.limit is None or offset < self.limit:
url = f'https://{self.server}/search?q={self.word}&s={offset}'
if page:
await asyncio.sleep(self.REQUEST_DELAY_SECONDS)
response = await AsyncFetcher.fetch(
@@ -131,7 +174,13 @@ class SearchMojeek:
if 'results-standard' not in normalized_body:
self._stop('failed', 'invalid-response')
return
if response.body in seen_bodies:
self._stop('partial', 'repeated-page')
return
seen_bodies.add(response.body)
self.total_results += f' {response.body}'
offset += 10
page += 1
async def do_search(self) -> SourceExecutionReport | None:
self._report = None
theHarvester/discovery/netlas.py
+44 -5
View File
@@ -1,5 +1,6 @@
from __future__ import annotations
import math
from typing import Any
from theHarvester.discovery.constants import MissingKey
@@ -10,10 +11,14 @@ from theHarvester.lib.source_execution import SourceExecutionReport
class SearchNetlas:
def __init__(self, word: str, limit: int) -> None:
if isinstance(limit, bool) or not isinstance(limit, int) or limit <= 0:
raise ValueError('Netlas limit must be a positive integer')
COUNT_URL = 'https://app.netlas.io/api/domains_count/'
DOWNLOAD_URL = 'https://app.netlas.io/api/domains/download/'
def __init__(self, word: str, limit: int | None) -> None:
if limit is not None and (isinstance(limit, bool) or not isinstance(limit, int) or limit <= 0):
raise ValueError('Netlas limit must be a positive integer or None')
self.word = word
self.unlimited = limit is None
self.limit = limit
self.totalhosts: set[str] = set()
self.key = Core.netlas_key()
@@ -32,14 +37,35 @@ class SearchNetlas:
return None, SourceExecutionReport('failed', 'invalid-response')
return response.body, None
def _query(self) -> str:
return f'*.{self.word}'
async def _count(self, session: Any) -> tuple[int | None, SourceExecutionReport | None]:
response = await AsyncFetcher.fetch(
session=session,
url=self.COUNT_URL,
params={'q': self._query()},
json=True,
include_metadata=True,
)
body, report = self._response_body(response)
if report is not None:
return None, report
if not isinstance(body, dict):
return None, SourceExecutionReport('failed', 'invalid-response')
count = body.get('count')
if isinstance(count, bool) or not isinstance(count, int) or count < 0:
return None, SourceExecutionReport('failed', 'invalid-response')
return count, None
async def do_search(self, session: Any, size: int) -> SourceExecutionReport | None:
response = await AsyncFetcher.post_fetch(
'https://app.netlas.io/api/domains/download/',
self.DOWNLOAD_URL,
session=session,
json=True,
include_metadata=True,
json_body={
'q': f'*.{self.word}',
'q': self._query(),
'size': size,
'fields': ['domain'],
'source_type': 'include',
@@ -76,6 +102,19 @@ class SearchNetlas:
headers={'Authorization': f'Bearer {self.key}'},
proxy=proxy,
) as session:
if self.unlimited:
size, report = await self._count(session)
if report is not None:
return report
if size is None:
return SourceExecutionReport('failed', 'invalid-response')
if size == 0:
return None
# Netlas documents counts above 1,000 as estimates within 3%.
if size > 1000:
size = math.ceil(size / 0.97)
return await self.do_search(session, size)
assert self.limit is not None
return await self.do_search(session, self.limit)
except Exception:
return SourceExecutionReport('failed', 'transport-error')
theHarvester/discovery/onyphe.py
+13 -6
View File
@@ -26,9 +26,9 @@ class SearchOnyphe:
MAX_RESULTS = 10_000
def __init__(self, word: str, limit: int) -> None:
if isinstance(limit, bool) or not isinstance(limit, int) or limit <= 0:
raise ValueError('ONYPHE limit must be a positive integer')
def __init__(self, word: str, limit: int | None) -> None:
if limit is not None and (isinstance(limit, bool) or not isinstance(limit, int) or limit <= 0):
raise ValueError('ONYPHE limit must be a positive integer or None')
self.word = word
self.limit = limit
self.response: object = {}
@@ -50,7 +50,7 @@ class SearchOnyphe:
}
page = 1
records_seen = 0
result_limit = min(self.limit, self.MAX_RESULTS)
result_limit = min(self.limit, self.MAX_RESULTS) if self.limit is not None else self.MAX_RESULTS
page_size = min(result_limit, self.MAX_RESULTS)
last_total = 0
report = None
@@ -104,8 +104,15 @@ class SearchOnyphe:
logger.info('Onyphe request failed: %s', type(error).__name__)
return SourceExecutionReport('failed', 'transport-error')
if self.limit > self.MAX_RESULTS and last_total > self.MAX_RESULTS and records_seen >= self.MAX_RESULTS:
return SourceExecutionReport('failed', 'provider-limit')
if self.limit is None and last_total > self.MAX_RESULTS and records_seen >= self.MAX_RESULTS:
return SourceExecutionReport('partial', 'provider-limit')
if (
self.limit is not None
and self.limit > self.MAX_RESULTS
and last_total > self.MAX_RESULTS
and records_seen >= self.MAX_RESULTS
):
return SourceExecutionReport('partial', 'provider-limit')
return report
async def parse_onyphe_resp_json(self) -> bool:
theHarvester/discovery/pentesttools.py
+24 -15
View File
@@ -5,6 +5,7 @@ from ipaddress import ip_address
from theHarvester.discovery.constants import MissingKey
from theHarvester.lib.core import AsyncFetcher, Core
from theHarvester.lib.hostnames import normalize_scoped_hostname
from theHarvester.lib.source_execution import SourceExecutionReport
logger = logging.getLogger(__name__)
@@ -32,7 +33,7 @@ class SearchPentestTools:
return None
return data
async def poll(self, scan_id: int) -> None:
async def poll(self, scan_id: int) -> SourceExecutionReport | None:
for _attempt in range(10):
await asyncio.sleep(3)
status = self._response_data(
@@ -44,11 +45,11 @@ class SearchPentestTools:
)
)
if status is None:
return
return SourceExecutionReport('failed', 'invalid-response')
status_name = status.get('status_name')
if not isinstance(status_name, str):
logger.info('Pentest-Tools returned a malformed status response')
return
return SourceExecutionReport('failed', 'invalid-response')
if status_name in {'waiting', 'running'}:
continue
if status_name == 'finished':
@@ -61,21 +62,23 @@ class SearchPentestTools:
)
)
if output is not None:
await self.parse_json(output)
return await self.parse_json(output)
return SourceExecutionReport('failed', 'invalid-response')
else:
logger.info('Pentest-Tools scan did not finish successfully')
return
return SourceExecutionReport('failed', 'provider-error')
logger.info('Pentest-Tools scan is still waiting after 10 status checks')
return SourceExecutionReport('partial', 'runtime-limit')
async def parse_json(self, json_results) -> None:
async def parse_json(self, json_results) -> SourceExecutionReport | None:
if json_results.get('output_type') != 'subdomain_list':
return
return SourceExecutionReport('failed', 'invalid-response')
try:
output_data = json_results['output_data']['subdomains']
except KeyError, TypeError:
return
return SourceExecutionReport('failed', 'invalid-response')
if not isinstance(output_data, list):
return
return SourceExecutionReport('failed', 'invalid-response')
for result in output_data:
if not isinstance(result, dict):
continue
@@ -92,6 +95,7 @@ class SearchPentestTools:
except ValueError:
continue
self.totalips.add(address)
return None
async def get_hostnames(self) -> set[str]:
return self.totalhosts
@@ -99,7 +103,7 @@ class SearchPentestTools:
async def get_ips(self) -> set[str]:
return self.totalips
async def do_search(self) -> None:
async def do_search(self) -> SourceExecutionReport | None:
# Pentest-Tools documents Subdomain Finder as tool 20:
# https://pentest-tools.com/docs/api-reference/scans/start-a-scan
subdomain_payload = {
@@ -121,13 +125,18 @@ class SearchPentestTools:
)
)
if response is None:
return
return SourceExecutionReport('failed', 'invalid-response')
scan_id = response.get('created_id')
if not isinstance(scan_id, int) or isinstance(scan_id, bool):
logger.info('Pentest-Tools returned a malformed start response')
return
await self.poll(scan_id)
return SourceExecutionReport('failed', 'invalid-response')
return await self.poll(scan_id)
async def process(self, proxy: bool = False) -> None:
async def process(self, proxy: bool = False) -> SourceExecutionReport | None:
self.proxy = proxy
await self.do_search() # Only need to do it once.
try:
return await self.do_search() # Only need to do it once.
except asyncio.CancelledError:
raise
except Exception:
return SourceExecutionReport('failed', 'transport-error')
theHarvester/discovery/rocketreach.py
+6 -5
View File
@@ -8,7 +8,7 @@ logger = logging.getLogger(__name__)
class SearchRocketReach:
def __init__(self, word, limit) -> None:
def __init__(self, word, limit: int | None) -> None:
self.ips: set = set()
self.word = word
self.key = Core.rocketreach_key()
@@ -23,7 +23,7 @@ class SearchRocketReach:
async def do_search(self) -> None:
try:
if self.limit <= 0:
if self.limit is not None and self.limit <= 0:
return
headers = {
@@ -34,8 +34,8 @@ class SearchRocketReach:
start = 0
remaining = self.limit
while remaining > 0:
page_size = min(100, remaining)
while remaining is None or remaining > 0:
page_size = min(100, remaining) if remaining is not None else 100
data = {
'query': {'current_employer_domain': [self.word]},
'start': start,
@@ -71,7 +71,8 @@ class SearchRocketReach:
self.emails.add(email['email'])
found = len(profiles)
remaining -= found
if remaining is not None:
remaining -= found
start += found
pagination = result.get('pagination', {})
theHarvester/discovery/search_dehashed.py
+7 -6
View File
@@ -10,7 +10,7 @@ logger = logging.getLogger(__name__)
class SearchDehashed:
def __init__(self, word: str, limit: int = 500) -> None:
def __init__(self, word: str, limit: int | None = 500) -> None:
self.word = word
self.key = (Core.dehashed_key() or '').strip()
if not self.key:
@@ -20,7 +20,7 @@ class SearchDehashed:
'Dehashed-Api-Key': self.key,
'User-Agent': Core.get_user_agent(),
}
self.limit = max(limit, 0)
self.limit = max(limit, 0) if limit is not None else None
self.emails: set[str] = set()
self.ips: set[str] = set()
self.proxy: bool = False
@@ -72,8 +72,8 @@ class SearchDehashed:
logger.info(f'\t[+] Performing Dehashed search for: {self.word}')
page = 1
remaining = self.limit
while remaining > 0:
size = min(100, remaining)
while remaining is None or remaining > 0:
size = min(100, remaining) if remaining is not None else 100
payload = {'query': self.word, 'page': page, 'size': size, 'wildcard': False, 'regex': False, 'de_dupe': False}
try:
response = await self._fetch_page(payload)
@@ -89,9 +89,10 @@ class SearchDehashed:
break
if not entries:
break
retained_entries = entries[:remaining]
retained_entries = entries[:remaining] if remaining is not None else entries
self._retain_evidence(retained_entries)
remaining -= len(retained_entries)
if remaining is not None:
remaining -= len(retained_entries)
logger.info(f'\t[+] Page {page} - Retrieved {len(retained_entries)} entries.')
if len(entries) < size:
break
theHarvester/discovery/searchhunterhow.py
+9 -8
View File
@@ -1,6 +1,6 @@
import asyncio
import base64
from datetime import UTC, datetime
from datetime import UTC, date, datetime
from typing import Any
from dateutil.relativedelta import relativedelta
@@ -14,9 +14,10 @@ from theHarvester.lib.source_execution import SourceExecutionReport
class SearchHunterHow:
REQUEST_DELAY_SECONDS = 2.0
ALL_HISTORY_START = date(1970, 1, 1)
def __init__(self, word: str, limit: int = 500) -> None:
if isinstance(limit, bool) or not isinstance(limit, int) or limit <= 0:
def __init__(self, word: str, limit: int | None = 500) -> None:
if limit is not None and (isinstance(limit, bool) or not isinstance(limit, int) or limit <= 0):
raise ValueError('Hunter.how limit must be a positive integer')
self.word = word
self.limit = limit
@@ -36,7 +37,7 @@ class SearchHunterHow:
async def do_search(self) -> SourceExecutionReport | None:
query = base64.urlsafe_b64encode(f'domain.suffix="{self.word}"'.encode()).decode('ascii')
end = datetime.now(UTC).date()
start = end - relativedelta(days=364)
start = self.ALL_HISTORY_START if self.limit is None else end - relativedelta(days=364)
page = 1
returned = 0
params: dict[str, Any] = {
@@ -52,11 +53,11 @@ class SearchHunterHow:
headers={'User-Agent': Core.get_user_agent()},
proxy=self.proxy,
) as session:
while returned < self.limit:
while self.limit is None or returned < self.limit:
request_params = {
**params,
'page': page,
'page_size': self._page_size(self.limit - returned),
'page_size': self._page_size(self.limit - returned) if self.limit is not None else 1000,
}
response = await AsyncFetcher.fetch(
session=session,
@@ -82,7 +83,7 @@ class SearchHunterHow:
if isinstance(total, bool) or not isinstance(total, int) or total < 0 or not isinstance(rows, list):
return SourceExecutionReport('failed', 'invalid-response')
remaining = self.limit - returned
remaining = self.limit - returned if self.limit is not None else len(rows)
malformed = False
for row in rows[:remaining]:
if not isinstance(row, dict) or not isinstance(row.get('domain'), str):
@@ -94,7 +95,7 @@ class SearchHunterHow:
report = SourceExecutionReport('failed', 'invalid-response')
returned += len(rows)
if not rows or returned >= min(total, self.limit):
if not rows or returned >= (min(total, self.limit) if self.limit is not None else total):
break
page += 1
await asyncio.sleep(self.REQUEST_DELAY_SECONDS)
theHarvester/discovery/securityscorecard.py
+13 -6
View File
@@ -1,5 +1,6 @@
from __future__ import annotations
import json
from ipaddress import ip_address
from typing import Any
@@ -13,8 +14,8 @@ from theHarvester.lib.source_execution import SourceExecutionReport, SourceRepor
class SearchSecurityScorecard:
PAGE_SIZE = 50
def __init__(self, word: str, limit: int) -> None:
if isinstance(limit, bool) or not isinstance(limit, int) or limit <= 0:
def __init__(self, word: str, limit: int | None) -> None:
if limit is not None and (isinstance(limit, bool) or not isinstance(limit, int) or limit <= 0):
raise ValueError('SecurityScorecard limit must be a positive integer')
self.word = word
self.limit = limit
@@ -75,8 +76,9 @@ class SearchSecurityScorecard:
async def _collect_assets(self, session: Any, route: str, field: str) -> bool:
page = 0
records_seen = 0
page_size = min(self.PAGE_SIZE, self.limit)
while records_seen < self.limit:
seen_pages: set[str] = set()
page_size = min(self.PAGE_SIZE, self.limit) if self.limit is not None else self.PAGE_SIZE
while self.limit is None or records_seen < self.limit:
response = await AsyncFetcher.post_fetch(
f'{self.base_url}/parent-domains/{self.word}/{route}',
session=session,
@@ -92,8 +94,13 @@ class SearchSecurityScorecard:
if not isinstance(entries, list) or isinstance(size, bool) or not isinstance(size, int | float) or size < 0:
self._stop('failed', 'invalid-response')
return False
signature = json.dumps(entries, sort_keys=True, separators=(',', ':'))
if signature in seen_pages:
self._stop('partial' if self.hosts or self.ips else 'failed', 'repeated-page')
return False
seen_pages.add(signature)
remaining = self.limit - records_seen
remaining = self.limit - records_seen if self.limit is not None else len(entries)
page_entries = entries[:remaining]
records_seen += len(page_entries)
malformed = False
@@ -112,7 +119,7 @@ class SearchSecurityScorecard:
malformed = True
if malformed:
self._stop('failed', 'invalid-response')
if records_seen >= self.limit or len(entries) < page_size:
if (self.limit is not None and records_seen >= self.limit) or len(entries) < page_size:
return True
page += 1
return True
theHarvester/discovery/shodansearch.py
+13 -2
View File
@@ -1,4 +1,5 @@
import asyncio
import json
import logging
import re
import socket
@@ -15,7 +16,7 @@ from theHarvester.lib.core import AsyncFetcher, Core, FetcherResponse, ResponseS
from theHarvester.lib.hostchecker import resolve_ip_addresses
from theHarvester.lib.hostnames import normalize_scoped_hostname
from theHarvester.lib.shodan_evidence import ShodanHostObservation, canonical_shodan_hosts
from theHarvester.lib.source_execution import SourceExecutionReport
from theHarvester.lib.source_execution import SourceExecutionReport, SourceReportStatus
logger = logging.getLogger(__name__)
@@ -375,6 +376,7 @@ class SearchShodan:
for query in (f'hostname:{self.scope}', f'ssl:{self.scope}'):
page = 1
received = 0
seen_pages: set[str] = set()
while True:
try:
response = await self._fetch_json(
@@ -411,6 +413,11 @@ class SearchShodan:
if not isinstance(matches, list) or isinstance(total, bool) or not isinstance(total, int) or total < 0:
error_types.add('InvalidResponseError')
break
signature = json.dumps(matches, sort_keys=True, separators=(',', ':'))
if signature in seen_pages:
error_types.add('RepeatedPageError')
break
seen_pages.add(signature)
for match in matches:
if not isinstance(match, dict):
error_types.add('InvalidResponseError')
@@ -518,8 +525,12 @@ class SearchShodan:
self.error_type = next(iter(sorted(provider_error_types)), None)
if dns_stop_reason is not None:
return SourceExecutionReport('failed', dns_stop_reason)
status: SourceReportStatus = 'partial' if self.shodan_hosts or self.totalhosts else 'failed'
return SourceExecutionReport(status, dns_stop_reason)
if provider_error_types:
if 'RepeatedPageError' in provider_error_types:
status = 'partial' if self.shodan_hosts or self.totalhosts else 'failed'
return SourceExecutionReport(status, 'repeated-page')
if provider_error_types <= {'HTTP401Error', 'HTTP403Error'}:
return SourceExecutionReport('failed', 'access-denied')
if provider_error_types == {'HTTP429Error'}:
theHarvester/discovery/sourcegraph.py
+13 -18
View File
@@ -74,29 +74,29 @@ def _parse_event(record: str) -> tuple[str, Any]:
class SearchSourcegraph:
"""Collect descendant-hostname candidates mentioned in Sourcegraph code.
One query to Sourcegraph requests up to 5,000 matches; ``--limit`` does not
change it, and this source never contacts the target. A code mention does not
prove ownership, scope, or liveness. Repository and shard limits, along with
unstable result ordering, can make the results partial and non-exhaustive.
One query to Sourcegraph requests up to 5,000 matches; ``--limit`` caps
emitted hostnames but does not change that provider query. This source never
contacts the target. A code mention does not prove ownership, scope, or
liveness. Repository and shard limits, along with unstable result ordering,
can make the results partial and non-exhaustive.
"""
ENDPOINT = 'https://sourcegraph.com/.api/search/stream'
MATCH_COUNT = 5000
MAX_EVENTS = 10_000
MAX_HOSTNAMES = 10_000
MAX_LINE_LENGTH = 4096
def __init__(self, word: str, limit: int) -> None:
del limit # Sourcegraph uses one fixed provider query; global --limit is unrelated.
def __init__(self, word: str, limit: int | None) -> None:
self.word = _normalize_hostname(word) or ''
if '.' not in self.word:
self.word = ''
self.limit = max(limit, 0) if limit is not None else None
self.totalhosts: set[str] = set()
self.proxy: bool | str = False
self._report: SourceExecutionReport | None = None
self._saw_done = False
self._saw_terminal_progress = False
self._final_progress_skipped = False
self._result_limit_reached = False
def _stop(self, reason: str, status: SourceReportStatus = 'failed') -> None:
self._report = SourceExecutionReport(status, reason)
@@ -105,8 +105,9 @@ class SearchSourcegraph:
for match in _HOST_TOKEN.finditer(content):
hostname = _normalize_hostname(match.group())
if hostname and hostname != self.word and hostname.endswith(f'.{self.word}'):
if len(self.totalhosts) >= self.MAX_HOSTNAMES and hostname not in self.totalhosts:
raise OverflowError
if self.limit is not None and len(self.totalhosts) >= self.limit and hostname not in self.totalhosts:
self._result_limit_reached = True
continue
self.totalhosts.add(hostname)
def _consume_matches(self, payload: object) -> None:
@@ -189,20 +190,12 @@ class SearchSourcegraph:
self._stop(f'http-{response.status}')
return
event_count = 0
async for record in response:
event_count += 1
if event_count > self.MAX_EVENTS:
self._stop('response-limit')
return
try:
failure = self._consume_event(record)
except json.JSONDecodeError, RecursionError, TypeError, ValueError:
self._stop('invalid-response')
return
except OverflowError:
self._stop('response-limit')
return
if failure:
self._stop(failure)
return
@@ -214,6 +207,8 @@ class SearchSourcegraph:
self._stop('invalid-response')
elif self._final_progress_skipped:
self._stop('provider-limited', 'partial')
elif self._result_limit_reached:
self._stop('result-limit', 'completed')
async def get_hostnames(self) -> list[str]:
return sorted(self.totalhosts)
theHarvester/discovery/subdomainapi.py
+71
View File
@@ -0,0 +1,71 @@
from urllib.parse import urlencode
from theHarvester.discovery.provider_response import provider_http_error
from theHarvester.lib.core import AsyncFetcher, Core, FetcherResponse
from theHarvester.lib.hostnames import normalize_hostname
from theHarvester.lib.source_execution import SourceExecutionReport
class SearchSubdomainApi:
"""Collect passive hostname evidence from Subdomain API."""
BASE_URL = 'https://api.subdomain.app/v1/query'
def __init__(self, word: str) -> None:
self.word = normalize_hostname(word)
self.totalhosts: set[str] = set()
self.proxy: bool | str = False
async def do_search(self) -> SourceExecutionReport | None:
url = f'{self.BASE_URL}?{urlencode({"domain": self.word})}'
responses = await AsyncFetcher.fetch_all(
[url],
headers={'User-Agent': Core.get_user_agent()},
proxy=self.proxy,
json=True,
include_metadata=True,
)
response = responses[0] if responses else None
if error := provider_http_error(response):
return SourceExecutionReport(*error)
assert isinstance(response, FetcherResponse)
payload = response.body
if not isinstance(payload, dict):
return SourceExecutionReport('failed', 'invalid-response')
count = payload.get('count')
total = payload.get('total')
subdomains = payload.get('subdomains')
provider_domain = payload.get('domain')
valid_count = isinstance(count, int) and not isinstance(count, bool) and count >= 0
valid_total = isinstance(total, int) and not isinstance(total, bool) and total >= 0
if (
provider_domain != self.word
or not valid_count
or not valid_total
or not isinstance(subdomains, list)
or count != len(subdomains)
or total < count
):
return SourceExecutionReport('failed', 'invalid-response')
for candidate in subdomains:
if not isinstance(candidate, str) or '*' in candidate:
continue
try:
hostname = normalize_hostname(candidate)
except ValueError:
continue
if hostname != self.word and hostname.endswith(f'.{self.word}'):
self.totalhosts.add(hostname)
return SourceExecutionReport('partial', 'provider-limit') if total > count else None
async def get_hostnames(self) -> set[str]:
return self.totalhosts
async def process(self, proxy: bool | str = False) -> SourceExecutionReport | None:
self.proxy = proxy
try:
return await self.do_search()
except Exception:
return SourceExecutionReport('failed', 'transport-error')
theHarvester/discovery/thc.py
+27 -15
View File
@@ -1,9 +1,12 @@
import asyncio
import logging
from urllib.parse import urlencode
import aiohttp
from theHarvester.lib.core import Core
from theHarvester.lib.hostnames import normalize_scoped_hostname
from theHarvester.lib.source_execution import SourceExecutionReport
logger = logging.getLogger(__name__)
@@ -11,15 +14,22 @@ logger = logging.getLogger(__name__)
class SearchThc:
"""Search THC (ip.thc.org) for subdomains."""
def __init__(self, word: str) -> None:
PROVIDER_MAX_RESULTS = 50_000
def __init__(self, word: str, limit: int | None = None) -> None:
if limit is not None and (isinstance(limit, bool) or not isinstance(limit, int) or limit <= 0):
raise ValueError('THC limit must be a positive integer or None')
self.word = word
self.limit = limit
self.results: set = set()
self.proxy = False
self.max_retries = 3
self.base_delay = 2
async def do_search(self) -> None:
url = f'https://ip.thc.org/api/v1/subdomains/download?domain={self.word}&limit=10000&hide_header=true'
async def do_search(self) -> SourceExecutionReport | None:
requested = self.PROVIDER_MAX_RESULTS if self.limit is None else min(self.limit, self.PROVIDER_MAX_RESULTS)
query = urlencode({'domain': self.word, 'limit': requested, 'hide_header': 'true'})
url = f'https://ip.thc.org/api/v1/subdomains/download?{query}'
headers = {'User-Agent': Core.get_user_agent()}
for attempt in range(self.max_retries):
@@ -31,7 +41,7 @@ class SearchThc:
rate_remaining = response.headers.get('x-ratelimit-remaining', '0')
if attempt == self.max_retries - 1:
logger.info(f'THC returned status 429 after {self.max_retries} attempts')
return
return SourceExecutionReport('rate-limited', 'http-429')
wait_time = self.base_delay * (attempt + 1)
logger.info(f'THC rate limit hit (remaining: {rate_remaining}). Waiting {wait_time}s before retry...')
await asyncio.sleep(wait_time)
@@ -39,32 +49,34 @@ class SearchThc:
if response.status != 200:
logger.info(f'THC returned status {response.status}')
return
return SourceExecutionReport('failed', f'http-{response.status}')
text = await response.text()
if text:
for line in text.splitlines():
hostname = line.strip().lower()
if hostname and self.word.lower() in hostname:
self.results.add(hostname)
return
lines = text.splitlines()
for line in lines:
if hostname := normalize_scoped_hostname(line, self.word):
self.results.add(hostname)
if len(lines) >= requested and (self.limit is None or self.limit > self.PROVIDER_MAX_RESULTS):
return SourceExecutionReport('partial', 'provider-limit')
return None
except Exception as e:
error_msg = str(e).lower()
if '429' in error_msg or 'rate' in error_msg:
if attempt == self.max_retries - 1:
logger.info(f'THC rate limit failure after {self.max_retries} attempts')
return
return SourceExecutionReport('rate-limited', 'provider-rate-limit')
wait_time = self.base_delay * (attempt + 1)
logger.info(f'THC rate limit detected. Waiting {wait_time}s before retry...')
await asyncio.sleep(wait_time)
continue
logger.info(f'An exception has occurred in THC: {e}')
return
return SourceExecutionReport('failed', 'transport-error')
return SourceExecutionReport('failed', 'transport-error')
async def get_hostnames(self) -> set:
return self.results
async def process(self, proxy: bool = False) -> None:
async def process(self, proxy: bool = False) -> SourceExecutionReport | None:
self.proxy = proxy
await self.do_search()
return await self.do_search()
theHarvester/discovery/tombasearch.py
+26 -12
View File
@@ -3,15 +3,16 @@ import logging
from theHarvester.discovery.constants import MissingKey
from theHarvester.lib.core import AsyncFetcher, Core, FetcherResponse
from theHarvester.lib.source_execution import SourceExecutionReport
logger = logging.getLogger(__name__)
class SearchTomba:
def __init__(self, word, limit, start) -> None:
def __init__(self, word, limit: int | None, start) -> None:
self.word = word
self.requested_limit = limit
self.limit = min(limit, 10)
self.limit = min(limit, 10) if limit is not None else 10
self.start = start
key, secret = Core.tomba_key()
self.key = (key.strip() if key else '', secret.strip() if secret else '')
@@ -43,7 +44,7 @@ class SearchTomba:
return None
return metadata.body
async def do_search(self) -> None:
async def do_search(self) -> SourceExecutionReport | None:
# First determine if a user account is not a free account, this call is free
is_free = True
headers = {
@@ -54,7 +55,7 @@ class SearchTomba:
acc_info_url = 'https://api.tomba.io/v1/me'
response = await self._fetch_json(acc_info_url, headers)
if response is None:
return
return None
is_free = (
is_free
if 'name' in response['data']['pricing'].keys() and response['data']['pricing']['name'].lower() == 'free'
@@ -73,24 +74,31 @@ class SearchTomba:
tomba_counter = f'https://api.tomba.io/v1/email-count?domain={self.word}'
response = await self._fetch_json(tomba_counter, headers)
if response is None:
return
total_results = min(max(0, response['data']['total'] - self.start), self.requested_limit)
return None
available_results = max(0, response['data']['total'] - self.start)
total_results = (
min(available_results, self.requested_limit) if self.requested_limit is not None else available_results
)
page_size = 50
first_page = self.start // page_size + 1
first_page_skip = self.start % page_size
total_number_reqs = (first_page_skip + total_results + page_size - 1) // page_size if total_results else 0
if total_requests_avail < total_number_reqs:
quota_exhausted = total_requests_avail < total_number_reqs
if quota_exhausted:
logger.info('WARNING: The account does not have enough requests to gather all the emails.')
return
remaining = total_results
for page in range(first_page, first_page + total_number_reqs):
provider_limit_reached = False
pages_to_fetch = min(total_number_reqs, max(total_requests_avail, 0))
for page in range(first_page, first_page + pages_to_fetch):
req_url = f'https://api.tomba.io/v1/domain-search?domain={self.word}&limit={page_size}&page={page}'
response = await self._fetch_json(req_url, headers)
if response is None:
return
return None
skip = first_page_skip if page == first_page else 0
raw_entries = response['data']['emails']
provider_limit_reached = is_free and isinstance(raw_entries, list) and len(raw_entries) >= page_size
response['data']['emails'] = response['data']['emails'][skip : skip + remaining]
temp_emails, temp_hostnames = await self.parse_resp(response)
self.emails.extend(temp_emails)
@@ -98,6 +106,11 @@ class SearchTomba:
remaining -= len(response['data']['emails'])
if not is_free:
await asyncio.sleep(1)
if quota_exhausted:
return SourceExecutionReport('partial', 'quota-exhausted')
if provider_limit_reached and (self.requested_limit is None or self.requested_limit > page_size):
return SourceExecutionReport('partial', 'provider-limit')
return None
async def parse_resp(self, json_resp):
emails = list(sorted({email['email'] for email in json_resp['data']['emails']}))
@@ -113,12 +126,13 @@ class SearchTomba:
)
return emails, domains
async def process(self, proxy: bool = False) -> None:
async def process(self, proxy: bool = False) -> SourceExecutionReport | None:
self.proxy = proxy
try:
await self.do_search() # Only need to do it once.
return await self.do_search() # Only need to do it once.
except AttributeError, KeyError, TypeError:
logger.info('Tomba returned malformed data')
return SourceExecutionReport('failed', 'invalid-response')
async def get_emails(self):
return self.emails
theHarvester/discovery/urlscan.py
+10 -7
View File
@@ -16,8 +16,8 @@ logger = logging.getLogger(__name__)
class SearchUrlscan:
MAX_PAGE_SIZE = 10_000
def __init__(self, word: str, limit: int) -> None:
if isinstance(limit, bool) or not isinstance(limit, int) or limit <= 0:
def __init__(self, word: str, limit: int | None) -> None:
if limit is not None and (isinstance(limit, bool) or not isinstance(limit, int) or limit <= 0):
raise ValueError('URLScan limit must be a positive integer')
self.word = word
self.limit = limit
@@ -143,8 +143,8 @@ class SearchUrlscan:
malformed = False
try:
async with AsyncFetcher.open_session(proxy=self.proxy) as session:
while records_seen < self.limit:
remaining = self.limit - records_seen
while self.limit is None or records_seen < self.limit:
remaining = self.limit - records_seen if self.limit is not None else self.MAX_PAGE_SIZE
params: dict[str, str | int] = {
'q': f'domain:{self.word}',
'size': min(self.MAX_PAGE_SIZE, remaining),
@@ -169,16 +169,19 @@ class SearchUrlscan:
if not results:
return SourceExecutionReport('failed', 'invalid-response') if malformed else None
page_results = results[:remaining]
page_results = results[:remaining] if self.limit is not None else results
records_seen += len(page_results)
malformed = self._parse_results(page_results, collected_at) or malformed
if records_seen >= self.limit:
if self.limit is not None and records_seen >= self.limit:
break
next_cursor = self._cursor(page_results[-1])
if next_cursor is None:
return SourceExecutionReport('failed', 'invalid-cursor')
if next_cursor in seen_cursors:
return SourceExecutionReport('failed', 'repeated-cursor')
return SourceExecutionReport(
'partial' if any((self.totalhosts, self.totalips, self.urls, self.totalasns)) else 'failed',
'repeated-cursor',
)
seen_cursors.add(next_cursor)
cursor = next_cursor
except Exception as error:
theHarvester/discovery/virustotal.py
+7 -7
View File
@@ -10,8 +10,8 @@ from theHarvester.lib.source_execution import SourceExecutionReport
class SearchVirustotal:
def __init__(self, word: str, limit: int) -> None:
if isinstance(limit, bool) or not isinstance(limit, int) or limit <= 0:
def __init__(self, word: str, limit: int | None) -> None:
if limit is not None and (isinstance(limit, bool) or not isinstance(limit, int) or limit <= 0):
raise ValueError('VirusTotal limit must be a positive integer')
self.key = Core.virustotal_key()
if not isinstance(self.key, str) or not self.key.strip():
@@ -29,8 +29,8 @@ class SearchVirustotal:
report = None
try:
async with AsyncFetcher.open_session(headers=headers, proxy=self.proxy) as session:
while records_seen < self.limit:
remaining = self.limit - records_seen
while self.limit is None or records_seen < self.limit:
remaining = self.limit - records_seen if self.limit is not None else 40
params: dict[str, int | str] = {'limit': min(40, remaining)}
if cursor:
params['cursor'] = cursor
@@ -50,11 +50,11 @@ class SearchVirustotal:
meta = response.body.get('meta', {})
if not isinstance(data, list) or not isinstance(meta, dict):
return SourceExecutionReport('failed', 'invalid-response')
page_data = data[:remaining]
page_data = data[:remaining] if self.limit is not None else data
records_seen += len(page_data)
hostnames, malformed = self.parse_hostnames(page_data, self.word)
for hostname in sorted(hostnames):
if len(self.hostnames) >= self.limit:
if self.limit is not None and len(self.hostnames) >= self.limit:
break
self.hostnames.add(hostname)
if malformed:
@@ -63,7 +63,7 @@ class SearchVirustotal:
if not data or not isinstance(next_cursor, str) or not next_cursor:
break
if next_cursor in seen_cursors:
return SourceExecutionReport('failed', 'repeated-cursor')
return SourceExecutionReport('partial' if self.hostnames else 'failed', 'repeated-cursor')
seen_cursors.add(next_cursor)
cursor = next_cursor
except Exception:
theHarvester/discovery/waybackarchive.py
+12 -16
View File
@@ -3,7 +3,7 @@ import logging
from urllib.parse import unquote_plus, urlencode, urlsplit
from theHarvester.lib.core import AsyncFetcher, Core
from theHarvester.lib.source_execution import SourceExecutionReport, SourceReportStatus
from theHarvester.lib.source_execution import SourceExecutionReport
logger = logging.getLogger(__name__)
@@ -16,12 +16,10 @@ class SearchWaybackarchive:
PAGE_SIZE = 1000
RUNTIME_SECONDS = 30.0
# ponytail: hard cap protects against endless cursors; raise only if real targets exceed one million rows.
MAX_PAGES_PER_QUERY = 1000
def __init__(self, word, limit: int = 500) -> None:
def __init__(self, word, limit: int | None = 500) -> None:
self.word = word.strip().rstrip('.').lower()
self.limit = max(limit, 0)
self.limit = max(limit, 0) if limit is not None else None
self.totalhosts: set = set()
self.proxy = False
self.hostname = 'https://web.archive.org'
@@ -61,7 +59,9 @@ class SearchWaybackarchive:
async def _search_pattern(self, pattern: str, headers: dict[str, str]) -> str | None:
resume_key: str | None = None
seen_resume_keys: set[str] = set()
for page_number in range(1, self.MAX_PAGES_PER_QUERY + 1):
page_number = 0
while True:
page_number += 1
query = {
'url': pattern,
'fl': 'original',
@@ -92,18 +92,18 @@ class SearchWaybackarchive:
domain = self._extract_domain_from_url(line.strip())
if domain.endswith(f'.{self.word}') or domain == self.word:
self.totalhosts.add(domain)
if len(self.totalhosts) >= self.limit:
if self.limit is not None and len(self.totalhosts) >= self.limit:
return 'result-limit'
if page_number == 1 or page_number % 10 == 0:
logger.info(f'Wayback Archive page {page_number}: hosts={len(self.totalhosts)}')
if next_resume_key is None or next_resume_key in seen_resume_keys:
if next_resume_key is None:
return None
if next_resume_key in seen_resume_keys:
return 'repeated-cursor'
seen_resume_keys.add(next_resume_key)
resume_key = next_resume_key
logger.info(f'Wayback Archive page limit reached for pattern {pattern}; results may be incomplete')
return 'page-limit'
async def do_search(self) -> SourceExecutionReport | None:
if self.limit == 0:
@@ -124,9 +124,6 @@ class SearchWaybackarchive:
if degraded_reason is None:
return SourceExecutionReport('completed', 'result-limit')
break
if outcome == 'page-limit':
degraded_reason = degraded_reason or outcome
break
if outcome is not None:
degraded_reason = degraded_reason or outcome
except TimeoutError:
@@ -134,10 +131,9 @@ class SearchWaybackarchive:
f'Wayback Archive runtime limit reached after {self.RUNTIME_SECONDS:g}s; '
f'preserved {len(self.totalhosts)} hosts'
)
return SourceExecutionReport('failed', 'runtime-limit')
return SourceExecutionReport('partial' if self.totalhosts else 'failed', 'runtime-limit')
if degraded_reason is not None:
status: SourceReportStatus = 'partial' if degraded_reason == 'page-limit' else 'failed'
return SourceExecutionReport(status, degraded_reason)
return SourceExecutionReport('failed', degraded_reason)
except Exception as e:
logger.info(f'Wayback Archive API error: {e}')
return SourceExecutionReport('failed', 'unexpected-error')
theHarvester/discovery/whoisxml.py
+6 -6
View File
@@ -8,8 +8,8 @@ from theHarvester.lib.source_execution import SourceExecutionReport
class SearchWhoisXML:
def __init__(self, word: str, limit: int) -> None:
if isinstance(limit, bool) or not isinstance(limit, int) or limit <= 0:
def __init__(self, word: str, limit: int | None) -> None:
if limit is not None and (isinstance(limit, bool) or not isinstance(limit, int) or limit <= 0):
raise ValueError('WhoisXML limit must be a positive integer')
self.word = word
self.limit = limit
@@ -25,7 +25,7 @@ class SearchWhoisXML:
records_seen = 0
report = None
async with AsyncFetcher.open_session(proxy=self.proxy) as session:
while records_seen < self.limit:
while self.limit is None or records_seen < self.limit:
params = {'apiKey': self.key, 'domainName': self.word}
if cursor is not None:
params['searchAfter'] = cursor
@@ -48,7 +48,7 @@ class SearchWhoisXML:
if not isinstance(next_cursor, str):
return SourceExecutionReport('failed', 'invalid-response')
remaining = self.limit - records_seen
remaining = self.limit - records_seen if self.limit is not None else len(result['records'])
records = result['records'][:remaining]
records_seen += len(records)
malformed = False
@@ -60,10 +60,10 @@ class SearchWhoisXML:
self.total_results.add(hostname)
if malformed:
report = SourceExecutionReport('failed', 'invalid-response')
if records_seen >= self.limit or not next_cursor:
if (self.limit is not None and records_seen >= self.limit) or not next_cursor:
break
if next_cursor in seen_cursors:
return SourceExecutionReport('failed', 'repeated-cursor')
return SourceExecutionReport('partial' if self.total_results else 'failed', 'repeated-cursor')
seen_cursors.add(next_cursor)
cursor = next_cursor
return report
theHarvester/discovery/windvane.py
+140 -160
View File
@@ -1,8 +1,16 @@
from __future__ import annotations
import json
import logging
from typing import TYPE_CHECKING
from theHarvester.lib.core import AsyncFetcher, Core
from theHarvester.lib.hostnames import normalize_scoped_hostname
from theHarvester.lib.source_execution import SourceExecutionReport
if TYPE_CHECKING:
from collections.abc import Callable
from typing import Any
logger = logging.getLogger(__name__)
@@ -24,8 +32,11 @@ class SearchWindvane:
Set the key with ``WINDVANE_API_KEY`` or ``search.set_api_key("your-key")``.
"""
def __init__(self, word) -> None:
def __init__(self, word: str, limit: int | None = None) -> None:
if limit is not None and (isinstance(limit, bool) or not isinstance(limit, int) or limit <= 0):
raise ValueError('Windvane limit must be a positive integer')
self.word = word.strip().lower().rstrip('.')
self.limit = limit
self.totalhosts: set = set()
self.totalips: set = set()
self.totalemails: set = set()
@@ -65,7 +76,85 @@ class SearchWindvane:
return {}
return {}
async def do_search(self) -> None:
@staticmethod
def _next_page(data: dict[str, Any], page: int, count: int, records: list[object]) -> int | None:
metadata = [data]
metadata.extend(
value for key in ('pagination', 'page_response', 'page_info') if isinstance((value := data.get(key)), dict)
)
for section in metadata:
for key in ('next_page', 'nextPage'):
if key in section:
value = section[key]
return int(value) if isinstance(value, int | str) and str(value).isdigit() and int(value) > 0 else None
for key in ('has_more', 'hasMore'):
if isinstance(section.get(key), bool):
return page + 1 if section[key] else None
for key in ('total', 'total_count', 'totalCount'):
if isinstance(section.get(key), int):
return page + 1 if page * count < section[key] else None
return page + 1 if len(records) >= count else None
async def _paginate(
self,
headers: dict[str, str],
endpoint: str,
query: dict[str, str],
page_size: int,
consume: Callable[[object], object],
) -> SourceExecutionReport | None:
page = 1
records_seen = 0
seen_pages: set[str] = set()
seen_cursors: set[int] = set()
url = f'{self.hostname}/{endpoint}'
while self.limit is None or records_seen < self.limit:
count = min(page_size, self.limit - records_seen) if self.limit is not None else page_size
request_data: dict[str, object] = {**query, 'page_request': {'page': page, 'count': count}}
response = await AsyncFetcher.post_fetch(
url,
headers=headers,
data=json.dumps(request_data, separators=(',', ':')),
proxy=self.proxy,
)
if not response:
return SourceExecutionReport('failed', 'transport-error')
response_data = self._safe_parse_json(response)
if not response_data:
return SourceExecutionReport('failed', 'invalid-response')
if response_data.get('code') != 0:
logger.info(f'Windvane {endpoint} API returned code {response_data.get("code")}')
return (
SourceExecutionReport('partial', 'provider-limit')
if records_seen
else SourceExecutionReport('failed', 'provider-error')
)
data = response_data.get('data')
if not isinstance(data, dict) or not isinstance(data.get('list'), list):
return SourceExecutionReport('failed', 'invalid-response')
records = data['list']
if not records:
return None
signature = json.dumps(records, sort_keys=True, default=str)
if signature in seen_pages:
return SourceExecutionReport('partial', 'repeated-page')
seen_pages.add(signature)
accepted = records[: self.limit - records_seen] if self.limit is not None else records
for record in accepted:
consume(record)
records_seen += len(accepted)
if self.limit is not None and records_seen >= self.limit:
return SourceExecutionReport('completed', 'result-limit')
next_page = self._next_page(data, page, count, records)
if next_page is None:
return None
if next_page == page or next_page in seen_cursors:
return SourceExecutionReport('partial', 'repeated-cursor')
seen_cursors.add(next_page)
page = next_page
return None
async def do_search(self) -> SourceExecutionReport | None:
"""Query the Windvane endpoints used by this source."""
try:
headers = {'User-agent': Core.get_user_agent(), 'Content-Type': 'application/json', 'Accept': 'application/json'}
@@ -75,180 +164,71 @@ class SearchWindvane:
headers['X-Api-Key'] = self.api_key
# With API key, use full API endpoints
await self._search_subdomains(headers)
await self._search_dns_history(headers)
await self._search_emails(headers)
reports = [
await self._search_subdomains(headers),
await self._search_dns_history(headers),
await self._search_emails(headers),
]
else:
# Without API key, use the provider's limited endpoint only.
logger.info('[*] Windvane API key not found. Using limited unauthenticated access.')
await self._search_subdomains_limited(headers)
reports = [await self._search_subdomains_limited(headers)]
retained = [report for report in reports if report is not None]
return next((report for report in retained if report.status != 'completed'), retained[0] if retained else None)
except Exception as e:
logger.info(f'Windvane API error: {e}')
return SourceExecutionReport('failed', 'transport-error')
async def _search_subdomains(self, headers: dict) -> None:
async def _search_subdomains(self, headers: dict[str, str]) -> SourceExecutionReport | None:
"""Search for subdomains with ``/ListSubDomain``."""
try:
url = f'{self.hostname}/ListSubDomain'
return await self._paginate(
headers,
'ListSubDomain',
{'domain': self.word},
30,
lambda item: self._add_host(item.get('domain')) if isinstance(item, dict) else None,
)
# Use pagination to get more results
for page in range(1, 4): # Get first 3 pages (up to 90 results)
data = {'domain': self.word, 'page_request': {'page': page, 'count': 30}}
try:
response = await AsyncFetcher.post_fetch(
url,
headers=headers,
data=json.dumps(data, separators=(',', ':')),
proxy=self.proxy,
)
if response:
response_data = self._safe_parse_json(response)
# Check if response is successful
if response_data.get('code') == 0:
data_section = response_data.get('data', {})
subdomains = data_section.get('list', [])
if not subdomains:
break # No more results
for item in subdomains:
if isinstance(item, dict):
self._add_host(item.get('domain'))
else:
# API error - stop pagination
if response_data.get('code') != 0:
logger.info(f'Windvane subdomain API returned code {response_data.get("code")}')
break
except Exception as e:
logger.info(f'Windvane subdomain request failed: {e}')
break
except Exception as e:
logger.info(f'Windvane subdomain search error: {e}')
async def _search_dns_history(self, headers: dict) -> None:
async def _search_dns_history(self, headers: dict[str, str]) -> SourceExecutionReport | None:
"""Collect subdomains and IP addresses from ``/ListDNS`` history."""
try:
url = f'{self.hostname}/ListDNS'
# Get DNS history records
for page in range(1, 3): # Get first 2 pages
data = {'domain': self.word, 'page_request': {'page': page, 'count': 30}}
def consume(record: object) -> None:
if not isinstance(record, dict):
return
answer = record.get('answer', '')
if (
self._add_host(record.get('domain'))
and record.get('answer_type') == 'A'
and isinstance(answer, str)
and self._is_valid_ip(answer)
):
self.totalips.add(answer)
try:
response = await AsyncFetcher.post_fetch(
url,
headers=headers,
data=json.dumps(data, separators=(',', ':')),
proxy=self.proxy,
)
if response:
response_data = self._safe_parse_json(response)
return await self._paginate(headers, 'ListDNS', {'domain': self.word}, 30, consume)
if response_data.get('code') == 0:
data_section = response_data.get('data', {})
dns_records = data_section.get('list', [])
if not dns_records:
break
for record in dns_records:
if isinstance(record, dict):
answer = record.get('answer', '')
answer_type = record.get('answer_type', '')
domain_is_scoped = self._add_host(record.get('domain'))
# Add IP addresses from A records
if domain_is_scoped and answer and answer_type == 'A' and self._is_valid_ip(answer):
self.totalips.add(answer)
else:
break
except Exception as e:
logger.info(f'Windvane DNS history request failed: {e}')
break
except Exception as e:
logger.info(f'Windvane DNS history search error: {e}')
async def _search_emails(self, headers: dict) -> None:
async def _search_emails(self, headers: dict[str, str]) -> SourceExecutionReport | None:
"""Search for email addresses with ``/ListEmail``."""
try:
url = f'{self.hostname}/ListEmail'
data = {'email': self.word, 'page_request': {'page': 1, 'count': 50}}
def consume(item: object) -> None:
if isinstance(item, dict):
self._add_email(item.get('email'))
self._add_host(item.get('domain'))
try:
response = await AsyncFetcher.post_fetch(
url,
headers=headers,
data=json.dumps(data, separators=(',', ':')),
proxy=self.proxy,
)
if response:
response_data = self._safe_parse_json(response)
return await self._paginate(headers, 'ListEmail', {'email': self.word}, 50, consume)
if response_data.get('code') == 0:
data_section = response_data.get('data', {})
email_results = data_section.get('list', [])
for item in email_results:
if isinstance(item, dict):
self._add_email(item.get('email'))
self._add_host(item.get('domain'))
except Exception as e:
logger.info(f'Windvane email search request failed: {e}')
except Exception as e:
logger.info(f'Windvane email search error: {e}')
async def _search_subdomains_limited(self, headers: dict) -> None:
async def _search_subdomains_limited(self, headers: dict[str, str]) -> SourceExecutionReport | None:
"""Search the unauthenticated subdomain endpoints."""
try:
# Try basic subdomain endpoint with minimal parameters
url = f'{self.hostname}/ListSubDomain'
# Simple request with just domain - limited to 5 calls
data = {
'domain': self.word,
'page_request': {
'page': 1,
'count': 10, # Smaller count for unauthenticated
},
}
try:
response = await AsyncFetcher.post_fetch(
url,
headers=headers,
data=json.dumps(data, separators=(',', ':')),
proxy=self.proxy,
)
if response:
response_data = self._safe_parse_json(response)
if isinstance(response_data, dict) and response_data.get('code') == 0:
data_section = response_data.get('data', {})
subdomains = data_section.get('list', [])
for item in subdomains:
if isinstance(item, dict):
self._add_host(item.get('domain'))
logger.info(f'[*] Found {len(subdomains)} subdomains with limited access')
else:
logger.info(f'Windvane limited API returned code {response_data.get("code")}')
except Exception as e:
logger.info(f'Windvane limited API failed: {e}')
except Exception as e:
logger.info(f'Windvane limited search error: {e}')
report = await self._paginate(
headers,
'ListSubDomain',
{'domain': self.word},
10,
lambda item: self._add_host(item.get('domain')) if isinstance(item, dict) else None,
)
logger.info(f'[*] Found {len(self.totalhosts)} subdomains with limited access')
return report
def set_api_key(self, api_key: str) -> None:
"""Set the API key for authenticated requests.
@@ -276,7 +256,7 @@ class SearchWindvane:
async def get_emails(self) -> set:
return self.totalemails
async def process(self, proxy: bool = False) -> None:
async def process(self, proxy: bool = False) -> SourceExecutionReport | None:
"""Run the Windvane search.
Args:
@@ -287,4 +267,4 @@ class SearchWindvane:
# API key is already set via _get_api_key() method
await self.do_search()
return await self.do_search()
theHarvester/discovery/yahoosearch.py
+61 -8
View File
@@ -1,26 +1,79 @@
from theHarvester.lib.core import AsyncFetcher, Core
from __future__ import annotations
from theHarvester.lib.core import AsyncFetcher, Core, FetcherResponse
from theHarvester.lib.source_execution import SourceExecutionReport
from theHarvester.parsers import myparser
class SearchYahoo:
def __init__(self, word, limit) -> None:
def __init__(self, word, limit: int | None) -> None:
self.word = word
self.total_results = ''
self.server = 'search.yahoo.com'
self.limit = limit
self.proxy = False
async def do_search(self) -> None:
def _page(self, response: object) -> tuple[str | None, SourceExecutionReport | None]:
if response is None:
return None, SourceExecutionReport('partial' if self.total_results else 'failed', 'transport-error')
if isinstance(response, FetcherResponse):
if not 200 <= response.status < 300:
return None, SourceExecutionReport('partial' if self.total_results else 'failed', f'http-{response.status}')
response = response.body
if not isinstance(response, str):
return None, SourceExecutionReport('partial' if self.total_results else 'failed', 'invalid-response')
normalized = response.casefold()
if not response.strip() or 'no results' in normalized or 'no-results' in normalized:
return '', None
if 'captcha' in normalized or 'verify you are human' in normalized:
return None, SourceExecutionReport('partial' if self.total_results else 'failed', 'security-verification')
if 'access denied' in normalized or 'temporarily blocked' in normalized:
return None, SourceExecutionReport('partial' if self.total_results else 'failed', 'access-denied')
return response, None
async def do_search(self) -> SourceExecutionReport | None:
base_url = f'https://{self.server}/search?p=%40{self.word}&b=xx&pz=10'
headers = {'Host': self.server, 'User-Agent': Core.get_browser_user_agent()}
urls = [base_url.replace('xx', str(num)) for num in range(0, self.limit, 10) if num <= self.limit]
responses = await AsyncFetcher.fetch_all(urls, headers=headers, proxy=self.proxy)
if self.limit is None:
seen_pages: set[str] = set()
offset = 0
while True:
response = await AsyncFetcher.fetch(
url=base_url.replace('xx', str(offset)),
headers=headers,
proxy=self.proxy,
include_metadata=True,
)
body, report = self._page(response)
if report is not None:
return report
if not body:
return None
if body in seen_pages:
return SourceExecutionReport('partial', 'repeated-page')
seen_pages.add(body)
self.total_results += f' {body}'
offset += 10
urls = [base_url.replace('xx', str(num)) for num in range(0, self.limit, 10)]
responses = await AsyncFetcher.fetch_all(urls, headers=headers, proxy=self.proxy, include_metadata=True)
if urls and not responses:
return SourceExecutionReport('failed', 'transport-error')
seen_finite_pages: set[str] = set()
for response in responses:
self.total_results += f' {response}'
body, report = self._page(response)
if report is not None:
return report
if not body:
break
if body in seen_finite_pages:
return SourceExecutionReport('partial', 'repeated-page')
seen_finite_pages.add(body)
self.total_results += f' {body}'
return None
async def process(self, proxy: bool = False) -> None:
async def process(self, proxy: bool = False) -> SourceExecutionReport | None:
self.proxy = proxy
await self.do_search()
return await self.do_search()
async def get_emails(self):
rawres = myparser.Parser(self.total_results, self.word)
theHarvester/discovery/zoomeyesearch.py
+6 -5
View File
@@ -33,8 +33,8 @@ class SearchZoomEye:
)
URL_PATTERN = re.compile(r'https?://[^\s"\'<>]+')
def __init__(self, word: str, limit: int) -> None:
if isinstance(limit, bool) or not isinstance(limit, int) or limit <= 0:
def __init__(self, word: str, limit: int | None) -> None:
if limit is not None and (isinstance(limit, bool) or not isinstance(limit, int) or limit <= 0):
raise ValueError('ZoomEye limit must be a positive integer')
key = Core.zoomeye_key()
if not isinstance(key, str) or not key.strip():
@@ -106,14 +106,15 @@ class SearchZoomEye:
return response.body
async def do_search(self, session: Any) -> None:
page_size = min(self.PAGE_SIZE, self.limit)
page_size = min(self.PAGE_SIZE, self.limit) if self.limit is not None else self.PAGE_SIZE
first = await self._fetch_page(session, 1, page_size)
if first is None:
return
await self._store_matches(first['data'][:page_size])
page_limit = math.ceil(min(first['total'], self.limit) / page_size) if first['total'] else 1
total = min(first['total'], self.limit) if self.limit is not None else first['total']
page_limit = math.ceil(total / page_size) if total else 1
for page in range(2, page_limit + 1):
remaining = self.limit - ((page - 1) * page_size)
remaining = self.limit - ((page - 1) * page_size) if self.limit is not None else page_size
response = await self._fetch_page(session, page, page_size)
if response is None:
return
theHarvester/lib/api/run_models.py
+15 -3
View File
@@ -80,9 +80,11 @@ class RunRequest(BaseModel):
)
limit: int = Field(
default=500,
ge=1,
le=10_000,
description='Maximum results requested from each source when that provider supports a limit.',
ge=0,
description=(
'Maximum results requested from each source when supported; 0 continues to provider exhaustion with '
'no local result or page-count cap.'
),
)
start: int = Field(
default=DEFAULT_RESULT_START,
@@ -587,6 +589,15 @@ class ScreenshotRecord(BaseModel):
url: str
class SourceYieldSummary(BaseModel):
source: str
observed_result_count: int = Field(ge=0)
unique_result_count: int = Field(ge=0)
shared_result_count: int = Field(ge=0)
resolved_hostname_count: int = Field(ge=0)
unique_resolved_hostname_count: int = Field(ge=0)
RunStatus = Literal['queued', 'running', 'cancelling', 'cancelled', 'completed', 'failed']
Activity = Literal['P0', 'P1', 'P2']
@@ -618,6 +629,7 @@ class RunDetail(RunSummary):
request: RunRequest | ImportedRunRequest
results: list[RunResult]
source_executions: list[dict[str, Any]]
source_yields: list[SourceYieldSummary]
action_executions: list[dict[str, Any]]
artifacts: list[dict[str, Any]]
screenshots: list[ScreenshotRecord]
theHarvester/lib/api/run_store.py
+12
View File
@@ -238,11 +238,23 @@ class RunStore:
),
}
if detail:
source_yields = (
[
item.to_dict()
for item in await self.results.source_yields(
UUID(str(record['evidence_run_id'])),
kind='hostname',
)
]
if evidence
else []
)
result.update(
request=request,
evidence=evidence,
results=normalized_results(evidence),
source_executions=source_executions(evidence),
source_yields=source_yields,
action_executions=evidence.get('action_executions', []) if evidence else [],
artifacts=evidence.get('artifacts', []) if evidence else [],
screenshots=screenshots(evidence, str(record['run_id']), self.artifact_directory(str(record['run_id']))),
theHarvester/lib/api/static/harvestview/app.js
+1 -1
View File
@@ -314,7 +314,7 @@
const request = run.request || {};
const sources = request.sources?.join(', ') || 'Not recorded';
const options = [
['Sources', sources], ['Result limit', request.limit ?? 'Imported evidence'],
['Sources', sources], ['Result limit', request.limit === 0 ? 'Unlimited' : request.limit ?? 'Imported evidence'],
['Result start offset', request.start ?? 'Not recorded'],
['Discovery source workers', request.source_workers ?? 'Not recorded'],
['Whole-run deadline', request.deadline_seconds === null ? 'Unlimited' : request.deadline_seconds === undefined ? 'Not recorded' : `${request.deadline_seconds} seconds`],
theHarvester/lib/api/static/harvestview/index.html
+2 -2
View File
@@ -217,8 +217,8 @@
<small>Use an explicitly authorized target.</small>
</label>
<label>Result limit
<input id="run-limit" name="limit" type="number" min="1" max="10000" value="500" required>
<small>Applied per selected source.</small>
<input id="run-limit" name="limit" type="number" min="0" value="500" required>
<small>0 means unlimited; positive values apply per selected source.</small>
</label>
<label>Whole-run deadline
<input id="run-deadline" name="deadline_seconds" type="number" min="30" max="86400">
theHarvester/lib/api/static/harvestview/schedules.html
+2 -1
View File
@@ -94,7 +94,8 @@
<summary>Run limits and optional activity</summary>
<div class="schedule-grid three">
<label>Results per source
<input id="run-limit" type="number" min="1" max="10000" value="500" required>
<input id="run-limit" type="number" min="0" value="500" required>
<small>0 means unlimited; positive values apply per selected source.</small>
</label>
<label>Source workers
<input id="source-workers" type="number" min="1" value="{{SOURCE_WORKERS}}" required>
theHarvester/lib/completed_result.py
+4
View File
@@ -219,6 +219,8 @@ class SourceYield:
observed_result_count: int
unique_result_count: int
shared_result_count: int
resolved_hostname_count: int
unique_resolved_hostname_count: int
def to_dict(self) -> dict[str, str | int]:
return {
@@ -226,6 +228,8 @@ class SourceYield:
'observed_result_count': self.observed_result_count,
'unique_result_count': self.unique_result_count,
'shared_result_count': self.shared_result_count,
'resolved_hostname_count': self.resolved_hostname_count,
'unique_resolved_hostname_count': self.unique_resolved_hostname_count,
}
theHarvester/lib/database.py
+46 -18
View File
@@ -1199,21 +1199,24 @@ class ResultStore:
except Exception as error:
logger.info(f'Unexpected error while storing result: {error}')
async def source_yields(self, run_id: UUID) -> list[SourceYield]:
async def source_yields(self, run_id: UUID, *, kind: ResultKind | None = None) -> list[SourceYield]:
"""Count each source's observed, unique, and shared results for a run.
A result is unique when one source reported it and shared when more than one
source reported it. Sources that ran without results still appear with zero counts.
source reported it. Sources that ran without matching results still appear with
zero counts.
"""
yields = await self._producer_yields(run_id, 'source')
yields = await self._producer_yields(run_id, 'source', kind=kind)
return [
SourceYield(
source=name,
observed_result_count=observed,
unique_result_count=unique,
shared_result_count=shared,
resolved_hostname_count=resolved,
unique_resolved_hostname_count=unique_resolved,
)
for name, observed, unique, shared in yields
for name, observed, unique, shared, resolved, unique_resolved in yields
]
async def action_yields(self, run_id: UUID) -> list[ActionYield]:
@@ -1225,23 +1228,35 @@ class ResultStore:
unique_result_count=unique,
shared_result_count=shared,
)
for name, observed, unique, shared in yields
for name, observed, unique, shared, _resolved, _unique_resolved in yields
]
async def _producer_yields(self, run_id: UUID, producer_kind: str) -> list[tuple[str, int, int, int]]:
async def _producer_yields(
self,
run_id: UUID,
producer_kind: str,
*,
kind: ResultKind | None = None,
) -> list[tuple[str, int, int, int, int, int]]:
async with self._session() as session:
execution_rows = (
await session.scalars(
select(_ExecutionRow).where(
_ExecutionRow.run_id == str(run_id),
_ExecutionRow.producer_kind == producer_kind,
)
)
).all()
result_rows = (await session.scalars(select(_ResultRow).where(_ResultRow.run_id == str(run_id)))).all()
execution_rows = (await session.scalars(select(_ExecutionRow).where(_ExecutionRow.run_id == str(run_id)))).all()
result_query = select(_ResultRow).where(_ResultRow.run_id == str(run_id))
if kind is not None:
result_query = result_query.where(_ResultRow.kind == kind)
result_rows = (await session.scalars(result_query)).all()
origin_rows = (await session.scalars(select(_ResultOriginRow).where(_ResultOriginRow.run_id == str(run_id)))).all()
producer_by_position = {row.position: row.name for row in execution_rows}
producer_by_position = {row.position: row.name for row in execution_rows if row.producer_kind == producer_kind}
dns_resolution_positions = {
row.position for row in execution_rows if row.producer_kind == 'action' and row.name == 'dns-resolve'
}
result_by_position = {row.position: (row.kind, row.value) for row in result_rows}
resolved_hostnames = {
result
for origin in origin_rows
if origin.execution_position in dns_resolution_positions
and (result := result_by_position.get(origin.result_position)) is not None
and result[0] == 'hostname'
}
producers_by_result: dict[tuple[str, str], set[str]] = {}
for origin in origin_rows:
producer = producer_by_position.get(origin.execution_position)
@@ -1251,12 +1266,25 @@ class ResultStore:
observed_counts: Counter[str] = Counter()
unique_counts: Counter[str] = Counter()
shared_counts: Counter[str] = Counter()
for producers in producers_by_result.values():
resolved_counts: Counter[str] = Counter()
unique_resolved_counts: Counter[str] = Counter()
for result, producers in producers_by_result.items():
for producer in producers:
observed_counts[producer] += 1
(unique_counts if len(producers) == 1 else shared_counts)[producer] += 1
if result in resolved_hostnames:
resolved_counts[producer] += 1
if len(producers) == 1:
unique_resolved_counts[producer] += 1
return [
(name, observed_counts[name], unique_counts[name], shared_counts[name])
(
name,
observed_counts[name],
unique_counts[name],
shared_counts[name],
resolved_counts[name],
unique_resolved_counts[name],
)
for name in sorted(producer_by_position.values())
]
theHarvester/lib/enumeration.py
+7 -1
View File
@@ -24,7 +24,7 @@ class EnumerationOptions:
domain: str
source: str | None = None
limit: int = DEFAULT_RESULT_LIMIT
limit: int | None = DEFAULT_RESULT_LIMIT
start: int = DEFAULT_RESULT_START
source_workers: int = DEFAULT_SOURCE_WORKERS
proxies: bool = False
@@ -56,6 +56,12 @@ class EnumerationOptions:
quiet: bool = False
verbose: bool = False
def __post_init__(self) -> None:
if self.limit is not None and self.limit < 0:
raise ValueError('result limit cannot be negative')
if self.limit == 0:
object.__setattr__(self, 'limit', None)
@classmethod
def from_namespace(cls, value: Any) -> Self:
"""Copy CLI or REST-like inputs into the shared execution contract."""
theHarvester/lib/source_catalog.py
+1
View File
@@ -187,6 +187,7 @@ _SPECS = (
),
_spec('shodanct', ResultRoute.SUBDOMAINS),
_spec('sourcegraph', ResultRoute.SUBDOMAINS),
_spec('subdomainapi', ResultRoute.SUBDOMAINS),
_spec('subdomaincenter', ResultRoute.SUBDOMAINS),
_spec('subdomainfinderc99', ResultRoute.SUBDOMAINS, activity=ActivityClass.DNS),
_spec('thc', ResultRoute.SUBDOMAINS),
theHarvester/lib/source_execution.py
+1 -1
View File
@@ -6,7 +6,7 @@ SourceReportStatus = Literal['completed', 'partial', 'failed', 'rate-limited']
@dataclass(frozen=True, slots=True)
class SourceExecutionReport:
"""Provider-specific terminal details for one source execution."""
"""Provider stop details; the source runner determines the final evidence-aware status."""
status: SourceReportStatus
stop_reason: str
theHarvester/lib/source_runner.py
+7 -5
View File
@@ -57,6 +57,7 @@ from theHarvester.discovery import (
shodanct,
shodansearch,
sourcegraph,
subdomainapi,
subdomaincenter,
subdomainfinderc99,
thc,
@@ -95,7 +96,7 @@ class SourceRequest:
source: str
target: str
limit: int
limit: int | None
start: int
proxy: bool
include_hostnames: bool
@@ -144,14 +145,14 @@ SOURCE_FACTORIES: dict[str, SourceFactory] = {
'fofa': lambda request: fofa.SearchFofa(request.target, request.limit),
'fullhunt': lambda request: fullhuntsearch.SearchFullHunt(request.target),
'github-code': lambda request: githubcode.SearchGithubCode(request.target, request.limit),
'gitlab': lambda request: gitlabsearch.SearchGitlab(request.target),
'gitlab': lambda request: gitlabsearch.SearchGitlab(request.target, request.limit),
'hackertarget': lambda request: hackertarget.SearchHackerTarget(request.target),
'haveibeenpwned': lambda request: haveibeenpwned.SearchHaveIBeenPwned(request.target),
'hibpverified': lambda request: hibpverified.SearchHibpVerified(request.target),
'hudsonrock': lambda request: hudsonrocksearch.SearchHudsonRock(request.target),
'hunter': lambda request: huntersearch.SearchHunter(request.target, request.limit, request.start),
'hunterhow': lambda request: searchhunterhow.SearchHunterHow(request.target, request.limit),
'intelx': lambda request: intelxsearch.SearchIntelx(request.target),
'intelx': lambda request: intelxsearch.SearchIntelx(request.target, request.limit),
'leakix': lambda request: leakix.SearchLeakix(request.target),
'leaklookup': lambda request: leaklookup.SearchLeakLookup(request.target),
'mojeek': lambda request: mojeek.SearchMojeek(request.target, request.limit),
@@ -170,15 +171,16 @@ SOURCE_FACTORIES: dict[str, SourceFactory] = {
'shodanInternetDB': lambda request: shodan_internetdb.SearchShodanInternetDB(request.target),
'shodanct': lambda request: shodanct.SearchShodanCt(request.target),
'sourcegraph': lambda request: sourcegraph.SearchSourcegraph(request.target, request.limit),
'subdomainapi': lambda request: subdomainapi.SearchSubdomainApi(request.target),
'subdomaincenter': lambda request: subdomaincenter.SubdomainCenter(request.target),
'subdomainfinderc99': lambda request: subdomainfinderc99.SearchSubdomainfinderc99(request.target),
'thc': lambda request: thc.SearchThc(request.target),
'thc': lambda request: thc.SearchThc(request.target, request.limit),
'tomba': lambda request: tombasearch.SearchTomba(request.target, request.limit, request.start),
'urlscan': lambda request: urlscan.SearchUrlscan(request.target, request.limit),
'virustotal': lambda request: virustotal.SearchVirustotal(request.target, request.limit),
'waybackarchive': lambda request: waybackarchive.SearchWaybackarchive(request.target, request.limit),
'whoisxml': lambda request: whoisxml.SearchWhoisXML(request.target, request.limit),
'windvane': lambda request: windvane.SearchWindvane(request.target),
'windvane': lambda request: windvane.SearchWindvane(request.target, request.limit),
'yahoo': lambda request: yahoosearch.SearchYahoo(request.target, request.limit),
'zoomeye': lambda request: zoomeyesearch.SearchZoomEye(request.target, request.limit),
}
theHarvester/source_yields.py
+129
View File
@@ -0,0 +1,129 @@
from __future__ import annotations
import argparse
import asyncio
import json
import sys
from collections import Counter, defaultdict
from pathlib import Path
from typing import TYPE_CHECKING, cast
from uuid import UUID
from theHarvester.lib.database import ResultStore
from theHarvester.lib.evidence_types import RESULT_KINDS, ResultKind
if TYPE_CHECKING:
from collections.abc import Sequence
def _parser() -> argparse.ArgumentParser:
parser = argparse.ArgumentParser(description='Report per-source result yields from a theHarvester database.')
parser.add_argument(
'--database',
type=Path,
default=Path(ResultStore().database),
help='Existing theHarvester SQLite database (default: %(default)s).',
)
parser.add_argument('--kind', choices=sorted(RESULT_KINDS), default='hostname', help='Result kind to compare.')
parser.add_argument('--run-id', type=UUID, help='Report one completed run instead of aggregating every run.')
parser.add_argument('--format', choices=('table', 'json'), default='table', help='Output format.')
return parser
async def _collect(database: Path, kind: ResultKind, run_id: UUID | None) -> tuple[int, list[dict[str, str | int | float]]]:
store = ResultStore(database)
try:
if run_id is not None:
await store.load_run(run_id)
run_ids = [run_id]
else:
run_ids = [UUID(str(run['run_id'])) for run in await store.list_runs(limit=None)]
totals: defaultdict[str, Counter[str]] = defaultdict(Counter)
for run_id in run_ids:
for source_yield in await store.source_yields(run_id, kind=kind):
totals[source_yield.source].update(
runs=1,
observed=source_yield.observed_result_count,
unique=source_yield.unique_result_count,
shared=source_yield.shared_result_count,
resolved=source_yield.resolved_hostname_count,
unique_resolved=source_yield.unique_resolved_hostname_count,
)
rows = []
for source, counts in totals.items():
row: dict[str, str | int | float] = {
'source': source,
'run_count': counts['runs'],
'observed_result_count': counts['observed'],
'unique_result_count': counts['unique'],
'unique_result_count_per_run': counts['unique'] / counts['runs'],
'shared_result_count': counts['shared'],
}
if kind == 'hostname':
row['resolved_hostname_count'] = counts['resolved']
row['unique_resolved_hostname_count'] = counts['unique_resolved']
row['unique_resolved_hostname_count_per_run'] = counts['unique_resolved'] / counts['runs']
rows.append(row)
rows.sort(
key=lambda row: (
-float(row['unique_result_count_per_run']),
-int(row['unique_result_count']),
-int(row['observed_result_count']),
str(row['source']),
)
)
return len(run_ids), rows
finally:
await store.dispose()
def _table(kind: ResultKind, run_count: int, rows: list[dict[str, str | int | float]]) -> str:
columns = [
('source', 'SOURCE'),
('run_count', 'RUNS'),
('observed_result_count', 'OBSERVED'),
('unique_result_count', 'UNIQUE'),
('unique_result_count_per_run', 'UNIQUE/RUN'),
('shared_result_count', 'SHARED'),
]
if kind == 'hostname':
columns.extend(
(
('resolved_hostname_count', 'RESOLVED'),
('unique_resolved_hostname_count', 'UNIQUE-RESOLVED'),
('unique_resolved_hostname_count_per_run', 'UNIQUE-RESOLVED/RUN'),
)
)
formatted_rows = [
[f'{row[key]:.2f}' if key.endswith('_per_run') else str(row[key]) for key, _label in columns] for row in rows
]
widths = [
max([len(label), *(len(values[index]) for values in formatted_rows)]) for index, (_key, label) in enumerate(columns)
]
lines = [
f'Kind: {kind}',
f'Run count: {run_count}',
' '.join(label.ljust(widths[index]) for index, (_key, label) in enumerate(columns)).rstrip(),
]
lines.extend(
' '.join(value.ljust(widths[index]) for index, value in enumerate(values)).rstrip() for values in formatted_rows
)
return '\n'.join(lines) + '\n'
def main(argv: Sequence[str] | None = None) -> int:
parser = _parser()
args = parser.parse_args(argv)
if not args.database.is_file():
parser.error(f'database does not exist: {args.database}')
kind = cast('ResultKind', args.kind)
try:
run_count, rows = asyncio.run(_collect(args.database, kind, args.run_id))
except LookupError as error:
parser.error(str(error))
if args.format == 'json':
output = json.dumps({'kind': kind, 'run_count': run_count, 'source_yields': rows}, sort_keys=True) + '\n'
else:
output = _table(kind, run_count, rows)
sys.stdout.write(output)
return 0