diff --git a/CONTEXT.md b/CONTEXT.md index ac0c7801..ea594125 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -31,6 +31,7 @@ When a change alters one of these boundaries, update this document and the neare - The source catalog is the authority for canonical source names, aliases, credentials, result capabilities, and activity class. Explicit source names and capability selectors form a union; `all` selects every cataloged P0 source once. - A source adapter returns `None` for ordinary completion or an immutable `SourceExecutionReport` when it must preserve an explicit outcome or stop reason. The central source runner owns observation collection, result counting, no-result classification, exception handling, and final source status. +- A requested result limit of zero means no shared local result-count cap or arbitrary local page-count ceiling; adapters continue to provider exhaustion. Provider quotas, protocol maxima, response, and runtime safety bounds remain source-owned and must surface an explicit partial stop reason when they prevent natural completion. - Source execution statuses are `completed`, `partial`, `failed`, `rate-limited`, and `skipped`. Mutable adapter fields such as `execution_status` and `stop_reason` are outside this release contract and are rejected, including when an adapter raises or is cancelled. - Run lifecycle statuses are `queued`, `running`, `cancelling`, `cancelled`, `completed`, and `failed`. Terminal evidence status is independently `complete`, `partial`, or `failed`; retained evidence survives a later cancellation or process failure. - Run schedules support one-time, hourly, daily, weekly, and monthly recurrence. Daily, weekly, and monthly occurrences preserve the selected local wall-clock time; a monthly day that does not exist falls on that month’s final day. @@ -39,6 +40,7 @@ When a change alters one of these boundaries, update this document and the neare ### Evidence and portability - A merged result is canonical by result kind and value. Source and action provenance, typed observations, execution outcomes, timestamps, and artifact metadata remain attached through deduplication and persistence. +- API run details derive each source's observed, unique, shared, DNS-resolved, and unique DNS-resolved hostname counts from persisted normalized provenance. Unique means no other source in that run reported the hostname; DNS-resolved means that run retained an A, AAAA, or CNAME answer. These measure marginal coverage and current DNS evidence, not independent corroboration or service reachability. - JSONL is the primary single-run interchange format: one summary record followed by sorted finding records. It retains terminal evidence status, producer outcomes, provenance, and supported structured observations. - SQLite is the canonical local multi-run store. Portable SQLite export contains every finalized evidence record, preserves original run IDs and canonical structured evidence, and excludes queue, cancellation, worker-lease, and legacy-observation state. Screenshot metadata travels with evidence; screenshot files remain separately managed artifacts. - Legacy JSON and XML remain supported grouped reports for existing consumers. They are presentation formats and do not replace JSONL or SQLite when lossless provenance and structured evidence are required. @@ -53,6 +55,7 @@ When a change alters one of these boundaries, update this document and the neare ### Deferred boundaries - Cross-run change detection, alerts, and automatic reactions remain separate future product decisions. A scheduled occurrence only submits finite enumeration runs. +- Cross-run source ranking remains a reporting decision. One run's hostname-yield summary does not automatically select, disable, or rank sources. - Distributed workers, multi-host operation, PostgreSQL, and hosted multi-user authorization require measured demand and new decisions. The release remains SQLite-first and local-operator focused. - Automatic scope expansion remains deferred. Evidence can suggest a later target, while the operator controls every scope change. @@ -210,6 +213,10 @@ _Avoid_: Cancelled run, process killed One attempt to run one canonical discovery source within an enumeration run, with an explicit completion status and summary counts. _Avoid_: Source result, provider response +**Source hostname yield**: +One source's observed, unique, shared, DNS-resolved, and unique DNS-resolved normalized hostname counts within one enumeration run. A unique hostname is attributed to exactly one source in that run. A DNS-resolved hostname has an A, AAAA, or CNAME answer retained by that run's resolution action. The counts express marginal coverage and current DNS evidence rather than independent corroboration, ownership, or service reachability. +_Avoid_: Source quality score, authoritative result count, independent confirmation + **Source capability**: A declared class of normalized result that a source can contribute to consolidated enumeration output, independent of whether one source execution yields any data. _Avoid_: Guaranteed result, module return type, source category diff --git a/README.md b/README.md index 38163e18..5034e8e7 100644 --- a/README.md +++ b/README.md @@ -140,7 +140,7 @@ docker compose down | `GET /api/v1/sources` | List registered discovery sources and capabilities. | | `POST /api/v1/runs` | Submit a finite enumeration run. | | `GET /api/v1/runs` | List durable run records. | -| `GET /api/v1/runs/{run_id}` | Retrieve lifecycle state, normalized results, and source outcomes. | +| `GET /api/v1/runs/{run_id}` | Retrieve lifecycle state, normalized results, source outcomes, and hostname yields. | | `POST /api/v1/runs/{run_id}/cancel` | Cancel queued or running work. | | `POST /api/v1/runs/import` | Import JSONL evidence without executing discovery. | | `POST /api/v1/runs/import-database` | Import completed runs from a theHarvester SQLite database. | @@ -167,7 +167,7 @@ Result types in this table always appear in this order: `subdomains`, `emails`, The `shodan` source contributes subdomains. Shodan host enrichment through `-s` or `--shodan` is a separate action and is not a source result route.
-View all 58 discovery sources +View all 59 discovery sources | Source | Returns | Activity | API key | | --- | --- | :---: | :---: | @@ -218,6 +218,7 @@ The `shodan` source contributes subdomains. Shodan host enrichment through `-s` | [`shodanct`](https://ctl.shodan.io/) | subdomains only | P0 | No | | [`shodanInternetDB`](https://internetdb.shodan.io/) | subdomains, ips | P1 | No | | [`sourcegraph`](https://sourcegraph.com/search) | subdomains only | P0 | No | +| [`subdomainapi`](https://api.subdomain.app/) | subdomains only | P0 | No | | [`subdomaincenter`](https://www.subdomain.center/) | subdomains only | P0 | No | | [`subdomainfinderc99`](https://subdomainfinder.c99.nl/) | subdomains only | P1 | No | | [`thc`](https://ip.thc.org/) | subdomains only | P0 | No | diff --git a/docs/adr/0003-run-worker-lifecycle.md b/docs/adr/0003-run-worker-lifecycle.md index 87c4adc3..6fa02e86 100644 --- a/docs/adr/0003-run-worker-lifecycle.md +++ b/docs/adr/0003-run-worker-lifecycle.md @@ -6,7 +6,7 @@ Status: accepted The HTTP application owns a durable run record separate from theHarvester's optional terminal `RunResult` evidence. A submission receives its stable ID while queued. One local worker claims one queued run at a time and executes the finite theHarvester core in an isolated child process. -Lifecycle transitions are `queued -> running -> completed|failed`, `queued -> cancelled`, and `running -> cancelling -> cancelled`. A finite whole-run deadline applies when selected; resolution, reverse, and recursive DNS runs default to unlimited so their complete candidate sets can finish, while other runs retain the 1800-second default. Running cancellation first requests cooperative termination, waits a short grace period, and then forces termination if needed. Queued cancellation is an atomic transition that prevents the worker claim. +Lifecycle transitions are `queued -> running -> completed|failed`, `queued -> cancelled`, and `running -> cancelling -> cancelled`. A finite whole-run deadline applies when selected; resolution, reverse, and recursive DNS runs default to unlimited so their complete candidate sets can finish, while other runs retain the 1800-second default. A result limit of zero removes the shared per-source result-count cap and arbitrary local page-count ceilings without removing provider quotas, protocol maxima, response-size guards, or runtime safety bounds. Running cancellation first requests cooperative termination, waits a short grace period, and then forces termination if needed. Queued cancellation is an atomic transition that prevents the worker claim. Evidence already persisted remains attached after failure or cancellation. Terminal evidence status (`complete`, `partial`, or `failed`) is reported independently from orchestration lifecycle status. On service restart, queued runs may resume; orphaned running or cancelling records become failed because their process ownership cannot be proven. @@ -16,4 +16,6 @@ The existing core is a finite one-shot enumerator and its result object is creat ## Consequences -Run records need one small SQLite table and a lifecycle API. Child-process boundaries make deadline and forced cancellation reliable across blocking provider code. Work is serialized by design. Imported evidence enters as an already completed run record and never enters the queue. Portable database export copies finalized evidence with its original run IDs while omitting lifecycle, cancellation, and worker-lease state. +Run records need one small SQLite table and a lifecycle API. Child-process boundaries make deadline and forced cancellation reliable across blocking provider code. Work is serialized by design. Imported evidence enters as an already completed run record and never enters the queue. Portable database export copies finalized evidence with its original run IDs while omitting lifecycle, cancellation, and worker-lease state. A source stopped by a provider or safety boundary records partial evidence and its exact stop reason instead of presenting the run as complete. + +Run details derive per-source observed, unique, shared, DNS-resolved, and unique DNS-resolved hostname counts from persisted result origins. DNS resolution therefore retains hostname origins alongside returned IP evidence. This adds no tracker table or second evidence path: imports and ordinary runs use the same provenance, sources with no matching hostnames remain visible with zero counts, and cross-run ranking or automatic source selection remains deferred. diff --git a/docs/images/run-evidence-architecture.svg b/docs/images/run-evidence-architecture.svg index d4c0e6cf..9e7519d4 100644 --- a/docs/images/run-evidence-architecture.svg +++ b/docs/images/run-evidence-architecture.svg @@ -99,7 +99,7 @@ SOURCES - 58 discovery adapters + 59 discovery adapters search · CT · DNS data code · archives threat intelligence diff --git a/docs/wiki/Operator-Workflows.md b/docs/wiki/Operator-Workflows.md index 9314b3bf..eddad048 100644 --- a/docs/wiki/Operator-Workflows.md +++ b/docs/wiki/Operator-Workflows.md @@ -16,6 +16,8 @@ uv run theHarvester -d example.com -b crtsh,certspotter,commoncrawl Use the [README source matrix](https://github.com/laramies/theHarvester/blob/dev/README.md#discovery-sources) to choose complementary sources. Adding every source usually increases noise, rate-limit failures, and runtime more than it improves a focused run. +To remove the shared per-source result-count cap and arbitrary local page-count ceilings, pass `--limit 0`; adapters then continue to provider exhaustion. This does not disable provider quotas, protocol maxima, response-size guards, or runtime safety bounds; a source that reaches one records partial evidence with its stop reason. + ## Save results for automation Network activity: provider-facing discovery plus local report writes. diff --git a/docs/wiki/Rest-API.md b/docs/wiki/Rest-API.md index 0e01d70f..0c45a8fe 100644 --- a/docs/wiki/Rest-API.md +++ b/docs/wiki/Rest-API.md @@ -33,7 +33,7 @@ Treat the runtime OpenAPI document as the exact request and response reference. | `GET /api/v1/sources` | List discovery sources, capabilities, activity classes, and credential names. | | `POST /api/v1/runs` | Submit one finite enumeration run. | | `GET /api/v1/runs` | List run records with `limit` and `offset` pagination. | -| `GET /api/v1/runs/{run_id}` | Retrieve lifecycle state, options, results, source outcomes, and artifacts. | +| `GET /api/v1/runs/{run_id}` | Retrieve lifecycle state, options, results, source outcomes and yields, and artifacts. | | `POST /api/v1/runs/{run_id}/cancel` | Cancel queued work or request cancellation of running work. | | `POST /api/v1/runs/import` | Import a JSONL result file without executing discovery. | | `POST /api/v1/runs/import-database` | Import completed runs from a theHarvester SQLite database. | @@ -122,7 +122,7 @@ run_id="$(curl -s http://127.0.0.1:5000/api/v1/runs \ curl -s "http://127.0.0.1:5000/api/v1/runs/$run_id" \ -H "X-API-Key: $THEHARVESTER_API_KEY" \ - | jq '{status, evidence_status, results, source_executions, action_executions, artifacts}' + | jq '{status, evidence_status, results, source_executions, source_yields, action_executions, artifacts}' ``` Run submission is asynchronous. Read the two status fields separately: @@ -134,6 +134,10 @@ Run submission is asynchronous. Read the two status fields separately: `source_workers` is the same positive concurrency used by CLI `-j` or `--source-workers` and HarvestView. It defaults to three, is reduced when fewer sources are selected, and never skips sources or limits their results. +`limit` defaults to 500 per source. Set it to `0` for no shared local result-count cap or arbitrary local page-count ceiling; adapters continue to provider exhaustion and there is no numeric maximum. Provider quotas, protocol maxima, response-size guards, and runtime safety bounds still apply and appear as partial source executions when they prevent natural completion. + +`source_yields` counts normalized hostname contributions within this run. `unique_result_count` is a hostname found by exactly one selected source. When `dns_resolve` ran, `resolved_hostname_count` and `unique_resolved_hostname_count` report the source's A/AAAA/CNAME-answer subsets. Read all four beside that source's execution status and stop reason. The [results guide](Results-and-Local-Data#compare-source-hostname-yield) gives the fixed-run methodology for comparing sources over time. + P1 DNS and P2 direct options are fields on the same run request. The OpenAPI schema shows their current defaults, limits, and descriptions. The server uses the operator-selected target and does not impose a public-only egress policy. ### Query RouteViews diff --git a/docs/wiki/Results-and-Local-Data.md b/docs/wiki/Results-and-Local-Data.md index 4e8f380b..36b598af 100644 --- a/docs/wiki/Results-and-Local-Data.md +++ b/docs/wiki/Results-and-Local-Data.md @@ -154,6 +154,34 @@ Two operational tables support the API: `run_records` stores queue and lifecycle `GET /api/v1/runs/{run_id}` returns lifecycle state plus a normalized `results` array. Each result has `type`, `value`, `sources`, and `actions`. A `hostname` found through the `vhost` action has native endpoint observations; a `prefix` found through RouteViews has native origin, route, and RPKI observations with fixed external-relationship scope. Run-level source and action outcomes remain available in `source_executions` and `action_executions`, while file metadata is returned through `artifacts`. JSONL imports or exports one run. SQLite import and `GET /api/v1/runs/export-database` move completed runs in bulk without queue, cancellation, or worker-lease state. Treat runtime `/docs`, `/redoc`, and OpenAPI as the exact request and response reference. +### Compare source hostname yield + +Run details include `source_yields`, derived from persisted normalized hostname provenance. For every selected source it reports: + +- `observed_result_count`: distinct hostnames attributed to the source; +- `unique_result_count`: hostnames no other source in that run reported; +- `shared_result_count`: hostnames also reported by at least one other source; +- `resolved_hostname_count`: attributed hostnames for which the run's `dns-resolve` action retained an A, AAAA, or CNAME answer; +- `unique_resolved_hostname_count`: resolved hostnames attributed to only this source. + +Use `unique_result_count` as order-independent marginal coverage and `unique_resolved_hostname_count` as its current-DNS subset. Neither proves that a provider is authoritative or independent, and a DNS answer does not prove service reachability. Check the matching `source_executions` status and stop reason: a failed, rate-limited, skipped, or provider-limited source is not a comparable zero-yield source. Providers that draw on the same certificate-transparency or passive-DNS family can still overlap or repeat the same upstream evidence. + +For a reusable benchmark, keep the authorized target, source set, requested limit, release version, resolver set, and collection window fixed. Use requested limit `0` when the comparison requires no shared local result-count cap. Retain the completed runs in SQLite, repeat across several authorized targets and dates, then compare median unique count, median unique-resolved count, resolution rate, and successful-run rate. Treat provider and adapter ceilings as explicit execution evidence rather than silently truncating or ranking them as zero yield. Do not commit target results; add cross-run aggregation only when enough comparable runs exist to justify it. + +#### Analyze yields from SQLite + +The installed `harvest-yields` command reads an existing results database. It reports hostname yields by default; select another result kind or one completed run as needed: + +```console +harvest-yields --database results.sqlite --kind hostname +harvest-yields --database results.sqlite --kind ip +harvest-yields --database results.sqlite --kind asn +harvest-yields --database results.sqlite --run-id 11111111-1111-4111-8111-111111111111 +harvest-yields --database results.sqlite --format json +``` + +Without `--run-id`, counts are sums of each run's source yields. The top-level `run_count` states how many runs were selected; each source row's `run_count` states how many of those runs executed that source, including zero-yield executions. “Unique” always means unique within one run; aggregate unique totals sum those per-run counts rather than recomputing uniqueness across targets or dates. Hostname output adds resolved and unique-resolved counts. Other result kinds omit those DNS-specific fields. + ## Handling and sharing - Store results only where the engagement permits. diff --git a/pyproject.toml b/pyproject.toml index f8858442..32c3ade7 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -49,6 +49,7 @@ dev = [ [project.scripts] theHarvester = "theHarvester.theHarvester:main" harvestview = "theHarvester.harvestview:main" +harvest-yields = "theHarvester.source_yields:main" [tool.pytest.ini_options] minversion = "8.3.3" diff --git a/tests/discovery/test_apisguru.py b/tests/discovery/test_apisguru.py index c1d80aa6..46926972 100644 --- a/tests/discovery/test_apisguru.py +++ b/tests/discovery/test_apisguru.py @@ -563,7 +563,7 @@ async def test_malformed_matching_directory_entry_is_failed( @pytest.mark.asyncio -async def test_directory_entry_scan_is_bounded(monkeypatch: pytest.MonkeyPatch) -> None: +async def test_directory_entry_scan_has_no_local_entry_cap(monkeypatch: pytest.MonkeyPatch) -> None: spec_urls = [f'https://api.apis.guru/v2/specs/example.com/service-{index}/1.0/openapi.json' for index in range(2)] directory = { f'example.com:service-{index}': { @@ -584,6 +584,11 @@ async def test_directory_entry_scan_is_bounded(monkeypatch: pytest.MonkeyPatch) status=200, headers={}, ), + FetcherResponse( + body={'openapi': '3.0.3', 'servers': [{'url': 'https://second.example.com'}]}, + status=200, + headers={}, + ), ] requested_urls: list[str] = [] @@ -591,16 +596,14 @@ async def test_directory_entry_scan_is_bounded(monkeypatch: pytest.MonkeyPatch) requested_urls.append(kwargs['url']) return responses.pop(0) - monkeypatch.setattr(apisguru.SearchApisGuru, 'MAX_DIRECTORY_ENTRIES', 1) monkeypatch.setattr(apisguru.AsyncFetcher, 'fetch_json', fake_fetch) search = apisguru.SearchApisGuru('example.com', limit=5) report = await search.process() - assert requested_urls == ['https://api.apis.guru/v2/example.com.json', spec_urls[0]] - assert await search.get_hostnames() == {'api.example.com'} - assert report.status == 'failed' - assert report.stop_reason == 'directory-entry-limit' + assert requested_urls == ['https://api.apis.guru/v2/example.com.json', *spec_urls] + assert await search.get_hostnames() == {'api.example.com', 'second.example.com'} + assert report is None @pytest.mark.asyncio @@ -840,7 +843,9 @@ async def test_apex_only_hostname_is_not_counted_as_a_retained_result(monkeypatc @pytest.mark.asyncio -async def test_scalar_results_are_hard_bounded(monkeypatch: pytest.MonkeyPatch) -> None: +async def test_scalar_results_use_the_requested_limit_without_a_protective_clamp( + monkeypatch: pytest.MonkeyPatch, +) -> None: spec_url = 'https://api.apis.guru/v2/specs/example.com/1.0/openapi.json' responses = [ FetcherResponse( @@ -872,16 +877,14 @@ async def test_scalar_results_are_hard_bounded(monkeypatch: pytest.MonkeyPatch) async def fake_fetch(**_kwargs: Any) -> FetcherResponse: return responses.pop(0) - monkeypatch.setattr(apisguru.SearchApisGuru, 'MAX_RESULTS_PER_ROUTE', 1) monkeypatch.setattr(apisguru.AsyncFetcher, 'fetch_json', fake_fetch) search = apisguru.SearchApisGuru('example.com', limit=5) report = await search.process() - assert await search.get_hostnames() == {'one.example.com'} - assert await search.get_urls() == {'https://one.example.com'} - assert report.status == 'failed' - assert report.stop_reason == 'result-cap' + assert await search.get_hostnames() == {'one.example.com', 'two.example.com'} + assert await search.get_urls() == {'https://one.example.com', 'https://two.example.com'} + assert report is None @pytest.mark.asyncio diff --git a/tests/discovery/test_arquivo.py b/tests/discovery/test_arquivo.py index ceff39b5..e4c4d063 100644 --- a/tests/discovery/test_arquivo.py +++ b/tests/discovery/test_arquivo.py @@ -1,3 +1,4 @@ +import asyncio import logging from typing import Any from urllib.parse import parse_qs, urlparse @@ -6,6 +7,7 @@ import pytest from theHarvester.discovery import arquivo from theHarvester.lib.core import FetcherResponse +from theHarvester.lib.source_execution import SourceExecutionReport @pytest.mark.asyncio @@ -51,19 +53,58 @@ async def test_process_collects_scoped_hosts_from_one_cdx_request(monkeypatch: p @pytest.mark.asyncio -async def test_process_bounds_the_provider_limit(monkeypatch: pytest.MonkeyPatch) -> None: +async def test_process_paginates_to_the_operator_limit(monkeypatch: pytest.MonkeyPatch) -> None: requested_urls: list[str] = [] + responses = ['{"url": "https://one.example.com"}\n{"url": "https://two.example.com"}', '{"url": "https://three.example.com"}'] async def fake_fetch_all(urls: list[str], **_kwargs: Any) -> list[FetcherResponse]: requested_urls.extend(urls) - return [FetcherResponse(body='', status=200, headers={})] + return [FetcherResponse(body=responses.pop(0), status=200, headers={})] monkeypatch.setattr(arquivo.AsyncFetcher, 'fetch_all', fake_fetch_all) + monkeypatch.setattr(arquivo.SearchArquivo, 'PAGE_SIZE', 2) - search = arquivo.SearchArquivo('example.com', 100_001) + search = arquivo.SearchArquivo('example.com', 3) await search.process() - assert parse_qs(urlparse(requested_urls[0]).query)['limit'] == ['10000'] + queries = [parse_qs(urlparse(url).query) for url in requested_urls] + assert [query['limit'] for query in queries] == [['2'], ['1']] + assert 'offset' not in queries[0] + assert queries[1]['offset'] == ['2'] + assert await search.get_hostnames() == {'one.example.com', 'two.example.com', 'three.example.com'} + + +@pytest.mark.asyncio +async def test_unlimited_request_pages_until_the_provider_is_exhausted(monkeypatch: pytest.MonkeyPatch) -> None: + requested_urls: list[str] = [] + responses = ['{"url": "https://one.example.com"}\n{"url": "https://two.example.com"}', ''] + + async def fake_fetch_all(urls: list[str], **_kwargs: Any) -> list[FetcherResponse]: + requested_urls.extend(urls) + return [FetcherResponse(body=responses.pop(0), status=200, headers={})] + + monkeypatch.setattr(arquivo.AsyncFetcher, 'fetch_all', fake_fetch_all) + monkeypatch.setattr(arquivo.SearchArquivo, 'PAGE_SIZE', 2) + + report = await arquivo.SearchArquivo('example.com', None).process() + + queries = [parse_qs(urlparse(url).query) for url in requested_urls] + assert [query['limit'] for query in queries] == [['2'], ['2']] + assert queries[1]['offset'] == ['2'] + assert report is None + + +@pytest.mark.asyncio +async def test_unlimited_request_reports_repeated_page(monkeypatch: pytest.MonkeyPatch) -> None: + body = '{"url": "https://one.example.com"}' + + async def fake_fetch_all(_urls: list[str], **_kwargs: Any) -> list[FetcherResponse]: + return [FetcherResponse(body=body, status=200, headers={})] + + monkeypatch.setattr(arquivo.AsyncFetcher, 'fetch_all', fake_fetch_all) + monkeypatch.setattr(arquivo.SearchArquivo, 'PAGE_SIZE', 1) + + assert await arquivo.SearchArquivo('example.com', None).process() == SourceExecutionReport('partial', 'repeated-page') @pytest.mark.asyncio @@ -83,14 +124,45 @@ async def test_process_reports_http_and_malformed_responses( with caplog.at_level(logging.INFO, logger=arquivo.__name__): first = arquivo.SearchArquivo('example.com', 500) - await first.process() + first_report = await first.process() second = arquivo.SearchArquivo('example.com', 500) - await second.process() + second_report = await second.process() + assert first_report == SourceExecutionReport('failed', 'http-429') + assert second_report == SourceExecutionReport('failed', 'invalid-response') assert await first.get_hostnames() == set() assert await second.get_hostnames() == set() assert 'Arquivo.pt request failed with HTTP 429' in caplog.text assert 'Arquivo.pt returned malformed CDX data' in caplog.text +@pytest.mark.asyncio +async def test_later_arquivo_failure_preserves_partial_evidence(monkeypatch: pytest.MonkeyPatch) -> None: + responses = [ + FetcherResponse(body='{"url": "https://one.example.com"}', status=200, headers={}), + FetcherResponse(body='unavailable', status=503, headers={}), + ] + + async def fake_fetch_all(_urls: list[str], **_kwargs: Any) -> list[FetcherResponse]: + return [responses.pop(0)] + + monkeypatch.setattr(arquivo.AsyncFetcher, 'fetch_all', fake_fetch_all) + monkeypatch.setattr(arquivo.SearchArquivo, 'PAGE_SIZE', 1) + search = arquivo.SearchArquivo('example.com', None) + + assert await search.process() == SourceExecutionReport('partial', 'http-503') + assert await search.get_hostnames() == {'one.example.com'} + + +@pytest.mark.asyncio +async def test_arquivo_cancellation_propagates(monkeypatch: pytest.MonkeyPatch) -> None: + async def cancel(*_args: Any, **_kwargs: Any) -> list[FetcherResponse]: + raise asyncio.CancelledError('operator-stop') + + monkeypatch.setattr(arquivo.AsyncFetcher, 'fetch_all', cancel) + + with pytest.raises(asyncio.CancelledError, match='operator-stop'): + await arquivo.SearchArquivo('example.com', None).process() + + pytestmark = pytest.mark.provider_contract('arquivo') diff --git a/tests/discovery/test_baidusearch.py b/tests/discovery/test_baidusearch.py index d098013e..cf44e085 100644 --- a/tests/discovery/test_baidusearch.py +++ b/tests/discovery/test_baidusearch.py @@ -196,6 +196,25 @@ def patch_http(monkeypatch: pytest.MonkeyPatch, responses: list[baidusearch.Fetc class TestBaiduSearch: + @pytest.mark.asyncio + async def test_unlimited_stops_when_provider_repeats_a_page(self, monkeypatch: pytest.MonkeyPatch) -> None: + state = patch_browser( + monkeypatch, + [ + PageResponse('one.example.com'), + PageResponse('two.example.com'), + PageResponse('two.example.com'), + ], + ) + search = baidusearch.SearchBaidu(word='example.com', limit=None) + + report = await search.process() + + assert report == baidusearch.SourceExecutionReport('partial', 'repeated-page') + assert len(state.calls) == 3 + assert state.delays == [1.0, 1.0] + assert await search.get_hostnames() == ['one.example.com', 'two.example.com'] + @pytest.mark.asyncio async def test_process_queries_site_first_and_reuses_one_browser(self, monkeypatch: pytest.MonkeyPatch) -> None: state = patch_browser( @@ -267,7 +286,7 @@ class TestBaiduSearch: assert len(state.calls) == 2 assert await search.get_hostnames() == ['api.example.com'] - assert report.status == 'failed' + assert report.status == 'partial' assert report.stop_reason == 'security-verification' @pytest.mark.asyncio @@ -329,6 +348,24 @@ class TestBaiduSearch: assert http.delays == [1.0] assert http.closed + @pytest.mark.asyncio + async def test_http_fallback_reports_repeated_page(self, monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.setattr(baidusearch, 'playwright_api', None) + patch_http(monkeypatch, [http_response('api.example.com'), http_response('api.example.com')]) + + report = await baidusearch.SearchBaidu(word='example.com', limit=None).process() + + assert report == baidusearch.SourceExecutionReport('partial', 'repeated-page') + + @pytest.mark.asyncio + async def test_http_fallback_reports_malformed_response(self, monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.setattr(baidusearch, 'playwright_api', None) + patch_http(monkeypatch, [baidusearch.FetcherResponse(body={}, status=200, headers={})]) + + report = await baidusearch.SearchBaidu(word='example.com', limit=10).process() + + assert report == baidusearch.SourceExecutionReport('failed', 'invalid-response') + @pytest.mark.asyncio async def test_cancellation_survives_cleanup_failures(self, monkeypatch: pytest.MonkeyPatch) -> None: cancellation = asyncio.CancelledError('operator-stop') diff --git a/tests/discovery/test_brave.py b/tests/discovery/test_brave.py index 129b5001..9adf5ff1 100644 --- a/tests/discovery/test_brave.py +++ b/tests/discovery/test_brave.py @@ -440,7 +440,7 @@ async def test_brave_rate_limit_does_not_skip_to_the_next_page( @pytest.mark.asyncio -async def test_brave_reports_maximum_page_offset_as_truncation( +async def test_brave_reports_provider_offset_boundary_as_truncation_when_unlimited( monkeypatch: pytest.MonkeyPatch, brave_credentials: InMemoryCredentialAdapter, ) -> None: @@ -452,11 +452,11 @@ async def test_brave_reports_maximum_page_offset_as_truncation( return _response([_result(len(requests))], more=True) monkeypatch.setattr(bravesearch.AsyncFetcher, 'fetch_json', fake_fetch) - search = bravesearch.SearchBrave('example.com', 1_000, credential_adapter=brave_credentials) + search = bravesearch.SearchBrave('example.com', None, credential_adapter=brave_credentials) report = await search.process() assert [request['offset'] for request in requests] == [[str(offset)] for offset in range(10)] - assert report == SourceExecutionReport('partial', 'pagination-limit') + assert report == SourceExecutionReport('partial', 'provider-limit') pytestmark = pytest.mark.provider_contract('brave') diff --git a/tests/discovery/test_censys.py b/tests/discovery/test_censys.py index c2823dc7..2695ae0a 100644 --- a/tests/discovery/test_censys.py +++ b/tests/discovery/test_censys.py @@ -1,10 +1,11 @@ +from __future__ import annotations + import asyncio import contextlib import sys import types -from collections.abc import AsyncIterator from pathlib import Path -from typing import Any +from typing import TYPE_CHECKING, Any import pytest @@ -23,6 +24,9 @@ from theHarvester.discovery import censysearch from theHarvester.discovery.constants import MissingKey from theHarvester.lib.core import FetcherResponse +if TYPE_CHECKING: + from collections.abc import AsyncIterator + @pytest.mark.asyncio async def test_missing_platform_token_raises(monkeypatch) -> None: @@ -140,6 +144,67 @@ async def test_search_calls_platform_api_directly_and_follows_page_tokens(monkey assert report is None +@pytest.mark.asyncio +async def test_unlimited_search_follows_tokens_until_the_provider_terminates(monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.setattr(censysearch.Core, 'censys_key', lambda: ('platform-token', None)) + calls: list[dict[str, object]] = [] + responses = [ + FetcherResponse({'result': {'hits': [], 'next_page_token': 'next-page'}}, 200, {}), + FetcherResponse({'result': {'hits': [], 'next_page_token': ''}}, 200, {}), + ] + + async def fake_post_fetch(_url: str, **kwargs: object) -> FetcherResponse: + calls.append(kwargs) + return responses.pop(0) + + monkeypatch.setattr(censysearch.AsyncFetcher, 'post_fetch', fake_post_fetch) + search = censysearch.SearchCensys('example.com', limit=None) + + assert await search.process() is None + assert [call['json_body'] for call in calls] == [ + { + 'query': 'cert.names: "example.com"', + 'fields': ['cert.names', 'cert.parsed.subject.email_address'], + 'page_size': 100, + }, + { + 'query': 'cert.names: "example.com"', + 'fields': ['cert.names', 'cert.parsed.subject.email_address'], + 'page_size': 100, + 'page_token': 'next-page', + }, + ] + + +@pytest.mark.parametrize( + ('hit', 'expected_status'), + [ + ({'certificate_v1': {'resource': {'names': ['api.example.com']}}}, 'partial'), + ({'host_v1': {'resource': {'ip': '192.0.2.1'}}}, 'failed'), + ], +) +@pytest.mark.asyncio +async def test_repeated_cursor_status_reflects_retained_evidence( + monkeypatch: pytest.MonkeyPatch, + hit: dict[str, object], + expected_status: str, +) -> None: + monkeypatch.setattr(censysearch.Core, 'censys_key', lambda: ('platform-token', None)) + responses = [ + FetcherResponse({'result': {'hits': [hit], 'next_page_token': 'same'}}, 200, {}), + FetcherResponse({'result': {'hits': [], 'next_page_token': 'same'}}, 200, {}), + ] + + async def fake_post_fetch(*_args: object, **_kwargs: object) -> FetcherResponse: + return responses.pop(0) + + monkeypatch.setattr(censysearch.AsyncFetcher, 'post_fetch', fake_post_fetch) + report = await censysearch.SearchCensys('example.com', limit=None).process() + + assert report.status == expected_status + assert report.stop_reason == 'repeated-cursor' + + @pytest.mark.asyncio async def test_session_setup_failure_reports_transport_error(monkeypatch: pytest.MonkeyPatch) -> None: monkeypatch.setattr(censysearch.Core, 'censys_key', lambda: ('platform-token', 'org-id')) diff --git a/tests/discovery/test_certspotter.py b/tests/discovery/test_certspotter.py index 8efba775..485139a4 100644 --- a/tests/discovery/test_certspotter.py +++ b/tests/discovery/test_certspotter.py @@ -76,7 +76,7 @@ class TestCertspotterSearch: ] @pytest.mark.asyncio - async def test_search_preserves_results_at_page_limit( + async def test_search_continues_until_provider_exhaustion_without_a_page_cap( self, monkeypatch: pytest.MonkeyPatch, caplog: pytest.LogCaptureFixture, @@ -86,22 +86,23 @@ class TestCertspotterSearch: async def fake_fetch_all(*_args: Any, **_kwargs: Any) -> list[list[dict[str, Any]]]: nonlocal requests requests += 1 - if requests > 2: + if requests > 3: return [[]] return [[{'id': f'cursor-{requests}', 'dns_names': [f'host-{requests}.example.com']}]] monkeypatch.setattr(certspottersearch.AsyncFetcher, 'fetch_all', fake_fetch_all) - monkeypatch.setattr(certspottersearch.SearchCertspoter, 'MAX_PAGES', 2, raising=False) - search = certspottersearch.SearchCertspoter(TestCertspotter.domain()) with caplog.at_level(logging.WARNING, logger=certspottersearch.__name__): report = await search.process() - assert requests == 2 - assert await search.get_hostnames() == {'host-1.example.com', 'host-2.example.com'} - assert report.status == 'partial' - assert report.stop_reason == 'page-limit' - assert 'page limit reached; results may be incomplete' in caplog.text + assert requests == 4 + assert await search.get_hostnames() == { + 'host-1.example.com', + 'host-2.example.com', + 'host-3.example.com', + } + assert report is None + assert 'page limit reached' not in caplog.text @pytest.mark.asyncio async def test_search_returns_only_normalized_scoped_names(self, monkeypatch: pytest.MonkeyPatch) -> None: diff --git a/tests/discovery/test_commoncrawl.py b/tests/discovery/test_commoncrawl.py index d2f6b4b0..e0e76767 100644 --- a/tests/discovery/test_commoncrawl.py +++ b/tests/discovery/test_commoncrawl.py @@ -99,6 +99,25 @@ async def test_process_uses_unique_indexes_from_latest_catalog_year_window_and_s assert all(old_endpoint not in url for url in requested_urls) +def test_unlimited_index_selection_has_no_local_history_window() -> None: + catalog = [ + { + 'id': 'CC-MAIN-2026-30', + 'cdx-api': 'https://index.commoncrawl.org/CC-MAIN-2026-30-index', + 'to': '2026-07-12T00:00:00', + }, + { + 'id': 'CC-MAIN-2012', + 'cdx-api': 'https://index.commoncrawl.org/CC-MAIN-2012-index', + 'to': '2012-12-31T00:00:00', + }, + ] + + indexes = commoncrawl.SearchCommoncrawl._select_indexes(catalog, include_all=True) + + assert [index['id'] for index in indexes] == ['CC-MAIN-2026-30', 'CC-MAIN-2012'] + + @pytest.mark.asyncio async def test_process_rejects_untrusted_catalog_endpoints( monkeypatch: pytest.MonkeyPatch, @@ -161,10 +180,7 @@ async def test_process_requests_provider_pages_sequentially(monkeypatch: pytest. @pytest.mark.asyncio -async def test_process_caps_provider_page_counts_and_reports_truncation( - monkeypatch: pytest.MonkeyPatch, - caplog: pytest.LogCaptureFixture, -) -> None: +async def test_unlimited_process_exhausts_provider_reported_page_count(monkeypatch: pytest.MonkeyPatch) -> None: catalog = [ { 'id': 'CC-MAIN-2026-30', @@ -179,21 +195,18 @@ async def test_process_caps_provider_page_counts_and_reports_truncation( return [catalog] query = parse_qs(urlsplit(urls[0]).query) if query.get('showNumPages') == ['true']: - return ['{"pages": 1000000, "pageSize": 5, "blocks": 5000000}'] - requested_pages.extend(int(parse_qs(urlsplit(url).query)['page'][0]) for url in urls) - return ['{"url":"https://api.example.com/"}' for _url in urls] + return ['{"pages": 3, "pageSize": 5, "blocks": 3}'] + page = int(query['page'][0]) + requested_pages.append(page) + return [f'{{"url":"https://host-{page}.example.com/"}}'] monkeypatch.setattr(commoncrawl.AsyncFetcher, 'fetch_all', fake_fetch_all) - monkeypatch.setattr(commoncrawl.SearchCommoncrawl, 'MAX_PAGES_PER_QUERY', 2) + search = commoncrawl.SearchCommoncrawl('example.com', limit=None) + report = await search.process() - search = commoncrawl.SearchCommoncrawl('example.com', limit=50) - with caplog.at_level(logging.WARNING, logger=commoncrawl.__name__): - report = await search.process() - - assert requested_pages == [0, 1, 0, 1] - assert 'Common Crawl page limit reached for index CC-MAIN-2026-30; results may be incomplete' in caplog.text - assert report.status == 'partial' - assert report.stop_reason == 'page-limit' + assert requested_pages == [0, 1, 2, 0, 1, 2] + assert await search.get_hostnames() == {'host-0.example.com', 'host-1.example.com', 'host-2.example.com'} + assert report is None @pytest.mark.asyncio @@ -424,7 +437,7 @@ async def test_process_retains_partial_results_at_the_runtime_limit(monkeypatch: assert await search.get_hostnames() == {'api.example.com'} - assert report.status == 'failed' + assert report.status == 'partial' assert report.stop_reason == 'runtime-limit' diff --git a/tests/discovery/test_criminalip.py b/tests/discovery/test_criminalip.py index 64bc8366..20df027b 100644 --- a/tests/discovery/test_criminalip.py +++ b/tests/discovery/test_criminalip.py @@ -1,10 +1,11 @@ #!/usr/bin/env python3 -# coding=utf-8 +import asyncio import logging import pytest from theHarvester.discovery import criminalip +from theHarvester.lib.source_execution import SourceExecutionReport @pytest.mark.asyncio @@ -18,8 +19,9 @@ async def test_failed_response_body_is_not_logged(monkeypatch, caplog) -> None: monkeypatch.setattr(criminalip.AsyncFetcher, 'post_fetch', fake_post_fetch) caplog.set_level(logging.INFO, logger=criminalip.__name__) - await criminalip.SearchCriminalIP('example.com').process() + report = await criminalip.SearchCriminalIP('example.com').process() + assert report == SourceExecutionReport('failed', 'provider-error') assert 'provider-secret-payload' not in caplog.text assert '500' in caplog.text @@ -32,7 +34,9 @@ async def test_parser_handles_missing_legacy_fields(monkeypatch) -> None: payload = { 'data': { 'certificates': [{'subject': 'www.example.com'}], - 'connected_domain_subdomain': [{'main_domain': {'domain': 'example.com'}, 'subdomains': [{'domain': 'api.example.com'}]}], + 'connected_domain_subdomain': [ + {'main_domain': {'domain': 'example.com'}, 'subdomains': [{'domain': 'api.example.com'}]} + ], 'connected_ip': [{'ip': '192.0.2.34'}], 'connected_ip_info': [ { @@ -122,4 +126,66 @@ async def test_do_search_uses_v2_report_endpoint(monkeypatch) -> None: assert all('/v1/domain/report/' not in url for url in called_urls) +@pytest.mark.asyncio +async def test_waiting_scan_reports_runtime_limit(monkeypatch) -> None: + monkeypatch.setattr(criminalip.Core, 'criminalip_key', lambda: 'test-key') + monkeypatch.setattr(criminalip.Core, 'get_user_agent', lambda: 'test-agent') + status_calls = 0 + + async def fake_post_fetch(*_args, **_kwargs): + return {'status': 200, 'data': {'scan_id': 12345}} + + async def fake_fetch_all(*_args, **_kwargs): + nonlocal status_calls + status_calls += 1 + return [{'status': 200, 'data': {'scan_percentage': 50}}] + + async def no_sleep(*_args, **_kwargs): + return None + + monkeypatch.setattr(criminalip.AsyncFetcher, 'post_fetch', fake_post_fetch) + monkeypatch.setattr(criminalip.AsyncFetcher, 'fetch_all', fake_fetch_all) + monkeypatch.setattr(criminalip.asyncio, 'sleep', no_sleep) + + report = await criminalip.SearchCriminalIP('example.com').process() + + assert status_calls == 10 + assert report == SourceExecutionReport('partial', 'runtime-limit') + + +@pytest.mark.asyncio +async def test_polling_cancellation_propagates(monkeypatch) -> None: + monkeypatch.setattr(criminalip.Core, 'criminalip_key', lambda: 'test-key') + monkeypatch.setattr(criminalip.Core, 'get_user_agent', lambda: 'test-agent') + + async def fake_post_fetch(*_args, **_kwargs): + return {'status': 200, 'data': {'scan_id': 12345}} + + async def fake_fetch_all(*_args, **_kwargs): + return [{'status': 200, 'data': {'scan_percentage': 50}}] + + async def cancel(*_args, **_kwargs): + raise asyncio.CancelledError + + monkeypatch.setattr(criminalip.AsyncFetcher, 'post_fetch', fake_post_fetch) + monkeypatch.setattr(criminalip.AsyncFetcher, 'fetch_all', fake_fetch_all) + monkeypatch.setattr(criminalip.asyncio, 'sleep', cancel) + + with pytest.raises(asyncio.CancelledError): + await criminalip.SearchCriminalIP('example.com').process() + + +@pytest.mark.asyncio +async def test_provider_timeout_returns_explicit_transport_error(monkeypatch) -> None: + monkeypatch.setattr(criminalip.Core, 'criminalip_key', lambda: 'test-key') + monkeypatch.setattr(criminalip.Core, 'get_user_agent', lambda: 'test-agent') + + async def timeout(*_args, **_kwargs): + raise TimeoutError + + monkeypatch.setattr(criminalip.AsyncFetcher, 'post_fetch', timeout) + + assert await criminalip.SearchCriminalIP('example.com').process() == SourceExecutionReport('failed', 'transport-error') + + pytestmark = pytest.mark.provider_contract('criminalip') diff --git a/tests/discovery/test_fofa.py b/tests/discovery/test_fofa.py index f7c734f4..679a918b 100644 --- a/tests/discovery/test_fofa.py +++ b/tests/discovery/test_fofa.py @@ -1,8 +1,9 @@ +from __future__ import annotations + import asyncio import base64 import contextlib -from collections.abc import AsyncIterator -from typing import Any +from typing import TYPE_CHECKING, Any import pytest @@ -11,6 +12,9 @@ from theHarvester.discovery.constants import MissingKey from theHarvester.lib.core import FetcherResponse from theHarvester.lib.source_execution import SourceExecutionReport +if TYPE_CHECKING: + from collections.abc import AsyncIterator + @pytest.mark.provider_contract('fofa') @pytest.mark.asyncio @@ -122,12 +126,23 @@ async def test_failures_are_structured( assert report == SourceExecutionReport(status, reason) +@pytest.mark.parametrize( + ('results', 'expected_status'), + [ + ([['api.example.com', '192.0.2.1']], 'partial'), + ([['outside.test', 'not-an-ip']], 'failed'), + ], +) @pytest.mark.asyncio -async def test_repeated_cursor_stops_without_a_third_request(monkeypatch: pytest.MonkeyPatch) -> None: +async def test_repeated_cursor_status_reflects_retained_evidence( + monkeypatch: pytest.MonkeyPatch, + results: list[list[str]], + expected_status: str, +) -> None: monkeypatch.setattr(fofa.Core, 'fofa_key', lambda: ('test-key', 'operator@example.com')) responses = [ - FetcherResponse({'error': False, 'results': [['api.example.com', '192.0.2.1']], 'next': 'same'}, 200, {}), - FetcherResponse({'error': False, 'results': [['mail.example.com', '192.0.2.2']], 'next': 'same'}, 200, {}), + FetcherResponse({'error': False, 'results': results, 'next': 'same'}, 200, {}), + FetcherResponse({'error': False, 'results': results, 'next': 'same'}, 200, {}), ] @contextlib.asynccontextmanager @@ -142,7 +157,7 @@ async def test_repeated_cursor_stops_without_a_third_request(monkeypatch: pytest search = fofa.SearchFofa('example.com', 10) report = await search.process() - assert report == SourceExecutionReport('failed', 'repeated-cursor') + assert report == SourceExecutionReport(expected_status, 'repeated-cursor') assert responses == [] diff --git a/tests/discovery/test_githubcode_contract.py b/tests/discovery/test_githubcode_contract.py index 61a3a8f7..b03c71e8 100644 --- a/tests/discovery/test_githubcode_contract.py +++ b/tests/discovery/test_githubcode_contract.py @@ -1,11 +1,16 @@ +from __future__ import annotations + +import asyncio import contextlib -from collections.abc import AsyncIterator -from typing import Any +from typing import TYPE_CHECKING, Any import pytest from theHarvester.discovery import githubcode +if TYPE_CHECKING: + from collections.abc import AsyncIterator + class FakeResponse: status = 200 @@ -14,7 +19,7 @@ class FakeResponse: self.payload = payload self.links = links - async def __aenter__(self) -> 'FakeResponse': + async def __aenter__(self) -> FakeResponse: return self async def __aexit__(self, *_args: Any) -> None: @@ -38,7 +43,7 @@ def install_github_responses(monkeypatch: pytest.MonkeyPatch): def __init__(self, *, headers: dict[str, str]) -> None: pass - async def __aenter__(self) -> 'FakeSession': + async def __aenter__(self) -> FakeSession: return self async def __aexit__(self, *_args: Any) -> None: @@ -208,4 +213,104 @@ async def test_github_code_malformed_page_terminates_without_following_paginatio assert await search.get_hostnames() == [] +@pytest.mark.asyncio +async def test_github_code_unlimited_pagination_cycle_preserves_partial_evidence(install_github_responses) -> None: + requested_urls = install_github_responses( + FakeResponse( + {'items': [{'text_matches': [{'fragment': 'api.example.com'}]}]}, + {'next': {'url': 'https://api.github.com/search/code?q=example.com&page=2'}}, + ), + FakeResponse( + {'items': [{'text_matches': [{'fragment': 'docs.example.com'}]}]}, + {'next': {'url': 'https://api.github.com/search/code?q=example.com&page=1'}}, + ), + ) + search = githubcode.SearchGithubCode('example.com', limit=None) + + report = await search.process() + + assert requested_urls == [ + 'https://api.github.com/search/code?q="example.com"&page=1', + 'https://api.github.com/search/code?q="example.com"&page=2', + ] + assert await search.get_hostnames() == ['api.example.com', 'docs.example.com'] + assert report == githubcode.SourceExecutionReport('partial', 'repeated-page') + + +@pytest.mark.asyncio +async def test_github_code_unlimited_repeated_content_stops_before_counting_duplicates( + install_github_responses, +) -> None: + requested_urls = install_github_responses( + FakeResponse( + { + 'items': [ + {'text_matches': [{'fragment': 'api.example.com'}, {'fragment': 'docs.example.com'}]}, + ] + }, + {'next': {'url': 'https://api.github.com/search/code?q=example.com&page=2'}}, + ), + FakeResponse( + { + 'items': [ + {'text_matches': [{'fragment': 'docs.example.com'}, {'fragment': 'api.example.com'}]}, + ] + }, + {'next': {'url': 'https://api.github.com/search/code?q=example.com&page=3'}}, + ), + ) + search = githubcode.SearchGithubCode('example.com', limit=None) + + report = await search.process() + + assert requested_urls == [ + 'https://api.github.com/search/code?q="example.com"&page=1', + 'https://api.github.com/search/code?q="example.com"&page=2', + ] + assert search.counter == 2 + assert await search.get_hostnames() == ['api.example.com', 'docs.example.com'] + assert report == githubcode.SourceExecutionReport('partial', 'repeated-page') + + +@pytest.mark.parametrize('fragments', [[], ['api.example.com']], ids=['no-evidence', 'partial-evidence']) +@pytest.mark.asyncio +async def test_github_code_persistent_exceptions_stop_with_truthful_report( + monkeypatch: pytest.MonkeyPatch, + fragments: list[str], +) -> None: + monkeypatch.setattr(githubcode.Core, 'github_key', staticmethod(lambda: 'test-token')) + monkeypatch.setattr(githubcode, 'get_delay', lambda: 0) + search = githubcode.SearchGithubCode('example.com', limit=None) + search.total_results = ' '.join(fragments) + search.counter = len(fragments) + search.max_retries = 1 + calls = 0 + + async def fail_search(*_args: Any, **_kwargs: Any) -> tuple[str, dict, int, Any]: + nonlocal calls + calls += 1 + raise RuntimeError('persistent provider failure') + + monkeypatch.setattr(search, 'do_search', fail_search) + + report = await search.process() + + assert calls == 2 + assert report == githubcode.SourceExecutionReport('partial' if fragments else 'failed', 'transport-error') + + +@pytest.mark.asyncio +async def test_github_code_cancellation_propagates(monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.setattr(githubcode.Core, 'github_key', staticmethod(lambda: 'test-token')) + search = githubcode.SearchGithubCode('example.com', limit=None) + + async def cancel_search(*_args: Any, **_kwargs: Any) -> tuple[str, dict, int, Any]: + raise asyncio.CancelledError('operator-stop') + + monkeypatch.setattr(search, 'do_search', cancel_search) + + with pytest.raises(asyncio.CancelledError, match='operator-stop'): + await search.process() + + pytestmark = pytest.mark.provider_contract('github-code') diff --git a/tests/discovery/test_gitlabsearch.py b/tests/discovery/test_gitlabsearch.py index af4bda03..064b2abd 100644 --- a/tests/discovery/test_gitlabsearch.py +++ b/tests/discovery/test_gitlabsearch.py @@ -1,13 +1,21 @@ +from __future__ import annotations + import json import sys -from pathlib import Path -from typing import Any +from typing import TYPE_CHECKING, Any +from urllib.parse import parse_qs, urlparse import pytest from theHarvester import __main__ as theharvester_main from theHarvester.discovery import gitlabsearch -from theHarvester.lib.completed_result import CompletedResult +from theHarvester.lib.core import FetcherResponse +from theHarvester.lib.source_execution import SourceExecutionReport + +if TYPE_CHECKING: + from pathlib import Path + + from theHarvester.lib.completed_result import CompletedResult @pytest.mark.asyncio @@ -53,12 +61,12 @@ async def test_public_discovery_normalizes_evidence_and_uses_bounded_requests( url = next(iter(urls)) requests.append({'url': url, 'headers': headers, 'proxy': proxy}) responses = { - 'https://gitlab.com/api/v4/projects?search=example.test&per_page=20': json.dumps(projects), + 'https://gitlab.com/api/v4/projects?search=example.test&per_page=100&page=1': json.dumps(projects), 'https://gitlab.com/api/v4/projects/group%2Fproject/repository/files/README.md/raw?ref=feature%2Freadme': ( 'Contact Admin@Example.TEST. at docs.example.test; ignore admin@notexample.test' ), - 'https://gitlab.com/api/v4/projects?search=*.example.test&per_page=20': '[]', - 'https://gitlab.com/api/v4/users?search=example.test&per_page=10': json.dumps(users), + 'https://gitlab.com/api/v4/projects?search=*.example.test&per_page=100&page=1': '[]', + 'https://gitlab.com/api/v4/users?search=example.test&per_page=100&page=1': json.dumps(users), } if url not in responses: raise AssertionError(f'unexpected GitLab request: {url}') @@ -73,10 +81,10 @@ async def test_public_discovery_normalizes_evidence_and_uses_bounded_requests( assert requests == [ {'url': url, 'headers': {'User-agent': 'UA'}, 'proxy': True} for url in ( - 'https://gitlab.com/api/v4/projects?search=example.test&per_page=20', + 'https://gitlab.com/api/v4/projects?search=example.test&per_page=100&page=1', 'https://gitlab.com/api/v4/projects/group%2Fproject/repository/files/README.md/raw?ref=feature%2Freadme', - 'https://gitlab.com/api/v4/projects?search=*.example.test&per_page=20', - 'https://gitlab.com/api/v4/users?search=example.test&per_page=10', + 'https://gitlab.com/api/v4/projects?search=*.example.test&per_page=100&page=1', + 'https://gitlab.com/api/v4/users?search=example.test&per_page=100&page=1', ) ] assert await search.get_hostnames() == { @@ -129,6 +137,103 @@ async def test_decoded_pages_are_accepted_without_silent_slicing(monkeypatch: py assert 'user-11@example.test' in emails +@pytest.mark.asyncio +async def test_unlimited_search_follows_project_and_user_pagination(monkeypatch: pytest.MonkeyPatch) -> None: + requests: list[tuple[str, str, int, int]] = [] + + async def fake_fetch_all(urls: list[str] | set[str], **kwargs: Any) -> list[FetcherResponse]: + assert kwargs['include_metadata'] is True + assert kwargs['json'] is True + parsed = urlparse(next(iter(urls))) + query = parse_qs(parsed.query) + term = query['search'][0] + page = int(query['page'][0]) + per_page = int(query['per_page'][0]) + endpoint = parsed.path.rsplit('/', 1)[-1] + requests.append((endpoint, term, page, per_page)) + records: dict[tuple[str, str, int], tuple[list[dict[str, object]], str]] = { + ('projects', 'example.test', 1): ([{'description': 'first.example.test'}], '2'), + ('projects', 'example.test', 2): ([{'description': 'second.example.test'}], ''), + ('projects', '*.example.test', 1): ([{'description': 'wildcard.example.test'}], ''), + ('users', 'example.test', 1): ([{'public_email': 'first@example.test'}], '2'), + ('users', 'example.test', 2): ([{'public_email': 'second@example.test'}], ''), + } + body, next_page = records[(endpoint, term, page)] + return [FetcherResponse(body, 200, {'x-next-page': next_page})] + + monkeypatch.setattr(gitlabsearch.AsyncFetcher, 'fetch_all', fake_fetch_all) + search = gitlabsearch.SearchGitlab('example.test', None) + + assert await search.process() is None + assert await search.get_hostnames() == { + 'first.example.test', + 'second.example.test', + 'wildcard.example.test', + } + assert await search.get_emails() == {'first@example.test', 'second@example.test'} + assert requests == [ + ('projects', 'example.test', 1, 100), + ('projects', 'example.test', 2, 100), + ('projects', '*.example.test', 1, 100), + ('users', 'example.test', 1, 100), + ('users', 'example.test', 2, 100), + ] + + +@pytest.mark.asyncio +async def test_unlimited_search_reports_repeated_gitlab_cursor(monkeypatch: pytest.MonkeyPatch) -> None: + async def fake_fetch_all(*_args: Any, **_kwargs: Any) -> list[FetcherResponse]: + return [FetcherResponse([{'description': 'api.example.test'}], 200, {'x-next-page': '1'})] + + monkeypatch.setattr(gitlabsearch.AsyncFetcher, 'fetch_all', fake_fetch_all) + search = gitlabsearch.SearchGitlab('example.test', None) + + assert await search.process() == SourceExecutionReport('partial', 'repeated-cursor') + + +@pytest.mark.asyncio +async def test_unlimited_search_reports_gitlab_provider_bound(monkeypatch: pytest.MonkeyPatch) -> None: + calls = 0 + + async def fake_fetch_all(*_args: Any, **_kwargs: Any) -> list[FetcherResponse]: + nonlocal calls + calls += 1 + if calls == 1: + return [FetcherResponse([{'description': 'api.example.test'}], 200, {'x-next-page': '2'})] + return [FetcherResponse({'message': 'Pagination limit reached'}, 400, {})] + + monkeypatch.setattr(gitlabsearch.AsyncFetcher, 'fetch_all', fake_fetch_all) + search = gitlabsearch.SearchGitlab('example.test', None) + + assert await search.process() == SourceExecutionReport('partial', 'provider-limit') + assert await search.get_hostnames() == {'api.example.test'} + + +@pytest.mark.asyncio +async def test_finite_limit_stops_each_gitlab_query(monkeypatch: pytest.MonkeyPatch) -> None: + requests: list[tuple[str, str, int, int]] = [] + + async def fake_fetch_all(urls: list[str] | set[str], **_kwargs: Any) -> list[FetcherResponse]: + parsed = urlparse(next(iter(urls))) + query = parse_qs(parsed.query) + endpoint = parsed.path.rsplit('/', 1)[-1] + term = query['search'][0] + page = int(query['page'][0]) + per_page = int(query['per_page'][0]) + requests.append((endpoint, term, page, per_page)) + return [FetcherResponse([{}], 200, {'x-next-page': '2'})] + + monkeypatch.setattr(gitlabsearch.AsyncFetcher, 'fetch_all', fake_fetch_all) + search = gitlabsearch.SearchGitlab('example.test', 1) + + assert await search.process() == SourceExecutionReport('completed', 'result-limit') + assert requests == [ + ('projects', 'example.test', 1, 1), + ('projects', '*.example.test', 1, 1), + ('users', 'example.test', 1, 1), + ] + + @pytest.mark.asyncio async def test_gitlab_urls_reach_completed_jsonl( monkeypatch: pytest.MonkeyPatch, @@ -147,8 +252,8 @@ async def test_gitlab_urls_reach_completed_jsonl( completed_results.append(result) class FakeGitlab: - def __init__(self, domain: str) -> None: - assert domain == 'example.test' + def __init__(self, domain: str, limit: int | None) -> None: + assert (domain, limit) == ('example.test', 500) async def process(self, _proxy: bool) -> None: return None diff --git a/tests/discovery/test_hunterhow.py b/tests/discovery/test_hunterhow.py index b3b1dfcb..aa7b6af4 100644 --- a/tests/discovery/test_hunterhow.py +++ b/tests/discovery/test_hunterhow.py @@ -76,6 +76,28 @@ async def test_process_paginates_to_limit_and_keeps_scoped_hostnames(monkeypatch assert session_exited is True +@pytest.mark.asyncio +async def test_unlimited_search_has_no_local_history_window(monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.setattr(searchhunterhow.Core, 'hunterhow_key', staticmethod(lambda: 'test-key')) + calls: list[dict[str, Any]] = [] + + @contextlib.asynccontextmanager + async def fake_open_session(**_kwargs: Any) -> AsyncIterator[object]: + yield object() + + async def fake_fetch(**kwargs: Any) -> FetcherResponse: + calls.append(kwargs) + return FetcherResponse({'code': 200, 'data': {'total': 0, 'list': []}}, 200, {}) + + monkeypatch.setattr(searchhunterhow.AsyncFetcher, 'open_session', fake_open_session) + monkeypatch.setattr(searchhunterhow.AsyncFetcher, 'fetch', fake_fetch) + + report = await searchhunterhow.SearchHunterHow('example.com', limit=None).process() + + assert report is None + assert calls[0]['params']['start_time'] == '1970-01-01' + + @pytest.mark.parametrize('key', [None, '', ' ']) def test_missing_or_empty_key_fails_before_transport(monkeypatch: pytest.MonkeyPatch, key: str | None) -> None: monkeypatch.setattr(searchhunterhow.Core, 'hunterhow_key', staticmethod(lambda: key)) diff --git a/tests/discovery/test_huntersearch.py b/tests/discovery/test_huntersearch.py index 7c567295..01e14c80 100644 --- a/tests/discovery/test_huntersearch.py +++ b/tests/discovery/test_huntersearch.py @@ -6,6 +6,7 @@ import pytest from theHarvester.discovery import huntersearch from theHarvester.discovery.constants import MissingKey from theHarvester.lib.core import FetcherResponse +from theHarvester.lib.source_execution import SourceExecutionReport @pytest.mark.parametrize('key', [None, ' ']) @@ -34,7 +35,6 @@ async def test_hunter_http_failures_return_no_results(monkeypatch, caplog, statu assert await search.get_emails() == [] assert await search.get_hostnames() == [] - assert f'Hunter request failed with HTTP {status}' in caplog.text assert 'provider detail' not in caplog.text @@ -196,6 +196,35 @@ async def test_free_hunter_search_honors_limit_and_offset(monkeypatch) -> None: ] +@pytest.mark.asyncio +async def test_free_hunter_unlimited_reports_saturated_provider_boundary(monkeypatch) -> None: + responses = iter( + [ + {'data': {'plan_name': 'Free', 'requests': {'searches': {'available': 10, 'used': 0}}}}, + { + 'data': { + 'emails': [ + {'value': f'user{index}@example.test', 'sources': [{'domain': f'user{index}.example.test'}]} + for index in range(10) + ] + } + }, + ] + ) + + async def fake_fetch_all(*_args, **_kwargs): + return [FetcherResponse(body=next(responses), status=200, headers={})] + + monkeypatch.setattr(huntersearch.Core, 'hunter_key', lambda: 'test-key') + monkeypatch.setattr(huntersearch.Core, 'get_user_agent', lambda: 'test-agent') + monkeypatch.setattr(huntersearch.AsyncFetcher, 'fetch_all', fake_fetch_all) + + search = huntersearch.SearchHunter('example.test', None, 0) + + assert await search.process() == SourceExecutionReport('partial', 'provider-limit') + assert len(await search.get_hostnames()) == 10 + + @pytest.mark.asyncio async def test_paid_hunter_search_stops_before_exceeding_quota(monkeypatch) -> None: requests: list[str] = [] @@ -203,6 +232,13 @@ async def test_paid_hunter_search_stops_before_exceeding_quota(monkeypatch) -> N [ {'data': {'plan_name': 'Growth', 'requests': {'searches': {'available': 1, 'used': 0}}}}, {'data': {'total': 250}}, + { + 'data': { + 'emails': [ + {'value': 'alice@example.test', 'sources': [{'domain': 'api.example.test'}]}, + ] + } + }, ] ) @@ -215,14 +251,16 @@ async def test_paid_hunter_search_stops_before_exceeding_quota(monkeypatch) -> N monkeypatch.setattr(huntersearch.AsyncFetcher, 'fetch_all', fake_fetch_all) search = huntersearch.SearchHunter('example.test', 250, 0) - await search.process() + report = await search.process() assert requests == [ 'https://api.hunter.io/v2/account?api_key=test-key', 'https://api.hunter.io/v2/email-count?domain=example.test', + 'https://api.hunter.io/v2/domain-search?domain=example.test&api_key=test-key&limit=100&offset=0', ] - assert await search.get_emails() == [] - assert await search.get_hostnames() == [] + assert report == SourceExecutionReport('partial', 'quota-exhausted') + assert await search.get_emails() == ['alice@example.test'] + assert await search.get_hostnames() == ['api.example.test'] pytestmark = pytest.mark.provider_contract('hunter') diff --git a/tests/discovery/test_intelxsearch.py b/tests/discovery/test_intelxsearch.py index f45c400a..5d575da0 100644 --- a/tests/discovery/test_intelxsearch.py +++ b/tests/discovery/test_intelxsearch.py @@ -9,8 +9,9 @@ from theHarvester.lib.completed_result import CompletedResult, ResultObservation class _Response: - def __init__(self, payload: object) -> None: + def __init__(self, payload: object, status: int = 200) -> None: self.payload = payload + self.status = status async def __aenter__(self): return self @@ -24,8 +25,9 @@ class _Response: class _Session: def __init__(self, result: object, search_result: object | None = None) -> None: - self.result = result + self.result = list(result) if isinstance(result, list) else result self.search_result = {'success': True, 'id': 'search-id'} if search_result is None else search_result + self.search_request: dict[str, object] | None = None async def __aenter__(self): return self @@ -33,11 +35,12 @@ class _Session: async def __aexit__(self, *_args) -> None: return None - def post(self, *_args, **_kwargs) -> _Response: + def post(self, *_args, **kwargs) -> _Response: + self.search_request = kwargs.get('json') return _Response(self.search_result) def get(self, *_args, **_kwargs) -> _Response: - return _Response(self.result) + return _Response(self.result.pop(0) if isinstance(self.result, list) else self.result) def test_blank_key_is_missing(monkeypatch: pytest.MonkeyPatch) -> None: @@ -50,6 +53,7 @@ def test_blank_key_is_missing(monkeypatch: pytest.MonkeyPatch) -> None: @pytest.mark.asyncio async def test_process_exposes_flat_normalized_in_scope_results(monkeypatch: pytest.MonkeyPatch) -> None: result = { + 'status': 1, 'selectors': [ {'selectorvalue': 'ADMIN@Example.COM'}, {'selectorvalue': 'bad local@example.com'}, @@ -64,7 +68,7 @@ async def test_process_exposes_flat_normalized_in_scope_results(monkeypatch: pyt {'selectorvalue': 'http://['}, {'selectorvalue': None}, None, - ] + ], } session = _Session(result) @@ -83,6 +87,41 @@ async def test_process_exposes_flat_normalized_in_scope_results(monkeypatch: pyt assert await search.get_urls() == ['https://portal.example.com/path'] +@pytest.mark.asyncio +async def test_unlimited_process_collects_provider_pages_until_terminal_status(monkeypatch: pytest.MonkeyPatch) -> None: + pages = [ + {'status': 0, 'selectors': [{'selectorvalue': 'one.example.com'}]}, + {'status': 1, 'selectors': [{'selectorvalue': 'two.example.com'}]}, + ] + session = _Session(pages) + + monkeypatch.setattr(intelxsearch.Core, 'intelx_key', staticmethod(lambda: 'test-key')) + monkeypatch.setattr(intelxsearch.aiohttp, 'ClientSession', lambda: session) + + search = intelxsearch.SearchIntelx('example.com', limit=None) + + assert await search.process() is None + assert await search.get_hostnames() == ['one.example.com', 'two.example.com'] + assert session.search_request['maxresults'] == intelxsearch.SearchIntelx.UNLIMITED_QUERY_RESULTS + + +@pytest.mark.asyncio +async def test_finite_process_stops_after_requested_selector_count(monkeypatch: pytest.MonkeyPatch) -> None: + pages = [ + {'status': 0, 'selectors': [{'selectorvalue': 'one.example.com'}, {'selectorvalue': 'two.example.com'}]}, + {'status': 1, 'selectors': [{'selectorvalue': 'three.example.com'}]}, + ] + session = _Session(pages) + + monkeypatch.setattr(intelxsearch.Core, 'intelx_key', staticmethod(lambda: 'test-key')) + monkeypatch.setattr(intelxsearch.aiohttp, 'ClientSession', lambda: session) + + search = intelxsearch.SearchIntelx('example.com', limit=2) + + assert await search.process() is None + assert await search.get_hostnames() == ['one.example.com', 'two.example.com'] + + @pytest.mark.asyncio @pytest.mark.parametrize( ('search_result', 'result'), @@ -125,7 +164,7 @@ async def test_orchestrator_stores_intelx_subdomains_without_dns(monkeypatch: py completed_results.append(result) class _Intelx: - def __init__(self, _domain: str) -> None: + def __init__(self, _domain: str, _limit: int | None) -> None: pass async def process(self, _proxy: bool) -> None: diff --git a/tests/discovery/test_netlas.py b/tests/discovery/test_netlas.py index 5f7dc99d..1265d2ef 100644 --- a/tests/discovery/test_netlas.py +++ b/tests/discovery/test_netlas.py @@ -67,6 +67,57 @@ async def test_process_uses_current_api_contract_and_keeps_scoped_results(monkey assert session_exited is True +@pytest.mark.asyncio +async def test_unlimited_request_covers_the_provider_count_error_margin(monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.setattr(netlas.Core, 'netlas_key', staticmethod(lambda: 'test-key')) + + @contextlib.asynccontextmanager + async def fake_open_session(**_kwargs: Any) -> AsyncIterator[object]: + yield object() + + count_calls: list[dict[str, Any]] = [] + download_calls: list[dict[str, Any]] = [] + + async def fake_fetch(*args: Any, **kwargs: Any) -> FetcherResponse: + count_calls.append({'url': args[0] if args else kwargs['url'], **kwargs}) + return FetcherResponse({'count': 12_345}, 200, {}) + + async def fake_post_fetch(*_args: Any, **kwargs: Any) -> FetcherResponse: + download_calls.append(kwargs) + return FetcherResponse([], 200, {}) + + monkeypatch.setattr(netlas.AsyncFetcher, 'open_session', fake_open_session) + monkeypatch.setattr(netlas.AsyncFetcher, 'fetch', fake_fetch) + monkeypatch.setattr(netlas.AsyncFetcher, 'post_fetch', fake_post_fetch) + + report = await netlas.SearchNetlas('example.com', None).process() + + assert count_calls[0]['url'] == 'https://app.netlas.io/api/domains_count/' + assert count_calls[0]['params'] == {'q': '*.example.com'} + assert download_calls[0]['json_body']['size'] == 12_727 + assert report is None + + +@pytest.mark.asyncio +async def test_unlimited_request_rejects_an_invalid_provider_count(monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.setattr(netlas.Core, 'netlas_key', staticmethod(lambda: 'test-key')) + + @contextlib.asynccontextmanager + async def fake_open_session(**_kwargs: Any) -> AsyncIterator[object]: + yield object() + + async def fake_fetch(*_args: Any, **_kwargs: Any) -> FetcherResponse: + return FetcherResponse({'count': 'many'}, 200, {}) + + monkeypatch.setattr(netlas.AsyncFetcher, 'open_session', fake_open_session) + monkeypatch.setattr(netlas.AsyncFetcher, 'fetch', fake_fetch) + + report = await netlas.SearchNetlas('example.com', None).process() + + assert report.status == 'failed' + assert report.stop_reason == 'invalid-response' + + @pytest.mark.parametrize('key', [None, '', ' ']) def test_missing_or_blank_key_fails_closed(monkeypatch: pytest.MonkeyPatch, key: str | None) -> None: monkeypatch.setattr(netlas.Core, 'netlas_key', staticmethod(lambda: key)) diff --git a/tests/discovery/test_onyphe.py b/tests/discovery/test_onyphe.py index b2fc479f..8a449f8d 100644 --- a/tests/discovery/test_onyphe.py +++ b/tests/discovery/test_onyphe.py @@ -1,7 +1,8 @@ +from __future__ import annotations + import asyncio import contextlib -from collections.abc import AsyncIterator -from typing import Any +from typing import TYPE_CHECKING, Any import pytest @@ -9,6 +10,9 @@ from theHarvester.discovery import onyphe from theHarvester.discovery.constants import MissingKey from theHarvester.lib.core import FetcherResponse +if TYPE_CHECKING: + from collections.abc import AsyncIterator + @pytest.mark.provider_contract('onyphe') @pytest.mark.asyncio @@ -196,7 +200,8 @@ async def test_operator_limit_is_not_reported_as_a_provider_limit(monkeypatch: p @pytest.mark.asyncio -async def test_search_api_reports_its_documented_total_boundary(monkeypatch: pytest.MonkeyPatch) -> None: +@pytest.mark.parametrize('limit', [None, 10_001]) +async def test_search_api_reports_its_documented_total_boundary(monkeypatch: pytest.MonkeyPatch, limit: int | None) -> None: monkeypatch.setattr(onyphe.Core, 'onyphe_key', lambda: 'test-key') calls: list[dict[str, Any]] = [] response = FetcherResponse( @@ -220,13 +225,13 @@ async def test_search_api_reports_its_documented_total_boundary(monkeypatch: pyt monkeypatch.setattr(onyphe.AsyncFetcher, 'open_session', fake_open_session) monkeypatch.setattr(onyphe.AsyncFetcher, 'fetch', fake_fetch) - search = onyphe.SearchOnyphe('example.com', 10_001) + search = onyphe.SearchOnyphe('example.com', limit) report = await search.process() assert [call['params']['page'] for call in calls] == [1] assert [call['params']['size'] for call in calls] == [10_000] assert await search.get_hostnames() == {'api.example.com'} - assert report.status == 'failed' + assert report.status == 'partial' assert report.stop_reason == 'provider-limit' diff --git a/tests/discovery/test_pentesttools.py b/tests/discovery/test_pentesttools.py index 38cfbf17..261a248a 100644 --- a/tests/discovery/test_pentesttools.py +++ b/tests/discovery/test_pentesttools.py @@ -1,8 +1,10 @@ +import asyncio import logging import pytest from theHarvester.discovery import pentesttools +from theHarvester.lib.source_execution import SourceExecutionReport @pytest.mark.asyncio @@ -107,8 +109,9 @@ async def test_status_error_payload_is_not_logged(monkeypatch, caplog) -> None: monkeypatch.setattr(pentesttools.asyncio, 'sleep', no_sleep) caplog.set_level(logging.INFO, logger=pentesttools.__name__) - await pentesttools.SearchPentestTools('example.com').process() + report = await pentesttools.SearchPentestTools('example.com').process() + assert report == SourceExecutionReport('failed', 'provider-error') assert 'provider-secret-payload' not in caplog.text assert 'private target data' not in caplog.text assert 'did not finish successfully' in caplog.text @@ -134,7 +137,6 @@ async def test_malformed_start_response_completes_without_evidence(monkeypatch, assert not await search.get_hostnames() assert not await search.get_ips() - assert 'malformed' in caplog.text @@ -159,12 +161,42 @@ async def test_waiting_scan_stops_after_ten_status_checks(monkeypatch, caplog) - monkeypatch.setattr(pentesttools.asyncio, 'sleep', no_sleep) caplog.set_level(logging.INFO, logger=pentesttools.__name__) - await pentesttools.SearchPentestTools('example.test').process() + report = await pentesttools.SearchPentestTools('example.test').process() assert status_calls == 10 + assert report == SourceExecutionReport('partial', 'runtime-limit') assert 'still waiting after 10 status checks' in caplog.text +@pytest.mark.asyncio +async def test_polling_cancellation_propagates(monkeypatch) -> None: + monkeypatch.setattr(pentesttools.Core, 'pentest_tools_key', lambda: 'test-key') + + async def fake_post_fetch(**_kwargs): + return {'data': {'created_id': 420323}} + + async def cancel(*_args, **_kwargs): + raise asyncio.CancelledError + + monkeypatch.setattr(pentesttools.AsyncFetcher, 'post_fetch', fake_post_fetch) + monkeypatch.setattr(pentesttools.asyncio, 'sleep', cancel) + + with pytest.raises(asyncio.CancelledError): + await pentesttools.SearchPentestTools('example.test').process() + + +@pytest.mark.asyncio +async def test_provider_timeout_returns_explicit_transport_error(monkeypatch) -> None: + monkeypatch.setattr(pentesttools.Core, 'pentest_tools_key', lambda: 'test-key') + + async def timeout(**_kwargs): + raise TimeoutError + + monkeypatch.setattr(pentesttools.AsyncFetcher, 'post_fetch', timeout) + + assert await pentesttools.SearchPentestTools('example.test').process() == SourceExecutionReport('failed', 'transport-error') + + @pytest.mark.asyncio async def test_malformed_status_response_completes_without_evidence(monkeypatch, caplog) -> None: monkeypatch.setattr(pentesttools.Core, 'pentest_tools_key', lambda: 'test-key') diff --git a/tests/discovery/test_securityscorecard.py b/tests/discovery/test_securityscorecard.py index 694fae28..e6c4ef09 100644 --- a/tests/discovery/test_securityscorecard.py +++ b/tests/discovery/test_securityscorecard.py @@ -157,6 +157,49 @@ async def test_malformed_asset_rows_preserve_valid_partial_results(monkeypatch: assert report.stop_reason == 'invalid-response' +@pytest.mark.parametrize( + ('entry', 'expected_status', 'expected_hosts'), + [ + ({'domain': 'api.example.com'}, 'partial', {'api.example.com'}), + ({'domain': 'outside.test'}, 'failed', set()), + ], +) +@pytest.mark.asyncio +async def test_unlimited_repeated_asset_page_stops_with_truthful_report( + monkeypatch: pytest.MonkeyPatch, + entry: dict[str, str], + expected_status: str, + expected_hosts: set[str], +) -> None: + monkeypatch.setattr(securityscorecard.Core, 'securityscorecard_key', staticmethod(lambda: 'test-key')) + monkeypatch.setattr(securityscorecard.SearchSecurityScorecard, 'PAGE_SIZE', 1) + calls = 0 + + @contextlib.asynccontextmanager + async def fake_open_session(**_kwargs: Any) -> AsyncIterator[object]: + yield object() + + async def fake_fetch(**_kwargs: Any) -> FetcherResponse: + return FetcherResponse({}, 200, {}) + + async def fake_post_fetch(*_args: Any, **_kwargs: Any) -> FetcherResponse: + nonlocal calls + calls += 1 + return FetcherResponse({'entries': [entry], 'size': 2}, 200, {}) + + monkeypatch.setattr(securityscorecard.AsyncFetcher, 'open_session', fake_open_session) + monkeypatch.setattr(securityscorecard.AsyncFetcher, 'fetch', fake_fetch) + monkeypatch.setattr(securityscorecard.AsyncFetcher, 'post_fetch', fake_post_fetch) + search = securityscorecard.SearchSecurityScorecard('example.com', None) + + report = await search.process() + + assert calls == 2 + assert await search.get_hostnames() == expected_hosts + assert report.status == expected_status + assert report.stop_reason == 'repeated-page' + + @pytest.mark.asyncio async def test_cancellation_closes_provider_session(monkeypatch: pytest.MonkeyPatch) -> None: monkeypatch.setattr(securityscorecard.Core, 'securityscorecard_key', staticmethod(lambda: 'test-key')) diff --git a/tests/discovery/test_shodan_engine.py b/tests/discovery/test_shodan_engine.py index 10a746ec..29000f2a 100644 --- a/tests/discovery/test_shodan_engine.py +++ b/tests/discovery/test_shodan_engine.py @@ -393,6 +393,52 @@ class TestShodanEngine: assert hosts['198.51.100.21']['services'][0]['tls'] == {'subject_cn': 'cert.example.test'} assert report is None + @pytest.mark.asyncio + @pytest.mark.parametrize( + ('hostnames', 'expected_status'), + [(('api.example.test',), 'partial'), (('outside.invalid',), 'failed')], + ) + async def test_shodan_repeated_search_page_is_not_counted_toward_provider_total( + self, + monkeypatch, + hostnames, + expected_status, + ): + from theHarvester.discovery import shodansearch + from theHarvester.lib.core import FetcherResponse + + search_calls = [] + repeated_match = { + 'ip_str': '198.51.100.20', + 'hostnames': list(hostnames), + 'port': 443, + 'transport': 'tcp', + } + + async def fetch_json(url, *, params, **_kwargs): + if not url.endswith('/search'): + return FetcherResponse(body=None, status=404, headers={}) + search_calls.append((params['query'], params['page'])) + if params['query'] == 'hostname:example.test': + return FetcherResponse(body={'matches': [repeated_match], 'total': 3}, status=200, headers={}) + return FetcherResponse(body={'matches': [], 'total': 0}, status=200, headers={}) + + monkeypatch.setattr(shodansearch.Core, 'shodan_key', lambda: 'test-key') + monkeypatch.setattr(shodansearch.AsyncFetcher, 'fetch_json', fetch_json) + patch_resolution(monkeypatch, shodansearch) + search = shodansearch.SearchShodan('example.test') + + report = await search.process() + + assert search_calls == [ + ('hostname:example.test', 1), + ('hostname:example.test', 2), + ('ssl:example.test', 1), + ] + assert await search.get_hostnames() == ({'api.example.test'} if expected_status == 'partial' else set()) + assert report.status == expected_status + assert report.stop_reason == 'repeated-page' + @pytest.mark.asyncio async def test_shodan_discovery_counts_service_only_evidence_as_a_result(self, monkeypatch): from theHarvester.discovery import shodansearch diff --git a/tests/discovery/test_sourcegraph.py b/tests/discovery/test_sourcegraph.py index 30a6a872..348f7327 100644 --- a/tests/discovery/test_sourcegraph.py +++ b/tests/discovery/test_sourcegraph.py @@ -95,7 +95,7 @@ async def test_sourcegraph_uses_fixed_chunk_query_and_collects_descendants( event('done', {}), ) calls = install_stream(monkeypatch, records) - search = sourcegraph.SearchSourcegraph(' Scope.TEST. ', limit=1) + search = sourcegraph.SearchSourcegraph(' Scope.TEST. ', limit=500) report = await search.process(proxy=True) @@ -364,7 +364,7 @@ async def test_sourcegraph_rejects_events_after_done(monkeypatch: pytest.MonkeyP @pytest.mark.asyncio -async def test_sourcegraph_preserves_prefix_at_hostname_limit(monkeypatch: pytest.MonkeyPatch) -> None: +async def test_sourcegraph_honors_finite_result_limit_after_terminal_stream(monkeypatch: pytest.MonkeyPatch) -> None: install_stream( monkeypatch, ( @@ -381,36 +381,36 @@ async def test_sourcegraph_preserves_prefix_at_hostname_limit(monkeypatch: pytes event('done', {}), ), ) - monkeypatch.setattr(sourcegraph.SearchSourcegraph, 'MAX_HOSTNAMES', 1) - search = sourcegraph.SearchSourcegraph('scope.test', limit=500) + search = sourcegraph.SearchSourcegraph('scope.test', limit=1) report = await search.process() assert await search.get_hostnames() == ['one.scope.test'] - assert report.status == 'failed' - assert report.stop_reason == 'response-limit' + assert report.status == 'completed' + assert report.stop_reason == 'result-limit' @pytest.mark.asyncio -async def test_sourcegraph_preserves_prefix_at_event_limit(monkeypatch: pytest.MonkeyPatch) -> None: +async def test_sourcegraph_unlimited_mode_has_no_event_or_hostname_cap(monkeypatch: pytest.MonkeyPatch) -> None: + hostnames = ' '.join(f'host-{number}.scope.test' for number in range(10_001)) install_stream( monkeypatch, ( event( 'matches', - [{'type': 'content', 'chunkMatches': [{'content': 'api.scope.test'}]}], + [{'type': 'content', 'chunkMatches': [{'content': hostnames}]}], ), - event('progress', {'done': False, 'skipped': []}), + *(event('filters', []) for _ in range(10_000)), + event('progress', {'done': True, 'skipped': []}), + event('done', {}), ), ) - monkeypatch.setattr(sourcegraph.SearchSourcegraph, 'MAX_EVENTS', 1) - search = sourcegraph.SearchSourcegraph('scope.test', limit=500) + search = sourcegraph.SearchSourcegraph('scope.test', limit=None) report = await search.process() - assert await search.get_hostnames() == ['api.scope.test'] - assert report.status == 'failed' - assert report.stop_reason == 'response-limit' + assert len(await search.get_hostnames()) == 10_001 + assert report is None @pytest.mark.asyncio diff --git a/tests/discovery/test_subdomainapi.py b/tests/discovery/test_subdomainapi.py new file mode 100644 index 00000000..35965d0c --- /dev/null +++ b/tests/discovery/test_subdomainapi.py @@ -0,0 +1,116 @@ +import asyncio +from typing import Any + +import pytest + +from theHarvester.discovery import subdomainapi +from theHarvester.lib.core import FetcherResponse +from theHarvester.lib.source_execution import SourceExecutionReport + + +@pytest.mark.asyncio +async def test_process_keeps_every_returned_scoped_name_and_reports_the_provider_limit( + monkeypatch: pytest.MonkeyPatch, +) -> None: + captured: dict[str, Any] = {} + + async def fake_fetch_all(urls: list[str], **kwargs: Any) -> list[FetcherResponse]: + captured.update({'urls': urls, **kwargs}) + return [ + FetcherResponse( + body={ + 'domain': 'example.com', + 'count': 6, + 'total': 10_006, + 'subdomains': [ + 'API.Example.COM.', + 'api.example.com', + 'deep.api.example.com', + '*.wild.example.com', + 'example.com', + 'outside.test', + ], + }, + status=200, + headers={}, + ) + ] + + monkeypatch.setattr(subdomainapi.AsyncFetcher, 'fetch_all', fake_fetch_all) + search = subdomainapi.SearchSubdomainApi('example.com') + + report = await search.process(proxy=True) + + assert await search.get_hostnames() == {'api.example.com', 'deep.api.example.com'} + assert report == SourceExecutionReport('partial', 'provider-limit') + assert captured == { + 'urls': ['https://api.subdomain.app/v1/query?domain=example.com'], + 'headers': {'User-Agent': subdomainapi.Core.get_user_agent()}, + 'proxy': True, + 'json': True, + 'include_metadata': True, + } + + +@pytest.mark.asyncio +async def test_process_accepts_a_valid_empty_response(monkeypatch: pytest.MonkeyPatch) -> None: + async def fake_fetch_all(*_args: Any, **_kwargs: Any) -> list[FetcherResponse]: + return [ + FetcherResponse( + body={'domain': 'example.com', 'count': 0, 'total': 0, 'subdomains': []}, + status=200, + headers={}, + ) + ] + + monkeypatch.setattr(subdomainapi.AsyncFetcher, 'fetch_all', fake_fetch_all) + search = subdomainapi.SearchSubdomainApi('example.com') + + assert await search.process() is None + assert await search.get_hostnames() == set() + + +@pytest.mark.parametrize( + ('response', 'expected'), + [ + (None, SourceExecutionReport('failed', 'transport-error')), + (FetcherResponse({}, 403, {}), SourceExecutionReport('failed', 'access-denied')), + (FetcherResponse({}, 429, {}), SourceExecutionReport('rate-limited', 'http-429')), + (FetcherResponse({}, 503, {}), SourceExecutionReport('failed', 'http-503')), + (FetcherResponse([], 200, {}), SourceExecutionReport('failed', 'invalid-response')), + ( + FetcherResponse( + {'domain': 'example.com', 'count': 2, 'total': 2, 'subdomains': ['api.example.com']}, + 200, + {}, + ), + SourceExecutionReport('failed', 'invalid-response'), + ), + ], +) +@pytest.mark.asyncio +async def test_process_reports_provider_and_response_failures( + monkeypatch: pytest.MonkeyPatch, + response: FetcherResponse | None, + expected: SourceExecutionReport, +) -> None: + async def fake_fetch_all(*_args: Any, **_kwargs: Any) -> list[FetcherResponse | None]: + return [response] + + monkeypatch.setattr(subdomainapi.AsyncFetcher, 'fetch_all', fake_fetch_all) + + assert await subdomainapi.SearchSubdomainApi('example.com').process() == expected + + +@pytest.mark.asyncio +async def test_process_propagates_cancellation(monkeypatch: pytest.MonkeyPatch) -> None: + async def cancelled_fetch(*_args: Any, **_kwargs: Any) -> list[FetcherResponse]: + raise asyncio.CancelledError + + monkeypatch.setattr(subdomainapi.AsyncFetcher, 'fetch_all', cancelled_fetch) + + with pytest.raises(asyncio.CancelledError): + await subdomainapi.SearchSubdomainApi('example.com').process() + + +pytestmark = pytest.mark.provider_contract('subdomainapi') diff --git a/tests/discovery/test_thc.py b/tests/discovery/test_thc.py index 0cdb74f7..ea91fc4d 100644 --- a/tests/discovery/test_thc.py +++ b/tests/discovery/test_thc.py @@ -1,5 +1,4 @@ #!/usr/bin/env python3 -# coding=utf-8 """Tests for the THC (ip.thc.org) discovery source. THC provides multiple endpoints: @@ -9,8 +8,10 @@ THC provides multiple endpoints: API documentation: https://ip.thc.org/docs/ """ -from types import TracebackType -from typing import Any, Self + +from __future__ import annotations + +from typing import TYPE_CHECKING, Any, Self from urllib.parse import parse_qs, urlparse import httpx @@ -19,6 +20,9 @@ import pytest from theHarvester.discovery import thc from theHarvester.lib.core import Core +if TYPE_CHECKING: + from types import TracebackType + class FakeResponse: def __init__(self, text: str, status: int = 200, headers: dict[str, str] | None = None) -> None: @@ -176,6 +180,27 @@ class TestThcSubdomainSearch: result_list = list(result) assert len(result_list) == len(set(result_list)) + @pytest.mark.asyncio + async def test_unlimited_uses_provider_max_and_reports_saturation( + self, + monkeypatch: pytest.MonkeyPatch, + ) -> None: + requested_urls: list[str] = [] + + class RecordingSession(FakeSession): + def get(self, url: str) -> FakeResponse: + requested_urls.append(url) + return FakeResponse('one.example.com\ntwo.example.com\n') + + monkeypatch.setattr(thc.SearchThc, 'PROVIDER_MAX_RESULTS', 2) + monkeypatch.setattr(thc.aiohttp, 'ClientSession', RecordingSession) + + report = await thc.SearchThc(self.domain(), None).process() + + assert parse_qs(urlparse(requested_urls[0]).query)['limit'] == ['2'] + assert report.status == 'partial' + assert report.stop_reason == 'provider-limit' + @pytest.mark.asyncio @pytest.mark.parametrize( ('outcomes', 'message'), @@ -425,12 +450,14 @@ class TestThcIntegration: async def test_module_can_be_imported(self) -> None: """Import the THC discovery module.""" from theHarvester.discovery import thc as thc_module + assert thc_module is not None @pytest.mark.asyncio async def test_search_class_exists(self) -> None: """Expose the ``SearchThc`` adapter.""" from theHarvester.discovery import thc as thc_module + assert hasattr(thc_module, 'SearchThc') @pytest.mark.asyncio diff --git a/tests/discovery/test_tombasearch.py b/tests/discovery/test_tombasearch.py index 99554328..83530a09 100644 --- a/tests/discovery/test_tombasearch.py +++ b/tests/discovery/test_tombasearch.py @@ -6,6 +6,7 @@ import pytest from theHarvester.discovery import tombasearch from theHarvester.discovery.constants import MissingKey from theHarvester.lib.core import FetcherResponse +from theHarvester.lib.source_execution import SourceExecutionReport def tomba_page(first: int, count: int = 50) -> dict: @@ -56,7 +57,6 @@ async def test_tomba_http_failures_return_no_results(monkeypatch, caplog, status assert await search.get_emails() == [] assert await search.get_hostnames() == [] - assert f'Tomba request failed with HTTP {status}' in caplog.text assert 'provider detail' not in caplog.text @@ -264,6 +264,33 @@ async def test_free_tomba_search_honors_limit_and_start(monkeypatch) -> None: } +@pytest.mark.asyncio +async def test_free_tomba_unlimited_reports_saturated_provider_boundary(monkeypatch) -> None: + responses = iter( + [ + { + 'data': { + 'pricing': {'name': 'Free'}, + 'requests': {'domains': {'available': 10, 'used': 0}}, + } + }, + tomba_page(0, 10), + ] + ) + + async def fake_fetch_all(*_args, **_kwargs): + return [FetcherResponse(body=next(responses), status=200, headers={})] + + monkeypatch.setattr(tombasearch.Core, 'tomba_key', lambda: ('test-key', 'test-secret')) + monkeypatch.setattr(tombasearch.Core, 'get_user_agent', lambda: 'test-agent') + monkeypatch.setattr(tombasearch.AsyncFetcher, 'fetch_all', fake_fetch_all) + + search = tombasearch.SearchTomba('example.test', None, 0) + + assert await search.process() == SourceExecutionReport('partial', 'provider-limit') + assert len(await search.get_hostnames()) == 10 + + @pytest.mark.asyncio async def test_paid_tomba_search_stops_before_exceeding_quota(monkeypatch) -> None: requests: list[str] = [] @@ -276,6 +303,8 @@ async def test_paid_tomba_search_stops_before_exceeding_quota(monkeypatch) -> No } }, {'data': {'total': 120}}, + tomba_page(0), + tomba_page(50), ] ) @@ -288,14 +317,17 @@ async def test_paid_tomba_search_stops_before_exceeding_quota(monkeypatch) -> No monkeypatch.setattr(tombasearch.AsyncFetcher, 'fetch_all', fake_fetch_all) search = tombasearch.SearchTomba('example.test', 120, 0) - await search.process() + report = await search.process() assert requests == [ 'https://api.tomba.io/v1/me', 'https://api.tomba.io/v1/email-count?domain=example.test', + 'https://api.tomba.io/v1/domain-search?domain=example.test&limit=50&page=1', + 'https://api.tomba.io/v1/domain-search?domain=example.test&limit=50&page=2', ] - assert await search.get_emails() == [] - assert await search.get_hostnames() == [] + assert report == SourceExecutionReport('partial', 'quota-exhausted') + assert len(await search.get_emails()) == 100 + assert len(await search.get_hostnames()) == 100 pytestmark = pytest.mark.provider_contract('tomba') diff --git a/tests/discovery/test_urlscan.py b/tests/discovery/test_urlscan.py index 990ee275..38f07cd6 100644 --- a/tests/discovery/test_urlscan.py +++ b/tests/discovery/test_urlscan.py @@ -1,8 +1,9 @@ +from __future__ import annotations + import asyncio import contextlib -from collections.abc import AsyncIterator from datetime import UTC, datetime, timedelta -from typing import Any +from typing import TYPE_CHECKING, Any import pytest @@ -10,6 +11,9 @@ from theHarvester.discovery import urlscan from theHarvester.lib.core import FetcherResponse from theHarvester.lib.source_execution import SourceExecutionReport +if TYPE_CHECKING: + from collections.abc import AsyncIterator + class ProviderSession: def __init__(self) -> None: @@ -375,16 +379,28 @@ async def test_missing_cursor_stops_after_first_page(monkeypatch: pytest.MonkeyP assert report == SourceExecutionReport('failed', 'invalid-cursor') +@pytest.mark.parametrize( + ('domains', 'expected_status', 'expected_hostnames'), + [ + (('first.example.com', 'second.example.com'), 'partial', {'first.example.com', 'second.example.com'}), + (('outside.test', 'other.test'), 'failed', set()), + ], +) @pytest.mark.asyncio -async def test_repeated_cursor_stops_without_a_third_request(monkeypatch: pytest.MonkeyPatch) -> None: +async def test_repeated_cursor_status_reflects_retained_evidence( + monkeypatch: pytest.MonkeyPatch, + domains: tuple[str, str], + expected_status: str, + expected_hostnames: set[str], +) -> None: responses = [ FetcherResponse( - body={'results': [{'page': {'domain': 'first.example.com'}, 'sort': [1, 'same']}]}, + body={'results': [{'page': {'domain': domains[0]}, 'sort': [1, 'same']}]}, status=200, headers={}, ), FetcherResponse( - body={'results': [{'page': {'domain': 'second.example.com'}, 'sort': [1, 'same']}]}, + body={'results': [{'page': {'domain': domains[1]}, 'sort': [1, 'same']}]}, status=200, headers={}, ), @@ -402,8 +418,8 @@ async def test_repeated_cursor_stops_without_a_third_request(monkeypatch: pytest report = await search.process() assert calls == 2 - assert await search.get_hostnames() == {'first.example.com', 'second.example.com'} - assert report == SourceExecutionReport('failed', 'repeated-cursor') + assert await search.get_hostnames() == expected_hostnames + assert report == SourceExecutionReport(expected_status, 'repeated-cursor') @pytest.mark.asyncio diff --git a/tests/discovery/test_virustotal.py b/tests/discovery/test_virustotal.py index 0b2c7349..0055fc6d 100644 --- a/tests/discovery/test_virustotal.py +++ b/tests/discovery/test_virustotal.py @@ -180,12 +180,24 @@ async def test_later_rate_limit_preserves_partial_results(monkeypatch: pytest.Mo assert report.stop_reason == 'http-429' +@pytest.mark.parametrize( + ('identifiers', 'expected_status', 'expected_hostnames'), + [ + (('outside.test', 'api.example.com'), 'partial', {'api.example.com'}), + (('outside.test', 'other.test'), 'failed', set()), + ], +) @pytest.mark.asyncio -async def test_repeated_cursor_stops_without_spending_more_quota(monkeypatch: pytest.MonkeyPatch) -> None: +async def test_repeated_cursor_status_reflects_retained_evidence( + monkeypatch: pytest.MonkeyPatch, + identifiers: tuple[str, str], + expected_status: str, + expected_hostnames: set[str], +) -> None: monkeypatch.setattr(virustotal.Core, 'virustotal_key', staticmethod(lambda: 'test-key')) responses = [ - FetcherResponse({'data': [{'id': 'outside.test', 'attributes': {}}], 'meta': {'cursor': 'same'}}, 200, {}), - FetcherResponse({'data': [{'id': 'api.example.com', 'attributes': {}}], 'meta': {'cursor': 'same'}}, 200, {}), + FetcherResponse({'data': [{'id': identifiers[0], 'attributes': {}}], 'meta': {'cursor': 'same'}}, 200, {}), + FetcherResponse({'data': [{'id': identifiers[1], 'attributes': {}}], 'meta': {'cursor': 'same'}}, 200, {}), ] @contextlib.asynccontextmanager @@ -201,8 +213,8 @@ async def test_repeated_cursor_stops_without_spending_more_quota(monkeypatch: py report = await search.process() - assert await search.get_hostnames() == {'api.example.com'} - assert report.status == 'failed' + assert await search.get_hostnames() == expected_hostnames + assert report.status == expected_status assert report.stop_reason == 'repeated-cursor' assert responses == [] diff --git a/tests/discovery/test_waybackarchive.py b/tests/discovery/test_waybackarchive.py index 5044dc92..ac265361 100644 --- a/tests/discovery/test_waybackarchive.py +++ b/tests/discovery/test_waybackarchive.py @@ -167,7 +167,7 @@ async def test_process_ignores_empty_html_and_non_text_responses( @pytest.mark.asyncio -async def test_process_respects_the_per_query_page_bound( +async def test_process_continues_until_provider_exhaustion_without_a_page_bound( monkeypatch: pytest.MonkeyPatch, caplog: pytest.LogCaptureFixture, ) -> None: @@ -176,24 +176,24 @@ async def test_process_respects_the_per_query_page_bound( async def fake_fetch_all(urls: list[str], **_kwargs: object) -> list[str]: nonlocal wildcard_requests query = parse_qs(urlparse(urls[0]).query) - if query['url'] == ['*.example.com']: + if query['url'] == ['*.example.com'] and wildcard_requests < 3: wildcard_requests += 1 return [f'https://host-{wildcard_requests}.example.com/path\n\npage-{wildcard_requests}'] return [''] monkeypatch.setattr(waybackarchive.AsyncFetcher, 'fetch_all', fake_fetch_all) - monkeypatch.setattr(waybackarchive.SearchWaybackarchive, 'MAX_PAGES_PER_QUERY', 2) - search = waybackarchive.SearchWaybackarchive('example.com') with caplog.at_level(logging.INFO, logger=waybackarchive.__name__): report = await search.process() - assert wildcard_requests == 2 - assert await search.get_hostnames() == {'host-1.example.com', 'host-2.example.com'} - assert 'Wayback Archive page limit reached for pattern *.example.com; results may be incomplete' in caplog.text - assert report is not None - assert report.status == 'partial' - assert report.stop_reason == 'page-limit' + assert wildcard_requests == 3 + assert await search.get_hostnames() == { + 'host-1.example.com', + 'host-2.example.com', + 'host-3.example.com', + } + assert 'Wayback Archive page limit reached' not in caplog.text + assert report is None @pytest.mark.asyncio @@ -219,12 +219,28 @@ async def test_process_retains_partial_results_at_the_runtime_limit( report = await search.process() assert await search.get_hostnames() == {'api.example.com'} - assert report.status == 'failed' + assert report.status == 'partial' assert report.stop_reason == 'runtime-limit' assert 'Wayback Archive page 1: hosts=1' in caplog.text assert 'example.com' not in caplog.text +@pytest.mark.asyncio +async def test_process_reports_a_runtime_limit_before_collecting_results(monkeypatch: pytest.MonkeyPatch) -> None: + async def fake_fetch_all(_urls: list[str], **_kwargs: object) -> list[str]: + await asyncio.Event().wait() + + monkeypatch.setattr(waybackarchive.AsyncFetcher, 'fetch_all', fake_fetch_all) + monkeypatch.setattr(waybackarchive.SearchWaybackarchive, 'RUNTIME_SECONDS', 0.01) + search = waybackarchive.SearchWaybackarchive('example.com') + + report = await asyncio.wait_for(search.process(), timeout=0.1) + + assert await search.get_hostnames() == set() + assert report.status == 'failed' + assert report.stop_reason == 'runtime-limit' + + @pytest.mark.asyncio async def test_process_stops_at_the_requested_result_limit(monkeypatch: pytest.MonkeyPatch) -> None: requested_urls: list[str] = [] diff --git a/tests/discovery/test_whoisxml.py b/tests/discovery/test_whoisxml.py index 57c1c89e..954a6289 100644 --- a/tests/discovery/test_whoisxml.py +++ b/tests/discovery/test_whoisxml.py @@ -192,6 +192,43 @@ async def test_later_page_failure_preserves_partial_results(monkeypatch: pytest. assert report.stop_reason == 'http-429' +@pytest.mark.parametrize( + ('domains', 'expected_status'), + [ + (('api.example.com', 'mail.example.com'), 'partial'), + (('outside.test', 'other.test'), 'failed'), + ], +) +@pytest.mark.asyncio +async def test_repeated_cursor_status_reflects_retained_evidence( + monkeypatch: pytest.MonkeyPatch, + domains: tuple[str, str], + expected_status: str, +) -> None: + monkeypatch.setattr(whoisxml.Core, 'whoisxml_key', staticmethod(lambda: 'test-key')) + responses = [ + FetcherResponse( + {'result': {'nextPageSearchAfter': 'same', 'records': [{'domain': domains[0]}]}}, + 200, + {}, + ), + FetcherResponse( + {'result': {'nextPageSearchAfter': 'same', 'records': [{'domain': domains[1]}]}}, + 200, + {}, + ), + ] + + async def fake_fetch(**_kwargs: Any) -> FetcherResponse: + return responses.pop(0) + + monkeypatch.setattr(whoisxml.AsyncFetcher, 'fetch', fake_fetch) + report = await whoisxml.SearchWhoisXML('example.com', 10).process() + + assert report.status == expected_status + assert report.stop_reason == 'repeated-cursor' + + @pytest.mark.asyncio async def test_cancellation_closes_provider_session( monkeypatch: pytest.MonkeyPatch, diff --git a/tests/discovery/test_windvane.py b/tests/discovery/test_windvane.py index 161da274..c2722ac2 100644 --- a/tests/discovery/test_windvane.py +++ b/tests/discovery/test_windvane.py @@ -4,6 +4,7 @@ import socket import pytest from theHarvester.discovery import windvane +from theHarvester.lib.source_execution import SourceExecutionReport @pytest.mark.asyncio @@ -81,4 +82,107 @@ async def test_keyless_provider_failure_does_not_guess_dns_names(monkeypatch) -> assert await search.get_ips() == set() +@pytest.mark.asyncio +async def test_authenticated_unlimited_search_follows_all_endpoint_pagination(monkeypatch) -> None: + monkeypatch.setattr(windvane.Core, 'windvane_key', lambda: 'test-key') + requests: list[tuple[str, int, int]] = [] + + async def fake_post_fetch(url, headers=None, data=None, proxy=False): + endpoint = url.rsplit('/', 1)[-1] + page_request = json.loads(data)['page_request'] + page = page_request['page'] + requests.append((endpoint, page, page_request['count'])) + last_pages = {'ListSubDomain': 4, 'ListDNS': 3, 'ListEmail': 2} + last_page = last_pages[endpoint] + if page > last_page: + return json.dumps({'code': 0, 'data': {'list': [], 'has_more': False}}) + if endpoint == 'ListSubDomain': + item = {'domain': f'sub-{page}.example.test'} + elif endpoint == 'ListDNS': + item = {'domain': f'dns-{page}.example.test', 'answer': f'203.0.113.{page}', 'answer_type': 'A'} + else: + item = {'email': f'user-{page}@example.test'} + return json.dumps({'code': 0, 'data': {'list': [item], 'has_more': page < last_page}}) + + monkeypatch.setattr(windvane.AsyncFetcher, 'post_fetch', fake_post_fetch) + search = windvane.SearchWindvane('example.test', None) + + assert await search.process() is None + assert 'sub-4.example.test' in await search.get_hostnames() + assert 'dns-3.example.test' in await search.get_hostnames() + assert 'user-2@example.test' in await search.get_emails() + assert requests == [ + ('ListSubDomain', 1, 30), + ('ListSubDomain', 2, 30), + ('ListSubDomain', 3, 30), + ('ListSubDomain', 4, 30), + ('ListDNS', 1, 30), + ('ListDNS', 2, 30), + ('ListDNS', 3, 30), + ('ListEmail', 1, 50), + ('ListEmail', 2, 50), + ] + + +@pytest.mark.asyncio +async def test_keyless_unlimited_search_reports_repeated_page(monkeypatch) -> None: + monkeypatch.setattr(windvane.Core, 'windvane_key', lambda: None) + + async def fake_post_fetch(*_args, **_kwargs): + return json.dumps( + { + 'code': 0, + 'data': {'list': [{'domain': 'api.example.test'}], 'has_more': True}, + } + ) + + monkeypatch.setattr(windvane.AsyncFetcher, 'post_fetch', fake_post_fetch) + search = windvane.SearchWindvane('example.test', None) + + assert await search.process() == SourceExecutionReport('partial', 'repeated-page') + assert await search.get_hostnames() == {'api.example.test'} + + +@pytest.mark.asyncio +async def test_keyless_unlimited_search_reports_provider_bound(monkeypatch) -> None: + monkeypatch.setattr(windvane.Core, 'windvane_key', lambda: None) + calls = 0 + + async def fake_post_fetch(*_args, **_kwargs): + nonlocal calls + calls += 1 + if calls == 1: + return json.dumps( + { + 'code': 0, + 'data': {'list': [{'domain': 'api.example.test'}], 'has_more': True}, + } + ) + return json.dumps({'code': 1, 'message': 'Unauthenticated request limit reached'}) + + monkeypatch.setattr(windvane.AsyncFetcher, 'post_fetch', fake_post_fetch) + search = windvane.SearchWindvane('example.test', None) + + assert await search.process() == SourceExecutionReport('partial', 'provider-limit') + assert await search.get_hostnames() == {'api.example.test'} + + +@pytest.mark.asyncio +async def test_finite_limit_stops_each_windvane_endpoint(monkeypatch) -> None: + monkeypatch.setattr(windvane.Core, 'windvane_key', lambda: 'test-key') + requests: list[tuple[str, int, int]] = [] + + async def fake_post_fetch(url, headers=None, data=None, proxy=False): + endpoint = url.rsplit('/', 1)[-1] + page_request = json.loads(data)['page_request'] + requests.append((endpoint, page_request['page'], page_request['count'])) + return json.dumps({'code': 0, 'data': {'list': [{}], 'has_more': True}}) + + monkeypatch.setattr(windvane.AsyncFetcher, 'post_fetch', fake_post_fetch) + search = windvane.SearchWindvane('example.test', 1) + + assert await search.process() == SourceExecutionReport('completed', 'result-limit') + assert requests == [('ListSubDomain', 1, 1), ('ListDNS', 1, 1), ('ListEmail', 1, 1)] + + pytestmark = pytest.mark.provider_contract('windvane') diff --git a/tests/discovery/test_yahoosearch.py b/tests/discovery/test_yahoosearch.py index dcf48f0d..f62d17d7 100644 --- a/tests/discovery/test_yahoosearch.py +++ b/tests/discovery/test_yahoosearch.py @@ -1,8 +1,11 @@ +import asyncio from typing import Any import pytest from theHarvester.discovery import yahoosearch +from theHarvester.lib.core import FetcherResponse +from theHarvester.lib.source_execution import SourceExecutionReport @pytest.mark.asyncio @@ -41,26 +44,82 @@ async def test_yahoo_uses_exact_pages_and_normalizes_evidence(monkeypatch: pytes assert await search.get_hostnames() == ['blog.example.com', 'example.com'] +@pytest.mark.asyncio +async def test_yahoo_unlimited_stops_when_provider_repeats_a_page(monkeypatch: pytest.MonkeyPatch) -> None: + requests: list[dict[str, Any]] = [] + responses = iter(['one.example.com', 'two.example.com', 'two.example.com']) + + async def fake_fetch(**kwargs: Any) -> FetcherResponse: + requests.append(kwargs) + return FetcherResponse(next(responses), 200, {}) + + monkeypatch.setattr(yahoosearch.AsyncFetcher, 'fetch', fake_fetch) + search = yahoosearch.SearchYahoo('example.com', None) + + report = await search.process() + + assert report == SourceExecutionReport('partial', 'repeated-page') + assert [request['url'] for request in requests] == [ + 'https://search.yahoo.com/search?p=%40example.com&b=0&pz=10', + 'https://search.yahoo.com/search?p=%40example.com&b=10&pz=10', + 'https://search.yahoo.com/search?p=%40example.com&b=20&pz=10', + ] + assert await search.get_hostnames() == ['one.example.com', 'two.example.com'] + + @pytest.mark.parametrize( - 'response', - ['', None, 'Access denied at api.example.net'], - ids=['empty', 'malformed', 'blocked'], + ('response', 'expected_report'), + [ + ('', None), + (None, SourceExecutionReport('failed', 'transport-error')), + (FetcherResponse({}, 200, {}), SourceExecutionReport('failed', 'invalid-response')), + ('Access denied at api.example.net', SourceExecutionReport('failed', 'access-denied')), + ], + ids=['empty', 'missing', 'malformed', 'blocked'], ) @pytest.mark.asyncio async def test_yahoo_unusable_responses_return_no_evidence( monkeypatch: pytest.MonkeyPatch, - response: str | None, + response: object, + expected_report: SourceExecutionReport | None, ) -> None: - async def fake_fetch_all(urls: list[str] | set[str], **_kwargs: Any) -> list[str | None]: + async def fake_fetch_all(urls: list[str] | set[str], **_kwargs: Any) -> list[object]: return [response] * len(urls) monkeypatch.setattr(yahoosearch.AsyncFetcher, 'fetch_all', fake_fetch_all) search = yahoosearch.SearchYahoo('example.com', 20) - await search.process() + report = await search.process() + assert report == expected_report assert await search.get_emails() == [] assert await search.get_hostnames() == [] +@pytest.mark.asyncio +async def test_yahoo_later_http_failure_is_partial(monkeypatch: pytest.MonkeyPatch) -> None: + async def fake_fetch_all(*_args: Any, **_kwargs: Any) -> list[FetcherResponse]: + return [ + FetcherResponse('one.example.com', 200, {}), + FetcherResponse('unavailable', 503, {}), + ] + + monkeypatch.setattr(yahoosearch.AsyncFetcher, 'fetch_all', fake_fetch_all) + + report = await yahoosearch.SearchYahoo('example.com', 20).process() + + assert report == SourceExecutionReport('partial', 'http-503') + + +@pytest.mark.asyncio +async def test_yahoo_cancellation_propagates(monkeypatch: pytest.MonkeyPatch) -> None: + async def cancel(**_kwargs: Any) -> FetcherResponse: + raise asyncio.CancelledError('operator-stop') + + monkeypatch.setattr(yahoosearch.AsyncFetcher, 'fetch', cancel) + + with pytest.raises(asyncio.CancelledError, match='operator-stop'): + await yahoosearch.SearchYahoo('example.com', None).process() + + pytestmark = pytest.mark.provider_contract('yahoo') diff --git a/tests/discovery/test_zoomeyesearch.py b/tests/discovery/test_zoomeyesearch.py index d1b9c4dd..6fb4d273 100644 --- a/tests/discovery/test_zoomeyesearch.py +++ b/tests/discovery/test_zoomeyesearch.py @@ -157,6 +157,33 @@ async def test_empty_pages_do_not_hide_later_provider_results(monkeypatch: pytes assert report is None +@pytest.mark.asyncio +async def test_unlimited_search_uses_provider_total(monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.setattr(zoomeyesearch.Core, 'zoomeye_key', staticmethod(lambda: 'test-key')) + calls: list[dict[str, Any]] = [] + responses = [ + FetcherResponse({'code': 60000, 'total': 2, 'data': [{'hostname': 'one.example.com'}]}, 200, {}), + FetcherResponse({'code': 60000, 'total': 2, 'data': [{'hostname': 'two.example.com'}]}, 200, {}), + ] + + @contextlib.asynccontextmanager + async def fake_open_session(**_kwargs: Any) -> AsyncIterator[object]: + yield object() + + async def fake_post_fetch(*_args: Any, **kwargs: Any) -> FetcherResponse: + calls.append(kwargs['json_body']) + return responses.pop(0) + + monkeypatch.setattr(zoomeyesearch.AsyncFetcher, 'open_session', fake_open_session) + monkeypatch.setattr(zoomeyesearch.AsyncFetcher, 'post_fetch', fake_post_fetch) + monkeypatch.setattr(zoomeyesearch.SearchZoomEye, 'PAGE_SIZE', 1) + search = zoomeyesearch.SearchZoomEye('example.com', None) + + assert await search.process() is None + assert [call['page'] for call in calls] == [1, 2] + assert await search.get_hostnames() == {'one.example.com', 'two.example.com'} + + @pytest.mark.asyncio async def test_numbered_pages_keep_a_stable_size_and_slice_the_final_page(monkeypatch: pytest.MonkeyPatch) -> None: monkeypatch.setattr(zoomeyesearch.Core, 'zoomeye_key', staticmethod(lambda: 'test-key')) diff --git a/tests/lib/test_api_v1.py b/tests/lib/test_api_v1.py index 6f8ee2e4..588f0d8b 100644 --- a/tests/lib/test_api_v1.py +++ b/tests/lib/test_api_v1.py @@ -509,6 +509,16 @@ def test_dns_limits_default_to_unlimited_and_keep_explicit_values() -> None: assert (explicit.dns_recursive_query_limit, explicit.dns_recursive_runtime_seconds) == (12, 1.5) +def test_result_limit_accepts_unlimited_and_has_no_numeric_ceiling() -> None: + from theHarvester.lib.api.run_models import RunRequest + + unlimited = RunRequest(target='example.test', sources=['crtsh'], limit=0) + large = RunRequest(target='example.test', sources=['crtsh'], limit=1_000_000) + + assert unlimited.limit == 0 + assert large.limit == 1_000_000 + + def test_run_detail_exposes_one_normalized_evidence_surface(tmp_path, monkeypatch) -> None: from theHarvester.lib.api import api @@ -966,6 +976,101 @@ def test_api_jsonl_round_trip_preserves_source_attribution(tmp_path, monkeypatch assert reimported.json()['source_executions'] == [source_execution] +def test_run_detail_reports_unique_hostname_contribution_per_source(tmp_path, monkeypatch) -> None: + from theHarvester.lib.api import api + + summary = { + 'type': 'summary', + 'run_id': '31ced47a-6354-4f18-b633-2583400c4aad', + 'target': 'example.test', + 'started_at': '2026-08-23T12:00:00Z', + 'completed_at': '2026-08-23T12:01:00Z', + 'evidence_status': 'partial', + 'result_count': 2, + 'counts': {'hostname': 2}, + 'source_executions': [ + { + 'source': 'crtsh', + 'status': 'completed', + 'duration_ms': 1, + 'result_count': 1, + 'error_type': None, + 'stop_reason': None, + }, + { + 'source': 'subdomainapi', + 'status': 'partial', + 'duration_ms': 2, + 'result_count': 2, + 'error_type': None, + 'stop_reason': 'provider-limit', + }, + ], + 'action_executions': [ + { + 'action': 'dns-resolve', + 'status': 'completed', + 'duration_ms': 3, + 'result_count': 1, + 'error_type': None, + 'stop_reason': None, + } + ], + } + payload = '\n'.join( + ( + json.dumps(summary), + json.dumps( + { + 'type': 'hostname', + 'value': 'shared.example.test', + 'sources': ['crtsh', 'subdomainapi'], + } + ), + json.dumps( + { + 'type': 'hostname', + 'value': 'unique.example.test', + 'sources': ['subdomainapi'], + 'actions': ['dns-resolve'], + } + ), + '', + ) + ) + monkeypatch.setenv('THEHARVESTER_API_KEY', 'test-key') + monkeypatch.setenv('THEHARVESTER_RUN_DB', str(tmp_path / 'runs.sqlite')) + monkeypatch.setenv('THEHARVESTER_RUN_WORKER', 'disabled') + + with TestClient(api.app) as client: + response = client.post( + '/api/v1/runs/import', + params={'filename': 'source-yields.jsonl'}, + headers={'X-API-Key': 'test-key'}, + content=payload, + ) + + assert response.status_code == 201 + assert response.json()['source_yields'] == [ + { + 'source': 'crtsh', + 'observed_result_count': 1, + 'unique_result_count': 0, + 'shared_result_count': 1, + 'resolved_hostname_count': 0, + 'unique_resolved_hostname_count': 0, + }, + { + 'source': 'subdomainapi', + 'observed_result_count': 2, + 'unique_result_count': 1, + 'shared_result_count': 1, + 'resolved_hostname_count': 1, + 'unique_resolved_hostname_count': 1, + }, + ] + + def test_api_jsonl_export_uses_evidence_timestamps_not_lifecycle_timestamps(tmp_path, monkeypatch) -> None: from theHarvester.lib.api import api from theHarvester.lib.api.run_models import RunRequest diff --git a/tests/lib/test_completed_persistence.py b/tests/lib/test_completed_persistence.py index cb8bd0ec..3e86bd10 100644 --- a/tests/lib/test_completed_persistence.py +++ b/tests/lib/test_completed_persistence.py @@ -1165,6 +1165,16 @@ async def test_source_yields_distinguish_unique_and_shared_results(tmp_path) -> SourceExecution('certspotter', 'completed', 8.0, 2), SourceExecution('empty-source', 'completed', 5.0, 0), ), + active_evidence=ActiveEvidence( + ( + ActionExecution.finish( + action='dns-resolve', + status='completed', + duration_ms=4, + groups={'hostname': ['api.example.com', 'mail.example.com']}, + ), + ) + ), ) await store.save_run(result) @@ -1176,18 +1186,68 @@ async def test_source_yields_distinguish_unique_and_shared_results(tmp_path) -> 'observed_result_count': 2, 'unique_result_count': 1, 'shared_result_count': 1, + 'resolved_hostname_count': 1, + 'unique_resolved_hostname_count': 0, }, { 'source': 'crtsh', 'observed_result_count': 2, 'unique_result_count': 1, 'shared_result_count': 1, + 'resolved_hostname_count': 2, + 'unique_resolved_hostname_count': 1, }, { 'source': 'empty-source', 'observed_result_count': 0, 'unique_result_count': 0, 'shared_result_count': 0, + 'resolved_hostname_count': 0, + 'unique_resolved_hostname_count': 0, + }, + ] + + +@pytest.mark.asyncio +async def test_source_yields_can_measure_hostname_contributions_only(tmp_path) -> None: + store = ResultStore(tmp_path / 'stash.sqlite') + await store.initialize() + result = CompletedResult.finish( + run_id=UUID('86eafad7-3308-4786-8e85-1e15b24afafc'), + target='example.com', + started_at=datetime(2026, 8, 23, 12, 0, tzinfo=UTC), + completed_at=datetime(2026, 8, 23, 12, 1, tzinfo=UTC), + groups={'hostname': ['shared.example.com'], 'url': ['https://unique.example.com/']}, + observations=( + ResultObservation('first', 'hostname', 'shared.example.com'), + ResultObservation('second', 'hostname', 'shared.example.com'), + ResultObservation('first', 'url', 'https://unique.example.com/'), + ), + source_executions=( + SourceExecution('first', 'completed', 10, 2), + SourceExecution('second', 'completed', 10, 1), + ), + ) + await store.save_run(result) + + yields = await store.source_yields(result.run_id, kind='hostname') + + assert [item.to_dict() for item in yields] == [ + { + 'source': 'first', + 'observed_result_count': 1, + 'unique_result_count': 0, + 'shared_result_count': 1, + 'resolved_hostname_count': 0, + 'unique_resolved_hostname_count': 0, + }, + { + 'source': 'second', + 'observed_result_count': 1, + 'unique_result_count': 0, + 'shared_result_count': 1, + 'resolved_hostname_count': 0, + 'unique_resolved_hostname_count': 0, }, ] diff --git a/tests/lib/test_enumeration.py b/tests/lib/test_enumeration.py index cb7a69c1..c1957c96 100644 --- a/tests/lib/test_enumeration.py +++ b/tests/lib/test_enumeration.py @@ -45,6 +45,17 @@ def test_enumeration_options_preserve_explicit_transport_values() -> None: assert options.source_workers == 7 +def test_zero_result_limit_means_unlimited() -> None: + options = EnumerationOptions(domain='example.com', source='crtsh', limit=0) + + assert options.limit is None + + +def test_negative_result_limit_is_rejected() -> None: + with pytest.raises(ValueError, match='result limit cannot be negative'): + EnumerationOptions(domain='example.com', source='crtsh', limit=-1) + + def test_routeviews_is_an_explicit_passive_action_independent_of_source_limits() -> None: options = EnumerationOptions.from_namespace(Namespace(domain='example.com', source=None, limit=25, routeviews=True)) diff --git a/tests/lib/test_harvestview_ui.py b/tests/lib/test_harvestview_ui.py index 5eb7c01f..2fdb3516 100644 --- a/tests/lib/test_harvestview_ui.py +++ b/tests/lib/test_harvestview_ui.py @@ -24,6 +24,9 @@ def test_harvestview_owns_root_and_issues_an_http_only_session(tmp_path, monkeyp assert f'value="{",".join(DEFAULT_DNS_RESOLVERS)}"' in root.text assert 'Resolve with the configured resolver addresses.' in root.text assert 'id="source-workers" name="source_workers" type="number" min="1"' in root.text + assert 'id="run-limit" name="limit" type="number" min="0" value="500"' in root.text + assert '0 means unlimited; positive values apply per selected source.' in root.text + assert 'id="run-limit" name="limit" type="number" min="0" max="10000"' not in root.text assert legacy.status_code == 404 cookie = root.headers['set-cookie'] assert 'theharvester-api-key=' in cookie @@ -55,6 +58,7 @@ def test_harvestview_assets_load_outside_the_repository_directory(tmp_path, monk assert "request.deadline_seconds === null ? 'Unlimited'" in response.text assert "deadline_seconds: form.get('deadline_seconds') ? Number(form.get('deadline_seconds')) : null" in response.text assert "source_workers: Number(form.get('source_workers'))" in response.text + assert "request.limit === 0 ? 'Unlimited'" in response.text def test_harvestview_exposes_local_schedule_page_and_assets(tmp_path, monkeypatch) -> None: @@ -77,6 +81,8 @@ def test_harvestview_exposes_local_schedule_page_and_assets(tmp_path, monkeypatc assert 'href="/schedules">Schedules' in root.text assert schedules.status_code == 200 assert '

Create a schedule

' in schedules.text + assert 'id="run-limit" type="number" min="0" value="500"' in schedules.text + assert 'id="run-limit" type="number" min="0" max="10000"' not in schedules.text assert '/static/harvestview/schedules.css?v=' in schedules.text assert '/static/harvestview/schedules.js?v=' in schedules.text assert stylesheet.status_code == 200 diff --git a/tests/lib/test_run_backend.py b/tests/lib/test_run_backend.py index a08fc16b..270351dd 100644 --- a/tests/lib/test_run_backend.py +++ b/tests/lib/test_run_backend.py @@ -990,6 +990,34 @@ def test_routeviews_child_receives_explicit_action_without_source_limit_controls assert received_options[0].limit == 9_999 +def test_persisted_unlimited_result_limit_becomes_no_worker_cap(tmp_path, monkeypatch) -> None: + from theHarvester import __main__ as main_module + from theHarvester.lib.api import run_worker + from theHarvester.lib.api.run_models import RunRequest + from theHarvester.lib.api.run_store import RunStore + from theHarvester.lib.completed_result import CompletedResult + + received_options = [] + + async def fake_start(options, **_kwargs): + received_options.append(options) + now = datetime.now(UTC) + return (CompletedResult.finish(target=options.domain, started_at=now, completed_at=now, groups={}),) + + monkeypatch.setattr(main_module, 'start', fake_start) + + async def scenario() -> None: + store = RunStore(tmp_path / 'runs.sqlite') + created = await store.create(RunRequest(target='example.test', sources=['crtsh'], limit=0)) + assert created['request']['limit'] == 0 + assert await store.claim_next() is not None + await run_worker._child_execute(created['run_id'], store.database) + + asyncio.run(scenario()) + + assert received_options[0].limit is None + + @pytest.mark.parametrize( ('field', 'value', 'option'), [ diff --git a/tests/lib/test_source_runner.py b/tests/lib/test_source_runner.py index 598bbd62..c690ebd4 100644 --- a/tests/lib/test_source_runner.py +++ b/tests/lib/test_source_runner.py @@ -99,7 +99,7 @@ def test_source_factories_match_the_catalog() -> None: ('fofa', 'theHarvester.lib.source_runner.fofa.SearchFofa', ('example.test', 25), {}), ('fullhunt', 'theHarvester.lib.source_runner.fullhuntsearch.SearchFullHunt', ('example.test',), {}), ('github-code', 'theHarvester.lib.source_runner.githubcode.SearchGithubCode', ('example.test', 25), {}), - ('gitlab', 'theHarvester.lib.source_runner.gitlabsearch.SearchGitlab', ('example.test',), {}), + ('gitlab', 'theHarvester.lib.source_runner.gitlabsearch.SearchGitlab', ('example.test', 25), {}), ( 'hackertarget', 'theHarvester.lib.source_runner.hackertarget.SearchHackerTarget', @@ -121,7 +121,7 @@ def test_source_factories_match_the_catalog() -> None: ('hudsonrock', 'theHarvester.lib.source_runner.hudsonrocksearch.SearchHudsonRock', ('example.test',), {}), ('hunter', 'theHarvester.lib.source_runner.huntersearch.SearchHunter', ('example.test', 25, 5), {}), ('hunterhow', 'theHarvester.lib.source_runner.searchhunterhow.SearchHunterHow', ('example.test', 25), {}), - ('intelx', 'theHarvester.lib.source_runner.intelxsearch.SearchIntelx', ('example.test',), {}), + ('intelx', 'theHarvester.lib.source_runner.intelxsearch.SearchIntelx', ('example.test', 25), {}), ('leakix', 'theHarvester.lib.source_runner.leakix.SearchLeakix', ('example.test',), {}), ('leaklookup', 'theHarvester.lib.source_runner.leaklookup.SearchLeakLookup', ('example.test',), {}), ('mojeek', 'theHarvester.lib.source_runner.mojeek.SearchMojeek', ('example.test', 25), {}), @@ -182,7 +182,7 @@ def test_source_factories_match_the_catalog() -> None: ('example.test',), {}, ), - ('thc', 'theHarvester.lib.source_runner.thc.SearchThc', ('example.test',), {}), + ('thc', 'theHarvester.lib.source_runner.thc.SearchThc', ('example.test', 25), {}), ('tomba', 'theHarvester.lib.source_runner.tombasearch.SearchTomba', ('example.test', 25, 5), {}), ('urlscan', 'theHarvester.lib.source_runner.urlscan.SearchUrlscan', ('example.test', 25), {}), ('virustotal', 'theHarvester.lib.source_runner.virustotal.SearchVirustotal', ('example.test', 25), {}), @@ -193,7 +193,7 @@ def test_source_factories_match_the_catalog() -> None: {}, ), ('whoisxml', 'theHarvester.lib.source_runner.whoisxml.SearchWhoisXML', ('example.test', 25), {}), - ('windvane', 'theHarvester.lib.source_runner.windvane.SearchWindvane', ('example.test',), {}), + ('windvane', 'theHarvester.lib.source_runner.windvane.SearchWindvane', ('example.test', 25), {}), ('yahoo', 'theHarvester.lib.source_runner.yahoosearch.SearchYahoo', ('example.test', 25), {}), ('zoomeye', 'theHarvester.lib.source_runner.zoomeyesearch.SearchZoomEye', ('example.test', 25), {}), ], @@ -218,6 +218,31 @@ def test_factory_constructor_shapes( assert calls == [(expected_args, expected_kwargs)] +@pytest.mark.parametrize( + ('source', 'patch_target'), + [ + ('gitlab', 'theHarvester.lib.source_runner.gitlabsearch.SearchGitlab'), + ('windvane', 'theHarvester.lib.source_runner.windvane.SearchWindvane'), + ], +) +def test_unlimited_limit_reaches_gitlab_and_windvane_factories( + monkeypatch: pytest.MonkeyPatch, + source: str, + patch_target: str, +) -> None: + calls: list[tuple[object, ...]] = [] + + def constructor(*args: object) -> object: + calls.append(args) + return object() + + monkeypatch.setattr(patch_target, constructor) + + create_source(SourceRequest(source, 'example.test', None, 0, False, True)) + + assert calls == [('example.test', None)] + + @pytest.mark.asyncio async def test_runner_normalizes_only_declared_apis_guru_routes(monkeypatch: pytest.MonkeyPatch) -> None: class FakeApisGuru: diff --git a/tests/test_harvestview_launcher.py b/tests/test_harvestview_launcher.py index 95fb02ca..81af5b04 100644 --- a/tests/test_harvestview_launcher.py +++ b/tests/test_harvestview_launcher.py @@ -7,12 +7,13 @@ import pytest from theHarvester import harvestview -def test_project_scripts_expose_harvestview() -> None: +def test_project_scripts_expose_commands() -> None: scripts = tomllib.loads(Path('pyproject.toml').read_text(encoding='utf-8'))['project']['scripts'] assert scripts == { 'theHarvester': 'theHarvester.theHarvester:main', 'harvestview': 'theHarvester.harvestview:main', + 'harvest-yields': 'theHarvester.source_yields:main', } diff --git a/tests/test_main.py b/tests/test_main.py index b07e778c..3e174447 100644 --- a/tests/test_main.py +++ b/tests/test_main.py @@ -23,6 +23,7 @@ from theHarvester.lib.hostchecker import HostDnsRecords from theHarvester.lib.network_evidence import PrefixOriginObservation, RpkiValidationObservation from theHarvester.lib.recursive_dns import RecursiveDNSClassification, RecursiveDNSFinding, RecursiveDNSResult from theHarvester.lib.routeviews import RouteViewsCancelled, RouteViewsResult +from theHarvester.lib.source_catalog import resolve_sources from theHarvester.lib.source_execution import SourceExecutionReport from theHarvester.lib.takeover_evidence import TakeoverCandidateOutcome from theHarvester.lib.virtual_host import ( @@ -67,6 +68,7 @@ async def test_cli_help_explains_proxy_and_direct_action_scope( assert 'Candidate names are never resolved through DNS.' in help_text assert '-j SOURCE_WORKERS' in help_text assert '--source-workers SOURCE_WORKERS' in help_text + assert '0 continues to provider exhaustion with no local result or page-count cap' in help_text assert 'Indicators are not confirmed takeovers.' in help_text @@ -676,6 +678,9 @@ async def test_rapiddns_hostnames_honor_explicit_dns_resolution(monkeypatch: pyt assert dns_execution.error_type == 'TimeoutError' assert dns_execution.stop_reason == 'query-errors' assert {(observation.kind, observation.value) for observation in dns_execution.observations} == { + ('hostname', 'api.example.com'), + ('hostname', 'crt.example.com'), + ('hostname', 'reported.example.com'), ('ip', '192.0.2.10'), ('ip', '192.0.2.21'), ('ip', '192.0.2.30'), @@ -1492,6 +1497,32 @@ async def test_source_progress_waits_for_runner_admission( assert '[*] Searching Dymo.' not in interim_output +@pytest.mark.asyncio +async def test_unlimited_subdomain_selection_passes_no_result_cap_to_every_source( + monkeypatch: pytest.MonkeyPatch, +) -> None: + captured: tuple[source_runner.SourceJob, ...] = () + + async def capture_jobs( + jobs: tuple[source_runner.SourceJob, ...], + **_kwargs: object, + ) -> tuple[source_runner.SourceOutcome, ...]: + nonlocal captured + captured = jobs + return () + + monkeypatch.setattr(theharvester_main, 'run_source_jobs', capture_jobs) + monkeypatch.setattr(theharvester_main, 'ResultStore', _NoopResultStore) + + await theharvester_main.start( + EnumerationOptions(domain='example.test', source='subdomains', limit=0, quiet=True), + return_completed_result=True, + ) + + assert [job.request.source for job in captured] == resolve_sources('subdomains') + assert all(job.request.limit is None for job in captured) + + @pytest.mark.asyncio async def test_source_completion_reports_verbose_terminal_summary( monkeypatch: pytest.MonkeyPatch, diff --git a/tests/test_mojeek.py b/tests/test_mojeek.py index ce7d90bd..22841a60 100644 --- a/tests/test_mojeek.py +++ b/tests/test_mojeek.py @@ -1,3 +1,4 @@ +import asyncio from typing import Any import pytest @@ -7,6 +8,120 @@ from theHarvester.lib.core import FetcherResponse class TestMojeekSearch: + @pytest.mark.asyncio + async def test_unlimited_keyless_stops_when_provider_repeats_a_page(self, monkeypatch: pytest.MonkeyPatch) -> None: + calls: list[dict[str, Any]] = [] + responses = iter( + [ + FetcherResponse(body='
    one.example.com
', status=200, headers={}), + FetcherResponse(body='
    two.example.com
', status=200, headers={}), + FetcherResponse(body='
    two.example.com
', status=200, headers={}), + ] + ) + + async def fake_fetch(**kwargs: Any) -> FetcherResponse: + calls.append(kwargs) + return next(responses) + + async def fake_sleep(_delay: float) -> None: + return None + + monkeypatch.setattr(mojeek.Core, 'mojeek_key', staticmethod(lambda: '')) + monkeypatch.setattr(mojeek.AsyncFetcher, 'fetch', fake_fetch) + monkeypatch.setattr(mojeek.asyncio, 'sleep', fake_sleep) + search = mojeek.SearchMojeek(word='example.com', limit=None) + + report = await search.process() + + assert report == mojeek.SourceExecutionReport('partial', 'repeated-page') + assert [call['url'] for call in calls] == [ + 'https://www.mojeek.com/search?q=example.com&s=0', + 'https://www.mojeek.com/search?q=example.com&s=10', + 'https://www.mojeek.com/search?q=example.com&s=20', + ] + assert await search.get_hostnames() == ['one.example.com', 'two.example.com'] + + @pytest.mark.asyncio + async def test_unlimited_keyed_api_stops_when_provider_exhausts_results(self, monkeypatch: pytest.MonkeyPatch) -> None: + requests: list[list[str]] = [] + responses = iter( + [ + FetcherResponse(body={'response': {'results': [{'url': 'https://one.example.com'}]}}, status=200, headers={}), + FetcherResponse(body={'response': {'results': [{'url': 'https://two.example.com'}]}}, status=200, headers={}), + FetcherResponse(body={'response': {'results': []}}, status=200, headers={}), + ] + ) + + async def fake_fetch_all(urls: list[str], **_kwargs: Any) -> list[FetcherResponse]: + requests.append(urls) + return [next(responses)] + + monkeypatch.setattr(mojeek.Core, 'mojeek_key', staticmethod(lambda: 'test-key')) + monkeypatch.setattr(mojeek.AsyncFetcher, 'fetch_all', fake_fetch_all) + search = mojeek.SearchMojeek(word='example.com', limit=None) + + report = await search.process() + + assert report is None + assert requests == [ + ['https://api.mojeek.com/search?api_key=test-key&q=example.com&fmt=json&s=1'], + ['https://api.mojeek.com/search?api_key=test-key&q=example.com&fmt=json&s=11'], + ['https://api.mojeek.com/search?api_key=test-key&q=example.com&fmt=json&s=21'], + ] + assert await search.get_hostnames() == ['one.example.com', 'two.example.com'] + + @pytest.mark.asyncio + async def test_unlimited_keyed_api_reports_repeated_page(self, monkeypatch: pytest.MonkeyPatch) -> None: + page = FetcherResponse( + body={'response': {'results': [{'url': 'https://one.example.com'}]}}, + status=200, + headers={}, + ) + + async def fake_fetch_all(_urls: list[str], **_kwargs: Any) -> list[FetcherResponse]: + return [page] + + monkeypatch.setattr(mojeek.Core, 'mojeek_key', staticmethod(lambda: 'test-key')) + monkeypatch.setattr(mojeek.AsyncFetcher, 'fetch_all', fake_fetch_all) + + report = await mojeek.SearchMojeek(word='example.com', limit=None).process() + + assert report == mojeek.SourceExecutionReport('partial', 'repeated-page') + + @pytest.mark.asyncio + async def test_keyless_later_http_failure_is_partial(self, monkeypatch: pytest.MonkeyPatch) -> None: + responses = iter( + [ + FetcherResponse(body='
    one.example.com
', status=200, headers={}), + FetcherResponse(body='unavailable', status=503, headers={}), + ] + ) + + async def fake_fetch(**_kwargs: Any) -> FetcherResponse: + return next(responses) + + async def fake_sleep(_delay: float) -> None: + return None + + monkeypatch.setattr(mojeek.Core, 'mojeek_key', staticmethod(lambda: '')) + monkeypatch.setattr(mojeek.AsyncFetcher, 'fetch', fake_fetch) + monkeypatch.setattr(mojeek.asyncio, 'sleep', fake_sleep) + + report = await mojeek.SearchMojeek(word='example.com', limit=None).process() + + assert report == mojeek.SourceExecutionReport('partial', 'http-503') + + @pytest.mark.asyncio + async def test_mojeek_cancellation_propagates(self, monkeypatch: pytest.MonkeyPatch) -> None: + async def cancel(**_kwargs: Any) -> FetcherResponse: + raise asyncio.CancelledError('operator-stop') + + monkeypatch.setattr(mojeek.Core, 'mojeek_key', staticmethod(lambda: '')) + monkeypatch.setattr(mojeek.AsyncFetcher, 'fetch', cancel) + + with pytest.raises(asyncio.CancelledError, match='operator-stop'): + await mojeek.SearchMojeek(word='example.com', limit=None).process() + @pytest.mark.asyncio async def test_keyless_pages_are_sequential_and_stop_after_first_empty_page( self, diff --git a/tests/test_readme.py b/tests/test_readme.py index 1faf7565..4d2b59a5 100644 --- a/tests/test_readme.py +++ b/tests/test_readme.py @@ -66,6 +66,7 @@ SOURCE_PROVIDER_LINKS = { 'shodanInternetDB': 'https://internetdb.shodan.io/', 'shodanct': 'https://ctl.shodan.io/', 'sourcegraph': 'https://sourcegraph.com/search', + 'subdomainapi': 'https://api.subdomain.app/', 'subdomaincenter': 'https://www.subdomain.center/', 'subdomainfinderc99': 'https://subdomainfinder.c99.nl/', 'thc': 'https://ip.thc.org/', @@ -102,7 +103,7 @@ def _declared_source_contracts() -> dict[str, set[str]]: def _source_matrix(readme: str) -> str: - return readme.split('View all 58 discovery sources', 1)[1].split('
', 1)[0] + return readme.split('View all 59 discovery sources', 1)[1].split('', 1)[0] def _documented_source_rows(readme: str) -> dict[str, list[str]]: @@ -151,8 +152,8 @@ def test_readme_matches_declared_source_contracts() -> None: assert _source_matrix(readme).count('| Source | Returns | Activity | API key |') == 1 assert 'Credentials |' not in _source_matrix(readme) - assert len(declared) == 58 - assert len(documented) == 58 + assert len(declared) == 59 + assert len(documented) == 59 assert documented == declared source_links = _documented_source_links(readme) assert len(source_links) == len(declared) @@ -224,7 +225,7 @@ def test_readme_architecture_diagrams_are_local_and_accessible() -> None: 'theHarvester discovery routes and enrichment', Path('docs/images/run-evidence-architecture.svg'), 'run-evidence-architecture', - ('58 discovery adapters', 'CompletedResult evidence contract', 'Terminal · JSONL · SQLite · REST'), + ('59 discovery adapters', 'CompletedResult evidence contract', 'Terminal · JSONL · SQLite · REST'), ), ( 'HarvestView run desk architecture', diff --git a/tests/test_source_yields_cli.py b/tests/test_source_yields_cli.py new file mode 100644 index 00000000..4ef67d54 --- /dev/null +++ b/tests/test_source_yields_cli.py @@ -0,0 +1,213 @@ +import asyncio +import json +import tomllib +from datetime import UTC, datetime, timedelta +from pathlib import Path +from uuid import UUID + +import pytest + +from theHarvester import source_yields +from theHarvester.lib.active_evidence import ActionExecution, ActiveEvidence +from theHarvester.lib.completed_result import CompletedResult, ResultObservation, SourceExecution +from theHarvester.lib.database import ResultStore +from theHarvester.lib.evidence_types import RESULT_KINDS + +RUN_ONE = UUID('11111111-1111-4111-8111-111111111111') +RUN_TWO = UUID('22222222-2222-4222-8222-222222222222') + + +def test_project_installs_harvest_yields_command() -> None: + project = tomllib.loads(Path('pyproject.toml').read_text(encoding='utf-8')) + + assert project['project']['scripts']['harvest-yields'] == 'theHarvester.source_yields:main' + + +def _completed_run( + run_id: UUID, + *, + observations: tuple[ResultObservation, ...], + resolved_hostnames: tuple[str, ...] = (), +) -> CompletedResult: + started_at = datetime(2026, 8, 23, 12, tzinfo=UTC) + timedelta(minutes=int(str(run_id)[0])) + sources = sorted({observation.source for observation in observations}) + active_evidence = ( + ActiveEvidence( + executions=( + ActionExecution.finish( + action='dns-resolve', + status='completed', + duration_ms=1, + groups={'hostname': resolved_hostnames}, + ), + ) + ) + if resolved_hostnames + else ActiveEvidence() + ) + return CompletedResult.finish( + run_id=run_id, + target='example.test', + started_at=started_at, + completed_at=started_at + timedelta(seconds=1), + groups={ + observation.kind: [item.value for item in observations if item.kind == observation.kind] + for observation in observations + }, + source_executions=tuple( + SourceExecution( + source=source, + status='completed', + duration_ms=1, + result_count=sum(observation.source == source for observation in observations), + ) + for source in sources + ), + observations=observations, + active_evidence=active_evidence, + ) + + +async def _create_database(database: Path) -> None: + store = ResultStore(database) + await store.initialize() + await store.save_run( + _completed_run( + RUN_ONE, + observations=( + ResultObservation('alpha', 'hostname', 'shared.example.test'), + ResultObservation('alpha', 'hostname', 'unique-alpha.example.test'), + ResultObservation('beta', 'hostname', 'shared.example.test'), + ResultObservation('beta', 'hostname', 'unique-beta.example.test'), + ), + resolved_hostnames=('shared.example.test', 'unique-alpha.example.test'), + ) + ) + await store.save_run( + _completed_run( + RUN_TWO, + observations=( + ResultObservation('alpha', 'hostname', 'second-alpha.example.test'), + ResultObservation('beta', 'hostname', 'second-beta.example.test'), + ResultObservation('gamma', 'hostname', 'second-gamma.example.test'), + ResultObservation('alpha', 'ip', '192.0.2.1'), + ResultObservation('beta', 'ip', '192.0.2.1'), + ResultObservation('gamma', 'ip', '198.51.100.2'), + ResultObservation('alpha', 'asn', 'AS64496'), + ), + resolved_hostnames=('second-beta.example.test',), + ) + ) + await store.dispose() + + +def test_missing_database_fails_without_creating_file( + tmp_path: Path, + capsys: pytest.CaptureFixture[str], +) -> None: + database = tmp_path / 'missing.sqlite' + + with pytest.raises(SystemExit) as error: + source_yields.main(['--database', str(database)]) + + assert error.value.code == 2 + assert 'database does not exist' in capsys.readouterr().err + assert not database.exists() + + +def test_default_table_sums_per_run_hostname_yields_and_sorts_sources( + tmp_path: Path, + capsys: pytest.CaptureFixture[str], +) -> None: + database = tmp_path / 'runs.sqlite' + asyncio.run(_create_database(database)) + + assert source_yields.main(['--database', str(database)]) == 0 + + lines = capsys.readouterr().out.splitlines() + assert lines[:3] == [ + 'Kind: hostname', + 'Run count: 2', + 'SOURCE RUNS OBSERVED UNIQUE SHARED RESOLVED UNIQUE-RESOLVED', + ] + assert [line.split() for line in lines[3:]] == [ + ['alpha', '2', '3', '2', '1', '2', '1'], + ['beta', '2', '3', '2', '1', '2', '1'], + ['gamma', '1', '1', '1', '0', '0', '0'], + ] + + +@pytest.mark.parametrize('kind', sorted(RESULT_KINDS)) +def test_kind_accepts_every_result_kind_and_only_hostname_shows_resolution_columns( + tmp_path: Path, + capsys: pytest.CaptureFixture[str], + kind: str, +) -> None: + database = tmp_path / 'runs.sqlite' + asyncio.run(_create_database(database)) + + assert source_yields.main(['--database', str(database), '--kind', kind]) == 0 + + lines = capsys.readouterr().out.splitlines() + assert lines[0] == f'Kind: {kind}' + assert ('RESOLVED' in lines[2]) is (kind == 'hostname') + if kind == 'ip': + assert [line.split() for line in lines[3:]] == [ + ['gamma', '1', '1', '1', '0'], + ['alpha', '2', '1', '0', '1'], + ['beta', '2', '1', '0', '1'], + ] + + +def test_run_id_selects_one_run( + tmp_path: Path, + capsys: pytest.CaptureFixture[str], +) -> None: + database = tmp_path / 'runs.sqlite' + asyncio.run(_create_database(database)) + + assert source_yields.main(['--database', str(database), '--run-id', str(RUN_ONE)]) == 0 + + lines = capsys.readouterr().out.splitlines() + assert lines[1] == 'Run count: 1' + assert [line.split() for line in lines[3:]] == [ + ['alpha', '1', '2', '1', '1', '2', '1'], + ['beta', '1', '2', '1', '1', '1', '0'], + ] + + +def test_unknown_run_id_fails_instead_of_reporting_an_empty_run( + tmp_path: Path, + capsys: pytest.CaptureFixture[str], +) -> None: + database = tmp_path / 'runs.sqlite' + asyncio.run(_create_database(database)) + missing_run = UUID('33333333-3333-4333-8333-333333333333') + + with pytest.raises(SystemExit) as error: + source_yields.main(['--database', str(database), '--run-id', str(missing_run)]) + + assert error.value.code == 2 + assert 'completed result not found' in capsys.readouterr().err + + +@pytest.mark.parametrize('kind', ['hostname', 'ip']) +def test_json_format_is_machine_readable_and_uses_kind_specific_fields( + tmp_path: Path, + capsys: pytest.CaptureFixture[str], + kind: str, +) -> None: + database = tmp_path / 'runs.sqlite' + asyncio.run(_create_database(database)) + + assert source_yields.main(['--database', str(database), '--kind', kind, '--format', 'json']) == 0 + + payload = json.loads(capsys.readouterr().out) + assert payload['kind'] == kind + assert payload['run_count'] == 2 + assert [row['source'] for row in payload['source_yields']] == ( + ['alpha', 'beta', 'gamma'] if kind == 'hostname' else ['gamma', 'alpha', 'beta'] + ) + resolution_fields = {'resolved_hostname_count', 'unique_resolved_hostname_count'} + assert all(resolution_fields <= row.keys() for row in payload['source_yields']) is (kind == 'hostname') + assert {row['source']: row['run_count'] for row in payload['source_yields']} == {'alpha': 2, 'beta': 2, 'gamma': 1} diff --git a/theHarvester/__main__.py b/theHarvester/__main__.py index 48dc99b6..7744b243 100644 --- a/theHarvester/__main__.py +++ b/theHarvester/__main__.py @@ -159,7 +159,10 @@ async def start( parser.add_argument( '-l', '--limit', - help='Maximum results requested from each source that supports result limits (default: 500).', + help=( + 'Maximum results requested from each source that supports result limits; 0 continues to provider ' + 'exhaustion with no local result or page-count cap (default: 500).' + ), default=DEFAULT_RESULT_LIMIT, type=int, ) @@ -504,11 +507,11 @@ async def start( engines: list = [] # If the user specifies full: list = [] - resolved_screenshot_hosts: set[str] = set() + resolved_hostnames: set[str] = set() reported_host_ip_pairs: set[tuple[str, str]] = set() ips: list = [] host_ip: list = [] - limit: int = args.limit + limit: int | None = args.limit routeviews_enabled = args.routeviews shodan = args.shodan start: int = args.start @@ -747,7 +750,7 @@ async def start( action='dns-resolve', status=status, duration_ms=dns_resolution_duration_ms, - groups={'ip': dns_resolution_ips}, + groups={'hostname': resolved_hostnames, 'ip': dns_resolution_ips}, error_type=error_type, stop_reason=stop_reason, ) @@ -837,7 +840,7 @@ async def start( dns_resolution_ips.update(_normalize_ip_addresses(temp_ips)) all_ip.extend(temp_ips) full.extend(resolved_pair) - resolved_screenshot_hosts.update(resolved_hosts) + resolved_hostnames.update(resolved_hosts) try: retain_results(await full_hosts_checker.check()) @@ -1026,7 +1029,7 @@ async def start( recursive_ips = [address for finding in recursive_result.findings for address in finding.records.addresses] all_hosts.extend(recursive_hosts) all_ip.extend(recursive_ips) - resolved_screenshot_hosts.update(recursive_hosts) + resolved_hostnames.update(recursive_hosts) for finding in recursive_result.findings: if finding.records.addresses: full.extend(f'{finding.hostname}:{address}' for address in finding.records.addresses) @@ -1253,7 +1256,7 @@ async def start( await checkpoint_completed_result() await persist_result(finish_completed_result()) raise - resolved_screenshot_hosts.update(hosts) + resolved_hostnames.update(hosts) normalized_brute_hosts = _normalize_hosts_for_storage(hosts, word) normalized_brute_ips = _normalize_ip_addresses(ips) temp = set() @@ -1676,9 +1679,9 @@ async def start( output_logger.info(f'\nScreenshots can be found in: {screen_shotter.output}{screen_shotter.slash}') output_logger.info('Filtering domains for ones we can reach') if not engines: - unique_resolved_domains = resolved_screenshot_hosts | {word} + unique_resolved_domains = resolved_hostnames | {word} elif dnsresolve != '': - unique_resolved_domains = resolved_screenshot_hosts + unique_resolved_domains = resolved_hostnames else: # Technically not resolved in this case, which is not ideal # You should always use dns resolve when doing screenshotting diff --git a/theHarvester/discovery/apisguru.py b/theHarvester/discovery/apisguru.py index 9b0ddf9f..1f26434c 100644 --- a/theHarvester/discovery/apisguru.py +++ b/theHarvester/discovery/apisguru.py @@ -3,7 +3,6 @@ import re from email.errors import HeaderParseError from email.headerregistry import Address from ipaddress import ip_address -from itertools import islice from urllib.parse import unquote, urlsplit, urlunsplit from theHarvester.lib.core import AsyncFetcher, FetcherResponse, ResponseStreamError @@ -16,32 +15,26 @@ class SearchApisGuru: Results include descendant hostnames, contact emails, concrete API base URLs, and related in-scope URLs. The adapter does not resolve IP addresses or expand - OpenAPI paths into operation endpoints. It checks up to 1,000 matching specs - for up to ten minutes; ``--limit`` caps stored results, not spec traversal. + OpenAPI paths into operation endpoints. It checks matching specs for up to + ten minutes; ``--limit`` caps stored results, not spec traversal. The shared fetcher caps each JSON response at 16 MiB. Oversized specs are skipped and leave the source marked partial. """ DIRECTORY_ROOT = 'https://api.apis.guru/v2' - MAX_DIRECTORY_ENTRIES = 1000 - MAX_SPEC_ITEMS = 1000 - MAX_RESULTS_PER_ROUTE = 1000 MAX_URL_LENGTH = 4096 REQUEST_TIMEOUT = 60 MAX_RUNTIME_SECONDS = 600 - def __init__(self, word: str, limit: int) -> None: + def __init__(self, word: str, limit: int | None) -> None: self.word = self._domain(word) - requested_limit = max(0, limit) - self.result_limit = min(requested_limit, self.MAX_RESULTS_PER_ROUTE) - self.result_limit_is_protective = requested_limit > self.MAX_RESULTS_PER_ROUTE + self.result_limit = max(0, limit) if limit is not None else None self.totalhosts: set[str] = set() self.totalemails: set[str] = set() self.urls: set[str] = set() self.proxy = False self._report: SourceExecutionReport | None = None self.result_limit_reached = False - self.protective_limit_reached = False @staticmethod def _domain(value: str) -> str: @@ -95,11 +88,8 @@ class SearchApisGuru: return bool(domain and normalize_scoped_hostname(domain, self.word)) def _retain(self, values: set[str], value: str) -> None: - if value not in values and len(values) >= self.result_limit: - if self.result_limit_is_protective: - self.protective_limit_reached = True - else: - self.result_limit_reached = True + if self.result_limit is not None and value not in values and len(values) >= self.result_limit: + self.result_limit_reached = True else: values.add(value) @@ -175,9 +165,7 @@ class SearchApisGuru: self._add_host(host) if isinstance(host, str) and isinstance(schemes, list): path = base_path if isinstance(base_path, str) and base_path.startswith('/') else '' - if len(schemes) > self.MAX_SPEC_ITEMS: - self.protective_limit_reached = True - for scheme in islice(schemes, self.MAX_SPEC_ITEMS): + for scheme in schemes: if isinstance(scheme, str) and scheme.lower() in {'http', 'https'}: self._add_url(f'{scheme.lower()}://{host}{path}') elif not isinstance(scheme, str): @@ -185,9 +173,7 @@ class SearchApisGuru: servers = spec.get('servers') if isinstance(servers, list): - if len(servers) > self.MAX_SPEC_ITEMS: - self.protective_limit_reached = True - for server in islice(servers, self.MAX_SPEC_ITEMS): + for server in servers: if not isinstance(server, dict): malformed = True elif not isinstance(server.get('url'), str): @@ -283,11 +269,10 @@ class SearchApisGuru: self._stop('failed', 'invalid-response') return - directory_limit_reached = len(directory) > self.MAX_DIRECTORY_ENTRIES spec_urls: list[str] = [] malformed = False seen_spec_urls: set[str] = set() - for api_id, api in islice(directory.items(), self.MAX_DIRECTORY_ENTRIES): + for api_id, api in directory.items(): if not isinstance(api_id, str): continue provider_domain = api_id.partition(':')[0] @@ -354,12 +339,8 @@ class SearchApisGuru: self._stop('failed', 'invalid-response') elif spec_failure is not None: self._stop('failed', spec_failure) - elif self.protective_limit_reached: - self._stop('failed', 'result-cap') elif self.result_limit_reached: self._stop('completed', 'result-limit') - elif directory_limit_reached: - self._stop('failed', 'directory-entry-limit') async def get_hostnames(self) -> set[str]: return self.totalhosts @@ -374,7 +355,6 @@ class SearchApisGuru: self.proxy = proxy self._report = None self.result_limit_reached = False - self.protective_limit_reached = False try: async with asyncio.timeout(self.MAX_RUNTIME_SECONDS): await self.do_search() diff --git a/theHarvester/discovery/arquivo.py b/theHarvester/discovery/arquivo.py index c1647aea..d2e34099 100644 --- a/theHarvester/discovery/arquivo.py +++ b/theHarvester/discovery/arquivo.py @@ -1,68 +1,94 @@ +import asyncio import json import logging from urllib.parse import urlencode, urlsplit from theHarvester.lib.core import AsyncFetcher, Core, FetcherResponse from theHarvester.lib.hostnames import normalize_scoped_hostname +from theHarvester.lib.source_execution import SourceExecutionReport logger = logging.getLogger(__name__) class SearchArquivo: - MAX_RESULTS = 10_000 + PAGE_SIZE = 10_000 - def __init__(self, word: str, limit: int) -> None: + def __init__(self, word: str, limit: int | None) -> None: self.word = word.strip().lower().rstrip('.') - self.limit = min(max(limit, 1), self.MAX_RESULTS) + self.limit = limit self.totalhosts: set[str] = set() self.proxy = False - async def process(self, proxy: bool = False) -> None: + async def process(self, proxy: bool = False) -> SourceExecutionReport | None: self.proxy = proxy - query = urlencode( - { + offset = 0 + previous_page = None + report = None + while self.limit is None or offset < self.limit: + page_size = self.PAGE_SIZE if self.limit is None else min(self.PAGE_SIZE, self.limit - offset) + parameters = { 'url': self.word, 'matchType': 'domain', 'output': 'json', 'fields': 'url', - 'limit': self.limit, + 'limit': page_size, } - ) - try: - responses: list[FetcherResponse | None] = await AsyncFetcher.fetch_all( - [f'https://arquivo.pt/wayback/cdx?{query}'], - headers={'User-agent': Core.get_user_agent()}, - proxy=self.proxy, - include_metadata=True, - ) - except Exception as error: - logger.info(f'Arquivo.pt request failed: {error}') - return - - response = responses[0] if responses else None - if response is None: - logger.info('Arquivo.pt request failed') - return - if not 200 <= response.status < 300: - logger.info(f'Arquivo.pt request failed with HTTP {response.status}') - return - if not isinstance(response.body, str): - logger.info('Arquivo.pt returned malformed CDX data') - return - - for line in response.body.splitlines(): + if offset: + parameters['offset'] = offset + query = urlencode(parameters) try: - item = json.loads(line) - except json.JSONDecodeError, TypeError: - continue - if not isinstance(item, dict) or not isinstance(url := item.get('url'), str): - continue - try: - hostname = urlsplit(url).hostname - except ValueError: - continue - if (normalized := normalize_scoped_hostname(hostname, self.word)) and normalized != self.word: - self.totalhosts.add(normalized) + responses: list[FetcherResponse | None] = await AsyncFetcher.fetch_all( + [f'https://arquivo.pt/wayback/cdx?{query}'], + headers={'User-agent': Core.get_user_agent()}, + proxy=self.proxy, + include_metadata=True, + ) + except asyncio.CancelledError: + raise + except Exception as error: + logger.info(f'Arquivo.pt request failed: {error}') + return SourceExecutionReport('partial' if self.totalhosts else 'failed', 'transport-error') + + response = responses[0] if responses else None + if response is None: + logger.info('Arquivo.pt request failed') + return SourceExecutionReport('partial' if self.totalhosts else 'failed', 'transport-error') + if not 200 <= response.status < 300: + logger.info(f'Arquivo.pt request failed with HTTP {response.status}') + return SourceExecutionReport('partial' if self.totalhosts else 'failed', f'http-{response.status}') + if not isinstance(response.body, str): + logger.info('Arquivo.pt returned malformed CDX data') + return SourceExecutionReport('partial' if self.totalhosts else 'failed', 'invalid-response') + if response.body == previous_page: + return SourceExecutionReport('partial', 'repeated-page') + previous_page = response.body + + lines = response.body.splitlines() + malformed = False + for line in lines: + try: + item = json.loads(line) + except json.JSONDecodeError, TypeError: + malformed = True + continue + if not isinstance(item, dict) or not isinstance(url := item.get('url'), str): + malformed = True + continue + try: + hostname = urlsplit(url).hostname + except ValueError: + malformed = True + continue + if (normalized := normalize_scoped_hostname(hostname, self.word)) and normalized != self.word: + self.totalhosts.add(normalized) + if malformed: + report = SourceExecutionReport('partial' if self.totalhosts else 'failed', 'invalid-response') + offset += len(lines) + if len(lines) < page_size: + break + if report is not None and self.totalhosts and report.status == 'failed': + return SourceExecutionReport('partial', report.stop_reason) + return report async def get_hostnames(self) -> set[str]: return self.totalhosts diff --git a/theHarvester/discovery/baidusearch.py b/theHarvester/discovery/baidusearch.py index 515a1c8d..1e1d252d 100644 --- a/theHarvester/discovery/baidusearch.py +++ b/theHarvester/discovery/baidusearch.py @@ -50,8 +50,9 @@ class SearchBaidu: return SourceExecutionReport('failed', 'no-response') return None - async def _http_search(self, urls: list[str], proxy: str | bool) -> SourceExecutionReport | None: + async def _http_search(self, urls, proxy: str | bool) -> SourceExecutionReport | None: headers = {'Host': self.server, 'User-Agent': Core.get_browser_user_agent()} + seen_bodies: set[str] = set() try: async with AsyncFetcher.open_session(headers=headers, proxy=proxy, request_timeout=60) as session: for page_number, url in enumerate(urls): @@ -64,16 +65,23 @@ class SearchBaidu: include_metadata=True, ) if not isinstance(response, FetcherResponse): - return SourceExecutionReport('failed', 'transport-error') - body = response.body if isinstance(response.body, str) else '' + return SourceExecutionReport('partial' if self.total_results else 'failed', 'transport-error') + if not isinstance(response.body, str): + return SourceExecutionReport('partial' if self.total_results else 'failed', 'invalid-response') + body = response.body if report := self._response_report(response.status, body, response.headers.get('location', '')): - return report + return SourceExecutionReport('partial', report.stop_reason) if self.total_results else report + if body in seen_bodies: + return SourceExecutionReport('partial', 'repeated-page') + seen_bodies.add(body) self.total_results += f' {body}' + except asyncio.CancelledError: + raise except Exception: - return SourceExecutionReport('failed', 'transport-error') + return SourceExecutionReport('partial' if self.total_results else 'failed', 'transport-error') return None - def __init__(self, word, limit) -> None: + def __init__(self, word, limit: int | None) -> None: self.word = word self.total_results = '' self.server = 'www.baidu.com' @@ -82,23 +90,27 @@ class SearchBaidu: async def do_search(self) -> SourceExecutionReport | None: base_url = f'https://{self.server}/s' - urls = [] - for offset in range(0, self.limit, 10): - params: dict[str, str | int] = { - 'ie': 'utf-8', - 'f': 8, - 'tn': 'baidu', - 'wd': f'site:{self.word}', - 'rqlang': 'en', - 'rsv_enter': 1, - 'rsv_dl': 'tb_enter', - } - if offset: - params['pn'] = offset - urls.append(f'{base_url}?{urlencode(params)}') + def urls(): + offset = 0 + while self.limit is None or offset < self.limit: + params: dict[str, str | int] = { + 'ie': 'utf-8', + 'f': 8, + 'tn': 'baidu', + 'wd': f'site:{self.word}', + 'rqlang': 'en', + 'rsv_enter': 1, + 'rsv_dl': 'tb_enter', + } + if offset: + params['pn'] = offset + yield f'{base_url}?{urlencode(params)}' + offset += 10 + + page_urls = urls() if playwright_api is None: - return await self._http_search(urls, self.proxy) + return await self._http_search(page_urls, self.proxy) proxy_url, _proxy_type = AsyncFetcher._resolve_proxy(self.proxy) manager = playwright_api.async_playwright() @@ -107,6 +119,7 @@ class SearchBaidu: report = None primary_error: BaseException | None = None cleanup_errors: list[BaseException] = [] + seen_bodies: set[str] = set() try: playwright = await manager.__aenter__() manager_entered = True @@ -116,16 +129,23 @@ class SearchBaidu: ) context = await browser.new_context(user_agent=Core.get_browser_user_agent()) page = await context.new_page() - for page_number, url in enumerate(urls): + for page_number, url in enumerate(page_urls): if page_number: await asyncio.sleep(self.REQUEST_DELAY_SECONDS) response = await page.goto(url, wait_until='domcontentloaded', timeout=60_000) if response is None: - report = SourceExecutionReport('failed', 'transport-error') + report = SourceExecutionReport('partial' if self.total_results else 'failed', 'transport-error') break body = await page.content() - if report := self._response_report(response.status, body, page.url): + if response_report := self._response_report(response.status, body, page.url): + report = ( + SourceExecutionReport('partial', response_report.stop_reason) if self.total_results else response_report + ) break + if body in seen_bodies: + report = SourceExecutionReport('partial', 'repeated-page') + break + seen_bodies.add(body) self.total_results += f' {body}' except BaseException as error: primary_error = error @@ -153,12 +173,12 @@ class SearchBaidu: isinstance(final_error, ValueError) and str(final_error) == 'startupinfo is not supported' ): if not self.total_results: - return await self._http_search(urls, proxy_url or False) - return SourceExecutionReport('failed', 'transport-error') + return await self._http_search(urls(), proxy_url or False) + return SourceExecutionReport('partial', 'transport-error') if final_error is not None: raise final_error if report is not None and report.stop_reason == 'transport-error' and not self.total_results: - return await self._http_search(urls, proxy_url or False) + return await self._http_search(urls(), proxy_url or False) return report async def process(self, proxy: bool = False) -> SourceExecutionReport | None: diff --git a/theHarvester/discovery/bravesearch.py b/theHarvester/discovery/bravesearch.py index 0ff7c1b1..491c8648 100644 --- a/theHarvester/discovery/bravesearch.py +++ b/theHarvester/discovery/bravesearch.py @@ -23,7 +23,11 @@ class SearchBrave: embedded use independent of operator configuration files. """ - def __init__(self, word: str, limit: int, credential_adapter: CredentialAdapter | None = None) -> None: + # Brave documents offsets 0 through 9; this is a provider boundary, not a + # theHarvester result cap. + MAX_OFFSET = 9 + + def __init__(self, word: str, limit: int | None, credential_adapter: CredentialAdapter | None = None) -> None: self.word = word self.results: list[dict[str, Any]] = [] self.totalresults = '' @@ -55,12 +59,12 @@ class SearchBrave: queries = [f'"{self.word}"', f'site:{self.word}'] for query in queries: - if len(self.results) >= self.limit: + if self.limit is not None and len(self.results) >= self.limit: break try: - for offset in range(10): - remaining = self.limit - len(self.results) - if remaining <= 0: + for offset in range(self.MAX_OFFSET + 1): + remaining = self.limit - len(self.results) if self.limit is not None else 20 + if self.limit is not None and remaining <= 0: break params = { 'q': query, @@ -139,14 +143,14 @@ class SearchBrave: self.totalresults += result_text + '\n' self.results.extend(results) - if len(self.results) >= self.limit: + if self.limit is not None and len(self.results) >= self.limit: return SourceExecutionReport('completed', 'result-limit') if not more_results_available: break await asyncio.sleep(get_delay()) else: - return SourceExecutionReport('partial', 'pagination-limit') + return SourceExecutionReport('partial', 'provider-limit') except ResponseStreamError as error: return SourceExecutionReport('failed', error.reason) diff --git a/theHarvester/discovery/censysearch.py b/theHarvester/discovery/censysearch.py index 7f465b96..3682e06e 100644 --- a/theHarvester/discovery/censysearch.py +++ b/theHarvester/discovery/censysearch.py @@ -13,7 +13,7 @@ class SearchCensys: MAX_RESULTS_PER_PAGE = 100 SERVER = 'https://api.platform.censys.io/v3/global/search/query' - def __init__(self, domain: str, limit: int = 500) -> None: + def __init__(self, domain: str, limit: int | None = 500) -> None: self.word = domain token, self.organization_id = Core.censys_key() if not isinstance(token, str) or not token.strip(): @@ -55,7 +55,7 @@ class SearchCensys: return False async def do_search(self) -> SourceExecutionReport | None: - if self.limit <= 0: + if self.limit is not None and self.limit <= 0: return None headers = {'Accept': 'application/json', 'Authorization': f'Bearer {self.token}'} @@ -70,11 +70,13 @@ class SearchCensys: malformed = False async with AsyncFetcher.open_session(headers=headers, proxy=self.proxy, request_timeout=720) as session: - while records_seen < self.limit: + while self.limit is None or records_seen < self.limit: body = { 'query': f'cert.names: "{self.word}"', 'fields': ['cert.names', 'cert.parsed.subject.email_address'], - 'page_size': min(self.MAX_RESULTS_PER_PAGE, self.limit - records_seen), + 'page_size': min(self.MAX_RESULTS_PER_PAGE, self.limit - records_seen) + if self.limit is not None + else self.MAX_RESULTS_PER_PAGE, } if page_token is not None: body['page_token'] = page_token @@ -108,11 +110,11 @@ class SearchCensys: return SourceExecutionReport('failed', 'invalid-response') for hit in hits: - if records_seen >= self.limit: + if self.limit is not None and records_seen >= self.limit: break malformed = self._parse_hit(hit) or malformed records_seen += 1 - if records_seen >= self.limit: + if self.limit is not None and records_seen >= self.limit: if malformed: return SourceExecutionReport('failed', 'invalid-response') return None @@ -121,7 +123,10 @@ class SearchCensys: return SourceExecutionReport('failed', 'invalid-response') return None if next_page_token in seen_tokens: - return SourceExecutionReport('failed', 'repeated-cursor') + return SourceExecutionReport( + 'partial' if self.totalhosts or self.emails else 'failed', + 'repeated-cursor', + ) seen_tokens.add(next_page_token) page_token = next_page_token return None diff --git a/theHarvester/discovery/certspottersearch.py b/theHarvester/discovery/certspottersearch.py index 893bfbb1..636918b7 100644 --- a/theHarvester/discovery/certspottersearch.py +++ b/theHarvester/discovery/certspottersearch.py @@ -13,9 +13,6 @@ class SearchCertspoter: API reference: https://sslmate.com/help/reference/ct_search_api_v1 """ - # ponytail: hard cap protects against endless unique cursors; raise only if real targets exceed 1,000 pages. - MAX_PAGES = 1000 - def __init__(self, word) -> None: self.word = word.strip().lower().rstrip('.') self.totalhosts: set = set() @@ -31,7 +28,7 @@ class SearchCertspoter: cursor = None seen_cursors: set[str] = set() try: - for _ in range(self.MAX_PAGES): + while True: params = { 'domain': self.word, 'include_subdomains': 'true', @@ -118,9 +115,6 @@ class SearchCertspoter: break seen_cursors.add(next_cursor) cursor = next_cursor - else: - self._mark_incomplete('page-limit') - logger.warning('Cert Spotter page limit reached; results may be incomplete.') except ConnectionError: self._mark_incomplete('connection-error') logger.warning('Cert Spotter network connection failed; results may be incomplete.') diff --git a/theHarvester/discovery/commoncrawl.py b/theHarvester/discovery/commoncrawl.py index 2813e456..4a25c38b 100644 --- a/theHarvester/discovery/commoncrawl.py +++ b/theHarvester/discovery/commoncrawl.py @@ -22,12 +22,10 @@ class SearchCommoncrawl: MAX_RECORDS_PER_REQUEST = 50 MAX_CONSECUTIVE_PAGE_ERRORS = 3 RUNTIME_SECONDS = 120.0 - # Protect the shared index service even when its page count is unexpectedly large. - MAX_PAGES_PER_QUERY = 100 - def __init__(self, word, limit: int = 500) -> None: + def __init__(self, word, limit: int | None = 500) -> None: self.word = word.lower().rstrip('.') - self.limit = max(limit, 0) + self.limit = max(limit, 0) if limit is not None else None self.totalhosts: set[str] = set() self.proxy = False self.hostname = 'https://index.commoncrawl.org' @@ -66,7 +64,7 @@ class SearchCommoncrawl: return '' @classmethod - def _select_indexes(cls, catalog: list[object]) -> list[dict]: + def _select_indexes(cls, catalog: list[object], *, include_all: bool = False) -> list[dict]: dated_indexes: list[tuple[datetime, dict]] = [] for entry in catalog: if not isinstance(entry, dict): @@ -98,12 +96,12 @@ class SearchCommoncrawl: if not dated_indexes: return [] - cutoff = max(timestamp for timestamp, _ in dated_indexes) - cls.INDEX_LOOKBACK + cutoff = None if include_all else max(timestamp for timestamp, _ in dated_indexes) - cls.INDEX_LOOKBACK selected: list[dict] = [] endpoints: set[str] = set() for timestamp, entry in sorted(dated_indexes, key=lambda item: item[0], reverse=True): endpoint = entry['cdx-api'] - if timestamp >= cutoff and endpoint not in endpoints: + if (cutoff is None or timestamp >= cutoff) and endpoint not in endpoints: endpoints.add(endpoint) selected.append(entry) return selected @@ -121,7 +119,7 @@ class SearchCommoncrawl: logger.error('Common Crawl API error: invalid index catalog') return SourceExecutionReport('failed', 'invalid-catalog') - indexes = self._select_indexes(catalog_response[0]) + indexes = self._select_indexes(catalog_response[0], include_all=self.limit is None) if not indexes: logger.error('Common Crawl API error: index catalog contains no usable entries') return SourceExecutionReport('failed', 'no-usable-indexes') @@ -136,7 +134,6 @@ class SearchCommoncrawl: successful_queries = 0 failed_queries = 0 - page_limit_reached = False query_number = 0 for index in indexes: endpoint = index['cdx-api'] @@ -158,12 +155,13 @@ class SearchCommoncrawl: if isinstance(page_count, bool) or not isinstance(page_count, int) or page_count < 0: raise ValueError('invalid page count') query_succeeded = page_count == 0 - page_limit = min(page_count, self.MAX_PAGES_PER_QUERY) first_page = 0 consecutive_page_errors = 0 - while first_page < page_limit: - remaining = self.limit - len(self.totalhosts) - if remaining == 0: + while first_page < page_count: + remaining = ( + self.limit - len(self.totalhosts) if self.limit is not None else self.MAX_RECORDS_PER_REQUEST + ) + if self.limit is not None and remaining == 0: return None page_url = f'{endpoint}?{urlencode({"url": query, "output": "json", "pageSize": self.PAGE_SIZE, "page": first_page, "limit": min(remaining, self.MAX_RECORDS_PER_REQUEST)})}' first_page += 1 @@ -179,7 +177,7 @@ class SearchCommoncrawl: domain = self._extract_domain_from_url(record.get('url', '')) if domain.endswith(f'.{self.word}') or domain == self.word: self.totalhosts.add(domain) - if len(self.totalhosts) >= self.limit: + if self.limit is not None and len(self.totalhosts) >= self.limit: return None except ValueError as error: message = str(error) @@ -194,12 +192,6 @@ class SearchCommoncrawl: consecutive_page_errors += 1 if consecutive_page_errors >= self.MAX_CONSECUTIVE_PAGE_ERRORS: break - if page_count > page_limit: - page_limit_reached = True - logger.warning( - f'Common Crawl page limit reached for index {index.get("id", "unknown")}; ' - 'results may be incomplete' - ) if query_had_errors: failed_queries += 1 except Exception as error: @@ -213,9 +205,6 @@ class SearchCommoncrawl: return SourceExecutionReport('partial', 'query-errors') logger.warning(f'Common Crawl failed all {query_total} queries') return SourceExecutionReport('failed', 'all-queries-failed') - if page_limit_reached: - return SourceExecutionReport('partial', 'page-limit') - except Exception as error: logger.error(f'Common Crawl API error: {error}') return SourceExecutionReport('failed', 'unexpected-error') @@ -233,4 +222,4 @@ class SearchCommoncrawl: logger.info( f'Common Crawl runtime limit reached after {self.RUNTIME_SECONDS:g}s; preserved {len(self.totalhosts)} hosts' ) - return SourceExecutionReport('failed', 'runtime-limit') + return SourceExecutionReport('partial' if self.totalhosts else 'failed', 'runtime-limit') diff --git a/theHarvester/discovery/criminalip.py b/theHarvester/discovery/criminalip.py index 1002bc30..7e68865e 100644 --- a/theHarvester/discovery/criminalip.py +++ b/theHarvester/discovery/criminalip.py @@ -5,6 +5,7 @@ from urllib.parse import urlparse from theHarvester.discovery.constants import MissingKey, get_delay from theHarvester.lib.core import AsyncFetcher, Core +from theHarvester.lib.source_execution import SourceExecutionReport logger = logging.getLogger(__name__) @@ -85,7 +86,7 @@ class SearchCriminalIP: for nested_value in value.values(): self._collect_hosts_from_value(nested_value) - async def do_search(self) -> None: + async def do_search(self) -> SourceExecutionReport | None: # https://www.criminalip.io/developer/api/post-domain-scan # https://www.criminalip.io/developer/api/get-domain-status-id # https://www.criminalip.io/developer/api/get-v2-domain-report-id @@ -103,15 +104,19 @@ class SearchCriminalIP: # {'data': {'scan_id': scan_id}, 'message': 'api success', 'status': 200} if not isinstance(response, dict): logger.info(f'CriminalIP scan response has unexpected type: {type(response).__name__}') - return + return SourceExecutionReport('failed', 'invalid-response') if response.get('status') != 200: logger.info(f'CriminalIP scan request failed with status {response.get("status")}') - return + if response.get('status') == 429: + return SourceExecutionReport('rate-limited', 'http-429') + if response.get('status') in {401, 403}: + return SourceExecutionReport('failed', 'access-denied') + return SourceExecutionReport('failed', 'provider-error') scan_id = response.get('data', {}).get('scan_id') if scan_id is None: logger.info('CriminalIP scan response did not include a scan_id') - return + return SourceExecutionReport('failed', 'invalid-response') scan_percentage = 0 counter = 0 @@ -127,25 +132,29 @@ class SearchCriminalIP: status = status_response[0] if isinstance(status_response, list) and len(status_response) > 0 else {} if not isinstance(status, dict): logger.info(f'CriminalIP status response has unexpected type: {type(status).__name__}') - return + return SourceExecutionReport('failed', 'invalid-response') if status.get('status') != 200: logger.info(f'CriminalIP status request failed with status {status.get("status")}') - return + if status.get('status') == 429: + return SourceExecutionReport('rate-limited', 'http-429') + if status.get('status') in {401, 403}: + return SourceExecutionReport('failed', 'access-denied') + return SourceExecutionReport('failed', 'provider-error') # Expected format: # {"data": {"scan_percentage": 100}, "message": "api success", "status": 200} scan_percentage = status.get('data', {}).get('scan_percentage') if scan_percentage is None: logger.info('CriminalIP status response did not include scan_percentage') - return + return SourceExecutionReport('failed', 'invalid-response') if scan_percentage == 100: break if scan_percentage == -2: logger.info(f'CriminalIP failed to scan: {self.word} does not exist, verify manually') - return + return None if scan_percentage == -1: logger.info('CriminalIP scan failed with scan_percentage -1') - return + return SourceExecutionReport('failed', 'provider-error') # Wait for scan to finish if counter >= 5: await asyncio.sleep(20 * get_delay()) @@ -156,7 +165,7 @@ class SearchCriminalIP: logger.info( 'Ten iterations have occurred in CriminalIP waiting for scan to finish, returning to prevent infinite loop.' ) - return + return SourceExecutionReport('partial', 'runtime-limit') report_url = f'https://api.criminalip.io/v2/domain/report/{scan_id}' scan_response = await AsyncFetcher.fetch_all( @@ -168,15 +177,21 @@ class SearchCriminalIP: scan = scan_response[0] if isinstance(scan_response, list) and len(scan_response) > 0 else {} if not isinstance(scan, dict): logger.info(f'CriminalIP report response has unexpected type: {type(scan).__name__}') - return + return SourceExecutionReport('failed', 'invalid-response') if scan.get('status') != 200: logger.info(f'CriminalIP report request failed with status {scan.get("status")}') - return + if scan.get('status') == 429: + return SourceExecutionReport('rate-limited', 'http-429') + if scan.get('status') in {401, 403}: + return SourceExecutionReport('failed', 'access-denied') + return SourceExecutionReport('failed', 'provider-error') try: await self.parser(scan) except Exception as e: logger.info(f'CriminalIP report parsing failed with {type(e).__name__}') + return SourceExecutionReport('failed', 'invalid-response') + return None async def parser(self, jlines): # TODO when new scope field is added to parse lines for potential new scope! @@ -314,6 +329,11 @@ class SearchCriminalIP: async def get_ips(self) -> set: return self.totalips - async def process(self, proxy: bool = False) -> None: + async def process(self, proxy: bool = False) -> SourceExecutionReport | None: self.proxy = proxy - await self.do_search() + try: + return await self.do_search() + except asyncio.CancelledError: + raise + except Exception: + return SourceExecutionReport('failed', 'transport-error') diff --git a/theHarvester/discovery/fofa.py b/theHarvester/discovery/fofa.py index ccb34edd..1cc95b98 100644 --- a/theHarvester/discovery/fofa.py +++ b/theHarvester/discovery/fofa.py @@ -15,8 +15,8 @@ class SearchFofa: MAX_PAGE_SIZE = 10_000 - def __init__(self, word: str, limit: int) -> None: - if isinstance(limit, bool) or not isinstance(limit, int) or limit <= 0: + def __init__(self, word: str, limit: int | None) -> None: + if limit is not None and (isinstance(limit, bool) or not isinstance(limit, int) or limit <= 0): raise ValueError('FOFA limit must be a positive integer') self.word = word self.limit = limit @@ -81,8 +81,8 @@ class SearchFofa: headers={'User-Agent': Core.get_user_agent()}, proxy=self.proxy, ) as session: - while records_seen < self.limit: - remaining = self.limit - records_seen + while self.limit is None or records_seen < self.limit: + remaining = self.limit - records_seen if self.limit is not None else self.MAX_PAGE_SIZE params: dict[str, str | int] = { 'email': self.email, 'key': self.api_key, @@ -110,7 +110,7 @@ class SearchFofa: if not isinstance(results, list): return SourceExecutionReport('failed', 'invalid-response') - page_results = results[:remaining] + page_results = results[:remaining] if self.limit is not None else results records_seen += len(page_results) if self._store_results(page_results): report = SourceExecutionReport('failed', 'invalid-response') @@ -118,7 +118,10 @@ class SearchFofa: if not results or not isinstance(next_cursor, str) or not next_cursor: break if next_cursor in seen_cursors: - return SourceExecutionReport('failed', 'repeated-cursor') + return SourceExecutionReport( + 'partial' if self.totalhosts or self.totalips else 'failed', + 'repeated-cursor', + ) seen_cursors.add(next_cursor) cursor = next_cursor except Exception: diff --git a/theHarvester/discovery/githubcode.py b/theHarvester/discovery/githubcode.py index 4963f912..17e75fe5 100644 --- a/theHarvester/discovery/githubcode.py +++ b/theHarvester/discovery/githubcode.py @@ -5,6 +5,7 @@ from typing import TYPE_CHECKING, Any, NamedTuple from theHarvester.discovery.constants import MissingKey, get_delay from theHarvester.lib.core import AsyncFetcher, Core +from theHarvester.lib.source_execution import SourceExecutionReport, SourceReportStatus from theHarvester.parsers import myparser if TYPE_CHECKING: @@ -29,7 +30,7 @@ class ErrorResult(NamedTuple): class SearchGithubCode: - def __init__(self, word, limit) -> None: + def __init__(self, word, limit: int | None) -> None: try: self.word = word self.total_results = '' @@ -127,35 +128,48 @@ class SearchGithubCode: logger.info(f'Error performing search: {e}') return '', {}, 500, {} - async def process(self, proxy: bool = False) -> None: + def _failure_report(self, reason: str, *, empty_status: SourceReportStatus = 'failed') -> SourceExecutionReport: + status = 'partial' if self.counter else empty_status + return SourceExecutionReport(status, reason) + + async def process(self, proxy: bool = False) -> SourceExecutionReport | None: try: self.proxy = proxy + visited_pages: set[int] = set() + seen_page_content: set[tuple[str, ...]] = set() async with AsyncFetcher.open_session(headers=self.headers, proxy=self.proxy) as session: - while self.counter < self.limit and self.page != 0: + while (self.limit is None or self.counter < self.limit) and self.page != 0: try: + visited_pages.add(self.page) api_response = await self.do_search(self.page, session) result = await self.handle_response(api_response) if isinstance(result, SuccessResult): + if self.limit is None and result.fragments: + page_content = tuple(sorted(result.fragments)) + if page_content in seen_page_content: + logger.info('\tRepeated result page detected; exiting to avoid infinite loop.') + self.page = 0 + return self._failure_report('repeated-page') + seen_page_content.add(page_content) # Reset retry counter on any successful response self.retry_count = 0 logger.info(f'\tSearching {self.counter} results.') - remaining = self.limit - self.counter + remaining = self.limit - self.counter if self.limit is not None else len(result.fragments) fragments = result.fragments[:remaining] if not fragments: self.page = 0 break self.total_results += f'{" ".join(fragments)} ' self.counter += len(fragments) - if self.counter >= self.limit: + if self.limit is not None and self.counter >= self.limit: self.page = 0 break next_or_last = result.next_page or result.last_page - # Break if pagination does not advance to avoid infinite loop - if next_or_last == self.page: - logger.info('\tNo page advancement detected; exiting to avoid infinite loop.') + if next_or_last in visited_pages: + logger.info('\tPagination cycle detected; exiting to avoid infinite loop.') self.page = 0 - break + return self._failure_report('repeated-page') self.page = next_or_last await asyncio.sleep(get_delay()) elif isinstance(result, RetryResult): @@ -163,7 +177,7 @@ class SearchGithubCode: if self.retry_count > self.max_retries: logger.info('\tMaximum retries reached; exiting to avoid infinite loop.') self.page = 0 - break + return self._failure_report('rate-limited', empty_status='rate-limited') sleepy_time = get_delay() + result.time logger.info(f'\tRetrying page in {sleepy_time} seconds...') await asyncio.sleep(sleepy_time) @@ -171,12 +185,19 @@ class SearchGithubCode: # On error, stop to avoid endless retries on a bad state logger.info(f'\tGitHub code API request failed with status {result.status_code}') self.page = 0 - break + reason = 'access-denied' if result.status_code in {401, 403} else 'provider-error' + return self._failure_report(reason) except Exception as e: logger.info(f'Error processing page: {e}') + self.retry_count += 1 + if self.retry_count > self.max_retries: + self.page = 0 + return self._failure_report('transport-error') await asyncio.sleep(get_delay()) except Exception as e: logger.info(f'An exception has occurred in githubcode process: {e}') + return self._failure_report('transport-error') + return None async def get_emails(self): try: diff --git a/theHarvester/discovery/gitlabsearch.py b/theHarvester/discovery/gitlabsearch.py index 0d63ef2f..bf9f33ed 100644 --- a/theHarvester/discovery/gitlabsearch.py +++ b/theHarvester/discovery/gitlabsearch.py @@ -3,8 +3,10 @@ import logging import re from urllib.parse import quote -from theHarvester.lib.core import AsyncFetcher, Core +from theHarvester.discovery.provider_response import provider_http_error +from theHarvester.lib.core import AsyncFetcher, Core, FetcherResponse from theHarvester.lib.hostnames import normalize_scoped_hostname +from theHarvester.lib.source_execution import SourceExecutionReport logger = logging.getLogger(__name__) @@ -12,8 +14,11 @@ logger = logging.getLogger(__name__) class SearchGitlab: """Search public GitLab project metadata, README files, and user profiles.""" - def __init__(self, word) -> None: + def __init__(self, word: str, limit: int | None = None) -> None: + if limit is not None and (isinstance(limit, bool) or not isinstance(limit, int) or limit <= 0): + raise ValueError('GitLab limit must be a positive integer') self.word = word + self.limit = limit self.totalhosts: set = set() self.totalemails: set = set() self.totalurls: set = set() @@ -32,6 +37,15 @@ class SearchGitlab: return {} return {} + @staticmethod + def _combine_reports( + current: SourceExecutionReport | None, + candidate: SourceExecutionReport | None, + ) -> SourceExecutionReport | None: + if current is None or (current.status == 'completed' and candidate is not None): + return candidate + return current + def _extract_domains_from_text(self, text: str) -> set: """Extract domain names that match the target domain.""" domains: set[str] = set() @@ -64,42 +78,74 @@ class SearchGitlab: self.totalemails.update(emails) return bool(hosts or emails) - async def search_projects(self) -> None: + async def _fetch_page( + self, + endpoint: str, + term: str, + page: int, + per_page: int, + ) -> tuple[list[object], str | None, SourceExecutionReport | None]: + url = f'{self.hostname}/api/v4/{endpoint}?search={term}&per_page={per_page}&page={page}' + response = await AsyncFetcher.fetch_all( + [url], + headers={'User-agent': Core.get_user_agent()}, + proxy=self.proxy, + json=True, + include_metadata=True, + ) + if not response: + return [], None, SourceExecutionReport('failed', 'transport-error') + payload = response[0] + headers: dict[str, str] | None = None + if isinstance(payload, FetcherResponse): + if error := provider_http_error(payload): + if page > 1 and payload.status == 400: + return [], None, SourceExecutionReport('partial', 'provider-limit') + return [], None, SourceExecutionReport(*error) + headers = payload.headers + payload = payload.body + records = self._safe_parse_json(payload) + if not isinstance(records, list): + return [], None, SourceExecutionReport('failed', 'invalid-response') + if headers is not None and 'x-next-page' in headers: + next_page = headers['x-next-page'].strip() or None + else: + next_page = str(page + 1) if len(records) >= per_page else None + return records, next_page, None + + async def search_projects(self) -> SourceExecutionReport | None: """Search GitLab projects for references to the target domain.""" try: headers = {'User-agent': Core.get_user_agent()} - - # Search for projects mentioning our domain search_terms = [self.word, f'*.{self.word}'] - + report = None for term in search_terms: - # Search projects - projects_url = f'{self.hostname}/api/v4/projects?search={term}&per_page=20' - response = await AsyncFetcher.fetch_all([projects_url], headers=headers, proxy=self.proxy) - - if not response or not isinstance(response, list) or not response[0]: - continue - - try: - projects = self._safe_parse_json(response[0]) - if not isinstance(projects, list): - continue - - for project in projects: + page = 1 + records_seen = 0 + seen_pages: set[str] = set() + seen_cursors: set[str] = set() + while self.limit is None or records_seen < self.limit: + per_page = min(100, self.limit - records_seen) if self.limit is not None else 100 + projects, next_page, page_report = await self._fetch_page('projects', term, page, per_page) + if page_report is not None: + report = self._combine_reports(report, page_report) + break + signature = json.dumps(projects, sort_keys=True, default=str) + if signature in seen_pages: + report = self._combine_reports(report, SourceExecutionReport('partial', 'repeated-page')) + break + seen_pages.add(signature) + accepted = projects[: self.limit - records_seen] if self.limit is not None else projects + records_seen += len(accepted) + for project in accepted: if not isinstance(project, dict): continue - - # Extract information from project metadata description = project.get('description', '') or '' name = project.get('name', '') or '' path = project.get('path_with_namespace', '') or '' web_url = project.get('web_url', '') or '' - - # Look for domains in description and name all_text = f'{description} {name} {path}' project_is_relevant = self._add_text_evidence(all_text) - - # Try to get README content for more detailed search project_id = project.get('id') default_branch = project.get('default_branch') if project_id and isinstance(default_branch, str) and default_branch: @@ -119,35 +165,46 @@ class SearchGitlab: if project_is_relevant and isinstance(web_url, str) and web_url.strip(): self.totalurls.add(web_url.strip()) - - except Exception as e: - logger.info(f'Failed to parse GitLab projects response: {e}') - + if self.limit is not None and records_seen >= self.limit: + report = self._combine_reports(report, SourceExecutionReport('completed', 'result-limit')) + break + if next_page is None: + break + if next_page in seen_cursors or next_page == str(page): + report = self._combine_reports(report, SourceExecutionReport('partial', 'repeated-cursor')) + break + seen_cursors.add(next_page) + try: + page = int(next_page) + except ValueError: + report = self._combine_reports(report, SourceExecutionReport('failed', 'invalid-response')) + break + return report except Exception as e: logger.info(f'GitLab API projects search error: {e}') + return SourceExecutionReport('failed', 'transport-error') - async def search_users(self) -> None: + async def search_users(self) -> SourceExecutionReport | None: """Search GitLab users for references to the target domain.""" try: - headers = {'User-agent': Core.get_user_agent()} - - # Search for users mentioning our domain - users_url = f'{self.hostname}/api/v4/users?search={self.word}&per_page=10' - response = await AsyncFetcher.fetch_all([users_url], headers=headers, proxy=self.proxy) - - if not response or not isinstance(response, list) or not response[0]: - return - - try: - users = self._safe_parse_json(response[0]) - if not isinstance(users, list): - return - - for user in users: + page = 1 + records_seen = 0 + seen_pages: set[str] = set() + seen_cursors: set[str] = set() + while self.limit is None or records_seen < self.limit: + per_page = min(100, self.limit - records_seen) if self.limit is not None else 100 + users, next_page, report = await self._fetch_page('users', self.word, page, per_page) + if report is not None: + return report + signature = json.dumps(users, sort_keys=True, default=str) + if signature in seen_pages: + return SourceExecutionReport('partial', 'repeated-page') + seen_pages.add(signature) + accepted = users[: self.limit - records_seen] if self.limit is not None else users + records_seen += len(accepted) + for user in accepted: if not isinstance(user, dict): continue - - # Extract information from user metadata name = user.get('name', '') or '' username = user.get('username', '') or '' bio = user.get('bio', '') or '' @@ -155,13 +212,10 @@ class SearchGitlab: website_url = user.get('website_url', '') or '' public_email = user.get('public_email', '') or '' - # Look for domains in user info user_hosts = self._extract_domains_from_text(f'{name} {username} {bio}') website_hosts = self._extract_domains_from_text(website_url) if isinstance(website_url, str) else set() user_hosts.update(website_hosts) self.totalhosts.update(user_hosts) - - # Check email user_emails: set[str] = set() if public_email: user_emails = self._extract_emails_from_text(public_email) @@ -173,16 +227,26 @@ class SearchGitlab: if user_is_relevant and isinstance(web_url, str) and web_url.strip(): self.totalurls.add(web_url.strip()) - - except Exception as e: - logger.info(f'Failed to parse GitLab users response: {e}') - + if self.limit is not None and records_seen >= self.limit: + return SourceExecutionReport('completed', 'result-limit') + if next_page is None: + return None + if next_page in seen_cursors or next_page == str(page): + return SourceExecutionReport('partial', 'repeated-cursor') + seen_cursors.add(next_page) + try: + page = int(next_page) + except ValueError: + return SourceExecutionReport('failed', 'invalid-response') + return None except Exception as e: logger.info(f'GitLab API users search error: {e}') + return SourceExecutionReport('failed', 'transport-error') - async def do_search(self) -> None: - await self.search_projects() - await self.search_users() + async def do_search(self) -> SourceExecutionReport | None: + project_report = await self.search_projects() + user_report = await self.search_users() + return self._combine_reports(project_report, user_report) async def get_hostnames(self) -> set: return self.totalhosts @@ -193,6 +257,6 @@ class SearchGitlab: async def get_urls(self) -> set: return self.totalurls - async def process(self, proxy: bool = False) -> None: + async def process(self, proxy: bool = False) -> SourceExecutionReport | None: self.proxy = proxy - await self.do_search() + return await self.do_search() diff --git a/theHarvester/discovery/huntersearch.py b/theHarvester/discovery/huntersearch.py index 5c1e9499..01992c14 100644 --- a/theHarvester/discovery/huntersearch.py +++ b/theHarvester/discovery/huntersearch.py @@ -3,15 +3,16 @@ import logging from theHarvester.discovery.constants import MissingKey from theHarvester.lib.core import AsyncFetcher, Core, FetcherResponse +from theHarvester.lib.source_execution import SourceExecutionReport logger = logging.getLogger(__name__) class SearchHunter: - def __init__(self, word, limit, start) -> None: + def __init__(self, word, limit: int | None, start) -> None: self.word = word self.requested_limit = limit - self.limit = min(limit, 10) + self.limit = min(limit, 10) if limit is not None else 10 self.start = start key = Core.hunter_key() self.key = key.strip() if key else '' @@ -46,14 +47,14 @@ class SearchHunter: return None return metadata.body - async def do_search(self) -> None: + async def do_search(self) -> SourceExecutionReport | None: # First determine if a user account is not a free account, this call is free is_free = True headers = {'User-Agent': Core.get_user_agent()} acc_info_url = f'https://api.hunter.io/v2/account?api_key={self.key}' response = await self._fetch_json(acc_info_url, headers) if response is None: - return + return None is_free = is_free if 'plan_name' in response['data'].keys() and response['data']['plan_name'].lower() == 'free' else False # Extract the total number of requests that are available for an account @@ -64,6 +65,13 @@ class SearchHunter: response = await self._fetch_json(self.database, headers) if response is not None: self.emails, self.hostnames = await self.parse_resp(json_resp=response) + entries = response.get('data', {}).get('emails', []) + if ( + isinstance(entries, list) + and len(entries) >= self.limit + and (self.requested_limit is None or self.requested_limit > self.limit) + ): + return SourceExecutionReport('partial', 'provider-limit') else: # Determine the total number of emails that are available # As the most emails you can get within one query are 100 @@ -71,31 +79,36 @@ class SearchHunter: hunter_dinfo_url = f'https://api.hunter.io/v2/email-count?domain={self.word}' response = await self._fetch_json(hunter_dinfo_url, headers) if response is None: - return - total_results = min(max(0, response['data']['total'] - self.start), self.requested_limit) + return None + available_results = max(0, response['data']['total'] - self.start) + total_results = ( + min(available_results, self.requested_limit) if self.requested_limit is not None else available_results + ) total_number_reqs = (total_results + 99) // 100 # Parse out meta field within initial JSON response to determine the total number of results - if total_requests_avail < total_number_reqs: + quota_exhausted = total_requests_avail < total_number_reqs + if quota_exhausted: logger.info('WARNING: account does not have enough requests to gather all emails') logger.info( f'Total requests available: {total_requests_avail}, total requests needed to be made: {total_number_reqs}' ) - logger.info('RETURNING current results, if you would still like to run this module comment out the if request') - return # max number of emails you can get per request is 100 # increments of 100 with offset determining where to start # See docs for more details: https://hunter.io/api-documentation/v2#domain-search - result_end = self.start + total_results + result_end = self.start + min(total_results, max(total_requests_avail, 0) * 100) for offset in range(self.start, result_end, 100): page_limit = min(100, result_end - offset) req_url = f'https://api.hunter.io/v2/domain-search?domain={self.word}&api_key={self.key}&limit={page_limit}&offset={offset}' response = await self._fetch_json(req_url, headers) if response is None: - return + return None temp_emails, temp_hostnames = await self.parse_resp(response) self.emails.extend(temp_emails) self.hostnames.extend(temp_hostnames) await asyncio.sleep(1) + if quota_exhausted: + return SourceExecutionReport('partial', 'quota-exhausted') + return None async def parse_resp(self, json_resp): emails = list(sorted({email['value'] for email in json_resp['data']['emails']})) @@ -111,12 +124,13 @@ class SearchHunter: ) return emails, domains - async def process(self, proxy: bool = False) -> None: + async def process(self, proxy: bool = False) -> SourceExecutionReport | None: self.proxy = proxy try: - await self.do_search() # Only need to do it once. + return await self.do_search() # Only need to do it once. except AttributeError, KeyError, TypeError: logger.info('Hunter returned malformed data') + return SourceExecutionReport('failed', 'invalid-response') async def get_emails(self): return self.emails diff --git a/theHarvester/discovery/intelxsearch.py b/theHarvester/discovery/intelxsearch.py index ae14efa4..cead6e39 100644 --- a/theHarvester/discovery/intelxsearch.py +++ b/theHarvester/discovery/intelxsearch.py @@ -1,8 +1,6 @@ import asyncio -import logging from email.errors import HeaderParseError from email.headerregistry import Address -from typing import Any from urllib.parse import urlparse import aiohttp @@ -10,79 +8,130 @@ import aiohttp from theHarvester.discovery.constants import MissingKey from theHarvester.lib.core import Core from theHarvester.lib.hostnames import normalize_scoped_hostname +from theHarvester.lib.source_execution import SourceExecutionReport from theHarvester.parsers import intelxparser -logger = logging.getLogger(__name__) - class SearchIntelx: - """Search the Intelligence X Phonebook API. + """Search the Intelligence X Phonebook API.""" - API documentation: https://github.com/IntelligenceX/SDK - """ + PAGE_SIZE = 1000 + UNLIMITED_QUERY_RESULTS = 2**31 - 1 + MAX_PENDING_POLLS = 30 + MAX_RUNTIME_SECONDS = 60.0 - def __init__(self, word) -> None: + def __init__(self, word: str, limit: int | None = None) -> None: + if limit is not None and (isinstance(limit, bool) or not isinstance(limit, int) or limit <= 0): + raise ValueError('IntelX limit must be a positive integer or None') self.word = word self.key = Core.intelx_key() if not isinstance(self.key, str) or not self.key.strip(): raise MissingKey('Intelx') self.database = 'https://2.intelx.io' - self.results: dict[str, Any] = {} + self.results: dict[str, list[object]] = {'selectors': []} self.emails: list[str] = [] self.hostnames: list[str] = [] self.urls: list[str] = [] - self.limit: int = 10000 + self.limit = limit self.proxy = False - self.offset = 0 - async def do_search(self) -> None: + async def do_search(self) -> SourceExecutionReport | None: + headers = {'x-key': self.key, 'User-Agent': Core.get_user_agent(), 'Content-Type': 'application/json'} + search_limit = self.limit if self.limit is not None else self.UNLIMITED_QUERY_RESULTS + data = { + 'term': self.word, + 'buckets': [], + 'lookuplevel': 0, + 'maxresults': search_limit, + 'timeout': 5, + 'datefrom': '', + 'dateto': '', + 'sort': 4, + 'media': 0, + 'terminate': [], + 'target': 0, + } + collected = 0 + pending_polls = 0 try: - headers = { - 'x-key': self.key, - 'User-Agent': Core.get_user_agent(), - 'Content-Type': 'application/json', - } - data = { - 'term': self.word, - 'buckets': [], - 'lookuplevel': 0, - 'maxresults': self.limit, - 'timeout': 5, - 'datefrom': '', - 'dateto': '', - 'sort': 4, # Sort by date descending for faster relevant results - 'media': 0, - 'terminate': [], - 'target': 0, - } - async with aiohttp.ClientSession() as session: - async with session.post(f'{self.database}/phonebook/search', headers=headers, json=data) as total_resp: - search_data = await total_resp.json() - if not search_data['success']: - logger.info('IntelX search request failed') - return + async with asyncio.timeout(self.MAX_RUNTIME_SECONDS): + async with aiohttp.ClientSession() as session: + async with session.post(f'{self.database}/phonebook/search', headers=headers, json=data) as response: + if response.status in {401, 403}: + return SourceExecutionReport('failed', 'access-denied') + if response.status == 429: + return SourceExecutionReport('rate-limited', 'http-429') + if not 200 <= response.status < 300: + return SourceExecutionReport('failed', f'http-{response.status}') + search_data = await response.json() + if ( + not isinstance(search_data, dict) + or search_data.get('success') is False + or not isinstance(search_data.get('id'), str) + or not search_data['id'] + ): + return SourceExecutionReport('failed', 'invalid-response') phonebook_id = search_data['id'] + while self.limit is None or collected < self.limit: + page_size = min(self.PAGE_SIZE, self.limit - collected) if self.limit is not None else self.PAGE_SIZE + async with session.get( + f'{self.database}/phonebook/search/result', + headers=headers, + params={'id': phonebook_id, 'limit': page_size}, + ) as response: + if response.status in {401, 403}: + return SourceExecutionReport('failed', 'access-denied') + if response.status == 429: + return SourceExecutionReport('rate-limited', 'http-429') + if not 200 <= response.status < 300: + return SourceExecutionReport('failed', f'http-{response.status}') + page = await response.json() + if ( + not isinstance(page, dict) + or isinstance(page.get('status'), bool) + or not isinstance(page.get('status'), int) + ): + return SourceExecutionReport('failed', 'invalid-response') + status = page['status'] + if status == 2: + return SourceExecutionReport('failed', 'search-not-found') + if status == 4: + return SourceExecutionReport('failed', 'provider-error') + if status not in {0, 1, 3}: + return SourceExecutionReport('failed', 'invalid-response') + selectors = page.get('selectors', []) + if not isinstance(selectors, list): + return SourceExecutionReport('failed', 'invalid-response') + if status == 3: + pending_polls += 1 + if pending_polls >= self.MAX_PENDING_POLLS: + return SourceExecutionReport('partial', 'runtime-limit') + await asyncio.sleep(1) + continue + if status == 0 and not selectors: + return SourceExecutionReport('failed', 'invalid-response') + pending_polls = 0 + retained = selectors[:page_size] + self.results['selectors'].extend(retained) + collected += len(retained) + if status == 1 or (self.limit is not None and collected >= self.limit): + return None + except TimeoutError: + return SourceExecutionReport('partial', 'runtime-limit') + except asyncio.CancelledError: + raise + except aiohttp.ClientError, OSError: + return SourceExecutionReport('failed', 'transport-error') + return None - await asyncio.sleep(2) # Reduced sleep time as 5s is excessive - - async with session.get( - f'{self.database}/phonebook/search/result?id={phonebook_id}&limit={self.limit}&offset={self.offset}', - headers=headers, - ) as resp: - self.results = await resp.json() - - except Exception as e: - logger.info(f'An exception has occurred in Intelx: {e}') - - async def process(self, proxy: bool = False): + async def process(self, proxy: bool = False) -> SourceExecutionReport | None: self.proxy = proxy - await self.do_search() + report = await self.do_search() intelx_parser = intelxparser.Parser() raw_emails, raw_selectors = await intelx_parser.parse_dictionaries(self.results) emails: set[str] = set() urls: set[str] = set() hostnames: set[str] = set() - for email in raw_emails: if email.count('@') != 1: continue @@ -92,7 +141,6 @@ class SearchIntelx: continue if address.username and (normalized_domain := normalize_scoped_hostname(address.domain, self.word)): emails.add(f'{address.username}@{normalized_domain}') - for selector in raw_selectors: selector = selector.strip() try: @@ -104,10 +152,10 @@ class SearchIntelx: hostnames.add(normalized_hostname) if parsed.scheme in {'http', 'https'} and parsed.netloc: urls.add(selector) - self.emails = sorted(emails) self.urls = sorted(urls) self.hostnames = sorted(hostnames) + return report async def get_emails(self) -> list[str]: return self.emails diff --git a/theHarvester/discovery/mojeek.py b/theHarvester/discovery/mojeek.py index 3096ff19..ce56c0f9 100644 --- a/theHarvester/discovery/mojeek.py +++ b/theHarvester/discovery/mojeek.py @@ -11,7 +11,7 @@ logger = logging.getLogger(__name__) class SearchMojeek: REQUEST_DELAY_SECONDS = 1.0 - def __init__(self, word, limit) -> None: + def __init__(self, word, limit: int | None) -> None: self.word = word self.limit = limit self.total_results = '' @@ -31,12 +31,80 @@ class SearchMojeek: logger.info('[*] Mojeek: No API key found, using default scraping mode.') def _stop(self, status: SourceReportStatus, reason: str) -> None: - self._report = SourceExecutionReport(status, reason) + self._report = SourceExecutionReport('partial' if self.total_results else status, reason) + + def _api_page_results(self, response: FetcherResponse) -> list[str] | None: + if response.status == 403: + self._stop('failed', 'access-denied') + return None + if response.status == 429: + self._stop('rate-limited', 'http-429') + return None + if not 200 <= response.status < 300: + self._stop('failed', f'http-{response.status}') + return None + data = response.body.get('response', response.body) if isinstance(response.body, dict) else None + if not isinstance(data, dict): + self._stop('failed', 'invalid-response') + return None + status = data.get('status') + if isinstance(status, str) and 'denied' in status.casefold(): + self._stop('failed', 'access-denied') + return None + if 'status' in data and not isinstance(status, str): + self._stop('failed', 'invalid-response') + return None + results = data.get('results') + if not isinstance(results, list): + self._stop('failed', 'invalid-response') + return None + parsed_results: list[str] = [] + for result in results: + if not isinstance(result, dict): + self._stop('failed', 'invalid-response') + return None + url_value = result.get('url') + title_value = result.get('title') + description_value = result.get('desc') + url = url_value.replace('\\/', '/') if isinstance(url_value, str) else '' + title = title_value if isinstance(title_value, str) else '' + description = description_value if isinstance(description_value, str) else '' + if not any((url, title, description)): + self._stop('failed', 'invalid-response') + return None + parsed_results.append(f'{url} {title} {description}') + return parsed_results async def _search_api(self, headers: dict[str, str]) -> None: + if self.limit is None: + seen_pages: set[tuple[str, ...]] = set() + offset = 1 + while True: + url = f'https://{self.api_server}/search?api_key={self.api_key}&q={self.word}&fmt=json&s={offset}' + responses = await AsyncFetcher.fetch_all( + [url], headers=headers, proxy=self.proxy, json=True, include_metadata=True + ) + if len(responses) != 1 or not isinstance(responses[0], FetcherResponse): + self._stop('failed', 'transport-error') + return + parsed_results = self._api_page_results(responses[0]) + if parsed_results is None: + return + if not parsed_results: + return + signature = tuple(parsed_results) + if signature in seen_pages: + self._stop('partial', 'repeated-page') + return + seen_pages.add(signature) + self.total_results += f' {" ".join(parsed_results)} ' + offset += 10 + return + + result_limit = self.limit urls = [ f'https://{self.api_server}/search?api_key={self.api_key}&q={self.word}&fmt=json&s={num}' - for num in range(1, self.limit, 10) + for num in range(1, result_limit, 10) ] responses = await AsyncFetcher.fetch_all( urls, @@ -45,55 +113,30 @@ class SearchMojeek: json=True, include_metadata=True, ) + seen_finite_pages: set[tuple[str, ...]] = set() for response in responses: if not isinstance(response, FetcherResponse): self._stop('failed', 'transport-error') return - if response.status == 403: - self._stop('failed', 'access-denied') + parsed_results = self._api_page_results(response) + if parsed_results is None: return - if response.status == 429: - self._stop('rate-limited', 'http-429') + if not parsed_results: + break + signature = tuple(parsed_results) + if signature in seen_finite_pages: + self._stop('partial', 'repeated-page') return - if not 200 <= response.status < 300: - self._stop('failed', f'http-{response.status}') - return - - data = response.body.get('response', response.body) if isinstance(response.body, dict) else None - if not isinstance(data, dict): - self._stop('failed', 'invalid-response') - return - status = data.get('status') - if isinstance(status, str) and 'denied' in status.casefold(): - self._stop('failed', 'access-denied') - return - if 'status' in data and not isinstance(status, str): - self._stop('failed', 'invalid-response') - return - results = data.get('results') - if not isinstance(results, list): - self._stop('failed', 'invalid-response') - return - for result in results: - if not isinstance(result, dict): - self._stop('failed', 'invalid-response') - return - url_value = result.get('url') - title_value = result.get('title') - description_value = result.get('desc') - url = url_value.replace('\\/', '/') if isinstance(url_value, str) else '' - title = title_value if isinstance(title_value, str) else '' - description = description_value if isinstance(description_value, str) else '' - if not any((url, title, description)): - self._stop('failed', 'invalid-response') - return - self.total_results += f' {url} {title} {description} ' - + seen_finite_pages.add(signature) + self.total_results += f' {" ".join(parsed_results)} ' logger.info('[*] Mojeek: API search completed successfully.') async def _search_keyless(self, headers: dict[str, str]) -> None: - urls = [f'https://{self.server}/search?q={self.word}&s={num}' for num in range(0, self.limit, 10)] - for page, url in enumerate(urls): + seen_bodies: set[str] = set() + offset = 0 + page = 0 + while self.limit is None or offset < self.limit: + url = f'https://{self.server}/search?q={self.word}&s={offset}' if page: await asyncio.sleep(self.REQUEST_DELAY_SECONDS) response = await AsyncFetcher.fetch( @@ -131,7 +174,13 @@ class SearchMojeek: if 'results-standard' not in normalized_body: self._stop('failed', 'invalid-response') return + if response.body in seen_bodies: + self._stop('partial', 'repeated-page') + return + seen_bodies.add(response.body) self.total_results += f' {response.body}' + offset += 10 + page += 1 async def do_search(self) -> SourceExecutionReport | None: self._report = None diff --git a/theHarvester/discovery/netlas.py b/theHarvester/discovery/netlas.py index 27c41f02..bb7ee0c4 100644 --- a/theHarvester/discovery/netlas.py +++ b/theHarvester/discovery/netlas.py @@ -1,5 +1,6 @@ from __future__ import annotations +import math from typing import Any from theHarvester.discovery.constants import MissingKey @@ -10,10 +11,14 @@ from theHarvester.lib.source_execution import SourceExecutionReport class SearchNetlas: - def __init__(self, word: str, limit: int) -> None: - if isinstance(limit, bool) or not isinstance(limit, int) or limit <= 0: - raise ValueError('Netlas limit must be a positive integer') + COUNT_URL = 'https://app.netlas.io/api/domains_count/' + DOWNLOAD_URL = 'https://app.netlas.io/api/domains/download/' + + def __init__(self, word: str, limit: int | None) -> None: + if limit is not None and (isinstance(limit, bool) or not isinstance(limit, int) or limit <= 0): + raise ValueError('Netlas limit must be a positive integer or None') self.word = word + self.unlimited = limit is None self.limit = limit self.totalhosts: set[str] = set() self.key = Core.netlas_key() @@ -32,14 +37,35 @@ class SearchNetlas: return None, SourceExecutionReport('failed', 'invalid-response') return response.body, None + def _query(self) -> str: + return f'*.{self.word}' + + async def _count(self, session: Any) -> tuple[int | None, SourceExecutionReport | None]: + response = await AsyncFetcher.fetch( + session=session, + url=self.COUNT_URL, + params={'q': self._query()}, + json=True, + include_metadata=True, + ) + body, report = self._response_body(response) + if report is not None: + return None, report + if not isinstance(body, dict): + return None, SourceExecutionReport('failed', 'invalid-response') + count = body.get('count') + if isinstance(count, bool) or not isinstance(count, int) or count < 0: + return None, SourceExecutionReport('failed', 'invalid-response') + return count, None + async def do_search(self, session: Any, size: int) -> SourceExecutionReport | None: response = await AsyncFetcher.post_fetch( - 'https://app.netlas.io/api/domains/download/', + self.DOWNLOAD_URL, session=session, json=True, include_metadata=True, json_body={ - 'q': f'*.{self.word}', + 'q': self._query(), 'size': size, 'fields': ['domain'], 'source_type': 'include', @@ -76,6 +102,19 @@ class SearchNetlas: headers={'Authorization': f'Bearer {self.key}'}, proxy=proxy, ) as session: + if self.unlimited: + size, report = await self._count(session) + if report is not None: + return report + if size is None: + return SourceExecutionReport('failed', 'invalid-response') + if size == 0: + return None + # Netlas documents counts above 1,000 as estimates within 3%. + if size > 1000: + size = math.ceil(size / 0.97) + return await self.do_search(session, size) + assert self.limit is not None return await self.do_search(session, self.limit) except Exception: return SourceExecutionReport('failed', 'transport-error') diff --git a/theHarvester/discovery/onyphe.py b/theHarvester/discovery/onyphe.py index c9a6901a..b3108e81 100644 --- a/theHarvester/discovery/onyphe.py +++ b/theHarvester/discovery/onyphe.py @@ -26,9 +26,9 @@ class SearchOnyphe: MAX_RESULTS = 10_000 - def __init__(self, word: str, limit: int) -> None: - if isinstance(limit, bool) or not isinstance(limit, int) or limit <= 0: - raise ValueError('ONYPHE limit must be a positive integer') + def __init__(self, word: str, limit: int | None) -> None: + if limit is not None and (isinstance(limit, bool) or not isinstance(limit, int) or limit <= 0): + raise ValueError('ONYPHE limit must be a positive integer or None') self.word = word self.limit = limit self.response: object = {} @@ -50,7 +50,7 @@ class SearchOnyphe: } page = 1 records_seen = 0 - result_limit = min(self.limit, self.MAX_RESULTS) + result_limit = min(self.limit, self.MAX_RESULTS) if self.limit is not None else self.MAX_RESULTS page_size = min(result_limit, self.MAX_RESULTS) last_total = 0 report = None @@ -104,8 +104,15 @@ class SearchOnyphe: logger.info('Onyphe request failed: %s', type(error).__name__) return SourceExecutionReport('failed', 'transport-error') - if self.limit > self.MAX_RESULTS and last_total > self.MAX_RESULTS and records_seen >= self.MAX_RESULTS: - return SourceExecutionReport('failed', 'provider-limit') + if self.limit is None and last_total > self.MAX_RESULTS and records_seen >= self.MAX_RESULTS: + return SourceExecutionReport('partial', 'provider-limit') + if ( + self.limit is not None + and self.limit > self.MAX_RESULTS + and last_total > self.MAX_RESULTS + and records_seen >= self.MAX_RESULTS + ): + return SourceExecutionReport('partial', 'provider-limit') return report async def parse_onyphe_resp_json(self) -> bool: diff --git a/theHarvester/discovery/pentesttools.py b/theHarvester/discovery/pentesttools.py index f6394619..b68a3c31 100644 --- a/theHarvester/discovery/pentesttools.py +++ b/theHarvester/discovery/pentesttools.py @@ -5,6 +5,7 @@ from ipaddress import ip_address from theHarvester.discovery.constants import MissingKey from theHarvester.lib.core import AsyncFetcher, Core from theHarvester.lib.hostnames import normalize_scoped_hostname +from theHarvester.lib.source_execution import SourceExecutionReport logger = logging.getLogger(__name__) @@ -32,7 +33,7 @@ class SearchPentestTools: return None return data - async def poll(self, scan_id: int) -> None: + async def poll(self, scan_id: int) -> SourceExecutionReport | None: for _attempt in range(10): await asyncio.sleep(3) status = self._response_data( @@ -44,11 +45,11 @@ class SearchPentestTools: ) ) if status is None: - return + return SourceExecutionReport('failed', 'invalid-response') status_name = status.get('status_name') if not isinstance(status_name, str): logger.info('Pentest-Tools returned a malformed status response') - return + return SourceExecutionReport('failed', 'invalid-response') if status_name in {'waiting', 'running'}: continue if status_name == 'finished': @@ -61,21 +62,23 @@ class SearchPentestTools: ) ) if output is not None: - await self.parse_json(output) + return await self.parse_json(output) + return SourceExecutionReport('failed', 'invalid-response') else: logger.info('Pentest-Tools scan did not finish successfully') - return + return SourceExecutionReport('failed', 'provider-error') logger.info('Pentest-Tools scan is still waiting after 10 status checks') + return SourceExecutionReport('partial', 'runtime-limit') - async def parse_json(self, json_results) -> None: + async def parse_json(self, json_results) -> SourceExecutionReport | None: if json_results.get('output_type') != 'subdomain_list': - return + return SourceExecutionReport('failed', 'invalid-response') try: output_data = json_results['output_data']['subdomains'] except KeyError, TypeError: - return + return SourceExecutionReport('failed', 'invalid-response') if not isinstance(output_data, list): - return + return SourceExecutionReport('failed', 'invalid-response') for result in output_data: if not isinstance(result, dict): continue @@ -92,6 +95,7 @@ class SearchPentestTools: except ValueError: continue self.totalips.add(address) + return None async def get_hostnames(self) -> set[str]: return self.totalhosts @@ -99,7 +103,7 @@ class SearchPentestTools: async def get_ips(self) -> set[str]: return self.totalips - async def do_search(self) -> None: + async def do_search(self) -> SourceExecutionReport | None: # Pentest-Tools documents Subdomain Finder as tool 20: # https://pentest-tools.com/docs/api-reference/scans/start-a-scan subdomain_payload = { @@ -121,13 +125,18 @@ class SearchPentestTools: ) ) if response is None: - return + return SourceExecutionReport('failed', 'invalid-response') scan_id = response.get('created_id') if not isinstance(scan_id, int) or isinstance(scan_id, bool): logger.info('Pentest-Tools returned a malformed start response') - return - await self.poll(scan_id) + return SourceExecutionReport('failed', 'invalid-response') + return await self.poll(scan_id) - async def process(self, proxy: bool = False) -> None: + async def process(self, proxy: bool = False) -> SourceExecutionReport | None: self.proxy = proxy - await self.do_search() # Only need to do it once. + try: + return await self.do_search() # Only need to do it once. + except asyncio.CancelledError: + raise + except Exception: + return SourceExecutionReport('failed', 'transport-error') diff --git a/theHarvester/discovery/rocketreach.py b/theHarvester/discovery/rocketreach.py index a57b160f..7105d4af 100644 --- a/theHarvester/discovery/rocketreach.py +++ b/theHarvester/discovery/rocketreach.py @@ -8,7 +8,7 @@ logger = logging.getLogger(__name__) class SearchRocketReach: - def __init__(self, word, limit) -> None: + def __init__(self, word, limit: int | None) -> None: self.ips: set = set() self.word = word self.key = Core.rocketreach_key() @@ -23,7 +23,7 @@ class SearchRocketReach: async def do_search(self) -> None: try: - if self.limit <= 0: + if self.limit is not None and self.limit <= 0: return headers = { @@ -34,8 +34,8 @@ class SearchRocketReach: start = 0 remaining = self.limit - while remaining > 0: - page_size = min(100, remaining) + while remaining is None or remaining > 0: + page_size = min(100, remaining) if remaining is not None else 100 data = { 'query': {'current_employer_domain': [self.word]}, 'start': start, @@ -71,7 +71,8 @@ class SearchRocketReach: self.emails.add(email['email']) found = len(profiles) - remaining -= found + if remaining is not None: + remaining -= found start += found pagination = result.get('pagination', {}) diff --git a/theHarvester/discovery/search_dehashed.py b/theHarvester/discovery/search_dehashed.py index 5cceb3d1..b941b7b5 100644 --- a/theHarvester/discovery/search_dehashed.py +++ b/theHarvester/discovery/search_dehashed.py @@ -10,7 +10,7 @@ logger = logging.getLogger(__name__) class SearchDehashed: - def __init__(self, word: str, limit: int = 500) -> None: + def __init__(self, word: str, limit: int | None = 500) -> None: self.word = word self.key = (Core.dehashed_key() or '').strip() if not self.key: @@ -20,7 +20,7 @@ class SearchDehashed: 'Dehashed-Api-Key': self.key, 'User-Agent': Core.get_user_agent(), } - self.limit = max(limit, 0) + self.limit = max(limit, 0) if limit is not None else None self.emails: set[str] = set() self.ips: set[str] = set() self.proxy: bool = False @@ -72,8 +72,8 @@ class SearchDehashed: logger.info(f'\t[+] Performing Dehashed search for: {self.word}') page = 1 remaining = self.limit - while remaining > 0: - size = min(100, remaining) + while remaining is None or remaining > 0: + size = min(100, remaining) if remaining is not None else 100 payload = {'query': self.word, 'page': page, 'size': size, 'wildcard': False, 'regex': False, 'de_dupe': False} try: response = await self._fetch_page(payload) @@ -89,9 +89,10 @@ class SearchDehashed: break if not entries: break - retained_entries = entries[:remaining] + retained_entries = entries[:remaining] if remaining is not None else entries self._retain_evidence(retained_entries) - remaining -= len(retained_entries) + if remaining is not None: + remaining -= len(retained_entries) logger.info(f'\t[+] Page {page} - Retrieved {len(retained_entries)} entries.') if len(entries) < size: break diff --git a/theHarvester/discovery/searchhunterhow.py b/theHarvester/discovery/searchhunterhow.py index f667c13f..0b98797e 100644 --- a/theHarvester/discovery/searchhunterhow.py +++ b/theHarvester/discovery/searchhunterhow.py @@ -1,6 +1,6 @@ import asyncio import base64 -from datetime import UTC, datetime +from datetime import UTC, date, datetime from typing import Any from dateutil.relativedelta import relativedelta @@ -14,9 +14,10 @@ from theHarvester.lib.source_execution import SourceExecutionReport class SearchHunterHow: REQUEST_DELAY_SECONDS = 2.0 + ALL_HISTORY_START = date(1970, 1, 1) - def __init__(self, word: str, limit: int = 500) -> None: - if isinstance(limit, bool) or not isinstance(limit, int) or limit <= 0: + def __init__(self, word: str, limit: int | None = 500) -> None: + if limit is not None and (isinstance(limit, bool) or not isinstance(limit, int) or limit <= 0): raise ValueError('Hunter.how limit must be a positive integer') self.word = word self.limit = limit @@ -36,7 +37,7 @@ class SearchHunterHow: async def do_search(self) -> SourceExecutionReport | None: query = base64.urlsafe_b64encode(f'domain.suffix="{self.word}"'.encode()).decode('ascii') end = datetime.now(UTC).date() - start = end - relativedelta(days=364) + start = self.ALL_HISTORY_START if self.limit is None else end - relativedelta(days=364) page = 1 returned = 0 params: dict[str, Any] = { @@ -52,11 +53,11 @@ class SearchHunterHow: headers={'User-Agent': Core.get_user_agent()}, proxy=self.proxy, ) as session: - while returned < self.limit: + while self.limit is None or returned < self.limit: request_params = { **params, 'page': page, - 'page_size': self._page_size(self.limit - returned), + 'page_size': self._page_size(self.limit - returned) if self.limit is not None else 1000, } response = await AsyncFetcher.fetch( session=session, @@ -82,7 +83,7 @@ class SearchHunterHow: if isinstance(total, bool) or not isinstance(total, int) or total < 0 or not isinstance(rows, list): return SourceExecutionReport('failed', 'invalid-response') - remaining = self.limit - returned + remaining = self.limit - returned if self.limit is not None else len(rows) malformed = False for row in rows[:remaining]: if not isinstance(row, dict) or not isinstance(row.get('domain'), str): @@ -94,7 +95,7 @@ class SearchHunterHow: report = SourceExecutionReport('failed', 'invalid-response') returned += len(rows) - if not rows or returned >= min(total, self.limit): + if not rows or returned >= (min(total, self.limit) if self.limit is not None else total): break page += 1 await asyncio.sleep(self.REQUEST_DELAY_SECONDS) diff --git a/theHarvester/discovery/securityscorecard.py b/theHarvester/discovery/securityscorecard.py index 395c04d3..488a0e1a 100644 --- a/theHarvester/discovery/securityscorecard.py +++ b/theHarvester/discovery/securityscorecard.py @@ -1,5 +1,6 @@ from __future__ import annotations +import json from ipaddress import ip_address from typing import Any @@ -13,8 +14,8 @@ from theHarvester.lib.source_execution import SourceExecutionReport, SourceRepor class SearchSecurityScorecard: PAGE_SIZE = 50 - def __init__(self, word: str, limit: int) -> None: - if isinstance(limit, bool) or not isinstance(limit, int) or limit <= 0: + def __init__(self, word: str, limit: int | None) -> None: + if limit is not None and (isinstance(limit, bool) or not isinstance(limit, int) or limit <= 0): raise ValueError('SecurityScorecard limit must be a positive integer') self.word = word self.limit = limit @@ -75,8 +76,9 @@ class SearchSecurityScorecard: async def _collect_assets(self, session: Any, route: str, field: str) -> bool: page = 0 records_seen = 0 - page_size = min(self.PAGE_SIZE, self.limit) - while records_seen < self.limit: + seen_pages: set[str] = set() + page_size = min(self.PAGE_SIZE, self.limit) if self.limit is not None else self.PAGE_SIZE + while self.limit is None or records_seen < self.limit: response = await AsyncFetcher.post_fetch( f'{self.base_url}/parent-domains/{self.word}/{route}', session=session, @@ -92,8 +94,13 @@ class SearchSecurityScorecard: if not isinstance(entries, list) or isinstance(size, bool) or not isinstance(size, int | float) or size < 0: self._stop('failed', 'invalid-response') return False + signature = json.dumps(entries, sort_keys=True, separators=(',', ':')) + if signature in seen_pages: + self._stop('partial' if self.hosts or self.ips else 'failed', 'repeated-page') + return False + seen_pages.add(signature) - remaining = self.limit - records_seen + remaining = self.limit - records_seen if self.limit is not None else len(entries) page_entries = entries[:remaining] records_seen += len(page_entries) malformed = False @@ -112,7 +119,7 @@ class SearchSecurityScorecard: malformed = True if malformed: self._stop('failed', 'invalid-response') - if records_seen >= self.limit or len(entries) < page_size: + if (self.limit is not None and records_seen >= self.limit) or len(entries) < page_size: return True page += 1 return True diff --git a/theHarvester/discovery/shodansearch.py b/theHarvester/discovery/shodansearch.py index 88e8ad61..39456892 100644 --- a/theHarvester/discovery/shodansearch.py +++ b/theHarvester/discovery/shodansearch.py @@ -1,4 +1,5 @@ import asyncio +import json import logging import re import socket @@ -15,7 +16,7 @@ from theHarvester.lib.core import AsyncFetcher, Core, FetcherResponse, ResponseS from theHarvester.lib.hostchecker import resolve_ip_addresses from theHarvester.lib.hostnames import normalize_scoped_hostname from theHarvester.lib.shodan_evidence import ShodanHostObservation, canonical_shodan_hosts -from theHarvester.lib.source_execution import SourceExecutionReport +from theHarvester.lib.source_execution import SourceExecutionReport, SourceReportStatus logger = logging.getLogger(__name__) @@ -375,6 +376,7 @@ class SearchShodan: for query in (f'hostname:{self.scope}', f'ssl:{self.scope}'): page = 1 received = 0 + seen_pages: set[str] = set() while True: try: response = await self._fetch_json( @@ -411,6 +413,11 @@ class SearchShodan: if not isinstance(matches, list) or isinstance(total, bool) or not isinstance(total, int) or total < 0: error_types.add('InvalidResponseError') break + signature = json.dumps(matches, sort_keys=True, separators=(',', ':')) + if signature in seen_pages: + error_types.add('RepeatedPageError') + break + seen_pages.add(signature) for match in matches: if not isinstance(match, dict): error_types.add('InvalidResponseError') @@ -518,8 +525,12 @@ class SearchShodan: self.error_type = next(iter(sorted(provider_error_types)), None) if dns_stop_reason is not None: - return SourceExecutionReport('failed', dns_stop_reason) + status: SourceReportStatus = 'partial' if self.shodan_hosts or self.totalhosts else 'failed' + return SourceExecutionReport(status, dns_stop_reason) if provider_error_types: + if 'RepeatedPageError' in provider_error_types: + status = 'partial' if self.shodan_hosts or self.totalhosts else 'failed' + return SourceExecutionReport(status, 'repeated-page') if provider_error_types <= {'HTTP401Error', 'HTTP403Error'}: return SourceExecutionReport('failed', 'access-denied') if provider_error_types == {'HTTP429Error'}: diff --git a/theHarvester/discovery/sourcegraph.py b/theHarvester/discovery/sourcegraph.py index 9ea1b8db..79577d04 100644 --- a/theHarvester/discovery/sourcegraph.py +++ b/theHarvester/discovery/sourcegraph.py @@ -74,29 +74,29 @@ def _parse_event(record: str) -> tuple[str, Any]: class SearchSourcegraph: """Collect descendant-hostname candidates mentioned in Sourcegraph code. - One query to Sourcegraph requests up to 5,000 matches; ``--limit`` does not - change it, and this source never contacts the target. A code mention does not - prove ownership, scope, or liveness. Repository and shard limits, along with - unstable result ordering, can make the results partial and non-exhaustive. + One query to Sourcegraph requests up to 5,000 matches; ``--limit`` caps + emitted hostnames but does not change that provider query. This source never + contacts the target. A code mention does not prove ownership, scope, or + liveness. Repository and shard limits, along with unstable result ordering, + can make the results partial and non-exhaustive. """ ENDPOINT = 'https://sourcegraph.com/.api/search/stream' MATCH_COUNT = 5000 - MAX_EVENTS = 10_000 - MAX_HOSTNAMES = 10_000 MAX_LINE_LENGTH = 4096 - def __init__(self, word: str, limit: int) -> None: - del limit # Sourcegraph uses one fixed provider query; global --limit is unrelated. + def __init__(self, word: str, limit: int | None) -> None: self.word = _normalize_hostname(word) or '' if '.' not in self.word: self.word = '' + self.limit = max(limit, 0) if limit is not None else None self.totalhosts: set[str] = set() self.proxy: bool | str = False self._report: SourceExecutionReport | None = None self._saw_done = False self._saw_terminal_progress = False self._final_progress_skipped = False + self._result_limit_reached = False def _stop(self, reason: str, status: SourceReportStatus = 'failed') -> None: self._report = SourceExecutionReport(status, reason) @@ -105,8 +105,9 @@ class SearchSourcegraph: for match in _HOST_TOKEN.finditer(content): hostname = _normalize_hostname(match.group()) if hostname and hostname != self.word and hostname.endswith(f'.{self.word}'): - if len(self.totalhosts) >= self.MAX_HOSTNAMES and hostname not in self.totalhosts: - raise OverflowError + if self.limit is not None and len(self.totalhosts) >= self.limit and hostname not in self.totalhosts: + self._result_limit_reached = True + continue self.totalhosts.add(hostname) def _consume_matches(self, payload: object) -> None: @@ -189,20 +190,12 @@ class SearchSourcegraph: self._stop(f'http-{response.status}') return - event_count = 0 async for record in response: - event_count += 1 - if event_count > self.MAX_EVENTS: - self._stop('response-limit') - return try: failure = self._consume_event(record) except json.JSONDecodeError, RecursionError, TypeError, ValueError: self._stop('invalid-response') return - except OverflowError: - self._stop('response-limit') - return if failure: self._stop(failure) return @@ -214,6 +207,8 @@ class SearchSourcegraph: self._stop('invalid-response') elif self._final_progress_skipped: self._stop('provider-limited', 'partial') + elif self._result_limit_reached: + self._stop('result-limit', 'completed') async def get_hostnames(self) -> list[str]: return sorted(self.totalhosts) diff --git a/theHarvester/discovery/subdomainapi.py b/theHarvester/discovery/subdomainapi.py new file mode 100644 index 00000000..40fcc2aa --- /dev/null +++ b/theHarvester/discovery/subdomainapi.py @@ -0,0 +1,71 @@ +from urllib.parse import urlencode + +from theHarvester.discovery.provider_response import provider_http_error +from theHarvester.lib.core import AsyncFetcher, Core, FetcherResponse +from theHarvester.lib.hostnames import normalize_hostname +from theHarvester.lib.source_execution import SourceExecutionReport + + +class SearchSubdomainApi: + """Collect passive hostname evidence from Subdomain API.""" + + BASE_URL = 'https://api.subdomain.app/v1/query' + + def __init__(self, word: str) -> None: + self.word = normalize_hostname(word) + self.totalhosts: set[str] = set() + self.proxy: bool | str = False + + async def do_search(self) -> SourceExecutionReport | None: + url = f'{self.BASE_URL}?{urlencode({"domain": self.word})}' + responses = await AsyncFetcher.fetch_all( + [url], + headers={'User-Agent': Core.get_user_agent()}, + proxy=self.proxy, + json=True, + include_metadata=True, + ) + response = responses[0] if responses else None + if error := provider_http_error(response): + return SourceExecutionReport(*error) + assert isinstance(response, FetcherResponse) + payload = response.body + if not isinstance(payload, dict): + return SourceExecutionReport('failed', 'invalid-response') + + count = payload.get('count') + total = payload.get('total') + subdomains = payload.get('subdomains') + provider_domain = payload.get('domain') + valid_count = isinstance(count, int) and not isinstance(count, bool) and count >= 0 + valid_total = isinstance(total, int) and not isinstance(total, bool) and total >= 0 + if ( + provider_domain != self.word + or not valid_count + or not valid_total + or not isinstance(subdomains, list) + or count != len(subdomains) + or total < count + ): + return SourceExecutionReport('failed', 'invalid-response') + + for candidate in subdomains: + if not isinstance(candidate, str) or '*' in candidate: + continue + try: + hostname = normalize_hostname(candidate) + except ValueError: + continue + if hostname != self.word and hostname.endswith(f'.{self.word}'): + self.totalhosts.add(hostname) + return SourceExecutionReport('partial', 'provider-limit') if total > count else None + + async def get_hostnames(self) -> set[str]: + return self.totalhosts + + async def process(self, proxy: bool | str = False) -> SourceExecutionReport | None: + self.proxy = proxy + try: + return await self.do_search() + except Exception: + return SourceExecutionReport('failed', 'transport-error') diff --git a/theHarvester/discovery/thc.py b/theHarvester/discovery/thc.py index 6ddbb5ef..30fdef2b 100644 --- a/theHarvester/discovery/thc.py +++ b/theHarvester/discovery/thc.py @@ -1,9 +1,12 @@ import asyncio import logging +from urllib.parse import urlencode import aiohttp from theHarvester.lib.core import Core +from theHarvester.lib.hostnames import normalize_scoped_hostname +from theHarvester.lib.source_execution import SourceExecutionReport logger = logging.getLogger(__name__) @@ -11,15 +14,22 @@ logger = logging.getLogger(__name__) class SearchThc: """Search THC (ip.thc.org) for subdomains.""" - def __init__(self, word: str) -> None: + PROVIDER_MAX_RESULTS = 50_000 + + def __init__(self, word: str, limit: int | None = None) -> None: + if limit is not None and (isinstance(limit, bool) or not isinstance(limit, int) or limit <= 0): + raise ValueError('THC limit must be a positive integer or None') self.word = word + self.limit = limit self.results: set = set() self.proxy = False self.max_retries = 3 self.base_delay = 2 - async def do_search(self) -> None: - url = f'https://ip.thc.org/api/v1/subdomains/download?domain={self.word}&limit=10000&hide_header=true' + async def do_search(self) -> SourceExecutionReport | None: + requested = self.PROVIDER_MAX_RESULTS if self.limit is None else min(self.limit, self.PROVIDER_MAX_RESULTS) + query = urlencode({'domain': self.word, 'limit': requested, 'hide_header': 'true'}) + url = f'https://ip.thc.org/api/v1/subdomains/download?{query}' headers = {'User-Agent': Core.get_user_agent()} for attempt in range(self.max_retries): @@ -31,7 +41,7 @@ class SearchThc: rate_remaining = response.headers.get('x-ratelimit-remaining', '0') if attempt == self.max_retries - 1: logger.info(f'THC returned status 429 after {self.max_retries} attempts') - return + return SourceExecutionReport('rate-limited', 'http-429') wait_time = self.base_delay * (attempt + 1) logger.info(f'THC rate limit hit (remaining: {rate_remaining}). Waiting {wait_time}s before retry...') await asyncio.sleep(wait_time) @@ -39,32 +49,34 @@ class SearchThc: if response.status != 200: logger.info(f'THC returned status {response.status}') - return + return SourceExecutionReport('failed', f'http-{response.status}') text = await response.text() - if text: - for line in text.splitlines(): - hostname = line.strip().lower() - if hostname and self.word.lower() in hostname: - self.results.add(hostname) - return + lines = text.splitlines() + for line in lines: + if hostname := normalize_scoped_hostname(line, self.word): + self.results.add(hostname) + if len(lines) >= requested and (self.limit is None or self.limit > self.PROVIDER_MAX_RESULTS): + return SourceExecutionReport('partial', 'provider-limit') + return None except Exception as e: error_msg = str(e).lower() if '429' in error_msg or 'rate' in error_msg: if attempt == self.max_retries - 1: logger.info(f'THC rate limit failure after {self.max_retries} attempts') - return + return SourceExecutionReport('rate-limited', 'provider-rate-limit') wait_time = self.base_delay * (attempt + 1) logger.info(f'THC rate limit detected. Waiting {wait_time}s before retry...') await asyncio.sleep(wait_time) continue logger.info(f'An exception has occurred in THC: {e}') - return + return SourceExecutionReport('failed', 'transport-error') + return SourceExecutionReport('failed', 'transport-error') async def get_hostnames(self) -> set: return self.results - async def process(self, proxy: bool = False) -> None: + async def process(self, proxy: bool = False) -> SourceExecutionReport | None: self.proxy = proxy - await self.do_search() + return await self.do_search() diff --git a/theHarvester/discovery/tombasearch.py b/theHarvester/discovery/tombasearch.py index 862c302f..d26cf770 100644 --- a/theHarvester/discovery/tombasearch.py +++ b/theHarvester/discovery/tombasearch.py @@ -3,15 +3,16 @@ import logging from theHarvester.discovery.constants import MissingKey from theHarvester.lib.core import AsyncFetcher, Core, FetcherResponse +from theHarvester.lib.source_execution import SourceExecutionReport logger = logging.getLogger(__name__) class SearchTomba: - def __init__(self, word, limit, start) -> None: + def __init__(self, word, limit: int | None, start) -> None: self.word = word self.requested_limit = limit - self.limit = min(limit, 10) + self.limit = min(limit, 10) if limit is not None else 10 self.start = start key, secret = Core.tomba_key() self.key = (key.strip() if key else '', secret.strip() if secret else '') @@ -43,7 +44,7 @@ class SearchTomba: return None return metadata.body - async def do_search(self) -> None: + async def do_search(self) -> SourceExecutionReport | None: # First determine if a user account is not a free account, this call is free is_free = True headers = { @@ -54,7 +55,7 @@ class SearchTomba: acc_info_url = 'https://api.tomba.io/v1/me' response = await self._fetch_json(acc_info_url, headers) if response is None: - return + return None is_free = ( is_free if 'name' in response['data']['pricing'].keys() and response['data']['pricing']['name'].lower() == 'free' @@ -73,24 +74,31 @@ class SearchTomba: tomba_counter = f'https://api.tomba.io/v1/email-count?domain={self.word}' response = await self._fetch_json(tomba_counter, headers) if response is None: - return - total_results = min(max(0, response['data']['total'] - self.start), self.requested_limit) + return None + available_results = max(0, response['data']['total'] - self.start) + total_results = ( + min(available_results, self.requested_limit) if self.requested_limit is not None else available_results + ) page_size = 50 first_page = self.start // page_size + 1 first_page_skip = self.start % page_size total_number_reqs = (first_page_skip + total_results + page_size - 1) // page_size if total_results else 0 - if total_requests_avail < total_number_reqs: + quota_exhausted = total_requests_avail < total_number_reqs + if quota_exhausted: logger.info('WARNING: The account does not have enough requests to gather all the emails.') - return remaining = total_results - for page in range(first_page, first_page + total_number_reqs): + provider_limit_reached = False + pages_to_fetch = min(total_number_reqs, max(total_requests_avail, 0)) + for page in range(first_page, first_page + pages_to_fetch): req_url = f'https://api.tomba.io/v1/domain-search?domain={self.word}&limit={page_size}&page={page}' response = await self._fetch_json(req_url, headers) if response is None: - return + return None skip = first_page_skip if page == first_page else 0 + raw_entries = response['data']['emails'] + provider_limit_reached = is_free and isinstance(raw_entries, list) and len(raw_entries) >= page_size response['data']['emails'] = response['data']['emails'][skip : skip + remaining] temp_emails, temp_hostnames = await self.parse_resp(response) self.emails.extend(temp_emails) @@ -98,6 +106,11 @@ class SearchTomba: remaining -= len(response['data']['emails']) if not is_free: await asyncio.sleep(1) + if quota_exhausted: + return SourceExecutionReport('partial', 'quota-exhausted') + if provider_limit_reached and (self.requested_limit is None or self.requested_limit > page_size): + return SourceExecutionReport('partial', 'provider-limit') + return None async def parse_resp(self, json_resp): emails = list(sorted({email['email'] for email in json_resp['data']['emails']})) @@ -113,12 +126,13 @@ class SearchTomba: ) return emails, domains - async def process(self, proxy: bool = False) -> None: + async def process(self, proxy: bool = False) -> SourceExecutionReport | None: self.proxy = proxy try: - await self.do_search() # Only need to do it once. + return await self.do_search() # Only need to do it once. except AttributeError, KeyError, TypeError: logger.info('Tomba returned malformed data') + return SourceExecutionReport('failed', 'invalid-response') async def get_emails(self): return self.emails diff --git a/theHarvester/discovery/urlscan.py b/theHarvester/discovery/urlscan.py index 9f679f0e..639e4e09 100644 --- a/theHarvester/discovery/urlscan.py +++ b/theHarvester/discovery/urlscan.py @@ -16,8 +16,8 @@ logger = logging.getLogger(__name__) class SearchUrlscan: MAX_PAGE_SIZE = 10_000 - def __init__(self, word: str, limit: int) -> None: - if isinstance(limit, bool) or not isinstance(limit, int) or limit <= 0: + def __init__(self, word: str, limit: int | None) -> None: + if limit is not None and (isinstance(limit, bool) or not isinstance(limit, int) or limit <= 0): raise ValueError('URLScan limit must be a positive integer') self.word = word self.limit = limit @@ -143,8 +143,8 @@ class SearchUrlscan: malformed = False try: async with AsyncFetcher.open_session(proxy=self.proxy) as session: - while records_seen < self.limit: - remaining = self.limit - records_seen + while self.limit is None or records_seen < self.limit: + remaining = self.limit - records_seen if self.limit is not None else self.MAX_PAGE_SIZE params: dict[str, str | int] = { 'q': f'domain:{self.word}', 'size': min(self.MAX_PAGE_SIZE, remaining), @@ -169,16 +169,19 @@ class SearchUrlscan: if not results: return SourceExecutionReport('failed', 'invalid-response') if malformed else None - page_results = results[:remaining] + page_results = results[:remaining] if self.limit is not None else results records_seen += len(page_results) malformed = self._parse_results(page_results, collected_at) or malformed - if records_seen >= self.limit: + if self.limit is not None and records_seen >= self.limit: break next_cursor = self._cursor(page_results[-1]) if next_cursor is None: return SourceExecutionReport('failed', 'invalid-cursor') if next_cursor in seen_cursors: - return SourceExecutionReport('failed', 'repeated-cursor') + return SourceExecutionReport( + 'partial' if any((self.totalhosts, self.totalips, self.urls, self.totalasns)) else 'failed', + 'repeated-cursor', + ) seen_cursors.add(next_cursor) cursor = next_cursor except Exception as error: diff --git a/theHarvester/discovery/virustotal.py b/theHarvester/discovery/virustotal.py index 1bf5e63a..5c8b0621 100644 --- a/theHarvester/discovery/virustotal.py +++ b/theHarvester/discovery/virustotal.py @@ -10,8 +10,8 @@ from theHarvester.lib.source_execution import SourceExecutionReport class SearchVirustotal: - def __init__(self, word: str, limit: int) -> None: - if isinstance(limit, bool) or not isinstance(limit, int) or limit <= 0: + def __init__(self, word: str, limit: int | None) -> None: + if limit is not None and (isinstance(limit, bool) or not isinstance(limit, int) or limit <= 0): raise ValueError('VirusTotal limit must be a positive integer') self.key = Core.virustotal_key() if not isinstance(self.key, str) or not self.key.strip(): @@ -29,8 +29,8 @@ class SearchVirustotal: report = None try: async with AsyncFetcher.open_session(headers=headers, proxy=self.proxy) as session: - while records_seen < self.limit: - remaining = self.limit - records_seen + while self.limit is None or records_seen < self.limit: + remaining = self.limit - records_seen if self.limit is not None else 40 params: dict[str, int | str] = {'limit': min(40, remaining)} if cursor: params['cursor'] = cursor @@ -50,11 +50,11 @@ class SearchVirustotal: meta = response.body.get('meta', {}) if not isinstance(data, list) or not isinstance(meta, dict): return SourceExecutionReport('failed', 'invalid-response') - page_data = data[:remaining] + page_data = data[:remaining] if self.limit is not None else data records_seen += len(page_data) hostnames, malformed = self.parse_hostnames(page_data, self.word) for hostname in sorted(hostnames): - if len(self.hostnames) >= self.limit: + if self.limit is not None and len(self.hostnames) >= self.limit: break self.hostnames.add(hostname) if malformed: @@ -63,7 +63,7 @@ class SearchVirustotal: if not data or not isinstance(next_cursor, str) or not next_cursor: break if next_cursor in seen_cursors: - return SourceExecutionReport('failed', 'repeated-cursor') + return SourceExecutionReport('partial' if self.hostnames else 'failed', 'repeated-cursor') seen_cursors.add(next_cursor) cursor = next_cursor except Exception: diff --git a/theHarvester/discovery/waybackarchive.py b/theHarvester/discovery/waybackarchive.py index 6d77e352..33cecb10 100644 --- a/theHarvester/discovery/waybackarchive.py +++ b/theHarvester/discovery/waybackarchive.py @@ -3,7 +3,7 @@ import logging from urllib.parse import unquote_plus, urlencode, urlsplit from theHarvester.lib.core import AsyncFetcher, Core -from theHarvester.lib.source_execution import SourceExecutionReport, SourceReportStatus +from theHarvester.lib.source_execution import SourceExecutionReport logger = logging.getLogger(__name__) @@ -16,12 +16,10 @@ class SearchWaybackarchive: PAGE_SIZE = 1000 RUNTIME_SECONDS = 30.0 - # ponytail: hard cap protects against endless cursors; raise only if real targets exceed one million rows. - MAX_PAGES_PER_QUERY = 1000 - def __init__(self, word, limit: int = 500) -> None: + def __init__(self, word, limit: int | None = 500) -> None: self.word = word.strip().rstrip('.').lower() - self.limit = max(limit, 0) + self.limit = max(limit, 0) if limit is not None else None self.totalhosts: set = set() self.proxy = False self.hostname = 'https://web.archive.org' @@ -61,7 +59,9 @@ class SearchWaybackarchive: async def _search_pattern(self, pattern: str, headers: dict[str, str]) -> str | None: resume_key: str | None = None seen_resume_keys: set[str] = set() - for page_number in range(1, self.MAX_PAGES_PER_QUERY + 1): + page_number = 0 + while True: + page_number += 1 query = { 'url': pattern, 'fl': 'original', @@ -92,7 +92,7 @@ class SearchWaybackarchive: domain = self._extract_domain_from_url(line.strip()) if domain.endswith(f'.{self.word}') or domain == self.word: self.totalhosts.add(domain) - if len(self.totalhosts) >= self.limit: + if self.limit is not None and len(self.totalhosts) >= self.limit: return 'result-limit' if page_number == 1 or page_number % 10 == 0: @@ -102,8 +102,6 @@ class SearchWaybackarchive: return None seen_resume_keys.add(next_resume_key) resume_key = next_resume_key - logger.info(f'Wayback Archive page limit reached for pattern {pattern}; results may be incomplete') - return 'page-limit' async def do_search(self) -> SourceExecutionReport | None: if self.limit == 0: @@ -124,9 +122,6 @@ class SearchWaybackarchive: if degraded_reason is None: return SourceExecutionReport('completed', 'result-limit') break - if outcome == 'page-limit': - degraded_reason = degraded_reason or outcome - break if outcome is not None: degraded_reason = degraded_reason or outcome except TimeoutError: @@ -134,10 +129,9 @@ class SearchWaybackarchive: f'Wayback Archive runtime limit reached after {self.RUNTIME_SECONDS:g}s; ' f'preserved {len(self.totalhosts)} hosts' ) - return SourceExecutionReport('failed', 'runtime-limit') + return SourceExecutionReport('partial' if self.totalhosts else 'failed', 'runtime-limit') if degraded_reason is not None: - status: SourceReportStatus = 'partial' if degraded_reason == 'page-limit' else 'failed' - return SourceExecutionReport(status, degraded_reason) + return SourceExecutionReport('failed', degraded_reason) except Exception as e: logger.info(f'Wayback Archive API error: {e}') return SourceExecutionReport('failed', 'unexpected-error') diff --git a/theHarvester/discovery/whoisxml.py b/theHarvester/discovery/whoisxml.py index a21e9042..502c771f 100644 --- a/theHarvester/discovery/whoisxml.py +++ b/theHarvester/discovery/whoisxml.py @@ -8,8 +8,8 @@ from theHarvester.lib.source_execution import SourceExecutionReport class SearchWhoisXML: - def __init__(self, word: str, limit: int) -> None: - if isinstance(limit, bool) or not isinstance(limit, int) or limit <= 0: + def __init__(self, word: str, limit: int | None) -> None: + if limit is not None and (isinstance(limit, bool) or not isinstance(limit, int) or limit <= 0): raise ValueError('WhoisXML limit must be a positive integer') self.word = word self.limit = limit @@ -25,7 +25,7 @@ class SearchWhoisXML: records_seen = 0 report = None async with AsyncFetcher.open_session(proxy=self.proxy) as session: - while records_seen < self.limit: + while self.limit is None or records_seen < self.limit: params = {'apiKey': self.key, 'domainName': self.word} if cursor is not None: params['searchAfter'] = cursor @@ -48,7 +48,7 @@ class SearchWhoisXML: if not isinstance(next_cursor, str): return SourceExecutionReport('failed', 'invalid-response') - remaining = self.limit - records_seen + remaining = self.limit - records_seen if self.limit is not None else len(result['records']) records = result['records'][:remaining] records_seen += len(records) malformed = False @@ -60,10 +60,10 @@ class SearchWhoisXML: self.total_results.add(hostname) if malformed: report = SourceExecutionReport('failed', 'invalid-response') - if records_seen >= self.limit or not next_cursor: + if (self.limit is not None and records_seen >= self.limit) or not next_cursor: break if next_cursor in seen_cursors: - return SourceExecutionReport('failed', 'repeated-cursor') + return SourceExecutionReport('partial' if self.total_results else 'failed', 'repeated-cursor') seen_cursors.add(next_cursor) cursor = next_cursor return report diff --git a/theHarvester/discovery/windvane.py b/theHarvester/discovery/windvane.py index f9e024a6..ae89c535 100644 --- a/theHarvester/discovery/windvane.py +++ b/theHarvester/discovery/windvane.py @@ -1,8 +1,16 @@ +from __future__ import annotations + import json import logging +from typing import TYPE_CHECKING from theHarvester.lib.core import AsyncFetcher, Core from theHarvester.lib.hostnames import normalize_scoped_hostname +from theHarvester.lib.source_execution import SourceExecutionReport + +if TYPE_CHECKING: + from collections.abc import Callable + from typing import Any logger = logging.getLogger(__name__) @@ -24,8 +32,11 @@ class SearchWindvane: Set the key with ``WINDVANE_API_KEY`` or ``search.set_api_key("your-key")``. """ - def __init__(self, word) -> None: + def __init__(self, word: str, limit: int | None = None) -> None: + if limit is not None and (isinstance(limit, bool) or not isinstance(limit, int) or limit <= 0): + raise ValueError('Windvane limit must be a positive integer') self.word = word.strip().lower().rstrip('.') + self.limit = limit self.totalhosts: set = set() self.totalips: set = set() self.totalemails: set = set() @@ -65,7 +76,85 @@ class SearchWindvane: return {} return {} - async def do_search(self) -> None: + @staticmethod + def _next_page(data: dict[str, Any], page: int, count: int, records: list[object]) -> int | None: + metadata = [data] + metadata.extend( + value for key in ('pagination', 'page_response', 'page_info') if isinstance((value := data.get(key)), dict) + ) + for section in metadata: + for key in ('next_page', 'nextPage'): + if key in section: + value = section[key] + return int(value) if isinstance(value, int | str) and str(value).isdigit() and int(value) > 0 else None + for key in ('has_more', 'hasMore'): + if isinstance(section.get(key), bool): + return page + 1 if section[key] else None + for key in ('total', 'total_count', 'totalCount'): + if isinstance(section.get(key), int): + return page + 1 if page * count < section[key] else None + return page + 1 if len(records) >= count else None + + async def _paginate( + self, + headers: dict[str, str], + endpoint: str, + query: dict[str, str], + page_size: int, + consume: Callable[[object], object], + ) -> SourceExecutionReport | None: + page = 1 + records_seen = 0 + seen_pages: set[str] = set() + seen_cursors: set[int] = set() + url = f'{self.hostname}/{endpoint}' + while self.limit is None or records_seen < self.limit: + count = min(page_size, self.limit - records_seen) if self.limit is not None else page_size + request_data: dict[str, object] = {**query, 'page_request': {'page': page, 'count': count}} + response = await AsyncFetcher.post_fetch( + url, + headers=headers, + data=json.dumps(request_data, separators=(',', ':')), + proxy=self.proxy, + ) + if not response: + return SourceExecutionReport('failed', 'transport-error') + response_data = self._safe_parse_json(response) + if not response_data: + return SourceExecutionReport('failed', 'invalid-response') + if response_data.get('code') != 0: + logger.info(f'Windvane {endpoint} API returned code {response_data.get("code")}') + return ( + SourceExecutionReport('partial', 'provider-limit') + if records_seen + else SourceExecutionReport('failed', 'provider-error') + ) + data = response_data.get('data') + if not isinstance(data, dict) or not isinstance(data.get('list'), list): + return SourceExecutionReport('failed', 'invalid-response') + records = data['list'] + if not records: + return None + signature = json.dumps(records, sort_keys=True, default=str) + if signature in seen_pages: + return SourceExecutionReport('partial', 'repeated-page') + seen_pages.add(signature) + accepted = records[: self.limit - records_seen] if self.limit is not None else records + for record in accepted: + consume(record) + records_seen += len(accepted) + if self.limit is not None and records_seen >= self.limit: + return SourceExecutionReport('completed', 'result-limit') + next_page = self._next_page(data, page, count, records) + if next_page is None: + return None + if next_page == page or next_page in seen_cursors: + return SourceExecutionReport('partial', 'repeated-cursor') + seen_cursors.add(next_page) + page = next_page + return None + + async def do_search(self) -> SourceExecutionReport | None: """Query the Windvane endpoints used by this source.""" try: headers = {'User-agent': Core.get_user_agent(), 'Content-Type': 'application/json', 'Accept': 'application/json'} @@ -75,180 +164,71 @@ class SearchWindvane: headers['X-Api-Key'] = self.api_key # With API key, use full API endpoints - await self._search_subdomains(headers) - await self._search_dns_history(headers) - await self._search_emails(headers) + reports = [ + await self._search_subdomains(headers), + await self._search_dns_history(headers), + await self._search_emails(headers), + ] else: # Without API key, use the provider's limited endpoint only. logger.info('[*] Windvane API key not found. Using limited unauthenticated access.') - await self._search_subdomains_limited(headers) + reports = [await self._search_subdomains_limited(headers)] + + retained = [report for report in reports if report is not None] + return next((report for report in retained if report.status != 'completed'), retained[0] if retained else None) except Exception as e: logger.info(f'Windvane API error: {e}') + return SourceExecutionReport('failed', 'transport-error') - async def _search_subdomains(self, headers: dict) -> None: + async def _search_subdomains(self, headers: dict[str, str]) -> SourceExecutionReport | None: """Search for subdomains with ``/ListSubDomain``.""" - try: - url = f'{self.hostname}/ListSubDomain' + return await self._paginate( + headers, + 'ListSubDomain', + {'domain': self.word}, + 30, + lambda item: self._add_host(item.get('domain')) if isinstance(item, dict) else None, + ) - # Use pagination to get more results - for page in range(1, 4): # Get first 3 pages (up to 90 results) - data = {'domain': self.word, 'page_request': {'page': page, 'count': 30}} - - try: - response = await AsyncFetcher.post_fetch( - url, - headers=headers, - data=json.dumps(data, separators=(',', ':')), - proxy=self.proxy, - ) - if response: - response_data = self._safe_parse_json(response) - - # Check if response is successful - if response_data.get('code') == 0: - data_section = response_data.get('data', {}) - subdomains = data_section.get('list', []) - - if not subdomains: - break # No more results - - for item in subdomains: - if isinstance(item, dict): - self._add_host(item.get('domain')) - else: - # API error - stop pagination - if response_data.get('code') != 0: - logger.info(f'Windvane subdomain API returned code {response_data.get("code")}') - break - - except Exception as e: - logger.info(f'Windvane subdomain request failed: {e}') - break - - except Exception as e: - logger.info(f'Windvane subdomain search error: {e}') - - async def _search_dns_history(self, headers: dict) -> None: + async def _search_dns_history(self, headers: dict[str, str]) -> SourceExecutionReport | None: """Collect subdomains and IP addresses from ``/ListDNS`` history.""" - try: - url = f'{self.hostname}/ListDNS' - # Get DNS history records - for page in range(1, 3): # Get first 2 pages - data = {'domain': self.word, 'page_request': {'page': page, 'count': 30}} + def consume(record: object) -> None: + if not isinstance(record, dict): + return + answer = record.get('answer', '') + if ( + self._add_host(record.get('domain')) + and record.get('answer_type') == 'A' + and isinstance(answer, str) + and self._is_valid_ip(answer) + ): + self.totalips.add(answer) - try: - response = await AsyncFetcher.post_fetch( - url, - headers=headers, - data=json.dumps(data, separators=(',', ':')), - proxy=self.proxy, - ) - if response: - response_data = self._safe_parse_json(response) + return await self._paginate(headers, 'ListDNS', {'domain': self.word}, 30, consume) - if response_data.get('code') == 0: - data_section = response_data.get('data', {}) - dns_records = data_section.get('list', []) - - if not dns_records: - break - - for record in dns_records: - if isinstance(record, dict): - answer = record.get('answer', '') - answer_type = record.get('answer_type', '') - - domain_is_scoped = self._add_host(record.get('domain')) - - # Add IP addresses from A records - if domain_is_scoped and answer and answer_type == 'A' and self._is_valid_ip(answer): - self.totalips.add(answer) - else: - break - - except Exception as e: - logger.info(f'Windvane DNS history request failed: {e}') - break - - except Exception as e: - logger.info(f'Windvane DNS history search error: {e}') - - async def _search_emails(self, headers: dict) -> None: + async def _search_emails(self, headers: dict[str, str]) -> SourceExecutionReport | None: """Search for email addresses with ``/ListEmail``.""" - try: - url = f'{self.hostname}/ListEmail' - data = {'email': self.word, 'page_request': {'page': 1, 'count': 50}} + def consume(item: object) -> None: + if isinstance(item, dict): + self._add_email(item.get('email')) + self._add_host(item.get('domain')) - try: - response = await AsyncFetcher.post_fetch( - url, - headers=headers, - data=json.dumps(data, separators=(',', ':')), - proxy=self.proxy, - ) - if response: - response_data = self._safe_parse_json(response) + return await self._paginate(headers, 'ListEmail', {'email': self.word}, 50, consume) - if response_data.get('code') == 0: - data_section = response_data.get('data', {}) - email_results = data_section.get('list', []) - - for item in email_results: - if isinstance(item, dict): - self._add_email(item.get('email')) - self._add_host(item.get('domain')) - - except Exception as e: - logger.info(f'Windvane email search request failed: {e}') - - except Exception as e: - logger.info(f'Windvane email search error: {e}') - - async def _search_subdomains_limited(self, headers: dict) -> None: + async def _search_subdomains_limited(self, headers: dict[str, str]) -> SourceExecutionReport | None: """Search the unauthenticated subdomain endpoints.""" - try: - # Try basic subdomain endpoint with minimal parameters - url = f'{self.hostname}/ListSubDomain' - - # Simple request with just domain - limited to 5 calls - data = { - 'domain': self.word, - 'page_request': { - 'page': 1, - 'count': 10, # Smaller count for unauthenticated - }, - } - - try: - response = await AsyncFetcher.post_fetch( - url, - headers=headers, - data=json.dumps(data, separators=(',', ':')), - proxy=self.proxy, - ) - if response: - response_data = self._safe_parse_json(response) - - if isinstance(response_data, dict) and response_data.get('code') == 0: - data_section = response_data.get('data', {}) - subdomains = data_section.get('list', []) - - for item in subdomains: - if isinstance(item, dict): - self._add_host(item.get('domain')) - - logger.info(f'[*] Found {len(subdomains)} subdomains with limited access') - else: - logger.info(f'Windvane limited API returned code {response_data.get("code")}') - - except Exception as e: - logger.info(f'Windvane limited API failed: {e}') - - except Exception as e: - logger.info(f'Windvane limited search error: {e}') + report = await self._paginate( + headers, + 'ListSubDomain', + {'domain': self.word}, + 10, + lambda item: self._add_host(item.get('domain')) if isinstance(item, dict) else None, + ) + logger.info(f'[*] Found {len(self.totalhosts)} subdomains with limited access') + return report def set_api_key(self, api_key: str) -> None: """Set the API key for authenticated requests. @@ -276,7 +256,7 @@ class SearchWindvane: async def get_emails(self) -> set: return self.totalemails - async def process(self, proxy: bool = False) -> None: + async def process(self, proxy: bool = False) -> SourceExecutionReport | None: """Run the Windvane search. Args: @@ -287,4 +267,4 @@ class SearchWindvane: # API key is already set via _get_api_key() method - await self.do_search() + return await self.do_search() diff --git a/theHarvester/discovery/yahoosearch.py b/theHarvester/discovery/yahoosearch.py index 84dd469e..1c7ba72d 100644 --- a/theHarvester/discovery/yahoosearch.py +++ b/theHarvester/discovery/yahoosearch.py @@ -1,26 +1,79 @@ -from theHarvester.lib.core import AsyncFetcher, Core +from __future__ import annotations + +from theHarvester.lib.core import AsyncFetcher, Core, FetcherResponse +from theHarvester.lib.source_execution import SourceExecutionReport from theHarvester.parsers import myparser class SearchYahoo: - def __init__(self, word, limit) -> None: + def __init__(self, word, limit: int | None) -> None: self.word = word self.total_results = '' self.server = 'search.yahoo.com' self.limit = limit self.proxy = False - async def do_search(self) -> None: + def _page(self, response: object) -> tuple[str | None, SourceExecutionReport | None]: + if response is None: + return None, SourceExecutionReport('partial' if self.total_results else 'failed', 'transport-error') + if isinstance(response, FetcherResponse): + if not 200 <= response.status < 300: + return None, SourceExecutionReport('partial' if self.total_results else 'failed', f'http-{response.status}') + response = response.body + if not isinstance(response, str): + return None, SourceExecutionReport('partial' if self.total_results else 'failed', 'invalid-response') + normalized = response.casefold() + if not response.strip() or 'no results' in normalized or 'no-results' in normalized: + return '', None + if 'captcha' in normalized or 'verify you are human' in normalized: + return None, SourceExecutionReport('partial' if self.total_results else 'failed', 'security-verification') + if 'access denied' in normalized or 'temporarily blocked' in normalized: + return None, SourceExecutionReport('partial' if self.total_results else 'failed', 'access-denied') + return response, None + + async def do_search(self) -> SourceExecutionReport | None: base_url = f'https://{self.server}/search?p=%40{self.word}&b=xx&pz=10' headers = {'Host': self.server, 'User-Agent': Core.get_browser_user_agent()} - urls = [base_url.replace('xx', str(num)) for num in range(0, self.limit, 10) if num <= self.limit] - responses = await AsyncFetcher.fetch_all(urls, headers=headers, proxy=self.proxy) + if self.limit is None: + seen_pages: set[str] = set() + offset = 0 + while True: + response = await AsyncFetcher.fetch( + url=base_url.replace('xx', str(offset)), + headers=headers, + proxy=self.proxy, + include_metadata=True, + ) + body, report = self._page(response) + if report is not None: + return report + if not body: + return None + if body in seen_pages: + return SourceExecutionReport('partial', 'repeated-page') + seen_pages.add(body) + self.total_results += f' {body}' + offset += 10 + urls = [base_url.replace('xx', str(num)) for num in range(0, self.limit, 10)] + responses = await AsyncFetcher.fetch_all(urls, headers=headers, proxy=self.proxy, include_metadata=True) + if urls and not responses: + return SourceExecutionReport('failed', 'transport-error') + seen_finite_pages: set[str] = set() for response in responses: - self.total_results += f' {response}' + body, report = self._page(response) + if report is not None: + return report + if not body: + break + if body in seen_finite_pages: + return SourceExecutionReport('partial', 'repeated-page') + seen_finite_pages.add(body) + self.total_results += f' {body}' + return None - async def process(self, proxy: bool = False) -> None: + async def process(self, proxy: bool = False) -> SourceExecutionReport | None: self.proxy = proxy - await self.do_search() + return await self.do_search() async def get_emails(self): rawres = myparser.Parser(self.total_results, self.word) diff --git a/theHarvester/discovery/zoomeyesearch.py b/theHarvester/discovery/zoomeyesearch.py index fc554989..fe0e969a 100644 --- a/theHarvester/discovery/zoomeyesearch.py +++ b/theHarvester/discovery/zoomeyesearch.py @@ -33,8 +33,8 @@ class SearchZoomEye: ) URL_PATTERN = re.compile(r'https?://[^\s"\'<>]+') - def __init__(self, word: str, limit: int) -> None: - if isinstance(limit, bool) or not isinstance(limit, int) or limit <= 0: + def __init__(self, word: str, limit: int | None) -> None: + if limit is not None and (isinstance(limit, bool) or not isinstance(limit, int) or limit <= 0): raise ValueError('ZoomEye limit must be a positive integer') key = Core.zoomeye_key() if not isinstance(key, str) or not key.strip(): @@ -106,14 +106,15 @@ class SearchZoomEye: return response.body async def do_search(self, session: Any) -> None: - page_size = min(self.PAGE_SIZE, self.limit) + page_size = min(self.PAGE_SIZE, self.limit) if self.limit is not None else self.PAGE_SIZE first = await self._fetch_page(session, 1, page_size) if first is None: return await self._store_matches(first['data'][:page_size]) - page_limit = math.ceil(min(first['total'], self.limit) / page_size) if first['total'] else 1 + total = min(first['total'], self.limit) if self.limit is not None else first['total'] + page_limit = math.ceil(total / page_size) if total else 1 for page in range(2, page_limit + 1): - remaining = self.limit - ((page - 1) * page_size) + remaining = self.limit - ((page - 1) * page_size) if self.limit is not None else page_size response = await self._fetch_page(session, page, page_size) if response is None: return diff --git a/theHarvester/lib/api/run_models.py b/theHarvester/lib/api/run_models.py index f2e41d77..6f47b59c 100644 --- a/theHarvester/lib/api/run_models.py +++ b/theHarvester/lib/api/run_models.py @@ -80,9 +80,11 @@ class RunRequest(BaseModel): ) limit: int = Field( default=500, - ge=1, - le=10_000, - description='Maximum results requested from each source when that provider supports a limit.', + ge=0, + description=( + 'Maximum results requested from each source when supported; 0 continues to provider exhaustion with ' + 'no local result or page-count cap.' + ), ) start: int = Field( default=DEFAULT_RESULT_START, @@ -587,6 +589,15 @@ class ScreenshotRecord(BaseModel): url: str +class SourceYieldSummary(BaseModel): + source: str + observed_result_count: int = Field(ge=0) + unique_result_count: int = Field(ge=0) + shared_result_count: int = Field(ge=0) + resolved_hostname_count: int = Field(ge=0) + unique_resolved_hostname_count: int = Field(ge=0) + + RunStatus = Literal['queued', 'running', 'cancelling', 'cancelled', 'completed', 'failed'] Activity = Literal['P0', 'P1', 'P2'] @@ -618,6 +629,7 @@ class RunDetail(RunSummary): request: RunRequest | ImportedRunRequest results: list[RunResult] source_executions: list[dict[str, Any]] + source_yields: list[SourceYieldSummary] action_executions: list[dict[str, Any]] artifacts: list[dict[str, Any]] screenshots: list[ScreenshotRecord] diff --git a/theHarvester/lib/api/run_store.py b/theHarvester/lib/api/run_store.py index f9935121..933fa06f 100644 --- a/theHarvester/lib/api/run_store.py +++ b/theHarvester/lib/api/run_store.py @@ -238,11 +238,23 @@ class RunStore: ), } if detail: + source_yields = ( + [ + item.to_dict() + for item in await self.results.source_yields( + UUID(str(record['evidence_run_id'])), + kind='hostname', + ) + ] + if evidence + else [] + ) result.update( request=request, evidence=evidence, results=normalized_results(evidence), source_executions=source_executions(evidence), + source_yields=source_yields, action_executions=evidence.get('action_executions', []) if evidence else [], artifacts=evidence.get('artifacts', []) if evidence else [], screenshots=screenshots(evidence, str(record['run_id']), self.artifact_directory(str(record['run_id']))), diff --git a/theHarvester/lib/api/static/harvestview/app.js b/theHarvester/lib/api/static/harvestview/app.js index 4bfacfac..d166741d 100644 --- a/theHarvester/lib/api/static/harvestview/app.js +++ b/theHarvester/lib/api/static/harvestview/app.js @@ -314,7 +314,7 @@ const request = run.request || {}; const sources = request.sources?.join(', ') || 'Not recorded'; const options = [ - ['Sources', sources], ['Result limit', request.limit ?? 'Imported evidence'], + ['Sources', sources], ['Result limit', request.limit === 0 ? 'Unlimited' : request.limit ?? 'Imported evidence'], ['Result start offset', request.start ?? 'Not recorded'], ['Discovery source workers', request.source_workers ?? 'Not recorded'], ['Whole-run deadline', request.deadline_seconds === null ? 'Unlimited' : request.deadline_seconds === undefined ? 'Not recorded' : `${request.deadline_seconds} seconds`], diff --git a/theHarvester/lib/api/static/harvestview/index.html b/theHarvester/lib/api/static/harvestview/index.html index 31051709..efa173f9 100644 --- a/theHarvester/lib/api/static/harvestview/index.html +++ b/theHarvester/lib/api/static/harvestview/index.html @@ -217,8 +217,8 @@ Use an explicitly authorized target.